Question: About the naming of meta-learner network, what strong connection does the proposed approach to the meta-learning paradigm?
Evidence: Time series data in the real world is high dimensional, unstructured, and complex with unique properties, leading to challenges for data modeling. In addition, without human recognizable patterns, it is much harder to label time series data than images and languages in realworld applications. These labeling limitations hinder deep learning methods, which typically require a huge amount of labeled data for training, been applied on time series data. Representation learning learns a fixed-dimension embedding from the original time series that keeps their inherent features. Comparing to the raw time series data, these representations are with better transferability and generalization capacity. To deal with labeling limitations, contrastive learning methods have been widely adopted in various domains for their soaring performance on representation learning, including vision, language, and graph-structured data. In a nutshell, contrastive learning methods typically train an encoder to map instances to an embedding space where dissimilar (negative) instances are easily distinguishable from similar (positive) ones and model predictions to be invariant to small noise applied to either input examples or hidden states.
Despite being effective and prevalent, contrastive learning has been less explored in the time series domain. Existing contrastive learning approaches often involve a specific data augmentation strategy that creates novel and realistic-looking training data without changing its label to construct positive alternatives for any input sample. Their success relies on carefully designed rules of thumb guided by domain expertise. Routinely used data augmentations for contrastive learning are mainly designed for image and language data, such as color distortion, flip, word replacement, and back-translation. These augmentation techniques generally do not apply to time series data.
Figure: InfoTS is composed of three parts: (1) candidate transformation that generates different augmentations of the original inputs, (2) a meta-network that selects the optimal augmentations, (3) an encoder that learns representations of time series instances. The meta-network is learned in tandem with contrastive encoder learning.
Recently, some researchers propose augmentations for time series to enhance the size and quality of the training data. For example, and propose to adopt jittering, scaling, and permutation strategies to generate augmented instances. extracts subsequences for data augmentation. In spite of the current progress, existing methods have two main limitations. First, unlike images with human recognizable features, time series data are often associated with inexplicable underlying patterns. Strong augmentation such as permutation may ruin such patterns and consequently, the model will mistake the negative handcrafts for positive ones. While weak augmentation methods such as jittering may generate augmented instances that are too similar to the raw inputs to be informative enough for contrastive learning. On the other hand, time series datasets from different domains may have diverse nature. Adapting a universal data augmentation method, such as subsequence in, for all datasets and tasks leads to sub-optimal performances. Other works follow empirical rules to select suitable augmentations from expensive trial-and-error. Akin to hand-crafting features, hand-picking choices of data augmentations are undesirable from the learning perspective. The diversity and heterogeneity of real-life time series data further hinder these methods away from wide applicability.
To address the challenges, we first introduce the criteria for selecting good data augmentations in contrastive learning. Data augmentation benefits generalizable, transferable, and robust representation learning by correctly extrapolating the input training space to a larger region. The positive instances enclose a discriminative zone in which all the data points should be similar to the original instance. The desired data augmentations for contrastive representation learning should have both high fidelity and high variety. High fidelity encourages the augmented data to maintain the semantic identity that is invariant to transformations. For example, if the downstream task is classification, then the generated augmentations of inputs should be class-preserving. Meanwhile, generating augmented samples with high variety benefits representation learning by increasing the generalization capacity. From the motivation, we theoretically analyze the information flows in data augmentations based upon information theory and derive the criteria for selecting desired time series augmentations. Due to the inexplicability in practical time series data, we assume that the semantic identity is presented by the target in the downstream task. Thus, high fidelity can be achieved by maximizing the mutual information between the downstream label and the augmented data. A one-hot pseudo label is assigned to each instance in the unsupervised setting when downstream labels are unavailable. These pseudo labels encourage augmentations of different instances to be distinguishable from each other. We show that data augmentations preserving these pseudo labels can add new information without decreasing the fidelity. Concurrently, we maximize the entropy of augmented data conditional on the original instances to increase the variety of data augmentations.
Based on the derived criteria, we propose an adaptive data augmentation method, InfoTS (as shown in Figure), by employing a meta-learning mechanism to avoid ad-hoc choices or painstakingly trial-and-error tuning. Specifically, we utilize a meta-network to learn the augmentation prior in tandem with contrastive learning. The meta-learner automatically selects optimal augmentations from candidate augmentations to generate feasible positive samples. Along with random sampled negative instances, augmented instances are then fed into a time series encoder to learn representations in a contrastive manner. With a reparameterization trick, the meta-network can be efficiently optimized with back-propagation based upon the proposed criteria. Therefore, the meta-network can automatically select data augmentations in a per dataset and per learning task manner without resorting to expert knowledge or tedious downstream validation. Our main contributions include:
• We propose criteria to guide the selection of data augmentations for contrastive time series representation learning without prefabricated knowledge. • We propose a meta-learning based method to automatically select feasible data augmentations for different time series datasets, which can be efficiently optimized with backpropagation. • We empirically verify the effectiveness of the proposed criteria to find optimal data augmentations. Extensive experiments demonstrate that InfoTS can achieve highly competitive performance with up to 11.4% reduction in MSE on forecasting task and up to 2.8% relative improvement in accuracy on classification task over the leading baselines.
We propose criteria to guide the selection of data augmentations for contrastive time series
representation learning without prefabricated knowledge. We propose a meta-learning based method to automatically select feasible data augmentations for different time series datasets, which can be efficiently optimized with backpropagation.**** In Figure 9, we show the relationship between forecasting performance and our proposed criteria. The results are consistent with the conclusion drawn from classification results that forecasting performance, evaluated with both MSE and MAE, is positively related to the proposed criteria. With the proposed criteria, we empirically show the advantage of the developed meta-network on learning optimal augmentations. The overall observations are consistent with ones from Section 4.4. Here, we provide more details. 1) “Random” is a variant that randomly selects a candidate augmentation method to generate augmented instances. “All” combines all candidate augmentations evenly to generate augmented instances. These two variances are routinely utilized in previous works (Chen et al., 2020; You et al., 2020). They work well when candidate augmentations are manually verified. Otherwise, their performances are heavily affected by the low-quality candidate augmentations, as shown in Tabel 8. On the other hand, by adaptively selecting optimal augmentations, InfoTS significantly improve the forecasting performances even with low-quality candidate augmentations. 2) By comparing InfoTS to its variant w/o Fidelity and w/o Variety, we further empirically demonstrate that both variety and fidelity make contributions to the superior performances of InfoTS.
Answer: In the previous review, the reviewer pointed out a phenomenon that the improvement is not significant compared to "Random" and our two variants. We analyzed the phenomena and added more experiments in Appendix 3.2 to further demonstrate the effectiveness of the proposed method in practice. We thank the reviewer for the insightful comments, such that we can make our paper more well-qualified.The reviewer still believes that our ablation studies are unsupportive. To take the reviewer's concern, we summarized our major and minor contributions as follows.The two main contributions of our paper are 1) proposing criteria of data augmentations for time series based on information theory that good augmentations should have high variety and fidelity 2) proposing a meta-network to adaptively select augmentations for each dataset based on the proposed criteria. For the first contribution, we have verified with Figure 3 in Section 3 and Figure 9 in Appendix 3.2. For the second contribution, we have verified with Table 3 and Table 8 that we outperform over "Random," which is used in previous works. The margins become more significant if there is no prior knowledge on the selection of augmentations (Appendix 3.2). This phenomenon shows that InfoTS is more practical because the quality of augmentations on time series data is not easy to check by humans virtually. Besides,  The comparison of InfoTS to its variants also demonstrated that both fidelity and variety are important for the selection of augmentations.  Another minor contribution in this paper is the contrastive loss. We have also verified its effectiveness with ablation studies. Thus, we believe that we have adequately verified our claims with extensive experiments. As we analyzed in the experimental part, experimental results also support our claims and we respectfully disagree with the reviewer's comment on "unsupportive ablation study".