Question: Table 3: these results are so close,are they really statistically significant? In particular, removing components from the meta objective seems to have minimal effect on final performance.
Evidence: To present deep insights into the proposed method, we verify key components in InfoTS with multiple ablation studies on the Electricity dataset. Results are shown in Table. 1) We verify the effectiveness of the adopted contrastive learning objective, Eq. (), by comparing InfoTS with its two variants. "w/o Local" removes the local contrastive loss between subsequences and "w/o Global" removes the instance-level contrastive loss. The comparison between these variants and InfoTS demonstrates that both instance-level and intra-temporal objectives are important in contrastive learning for time series data. 2) To demonstrate the advantage of adaptive selection of augmentations, we compare InfoTS with variants "Random" and "All". "Random" randomly selects an augmentation from candidate transformation functions each time and "All" sequentially applies transformations to generate augmented instances. Performance decreases are observed in these variances, verifying the key role of adaptive selection in our method. 3) To show the effects of variety and fidelity objectives in meta-network training, we include two variants, "w/o Fidelity" and "w/o Variety", which dismiss the fidelity or variety objective, respectively. The comparison between InfoTS and the variants empirically confirms both variety and fidelity are important for data augmentation in contrastive learning. Note that the adopted basic augmentations are manually tuned in a previous paper with high qualities for contrastive learning. Including low quality basic augmentations, we conduct more ablation studies are shown in Appendix D.3.
The comparison between InfoTS and the variants empirically confirms both variety and fidelity are important for data augmentation in contrastive learning. Note that the adopted basic augmentations are manually tuned in a previous paper (Fan et al., 2020) with high qualities for contrastive learning. Including low quality basic augmentations, we conduct more ablation studies are shown in Appendix D.3.****In Section 2.2, we propose criteria of data augmentations for time series based on information theory that good augmentations should have high variety and fidelity. With mutual information neural estimation and cross-entropy as approximations, we get the criteria in Eq. (5). To empirically verify the effectiveness of the proposed criteria, we adopt the dataset CricketX from the UCR archive (Dau et al., 2019; Fan et al., 2020), and conduct augmentations with different configurations, such as jittering with noise from N (0, 0.3) and cutout 10% time stamps. For each configuration, we calculate the criteria score and the corresponding classification accuracy within the setting in Section 4.2. As shown in Figure 3, in general, accuracy performance is positively related to the proposed criteria in both supervised and unsupervised settings, verifying the correctness of using the criteria as the objective in the meta-network training.
Answer: The main reason is that the basic augmentations are adopted from a previous paper [14], which have been manually tuned to generate relatively good performances for contrastive learning. For these reasons, significant improvements on different variants of our methods are impractical and unreasonable. In fact, in the initial version, we have conducted some experiments on classification which adopts ACC as the metric, using both good and bad basic augmentations. As shown in Figure 3, augmentations with high variety and high fidelity can achieve good accuracy performance with significant improvement. To further take the reviewers' comments into consideration, we have conducted more experiments in Appendix D.3.2 with more low-quality augmentations.  The gaps between different variants are more obvious, demonstrating that both variety and fidelity make contributions to the superior performances of InfoTS.