Question: Same remark for p8, where the authors argue that InfoTS can adaptively select the most suitable augmentations. At this point of reading, there is no proof of that yet. Later in the ablation studies the authors do however check for this, but the conclusion from Table 3 is mainly that all factors contributed (at least a bit), where the Fidelity and Variety criterion seemed to have contributed the least. So is it not super fair to state that the model's superior performance is (only) thanks to adaptive augmentation selection? In fact, from Fig. 7 it even seems that the model just learns to select one best augmentation, rather than a best combination of all of them.
Evidence: In Section 2.2, we propose criteria of data augmentations for time series based on information theory that good augmentations should have high variety and fidelity. With mutual information neural estimation and cross-entropy as approximations, we get the criteria in Eq. (5). To empirically verify the effectiveness of the proposed criteria, we adopt the dataset CricketX from the UCR archive (Dau et al., 2019; Fan et al., 2020), and conduct augmentations with different configurations, such as jittering with noise from N (0, 0.3) and cutout 10% time stamps. For each configuration, we calculate the criteria score and the corresponding classification accuracy within the setting in Section 4.2. As shown in Figure 3, in general, accuracy performance is positively related to the proposed criteria in both supervised and unsupervised settings, verifying the correctness of using the criteria as the objective in the meta-network training.****With the proposed criteria, we empirically show the advantage of the developed meta-network on learning optimal augmentations. The overall observations are consistent with ones from Section 4.4. Here, we provide more details. 1) “Random” is a variant that randomly selects a candidate augmentation method to generate augmented instances. “All” combines all candidate augmentations evenly to generate augmented instances. These two variances are routinely utilized in previous works (Chen et al., 2020; You et al., 2020). They work well when candidate augmentations are manually verified. Otherwise, their performances are heavily affected by the low-quality candidate augmentations, as shown in Tabel 8. On the other hand, by adaptively selecting optimal augmentations, InfoTS significantly improve the forecasting performances even with low-quality candidate augmentations. 2) By comparing InfoTS to its variant w/o Fidelity and w/o Variety, we further empirically demonstrate that both variety and fidelity make contributions to the superior performances of InfoTS.
Answer: The reason is that the basic augmentations are adopted from a previous paper [14], which have been manually tuned to generate relatively good performances for contrastive learning. For these reasons, significant improvements on different variants of our methods are impractical and unreasonable. In the initial version, we have conducted some experiments on classification, using both good and bad basic augmentations. As shown in Figure 3, augmentations with high variety and high fidelity can also achieve good accuracy performance with significant improvement. To further take the reviewers' comments into consideration, we have conducted more experiments to show the effectiveness of each component with both good and bad basic augmentations in Appendix D. As is shown In Table 8, comparing to "Random'', which is not trained with variety and fidelity objectives, our InfoTS significantly increases the performances. For example, "Random'' gets 0.384 averaged MAE, while our InfoTS achieves 0.370 average MAE achieved.  We also report other variants to further empirically demonstrate that both variety and fidelity make contributions to the superior performances of InfoTS.