Information-Aware Time Series Meta-Contrastive Learning
Authors: Dongsheng Luo
Wei Cheng
Yingheng Wang
Dongkuan Xu
Jingchao Ni
Wenchao Yu
Xuchao Zhang
Yanchi Liu
Haifeng Chen
Xiang Zhang
Venue: ICLR 2022 Submitted (2022)
Keywords: Information-Aware Time Series Meta-Contrastive Learning

Abstract: Various contrastive learning approaches have been proposed in recent years and achieve significant empirical success. While effective and prevalent, contrastive learning has been less explored for time series data. A key component of contrastive learning is to select appropriate augmentations imposing some priors to construct feasible positive samples, such that an encoder can be trained to learn robust and discriminative representations. Unlike image and language domains where ``desired'' augmented samples can be generated with the rule of thumb guided by prefabricated human priors, the ad-hoc manual selection of time series augmentations is hindered by their diverse and human-unrecognizable temporal structures. How to find the desired augmentations of time series data that are meaningful for given contrastive learning tasks and datasets remains an open question. In this work, we address the problem by encouraging both high fidelity and variety based upon information theory. A theoretical analysis leads to the criteria for selecting feasible data augmentations. On top of that, we employ the meta-learning mechanism and propose an information-aware approach, InfoTS, that adaptively selects optimal time series augmentations for contrastive representation learning. The meta-learner and the encoder are jointly optimized in an end-to-end manner to avoid sub-optimal solutions. Experiments on various datasets show highly competitive performance with up to 11.4%  reduction in MSE on the forecasting task and up to 2.8% relative improvement in accuracy on the classification task over the leading baselines.

Under review as a conference paper at ICLR 2022
INFORMATION-AWAR TIME SERIES
META-CONTRASTIVE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Various contrastive learning approaches have been proposed in recent years and achieve significant empirical success. While effective and prevalent, contrastive learning has been less explored for time series data. A key component of con- trastive learning is select appropriate augmentations imposing some priors to construct feasible positive samples, such that an encoder can be trained to learn robust and discriminative representations. Unlike image and language domains where "desired" augmented samples can be generated with the rule of thumb guided by prefabricated human priors, the ad-hoc manual selection of time se- ries augmentations is hindered by their diverse and human-unrecogen uuman-unrecognizable tem- poral structures. How to find the desired augmentations of time series data that are meaningful for given contrastive learning tasks and datasets remains an open question. In this work, address the problem by encouraging both high fidelity and variety based upon information theory. A theoretical analysis leads to the criteria for selecting feasible data augmentations. On top of that, we employ the meta- meta-learning mechanism and propose an information-awee that adaptively selects optimal time series augmentations for contrastive represen- tation learning. The meta-learne and the encoder are jointly optimized in an end- to-end manner to avoid sub-optimal solutions. Experiments on various datasets show highly competitive performance with up to 11.4% reduction in MSE on fore- casting task and up to 2.8% relative improvement in accuracy on classification task over the leading baselines.
approach, InfoTS,
INTRODUCTION
Time series data in the real world is high dimensional, unstructured, and complex with unique prop- erties, leading to challenges for data modeling (Yang & Wu [006). addition, without human recognizable patterns, it is much harder label time series data than images and languages in real- world applications. These labeling limitations hinder deep learning methods, which typically require huge amount of labeled data for training, been applied time series data (Eldele et al.) 1021). Representation learning learns a -xxedddimension embedding from the original time series that keeps their inherent features. Comparing the raw time series data, these representations transferability and generalization capacity. To deal with labeling limitations, contrastive learning methods have been widely adopted various domains for their soaring performance on represen- lation learning, including vision, language, and graph-structured data (Chen et al., 2020 Xie et al.) 2019 You al. 2020). In nutshell, contrastive learning methods typically train an encoder to map instances to an embedding space where dissimilar (negative) instances are easily distinguishable from similar (positive) ones and model predictions to be invariant to small noise applied to either input examples or hidden states.
with better
Despite being effective and prevalent, contrastive learning has been less explored the time series domain (Eldele et al. 2021 Franceschi et al. 2019 Fan et al. 2020 Tonekaboni et al. 2021). Exist- ing contrastive learning approaches often involve a specific data augmentation strategy that creates novel and realistic-looking training data without changing its label to construct positive alternatives for any input sample. Their success relies on carefully designed rules thumb guided by domain expertise. Routinely used data augmentations for contrastive learning mainly designed for image and language data, such as color distortion, flip. word replacement, and back- translation (Chen et al.) 2020
[uo et al. 2021). These augmentation techniques generally do not apply to time series data.

---

Under review as a conference paper at ICLR 2022
Meta-neliwork selects the optimaa data augmentations during training.
Augmeniation
Meta objective
Contrastive objective
Auginentaaiion
X
Inputs
Augmented data
Encoder
Figure 1: InfoTS is composed of three parts: (1) candidate transformation that generates different augmentations of the original inputs, (2) a meta-network that selects the optimal augmentations, (3) encoder that learns representations of time series instances. The meta-network is learned in tandem with contrastive encoder learning.
Recently, some researchers propose augmentations for time series to enhance the size and quality of the training data (Wen et al. 2021). For example, [Eldele et al. (202T) and et al. (2020) propose to adopt jittering. scaling, and permutation strategies to generate augmented instances. Franceschi al. (219)) extracts subsequences for data augmentation. In spite of the current progress, existing methods have two main limitations. First, unlike images with human recognizable features, time series data are often associated with inexplicable underlying patterns. Strong augmentation such as permutation may ruin such patterns and consequently, the model will mistake the negative handcrafts for positive ones. While weak augmentation methods such as jittering may generate augmented in- stances that are too similar to the raw inputs to be informative enough for contrastive learning. On the other hand, time series datasets from different domains may have diverse nature. Adapting a universal data augmentation method, such subsequence in Xie et al. (2019), for all datasets and tasks leads to sub-optimal performances. Other works follow empirical rules to select suitable aug- mentations from expensive -nrerord- Akin to hand-crafting features, hand-picking choices of data augmentations are undesirable from the learning perspective. The diversity and heterogeneity of real-life time series data further hinder these methods away from wide applicability.
To address the challenges, we first introduce the criteria for selecting good data augmentations in contrastive learning. Data augmentation benefits generalizable, transferable, and robust represen- tation learning by correctly extrapolating the input training space to a larger region (Wilk et al.] 2018). The positive instances enclose discriminative zone in which all the data points should be similar to the original instance. The desired data augmentations for contrastive representation learn- ing should have both high fidelity and high variety. High fidelity encourages the augmented data to maintain the semantic identity that is invariant to transformations (Wilk et al. 2018). For exam- ple, if the downstream task is classification, then the generated augmentations of inputs should be class-preserving, Meanwhile, generating augmented samples with high variety benefits representa- tion learning by increasing the generalization capacity (Chen et al., 2020). From the motivation, we theoretically analyze the information flows in data augmentations based upon information theory and derive the criteria for selecting desired time series augmentations. Due to the inexplicability in practical time series data, we assume that the semantic identity is presented by the target in the downstream task. Thus, high fidelity can be achieved by maximizing the mutual information be- tween the downstream label and the augmented data. A one-hot pseudo label is assigned to each instance in the unsupervised setting when downstream labels are unavailable. These pseudo labels encourage augmentations of different instances to be distinguishable from each other. We show that data augmentations preserving these pseudo labels can add new information without decreasing the fidelity. Concurrently, we maximize the entropy of augmented data conditional on the original instances to increase the variety of data augmentations.
Based on the derived criteria, we propose an adaptive data augmentation method. InfoTS (as shown in Figure []. by employing a meta-learning mechanism to avoid ad-hoc choices or painstakingly trial-and-error tuning. Specifically,, we utilize a meta-network to learn the augmentation prior in tandem with contrastive learning. The meta-learner automatically selects optimal augmentations from candidate augmentations to generate feasible positive samples. Along with random sampled negative instances, augmented instances are then fed into a time series encoder to learn representa- tions in a contrastive manner. With a reparameterization trick, the meta-network can be efficiently optimized with back propaarppagttoon based upon the proposed criteria. Therefore, the meta- nettone can automatically select data augmentations in a per dataset and per learning task manner without resorting to expert knowledge or tedious downstream validation. Our main contributions include:
2

---

Under review as a conference paper at ICLR 2022
We propose criteria to guide the selection of data augmentations for contrastive time series representation learning without prefabricated knowledge.
We propose a meta-learning based method automatically select feasible data augmen- tations for different time series datasets, which can be efficiently optimized with back- propagation.
We empirically verify the effectiveness of the proposed criteria to find optimal data aug- mentations. Extensive experiments demonstrate that InfoTS can achieve highly competitive performance with up 1..4 reduction in MSE on forecasting task and up 2.8% relative improvement in accuracy on classification task over the leading baselines.
2 METHODOLOGY
2.1 NOTATIONS AND PROBLEM DEFINITION
time series instance x: has dimension T F, where T the length of sequence and F is the dimension of features. Given a set of time series instances X. aim to learn an encoder fo(x) that maps each instance x to a fixed-length vector z E IR, where 0 is the learnable parameters of the encoder network and D is the dimension of representation vectors. In semi-supervised settings, each instance x in the labelled set X2 S X is associated with a label y for the downstream task. Specially, X,. X holds in the fully supervised setting. In the work, lowercase letters, such as x, to denote random time series variables and italic lowercase letters, such as x, for sampled instances.
the Sans-serif style
2.2
WHAT ARE GOOD AUGMENTATIONS FOR CONTRASTIVE LEARNING?
The goal of data augmentation for contrastive learning is to create realistically rational instances that maintain semantics through different transformation approaches. Unlike instances vision and lan- guage domains, the underlying semantics of time series data is not recognizable to human, making it hard, if not impossible, to include human knowledge to data augmentation for time series data. For example, rotating image will not change its content the label. While permuting time series instance may ruin its signal patterns and generates a meaningless time series instance. In ad- dition, the tremendous heterogeneity of real-life time series datasets further makes selections based on trial-and-errors impractical. Although multiple data augmentation methods have been proposed for time series data, there is less discussion on what is good augmentation that is meaningful for given learning task and dataset without prefabricated human priors. From our perspective, ideal data augmentations for contrastive representation should keep high fidelity, high variety, and adaptive to different datasets. The illustration and examples are shown in Figure
High Fidelity. Augmentations with high fidelity maintain the semantic identity that is invariant to transformations. Considering the inexplicability in practical time series data, challenging to visually check the fidelity of augmentations. Thus, we assume that the semantic identity of a time series instance is presented by its label in the downstream task, which might be either available or unavailable during the training period. Here, we start our analysis from the supervised case and will extend to the unsupervised case later. Based on the information theory (Tishby et al., 2000), define the objective that keeps high fidelity as the large mutual information (MI) between augmentation and the label y, ie.. MI(v:y).
We consider augmentation From the definition of mutual information, we have MI(v:y) = H() - Hylv),, where Hy) the (Shannon) entropy of y and H (ylvv) is the entropy of y conditioned on augmentation v. Since H Hy)) irrelevant to data augmentations, the objective is equivalent to minimizing the conditional entropy H (ylv). Considering the efficient optimization, to approximate it with cross-entropy between y and y, where y is the prediction with augmentation v as the input and calculated via
probabilistic function of x and random variable that v = g(x; e).
follow Ying al. (2019) and Luo et al. (2020)
gx;e))
= folv)
ý -= h (z),
where z is the representation and hn (-) is a prediction projector parameterized by w. The predic- tion projector is optimized by the classification objective. Then, the objective of high fidelity for
3

---

Under review as a conference paper at ICLR 2022
y: Label
Supervised Setting
Unsupervised Setting
A D
B
V. Augmented data
x: data
label
label1
Augmented samples inrinntance
(a) Information-aware criteria
(b) Examples
Figure 2: Illustration of the criteria. (a) The proposed criteria have two components: high fidelity, and variety. Fidelity is represented by the area of A+B, the mutual information between augmented data v and label y. Variety is denoted by A+D, the entropy of v conditioned on the raw input x. (b) In the supervised setting, good data augmentations generate instances in the area constrained by the label to enlarge the input training space. In the unsupervised setting, with one-hot-basd pseudo labels, the generated instances constrained the region around the raw input. Such that they still distinguishable from other instances.
supervised semi-supervised cases is to minimize
CE(y;)) = E P(y= c) P(y c),
where C the number of labels.
In the unsupervised settings where is unavailable, one-hot encoding y, E R X is utilized as the pseudo label to replace in Eq. (2). The motivation is that augmented instances are still distin- guishable from other instances with the classifier. We theoretically show that augmentations that preserving pseudo labels have the following properties.
Property (Preserving Fidelity). augmentation preserves the one-hot encoding pseudo label, the mutual information between v and the downstream task label y (although not visible to training) is equivalent to that between raw input x and y, i.e., MI(v;y) MI(x:y).
Property (Adding New Information). By preserving the one-hot encoding pseudo label, augmen- tation contains new information comparing to the raw input x, i.e., H(v) H()).
Detailed proofs are shown the Appendix These properties show that in the unsupervised setting, preserving the one-hot encoding pseudo label guarantees that the generated augmentations will not decrease the fidelity, regardless of the downstream tasks and variances inherent in the augmentations. Concurrently, may introduce new information for contrastive learning.
Since the number of labels is equal to the number of instances in dataset X in an unsupervised case, direct optimization of Eq. (2) is inefficient and unscalable. Thus, further relax it by approximat- ing with the batch-wist wise one-hot encoding yB, which decreases the number of labels C' from the dataset size the batch size.
High Variety. Sufficient variances in augmentations improve the generalization capacity of con- trastive learning models. In the information theory, the uncertainty inherent in the random variable's possible outcomes is described by its entropy. Considering that augmented instances are generated based on the raw input x, we maximize the entropy of v conditioned on H (v/x), to maintain high variety augmentations. From the definition of conditional entropy, we have
(vvxx H()) MI(v:x).
(3)
We dismiss the first part since the unconstrained entropy of be dominated by meaningless noise. Considering the continuity of both and x, we adopt a mutual information neural estimator, InfoNCE (Ord al. 2018 Tian et al. 2020) to approximately compute the mutual information for
its practical effectiveness. Other MI estimators, such as Jensen-Shannon (JSD) estimator (Nowozin ct al. 2016), normalized temperature- scaled cross-entropy (NT-Xent) (Chen al. 2020). and leave- one-out (Poole et al., 2019) can also conveniently be the plug-and-play component in our framework. Then, the objective to encourage high variety maximize the InfoNCE between v and x:
exp(sim(z.: 2,))
InfoNCH(v;x) -E log Errex exp(sim(2,,) Z.
(4)
4

---

Under review as a conference paper at ICLR 2022
where v' is augmented instance of input instance z'. Zy. Zu. and Z representations of instance 2, v, and v respectively. sim(z1, Zz) = z 2g is the inner product of vectors 21 and Z2.
Criteria, Combining the information aware definition of both high fidelity and variety, we propose the criteria for selecting good augmentations without prior knowledge.
min InfoNCE(v; x) + 3yE(y; (fo())))
(5)
where 8 is a parameerr to achieve the trade-ofi off between fidelity and variety, Note that in the unsupervised settings, y is replaced by one-hot encoding pseudo label..
Relation Information Bottleneck. Although the formation similar information bottleneck in data compression, mimp(e]s) Mx; e) 3MI(e:y), y), our criteria are different in the following as- pects. First, e the information bottleneck is representation of input x, while v Eq. represents the augmented instances. Second, information bottleneck aims to keep minimal and sufficient infor- mation for data compression, while our criteria are designed for data augmentations in contrastive learning. Third, in information bottleneck, the compressed representation e is deterministic func- tion of input with no variances. MI(e:y) and MI(e; x) are constraint by MI(x; y) and (x)) that MI(e:y) s MI(x; y) and MI(e: = (e), where (e) is the entropy of In our criteria, v is probabilistic function of input x. As a result, the variances of v makes the augmentation space much larger than the compression representation space in information bottleneck.
Relation to InfoMin. InfoMin is designed based on the information bottleneck that good views should keep minimal and sufficient information from the original input (Tian et al,, 2020). Similar to the information bottleneck, InfoMin assumes that augmented views are functions of the input, which heavily constrains the variance of data augmentations. Besides, high fidelity property is dismissed in the unsupervised setting. It works for image datasets due the availability of human knowledge. However, it may fail to generate reasonable augmentations for time series data.
2.3 TIME SERIES META-CONTRASTIVE LEARNING
We aim to design a learnable augmentation selector that learns select feasible augmentations in a data-driven manner. With such adaptive data augmentations, the contrastive loss is then used to train the encoder that learns representations from raw time series.
2.3.1 ARCHITECTURE
The adopted encoder fo(x): - RD consists of two components, a fully connected layer, and 10-layer dilated CNN module (Franceschi al. 2019; Yue al., (02T). To explore the inherent structure of time series, we include both global-wise (instance-level) and local-wise (subsequence- level) losses in the contrastive learning framework to train the encoder.
Global-wise contrastive loss designed to capture the instance level relations in a time series dataset. Formally, given a batch of time series instances XB S X, for each instance E X,, we generate an augmented instance v with an adaptively selected transformation, which will be introduced later. (ar, v) is regarded as a positive pair and other (B (B-1) combinations {(x, d')). where is an augmented instance of x' and x' # z, are considered as negative pairs. Following Chen et al. 2020): You et al. (2020), we design the global-wise contrastive loss based on InfoNCE (Hjelm et al. The wisc instance-level contrastive loss
2018).
Cg
1
Σ x
log
cxp(sim(zz (Zz, 2.))
(6)
Ex'Ex exp
))
Local-wise contrastive loss is proposed to explore the intra-temporal relations in time series. For an augmented instance v of a time series instance x, we first split it into a set of subsequences S, each with length L. For each subsequence E S, we follow "ooeeaaboni et al. (2021) to generate a positive pair (s,p) by selecting another subsequence close to it. Non-neighboring samples, N.. adopted generate negative pairs. Detailed descriptions can be found in Appendix B Then, the local-wise contrastive loss for an instance x is:
exp(sim(z,, zp))
Lez
log
S
exp( poossimzz Ze, 2p)) + EjEN,
2,))
2
5

---

Under review as a conference paper at ICLR 2022
Across all instances in a batch, we have L. [&al Exexs Lex. Then, the final contrastive objective
is:
min bg + aLcr
(8)
where a is hlyper-paaaette to achieve the trade-off between global and local contrastive losses.
2.3.2 META-LEARNER NETWORK
Previous time series contrastive learning methods (Franceschi et al., 2019 Fan et al., 2020 Eldele al. 2021 Tonekaboni et al., 2021) generate augmentations with either rule of thumb guided by prefabricated human priors or tedious trial-and-errors, which are designed for specific datasets and learning tasks. In this part, we discuss how to adaptively select the optimal augmentations with meta-learner network based on the proposed information-aware criteria. We can regard its choice of optimal augmentation as kind of prior selection. We first choose a set of candidate transformations T, such jittering and time warping. Each candidate transformation 4 E ' is associated with an important weight p; E (0, 1), inferring the probability of selecting transformation t,. For an instance x, the augmented instance e through transformation t can be computed by:
(ernoulli(p:)
t/2 = (1 a... + a.l((z).
Considering multiple transformations, we pad all 2s be with the same length. Then, the adaptive augmented instance can be achieved by combing candidate ones, ग.| Σ, Us.
To enable the efficient optimization with gradient-based methods, we approximate discrete Bernoulli processes with binary concrete distributions (Maddison et al. 2016). Specifically, we approximate a, in Eq. (9) with
e~ Unifonn(o.,
a = o[(loge - loog( c) + log 1 Pi
Pi
1/T),
(10)
where (.) is the sigmoid function and T the temperature controlling the approximation, The rationality of such approximation is given in Appendix [A] Moreover, with temperature 7 0, the
gradient de is well-defined. Therefore, our meta-network is end-to-end differentiable. Detailed op:
algorithm is shown in Appendix
RELATED WORK
3.1 CONTRASTIVE TIME SERIES REPRESENTATION LEARNING
Contrastive learning has been utilized widely in representation learning with superior performances
in various domains (Chen et al. 2020 Xie et al. 2019, You et al. 2020). Recently, some efforts have been devoted to applying contrastive learning to time series domain (Oord al., 2018 Franceschi et al., 2019 Fan et al. 2020 Eldele et al.) [021] Tonekaboni et al.) 2021 Yue et al.,
2021 Time Contrastive Learning trains a feature extractor with a multinomial logistic regression classifier to discriminate all segments in a time series (Hyvarinen & Morioka (016). Franceschi
ct al. (2119) generates positive and negative pairs based on subsequences. TNC employs a debiased contrastive objective ensure that in the representation space, signals in the local neighborhood
are distinguishable from non-neighboring signals (Tonekaboni et al. (021). SelfTime adopts multi- ple hand-crafted augmentations for unsupervised time series contrastive learning by exploring both inter-sample and intra-sample relations (Fan et al. (020)p. TS2Vec learns a representation for each time stamp and conducts contrastive learning in a hierarchical way (Yue et al. (202)) However, data augmentations in these methods are either universal or selected by error-and-trail, hindering them away from been widely applied in complex real-life datasets.
3.2 TIME SERIES FORECASTING
Forecasting is critical task in time series analysis. Deep learning architectures used in the literature include Recurrent Neural Networks (RNNs) (Salinas et al. 2020 Oreshkin et al. 2019). Convolu- tional Neural Networks (CNNs) (Bii et al. 2018). Transformers (Li et al. 2019 Zhou al. 202 and Graph Neural Networks (GNNs) (Cao et al. 202T). N-BEATS deeply stacks fully cully-connected layers with backward and forward residual links for univariate times series forecasting (Oreshkin

---

Under review as a conference paper at ICLR 2022
2019h. TCN utilizes deep CNN architecture with dilated causal convolutions (Bai et al. Considering both long-term dependencies and short-term trends in multivariate time series, Let combines both CNNs and RNNS in unified model (Lai et al. 2018). LogTrans brings the Transformer model to time series forecasting with causal convolution in its attention mechanism Li
et al.
2018
et al.] 2019). Informer further proposes sparse self-attention mechanism reduce the time com- plexity and memory usage (Zhou et al., (021). StemGNN is GNN based model that considers the intra-temporal and inter-series correlations simultaneously [Cao et al. (202I). Unlike these works, we aim to learn general representations for time series data that can not only be used for forecasting but also other tasks, such classification. Besides, the proposed framework is compatible with various architectures as encoders.
3.3 ADAPTIVE DATA AUGMENTATION
Data augmentation is an important component in contrastive learning. Existing researches reveal that the choices of optimal augmentation dependent downstream tasks and datasets (Chen iet al.) 2020; Fan et al. 2020). Some researchers have explored adaptive selections of optimal augmentations for contrastive learning in the vision field. AutoAugment automatically searches the combination of translation policies via a reinforcement learning method (Cubuk et al.) (2019). Faster-AA improves the searching pipeline for data augmentation using differentiable policy net- work (Hataya et al. 2020). DADA further introduces an unbiased gradient estimator for an efficient one-pass optimization strategy (Li al. 2020). Within contrastive learning frameworks, |Tian al.) (2020) applies the Information Bottleneck theory that optimal views should share minimal and suffi- cient information, guide the selection of good views for contrastive learning the vision domain. Considering the inexplicability of time series data, directly applying the InfoMin framework may keep insufficient information during augmentation. Different from Tian et al. (2020), we focus on the time series domain and propose end-to-end differentiable method to automatically select the optimal augmentations for each dataset.
EXPERIMENTS
In this section, we compare InfoTS with SOTA baselines on time series forecasting and classification tasks. We also conduct case studies to show insights into the proposed criteria and meta-learning framework. Detailed experimental setups are shown in Appendix C Full experimental results and extra experiments, such as parameter sensitivity studies, are presented in Appendix
4.1 TIME SERIES FORECASTING
Time series forecasting aims predict the future Ly time stamps, with the last L, observations. We follow Yue et al.] (2021) to train a linear model regularized with the L2 norm penalty to make predictions. The output has dimension Ty in the univariate case and Ly F for the multivariate case, where F is the feature dimension.
Datasets and Baselines. Four benchmark datasets for time series forecasting are adopted, includ- ing ETTh1, ETTh2, ETTml (Zhou et al., 2021), and the Electricity dataset (Dua & Graff 2017).
These datasets are used in both univariate and multivariate settings. We compare unsupervised In-
foTS to the SOTA baselines, including TS2Vec (uue et al. 2021). Informer (Zhou al. 202
StemGNN (Cao et al. 2021), TCN (Bai et al. 2018). LogTrans Trans (Li al. 2019). LSTnet (Lai et al. 2018). and N-BEATS Oreshkin et al.] 2019). Among these methods, N-BEATS is merely designed for the univariate and StemGNN is for multivariate only. We refer to Yue et al. (2021) to set up baselines for a fair comparison. Standard metrics for a regression problem, Mean Squared Error (MSE), and Mean Absolute Error (MAE) are utilized for evaluation. Evaluation results of univariate time series forecasting are shown in Table [ while multivariate forecasting results are reported in the Appendix D.4 due to the space limitation.
and Tabel comparison in both univariate and multivariate set- tings indicates that InfoTS consistently matches or outperforms the leading baselines. Some results of StemGNN are unavailable due the oft-of-memory issue (Yue et al. 2021). Specifically, have the following observations. TS2Vec, another contrastive learning method with data augmenta- tions, achieves the second-best performance in most cases. The consistent improvement of TS2Vec
Performance. As shown in Tabel and
7

---

Under review as a conference paper at ICLR 2022
Table I: Univariate time series forecasting results.
InfoTS
TS2Vec
Informer
Log Trans
N-BEATS
TCN
LSTnet
Dataset
Ly
MSE
MAE
MSB
MAE
MSE
MAB
MSE
MSE
MAE
MSE
MSE
MAE
24
0,039
0.148
0.039
0.152
0.098
0.247
0.103
0.259
0.094
0.238
0.075
0.210
0.108
0.284
48
0.056
0.179
0.158
0.319
0.367
0.227
0.175
0.424
ETTB
168
0.099
0.237
0.134
0.282
0.183
0,346
0.207
0.375
0.232
0,391
0,316
0.493
0.396
0.504
336
0.114
0.260
0.310
0.222
0.387
0.230
0.398
0.232
0.388
0.306
0.468
0.593
720
0.147
0.309
0.163
0.327
0.269
0.435
0.273
0.463
0.322
0.490
0.390
0.557
0.659
0.766
24
0.081
0.215
0.090
0.093
0.240
0.102
0.198
0.345
0.103
0,249
3.554
0.445
48
0.115
0.261
0.124
0.273
0.155
0.314
0.169
0.348
0.234
0,386
0.142
0.290
3.190
0.474
ETThg
0.171
0.327
0.360
0.232
0.389
0.246
0.381
0.453
0.227
2,800
0.595
336
0.183
0.341
0.213
0.369
0.263
0.417
0.267
0.437
0.431
0,508
0.296
0.430
2.753
0.738
720
0.194
0.357
0.374
0.277
0,431
0.437
0,517
0.325
2.878
1.044
24
0.014
0.085
0.015
0.092
0.030
0.137
0.065
0.202
0.054
0,184
0.041
0.157
0.090
0.206
48
0.024
0.113
0.027
0.126
0.069
0.203
0.078
0.220
0.190
0.36
0.101
0.257
0.179
0.306
ETTm;:
96
0.036
0.142
0.044
0.161
0.194
0.372
0.199
0.386
0,183
0.353
0.142
0.311
0.272
0.399
288
0.077
0.208
0.103
0.401
0.554
0.186
0.362
0.318
0.462
0.558
672
0.115
0.257
0.156
0.307
0.512
0.644
0.598
0.702
0.197
0.368
0.397
0.547
0.639
0.697
24
0.249
0.273
0.260
0.288
0.251
0,275
0,528
0,447
0.427
0,330
0.263
0.279
0.281
0.287
48
0.294
0.299
0.319
0.324
0.346
0.339
0.409
0.414
0
0.392
0.373
0.344
0.381
0.366
Electricity
0.401
0.368
0.394
0.544
0.424
0,959
0.612
0.893
0.538
0.609
0,462
0.599
0.500
336
0.536
0.453
0.565
0.474
0.713
0,512
1.079
0.639
1.035
0.669
0.855
0.606
0.823
0.624
Avg.
0.155
0.254
0.175
0.278
0.263
0.367
0.336
0.419
0.338
0.402
0.289
0.359
1.090
0.516
Table 2: Multivarite time series classification on 30 UEA datasets.
Method
InfoTS,
InfoTS
TS2
T-Loss
TNC
TS-TCC
TST
DTW
Avg. ACC
0.724
0.711
0.704
0.658
0.670
0.668
0.617
0.629
Avg. Rank
2.133
2.633
2.933
3.733
4.233
4.033
4.867
4.233
over other baselines indicates the effectiveness of contrastive learning for time series representa- tions learning. However, such universal data augmentations may not be the most informative ones to generate positive pairs. Comparing to TS2Vec, InfoTS decreases the average MSE by 11.4%, and the average MAE by 8.6% in the univariate setting. In the multivariate setting, the MSE and MAE decrease by 4.6% and 2.3%, respectively. The reason is that Info can adaptively select the most suitable augmentations in data-driven manner with high variety and high fidelity. Encoders trained with such informative augmentations learn representations with higher quality.
4.2 TIME SERIES CLASSIFICATION
Following the previous setting, we evaluate the quality of representations on time series classification in a standard supervised manner (Franceschi et al., 2019; Yue et al. 2021). We train an SVM classifier with a radial basis function kernel on top of representations in the training split and then compare the prediction in the test set.
Datasets and Baselines. Two types of benchmark datasets are used for evaluations. The UCR archive (Dau et al. 2019) is collection of 128 univariate time series datasets and the UEA archive (Bredin] 2017) consists of 30 multivariate datasets. We compare InfoTS with baselines including TS2Vec Yue al. 202T). T-Loss (Franceschi al. 2019). TS- (Eldele al. 2021), TST (Zerveas et al. (021), and DTW (Franceschi et al. 2019). For our methods, InfoTS,,, training labels are only used to train the meta-network to select suitable augmentations, and InfoTS is with a purely unsupervised setting for representation learning.
Performance, The results on the UEA datasets are summarized in Table 2 Full results are provided in Appendix [D.4] With the ground-truth label guiding the meta-network, InfoTS, substantially outperforms other baselines. On average, it improves the classification accuracy by 2.8% over the best baseline, TS2Vec, with an average rank value 2.333 on all 30 UEA datasets. Under the purely unsupervised setting, InfoTS preserves fidelity by adopting one-hot encoding as the pseudo labels. InfoTS achieves the second best average performance in T with an average rank value 2.633. Performances the 128 UCR datasets shown in Table 9 in Appendix D.4 These datasets are univariate with easily recognized patterns, where data augmentations have marginal or even negative effects (Yue et al. 2021). However, with adaptively selected augmentations for each dataset based on our criteria, InfoTS, and InfoTS still outperform the state-of-the-arts.
4.3 EVALUATION OF THE CRITERIA
In Section 2.2, we propose criteria of data augmentations for time series based on information the- ory that good augmentations should have high variety and fidelity. With mutual information neural
8

---

Under review as a conference paper at ICLR 2022
0.7
0.6
0.5
0.4
0.3
3.0
0.5
Variety Mydelty
Varisty Hoddly
(a) Supervised Setting
(b) Unsupervised Set- ting
Figure 3: Evaluation of the criteria.
Table 3: Ablation studies.
Contrastive Objoctive
Data Augmentation
Meta Objective
InfoTS
w/o Local
wlo Global
Randon
All
wlo Fidelity
woo Variety
Ly
MSB
MAE
MSE
MAB
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAB
24
0.249
0.273
0.249
0.273
0.261
0.385
0.255
0,280
0.255
0.276
0.248
0.274
0.251
0.273
0.294
0.299
0.298
0.304
0.307
0.313
0.301
0.308
0.303
0.308
0.294
0.301
0.293
0.299
168
0.401
0.368
0.405
0.373
0.420
0.385
0.413
0.378
0.410
0.376
0.402
0.371
0.403
0.369
336
0.536
0.453
0.545
0.457
0.557.
0.474
0.549
0.466
0.552
0.470
0.537
0.460
0.537
0.459
Avg.
0.370
0.348
0.380
0.358
0,380
0.357
0.374
0.352
0.386
0.365
0.370
0.351
0.371
0.350
estimation and cross-entropy as approximations, we get the criteria in Eq. 5 To empirically verify the effectiveness of the proposed criteria, we adopt the dataset CricketX from the UCR archive (Dau et al. 2019: Fan et al., 2020), and conduct augmentations with different configurations, such as jit- tering with noise from (0, 0.3) and cutout 10% time stamps. For each configuration, we calculate the criteria score and the corresponding classification accuracy within the setting in Section [.2 As shown in Figure 3 in general, aceuracy performance positively related to the proposed criteria in both supervised and unsupervised settings, verifying the correctness of using the criteria as the objective in the neta-network training.
4.4 ABLATION STUDIES
To present deep insights into the proposed method, we verify key components InfoTS with mul- tiple ablation studies on the Electricity dataset. Results are shown in Table 3 1) We verify the effectiveness of the adopted contrastive learning objective, Eq. (8). by comparing InfoTS with its two variants. "w/o Local" removes the local contrastive loss between subsequences and "woo Global" removes the instance-level contrastive loss. The comparison between these variants and InfoTS demonstrates that both instance-level and intra-temporal objectives are important in trastive learning for time series data. 2) To demonstrate the advantage of adaptive selection of augmentations, we compare InfoTS with variants "Random" and "All". "Random" randomly lects an augmentation from candidate transformation functions each time and "AIl" sequentially applies transformations to generate augmented instances. Performance decreases are observed in these variances, verifying the key role of adaptive selection in our method. 3) To show the effects of variety and fidelity objectives in meta-network training. we include two variants, "w/o Fidelity" and "wwo Variety", which dismiss the fidelity or variety objective, respectively. The comparison between InfoTS and the variants empirically confirms both variety and fidelity are important for data aug- mentation in contrastive learning. Note that the adopted basic augmentations are manually tuned in a previous paper (Fan et al.) 2020) with high qualities for contrastive learning. Including low quality basic augmentations, we conduct more ablation studies are shown in Appendix D.3
5 CONCLUSIONS
We propose criteria of data augmentations for time series data based on the information theory that good augmentations should preserve high variety and high fidelity. We approximate the criteria with mutual information neural estimation and cross-entropy estimation. Based on the approximated cri- teria, we present meta-learning method to adaptively select optimal augmentations for contrastive representation learning. Comprehensive experiments show that representations produced by our method are high qualified and easy in various downstream tasks, such time series forecast- ing and classification, with state-of-the-ar performances.

---

Under review as a conference paper at ICLR 2022
STATEMENTS
Reproducibility Statement. The training algorithm of the proposed InfoTS is shown in Ap- pendix B Detailed experimental settings, including architectures and hyper-parameters settings, are given Appendix Core source codes with an example of usages are attached in the supplemen- tary material. Full version will be made available upon acceptance.
REFERENCES
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: 2018.
Hervé Bredin. Tristounet: triplet loss for speaker turn embedding. In ICASSP, pp. 5430-5434, 2017.
Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Conguri Huang, Yunhai Tong, Bix- iong Xu, Jing Bai, Jie Tong, et al. Spectral temporal graph neural network for multivariate time- series forecasting. arXiv preprint 103:::203300719,, 2021.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pp. 1597-1607, 2020.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In CVPR, pp. 113-123, 2019.
Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi. Chotirat Ann Ratanamahatana, and Eamonn Kcogh. The ucr time series archive. IEEE/CAA Journal Automatica Sinica, 6(6): 2019.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. http://////ach
ics uci. edu/ml
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting, arXiv preprint MKXVv22006.11122 2021.
Haoyi Fan, Fengbin Zhang, and Yue Gao. Self-supervised time series representation learning by inter-intra relational reasoning. arXiv preprint arXiv:2011. 2020.
Jean- Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. arXiv preprint arXiv: 2019.
Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, and Hideki Nakayama. Faster autoaugment:
Learning augmentation strategies using backpropagation. In ECCV, pp. 1-16. Springer, 2020.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus).
arXiv preprint
160.006606.8815. 2016.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie- Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint
2018.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In NIPS, pp. 3765-3773, 2016.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriization with gumbel-softmax, arXiv preprint 10X.::6611.011000 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint /0.:::422.9900 2014.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In SIGIR, pp. 95-104, 2018.
10

---

Under review as a conference paper at ICLR 2022
Arthur Le Guennec, Simon Malinowski, and Romain Tavenard. Data Augmentation for Time Series Classification using Convolutional Neural Networks. In ECML/KKDD Workshop on Advanced Analyties and Learning Temporal Data, 2016.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In NeurIPS, pp. 5243-5253, 2019.
Yonggang Li, Guosheng Hu, Yongtao Wang. Timothy Hospedales, Neil M Robertson, and Yongxin Yang. Dada: Differentiable automatic data augmentation. arXiv preprint arXiv:2003.03 03770, 2020.
Dongsheng Luo, Wei Cheng, Dongkuan Xu, Wenchao Yu, Bo Zong, Haifeng Chen, and Xiang Zhang. Parameterized explainer for graph neural network. arXiv preprint arXiv:2011.04573. 2020.
Dongsheng Luo, Wei Cheng, Jingchao Ni, Wenchao Yu, Xuchao Zhang, Bo Zong, Yanchi Liu, Zhengzhang Chen, Dongjin Song. Haifeng Chen, et al. Unsupervised document embedding via contrastive augmentation. arXiv preprint arXiv:2103.14544 2021.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint urXiv:6611.00712 2016.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers
using variational divergence minimization, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1rXiv:1807.033455 2018.
Boris Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting, arXiv preprint 1905:1900.1040 2019.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-leamn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171- 5180. PMLR, 2019.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic fore- casting with autoregressive recurrent networks. International Journal of Forecasting, 1683):1181- 1191, 2020.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? arXiv preprint 2xXi::2005.1024: 2020.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. arXiv preprint arXiv:2106.00750 2021.
Qingsong Wen, Liang Sun, Xiaomin Song, Jingkun Gao, Xue Wang, and Huan Xu. Time series data augmentation for deep learning: A survey. AAAI, 2021.
Mark van der Wilk. Matthias Bauer, ST John, and James Hensman. Learning invariances using the marginal likelihood. In NeurlPS, pp. 9960-9970, 2018.
Qizhe Xie. Zihang Dai, Eduard Hovy, Minh- Minh-Thang Luong, and Quoe V Le. Unsupervised data augmentation for consistency training. arXiv preprint 1904, /2848, 2019.

---

Under review as a conference paper at ICLR 2022
Qiang Yang and Xindong Wu. 10 challenging problems in data mining research. International Journal of Information Technology & Decision Making, 5(04))597-60) 604, 2006.
Rex Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Gen- erating explanations for graph neural networks. In NewrIPS, volume 32, pp. 9240, 2019.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang. and Yang Shen. Graph contrastive learning with augmentations. In NeurlPS, pp. 5812-5823, 2020.
Zhihan Yue, Yujing Wang, Juanyong Duan, Tiammeng Yang. Congrui Huang, and Bixiong Xu.
Ts2vec: Towards universal representation of time series. arXiv preprint arXiv:2106. 2021.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. In SIGKDD, pp. 2114-2124, 2021.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer long sequence time-series forecasting. AAAI, 2021.
12

---

Under review as a conference paper at ICLR 2022
A DETAILED PROOFS
Properties of data augmentations that preserve pseudo labels.
We assume that iú and 0j are two augmented instances of inputs x; and zj, respectively. Preserving pseudo labels defined in one-hot encoding requires that the map between variable and is one-to- many. Formally, zi + Ij + v, A Vj. This can proved by contradiction. we have a pair of (f.5) that + J and 2; Dj. then we have (v: fo(r)), showing that augmentations cannot preserve pseudo labels.
Property (Preserving Fidelity). If augmentation preserves the one-hot encoding pseudo label, the mutual information between v and downstream task label y (although not visible to training) is equivalent to that between raw input x and y, i.e., MI(v;y) M)
Proof. From the definition mutual information, we have
p y) p(v)
MI(v.y) = H(y) - Hylvv) = H(y) + Eplu, log
p(o.)) log p(v.u) p(u)
= H Σ Σ
+
E. vEV(2)
where (<x) is the set of augmented instances of a time series instance x. In the unsupervised setting where the ground-truth label y is unknown, we assume that the augmentation v is (probabilistic) function of only. The only qualifier means p(v) x. y) = p(e|z). Since the mapping from x to is one-to-many. For each v E V(x) we have p(v, = p(e, xr v) and p(v) = p()l))((x) Thus, we
have
p(u,8) 2(u,,,1) p(olz,y)p(( v) p(ul=)p(a,)) p(a, v) p(u())p(x))
p(v) p(
p(u(z)p(()) p(x)
v(u,v) log p(a.v) v)
MI((:y) = H()) +
p(z)
N.y vEV(x)
p(u v) log p(r, y) p(z)
= -H( + Σ Σ
z.y WE ((x)
p(x,y)
Hy + Σ
p(z, log
p(z)
2.V
= =MI(xyy).
Property 2 (Adding New Information). By preserving the one-hot encoding pseudo label, augmen- tation contains new information comparing to the raw input x, i.e., H(v) Z Hx)).
In information theory, entropy describes the amount of information of random variable. For sim- plicity, we assume a finite number of augmented instances for each input, and each augmented instance is generated independently. Then, we have p(x)) = Evev(a) p(v). Then we have that the entropy of variable is larger than the entropy of v.
H(x) = Σ p(x) log p(x) = = Σ
Σ
((u) Σ p(u)] vEV(w)
2
a
vEV(w)
Σ Σ p(u) plu)logl logl Σ p(v)]
vEV(x)
eEV(x)
- Σ
Σ logp(()
2 DEV(2)
Σ log((())) H(v)
Rationality of approximation of Bernoulli distribution with binary concrete distribution in Eq. (10).
13

---

Under review as a conference paper at ICLR 2022
In the binary concrete distribution, parameter controls the temperature that achieves the trade-off between binary output and continuous optimization. When T - O, we have lim,- P(a. = 1) =
Ps, which is equivalent to the Bernoulli distribution.
Proof.
lim P(a: = lim P(a((loge log(1 -e) log 7-0
Pi
1/7) 1)
T-0
1-p- pi
P(loge - log((1 - c) + log
pi
> 0)
- ps
1 Pi
P(log 1-6 log
Pi
1 Pi
= P( 1 Ps
Since e, and ps both in (0, 1), and function
monotonically increasing in this region.
Thus, we have
lim P(a: = P(e pt) T-0
B ALGORITHMS
B. TRAINING ALGORITHM
The training algorithm InfoTS under both supervised and unsupervised settings described in We first randomly initiate parameters in the encoder, meta-network, and classifier (line 2). Given a batch of training instances XB C X, for each candidate transformation t,, we utilize binary concrete distribution get parameters a; (lines 6-7), which indicates whether the transfor- mation should applied (line 8). vli denotes the batch of augmented instances generated from transformation function f,. The final augmented instances are generated by adaptively considering all candidates transformations (line 10). Parameters the encoder are updated by minimizing the contrastive objective (lines 1 1-13), Meta-network is then optimized with information-awwe criteria (lines 16). Then, classifier hw is optimized with the classification objective (line 17).
Algorithm[] [
Algorithm 1: Algorithm for InfoTS in both supervised and unsupervised settings
I: Input: time series dataset the label set (supervised setting), a set candidate transformations hyper-parameters a and 3,
Initialize the encoder fe. parameters meta-network (q: [=) and the classifier how.
3: for each epoch do
4: for each training batch X C X do
5: for each transformation t; E do
6: Pi + a(g))
ag hinaryConcrete(pi)
v(() +- transform each E Xg with Eq.
9:
end for
10:
Get Ve by averaging (v(2)
11:
Compute global contrastive loss Cg with Eq. (6)
12:
Compute local contrastive loss L. with Eq.
13:
Update parameters in the encoder with Eq.
14:
Compute fidelity loss with Fq. (2)
15:
Compute variety loss with Eq.
16:
Update parameters {qs | in the meta--ettwer with Eq.
17:
Update parameters in the classifier bho with the cross-entropy loss
18:
end for
19:
end
B.2 IMPLEMENTATION OF LOCAL-WISE CONTRASTIVE
Local-wise contrastive loss aims to capture the intra-temporal relations each time series instance. For augmented instance v, first split it into multiple subsequences, shown in Figure
14

---

Under review as a conference paper at ICLR 2022
Subsequence length L
An time series instance
y MM
M m
m Ww Myy
Mm
S
p
anchor
Positive sample
Negative sample
Figure 4: Positive and negative samples for subsequence 8.
Each subsequence has length L. For each subsequence s, the neighboring subsequences within window size are considered as positive samples. If s locates at the end of e, then we choose the subsequence in front of as the positive pair p. Otherwise, we choose the subsequence following instead. Subsequences out of window size are considered as negative samples.
C EXPERIMENTAL SETTINGS
C.I DATA AUGMENTATIONS
We follow Fan al. (020) to set up candidate data augmentations, including jittering, scaling, cutout, time warping, window slicing, window warping and subsequence augmentation (Franceschi al. 2019). Detailed descriptions listed follows.
Jittering augmentation adds the random noise sampled from Gaussian distribution (00, 0.3) to the input time series.
Scaling augmentation multiplies the input time series by a scaling factor sampled from a Gaussian distribution ((000.5).
Cutout operation replaces features of 10% randomly sampled time stamps of the input with zeros.
Time warping random changes the speed of the timeline The number of speed changes is 100 and the maximal ratio max/min speed If necessary, over-sampling or sampling methods are adopted to ensure the length of the augmented instance is the same as the original one.
Window slicing randomly crops half the input time series and then linearly interpolates back to the original length (Le Guennec al. 2016)_
Window warping first randomly selects 30% of the input time series along the timeline and then warps the time dimension by 0.5 or 2. Finally, we adopt linear interpolation to transform it back to the original length (Le Guennee al. 2016).
Subsequence operation random selects subsequence from the input time series Yue al,
2021
With the first 100 time stamps in the univariate Electricity dataset as an example, we visualize the original time series and the augmented ones in Figure [5]
C.2 HARDWARE AND IMPLEMENTATIONS
experiments are conducted on a Linux machine with NVIDIA GeForce RTX 2080 GPUs, each with 1 IGB memory. CUDA version is 10.1 and Driver Version is 418.56. Our method InfoTS implemented with Python 3.7.7 and Pytorch 1.7.1.
15

---

Under review as a conference paper at ICLR 2022
Mi
ill the
Original
Jittering
(c) Scaling
Cutout
330
(e) Time Warping
(1) Window Slicing
(g) Window Warping
(h) Subsequence
Figure 5: Examples of candidate augmentations on Electricity univariate dataset. Blue lines are the original time series data and orange ones augmented instances.
C.3 HYPERPARAMETERS
We train and evaluate our methods with the following hyperparameters and configurations.
Optimizer: Adam optimizer (Kingma & Ba 2014) with learning rate and decay rates setting to 0.001 and (0.9,0.999), respectively.
SVM: scikit-learn implementation (Pedregosa et al. 2011) with penalty C' E {10%|a e Franceschi et al. 2019).
-4,44) U oc)
Encoder architecture: We follow Yue et al. 2021) to design the encoder. Specifically, the output dimension of the linear projection layer is set 64, the same for the number of channels in the following dilated CNN module. In the CNN module, GELU (Hendrycks) & Gimpel, 2016) is adopted as the activation function, and the kernel size is set 3. The dilation is set to 2' in the i-the block.
Classifier architecture: a fully connected layer that maps the representations to the label is adopted.
Trade-off hyperparameters:
in Eq. and in Eq. are searched in 0.51, 1.0, 5, 10). Parameter sensitivity studies are shown in Section D.I
Temperature binary concrete distribution: we follow the practice in (Jang et al. 2016 to adopt the strategy by starting the training with high temperature and anneal to a small value with a guided schedule.
D MORE EXPERIMENTAL RESULTS
D. PARAMETER SENSITIVITY STUDIES
In this part, we adopt the electricity dataset to analyze the effects of two important hyper-parameters in our method InfoTS. The hyperparameter in Eq. (8) controls the trade-off between local and global contrastive losses when training the encoder. 3 in Eq. (5) achieves the balance between high variety and high fidelity when training the meta-network. We tune these parameters in range 10.1, 0.5, 1.0, 5, 10) and show the results in Figure These figures show that our method achieves high performance with a wide range of selections, demonstrating the robustness of the proposed method. In general, setting trade-off parameters to 0.5 achieves good performance.
D.2 CASE STUDY ON SIGNAL DETECTION
In this part, we show the potential usage of InfoTS to detect the informative signals in the time series. We adopt the CricketX dataset example for the case study. Subsequence augmentations
16

---

Under review as a conference paper at ICLR 2022
Table 4: Multivariate time series forecasting results.
InfoTS
TS2Vec
Informer
StemGNN
TCN
Log'
LSTnet
Dalaset
1ng
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
24
0.564
0.520
0.599
0.534
0.577
0.549
0.614
0.571
0.767
0.612
0.686
0.604
1.293
0.901
48
0.607
0.553
0.629
0.555
0.685
0.625
0.748
0.618
0.713
0.617
0.766
0.757
1.456
0.960
ETTby
168
0.746
0.638
0.755
0.636
0.931
0.752
0.663
0.608
0.995
0.738
1.002
0.846
1.997
1.214
336
0.904
0.722
0.907
0.717
1.128
0.873
0.927
0.730
1.175
0.800
1.362
0.952
2.655
1.369
720
1.098
0.811
1,048
0.790
1,215
0.896
1.453
1,311
1.397
1.291
2.143
1,380
24
0.397
0.456
0.398
0.461
0,720
0.068
1.292
0.883
1,365
0.888
0.828
0,750
2.742
1.457
48
0.584
0.572
0,580
0.573
1.457
1.001
1.099
0.847
1.395
0.960
1,866
1,034
3.567
1.687
ETTb,
168
1.787
1,035
1,901
1.065
3.489
1.315
2.282
1.228
3.166
1.407
4.070
1.681
3.242
2.513
336
2.249
1,187
2.304
1.2115
2,723
1,340
3.086
1,351
3,256
1,481
3.875
1.763
2.544
2.591
720
2.511
1,316
2,650
1.373
3.467
1.473
3.690
1,588
3.913
1.552
4.625
3.709
24
0.391
0,408
0,443
0.436
0.323
0.369
0.620
0.570
0.324
0.374
0.419
0.412
1,968
1.170
48
0.503
0.475
0.582
0.515
0.494
0.503
0.744
0.628
0.477
0,450
0.507
0.583
1.999
1.215
ETTm;
96
0.537
0.503
0.622
0.678
0.614
0.709
0.624
0.636
0.602
0.768
2.762
288
0.653
0.579
0.709
0.609
1.056
0.786
0.843
0.683
1.270
1.351
1.462
1.320
1.257
2.076
672
0.757
0.642
0.786
0.655
1.192
0.926
1.381
1.467
1.609
1.461
2.941
24
0.255
0.350
0.287
0.374
0.312
0.387
0.439
0.388
0.305
0.384
0.297
0.374
0.356
0.419
48
0.279
0.392
Electricity
168
0.302
0.385
0.332
0.407
0,315
0.509
0.506
0.518
0.358
0.423
0.426
0.466
0.372
0.425
336
0.759
0.625
0.365
Avg.
0.813
0.627
0.852
0.645
1.164
0.781
0.977
0.706
1.243
0.854
1 1.402
1.032
1.836
1.374
0.39
MSE
0.39
...MEE
MAE
MAE
Error
0.36
0.5
1.0
10
0.1
1.0
5
(a) in Eq.
8
(b) B in Eq.
Figure 6: Parameter sensitivity studies.
on 0-100, 100-200, and 200-300 periods are adopted as candidate transformations. We observe that the one operated on 100-200 period has high fidelity and variety, leading to better accuracy performance, which is consistent with the visualization results in Figure
Th
Label
Label 2
Figure 7: The informative signals locate in the middle periods of time series in the CricketX dataset.
D.3 MORE ABLATION STUDIES
D.3.1 UPDATING PROCESS OF INFOTS
To show that our InfoTS can adaptively detect the most effective augmentation based the data distribution, we follow the setting in Section 4.4 and conduct more ablation studies to investigate comprchensively into the proposed model. We compare performances of variants that each applies a single transformation to generate augmented instances in Table [ From the table, we know that augmentation with subsequence benefits the most for the Electricity dataset. We visualize the weight
17

---

Under review as a conference paper at ICLR 2022
Cutout
Jittering
0.8
Scaling
Time Warping
Window Slicing
0.6
Window Warping
Subsequence
0.4
0.2
0
20
40
60
80
100
120
140
160
180
200
Epoch
Figure 8: Weight updating process of meta-network in InfoTS.
Table 5: Effectiveness of each candidate data transformation.
InfoTS
Cutout
Jittering
Scaling
Time Warp
Window Slice
Window Warp
Subsequence
1
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
MSB
MAE
0.275
48
0.294
0.299
0.304
0.309
0.297
0.302
0.302
0.307
0.305
0.309
0.310
0.314
0.307
0.310
0.295
0.301
168
0.401
0.368
0.412
0.381
0.4.3
0.373
0.407
0.377
0.415
0.382
0.415
0.381
0.416
0.382
0.405
0,372
336
0.536
0.453
0.555
0.465
0.545
0.458
0.552
0.461
0.555
0.469
0.551
0.470
0.554
466
0.546
0.456
Avg.
0.370
0.348
0.381
0.358
0.374
0.352
0.377
0.354
0.381
0.389
0.383
0.361
0,383
0.359
0.374
0,350
updating process of InfoTS in Figure 8 with each line representing the normalized importance score of the corresponding transformation. The weight for subsequence increase with the epoch, showing that InfoTSS tends to adopt subsequence as the optimal transformation. Consistency between accuracy performance and weight updating process demonstrates the effectiveness of InfoTS to adaptively select feasible transformations. Besides, shown in Table 5 InfoTS outperforms the variant that uses subsequence only. This comparison shows that the meta-network learns to consider the combinations, which is better than any (single) candidate augmentation.
D.3.2 INFOTS WITH CONTROLLABLE AUGMENTATIONS.
We observe that in Table 3 the improvement of InfoTS over its two variants without Variety and Fidelity objectives is not significant. The reason is that basic augmentations are adopted from previous paper (Fan et al. 2020), which have been manually tuned to generate relatively good per- formances for contrastive learning. To provide deep insights into the proposed criteria and model, this part, we consider more augmentations with controllable dataset.
for the Electricity
Two groups of augmentations, subsequence augmentations with diffcrent lengths and jitter augmen- tations with different standard deviations, considered in this part. Subsequence augmentations work on the temporal dimension, and jitter augmentations work on the feature dimension. For the subsequence augmentations, we range the ratio of subsequences r in the range [0.01, 0.99]. The subsequence augmentation with ratio r is denoted by Sub,. such as Subo. For the jitter augmen- tations, the standard deviations chosen from the range [0.01, 3.0]. The jitter augmentation with standard deviation std is denoted by Jitterstd, such as Jittero.u1-
Intuitively, with r increasing. Sub, generates augmented instances with lower variety and higher fidelity. For example, with = 0.01, Sub, generates subsequences that only keep 1% time stamps from the original input, leading high variety but extremely low fidelity. Similarly, for jitter aug- mentations, with std increasing, Jitter,ed generates augmented instances with higher variety but lower fidelity. We show their forecasting performances in Table 0 and Table 7 We observed that augmentations that only keeps high variety, such as Subo.o1. Jitters.o o achieve worse performances in terms of MSE and MAE compared to the ones considering both criteria, such as Subo.3 and Jittery.s- On the other hand, augmentations that only keep high fidelity, such Suba. Jittery and Jittery. cannot lead to satisfactory performances either. Thus, we empirically verify the importance both variety and fidelity for the selection of optimal data augmentations for contrastive learning. In Fig-
18

---

Under review as a conference paper at ICLR 2022
Table 6: Forecasting performances of Sub,.
0.01
0.5
0.7
0.99
Ly
MSB
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0.262
0.292
0.249
0.278
0.278
0.277
0.252
0.279
0.253
0.279
0.309
0.321
0.300
0.310
0.298
0.306
0.302
0.309
0.305
0.311
168
0.417
0.395
0.310
0.384
0,409
0.378
0.418
0.384
0,414
0.381
336
0.550
0.475
0.550
0.467
0.553
0.460
0.562
469
0.558
0.463
Avg.
0.384
0.371
0.379
0.360
0.377
0.355
0.384
0.362
0.383
0.358
Table Forecasting performances of Jitter std-
0.3
1.0
3.0
Ly
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
24
0.252
0.277
0.253
0.278
0.250
0.276
0.249
0.250
0.281
0.277
0,303
0.314
0.303
0.310
0.295
0.304
0.300
0.302
0.314
0,306
0.416
0.385
0.413
0.383
0.401
0.373
0.418
0.418
0.391
0.378
0.559
0.473
0.554
0.472
0.545
0.458
0.559
0.570
0.476
0.460
Avg.
0.383
0.362
0.381
0.361
0.373
0.353
0.381
0.366
0.385
0.366
ure we show the relationship between forecasting performance and our proposed criteria. The results are consistent with the conclusion drawn from classification results that forecasting perfor- mance, evaluated with both MSE and MAE, is positively related to the proposed criteria.
0.300
0.570
g 0.380
0.75
0.355
0310
0.350
Hideltyy
Hidelity
(a) MSE
(b) MAE
Figure Evaluation of the criteria on forecasting.
With the proposed criteria, we empirically show the advantage of the developed meta nettokk on learning optimal augmentations. The overall observations are consistent with ones from Section 4.4
Here, we provide more details. "Random" is variant that randomly selects candidate augmen- tation method to generate augmented instances. "All" combines all candidate augmentations evenly to generate augmented instances. These two variances are routinely utilized previous works (Chen) cet al. 2020 You et al. 2020). They work well when candidate augmentations manually verified. Otherwise, their performances are heavily affected by the low-quality candidate augmentations, as shown in T On the other hand, by adaptively selecting optimal augmentations, InfoTS signifi- cantly improve the forecasting performances even with low- quality candidate augmentations. 2) By comparing InfoTS its variant w/o Fidelity and w/o Variety, further empirically demonstrate that both variety and fidelity make contributions to the superior performances of InfoTTS.
D.4 FULL RESULTS OF TIME SERIES PREDICTION AND CLASSIFICATION
The full results multivariate time series forecasting are shown Tabel Results StemGNN with L, 720 are not available due to the out-of-memory error (Yue al. 2021 Full results of univariate time series classification on 128 UCR datasets are shown in Tabel 9 Results of T-Loss, TS-TCC, and TNC are not reported several datasets because they not able to deal with missing observations in time series data. These unavailable accuracy scores are dismissed when computing average accuracy and considered as O when calculating the average rank. Results of multivariate
19

---

Under review as a conference paper at ICLR 2022
Table 8: Ablation studies with controllable augmentations.
InfoTS
Data Augmentation
Meta Objective
Random
All
wio Fidelity
w/o Variety
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAE
24
0.246
0.269
0.252
0.284
0.249
0.279
0.250
0,274
0.251
0.274
0.316
168
0.400
0.369
0.414
0.386
0.414
0.388
0.404
0,369
0.409
0.376
336
0.542
0.455
0.565
0.477
0.563
0.480
0.554
0.456
0.542
0.353
Avg.
0.370
0.348
0.384
0.366
0.382
0.365
0.377
0.352
0,375
0.353
classification on 30 UEA datasets are listed in Tabel 10 Computations of average accuracy scores and ranks follow the ones in 128 UCR datasets.
20

---

Under review as a conference paper at ICLR 2022
InfoTS.
InfOTS
TS2Vee
T-Loss
TNC
TS-TOC
TST
DTW
Adiac
0.795
0.788
0.775
0.675
0.726
0.767
0.550
0.604
Arrowleaud
0.874
0.874
0.857
0.766
0.703
0.737
0.771
0.703
Beef
0.900
0.833
0.767
0.667
0.733
0.600
0.500
0.633
BectleFly
0.950
0.950
0.900
0,800
0,850
0.800
1,000
0.700
BirdChicken
0.850
0,900
0,800
0.850
0.750
0.650
0.650
0.750
Car
0.900
0,883
0.883
0.833
0.683
0.583
0,550
0.733
CBF
1,000
0.999
1.000
0.983
0. 0.983
0.998
0.898
0.997
ChlorineConcennration
0.825
0.822
0.832
0.749
0. 0.760
0.753
0.562
0.648
CinCECG
0.896
0.928
0.827
0.713
0.669
0.671
0.508
0.651
Coffee
1,000
1.000
1,000
1.000
1,000
1.000
0.821
1,000
Computers
0.720
0.748
0.660
0.664
0.684
0.704
0.696
0.700
CricketX
0.780
0.774
0.805
0.713
0.623
0.731
0.385
0.754
CricketY
0.774
0.774
0.769
0.728
0.597
0.718
0.467
0.744
Cricket7
0.792
0.787
0.792
0.708
0.682
0.713
0.403
0.754
DisbennSoocedduction
0.997
0.997
0,987
0.984
0.993
0.977
0,961
0.967
DistalPhalann CostlPPhlllantttliiooCCneeett
0.808
0.801
0.775
0.775
0.754
0.754
0.728
0.717
DistalPhalanx Outiinegnnnn
0.763
0.763
0.727
0.727
0.741
0.755
0.741
0.770
DistalPhalanx TW
0.720
0.727
0.698
0.676
0.669
0.676
0,568
0.590
Earthquakes
0.821
0.821
0.748
0.748
0.748
0.748
0.748
0.719
ECG200
0.950
0,930
0.920
0.940
0.830
0.880
0.830
0.770
BCG5000
0.945
0,945
0.935
0.933
0.937
0.941
0.928
0.924
BCGFiveDays
1,000
1.000
1,000
1,000
0,999
0.878
0,763
0.768
ElectrieDevioes
0.691
0.702
0,721
0.707
0.700
0.686
0.676
0.602
FaceAll
0.929
0,929
0.805
0.786
0.766
0.813
0.504
0.808
FaceFour
0.864
0.818
0,932
0.920
0.659
0.773
0,511
0.830
FaceSUCR
0.917
0.913
0.930
0.884
0.789
0.863
0.543
0.905
FiftyWorls
0.509
0.793
0.774
0.732
0.653
0.653
0.525
0.690
Fish
0.949
0.957
0.937
0.891
0.817
0.817
0.720
0.920
FordA
0.925
0.915
0,948
0.928
0,902
0.930
FordB
0.795
0.785
0.807
0.793
0.733
0,815
0.507
0.620
GunPoint
1.000
1.000
0.987
0.980
0.967
0.993
0.827
0.907
Ham
0,848
0.858
0.724
0.724
0.752
0.743
0,524
0.467
HandOutlines
0.946
0,946
0.930
0.922
0.930
0.724
0.735
0.881
Haptics
0.545
0.546
0.536
0,490
0.474
0.396
0.357
0.377
Herring
0,703
0.656
0.641
0.594
0.594
0.594
0.594
0.531
InlineSkate
0.420
0.424
0.415
0.371
0.378
0.347
0.287
0.384
InseeeWingheatSoum
0.664
0.639
0.630
0.597
0.549
0.415
0.266
0.355
PeweeDDndDed
0
0,966
0.961
0.954
0.928
0.955
0.845
0.950
LargekicchenAApllaane
0.851
0,853
0,875
0.789
0.776
0.848
0.595
0.795
Lightning?
0.934
0,934
0. 0,869
0.869
0. 869
0.836
0.705
0.869
Lightning?
0.863
0.877
0,863
0.795
0. 0.767
0.685
0,41
0.726
Mallat
0.967
0.974
0.915
0.951
0.871
0.922
0.713
0.934
Meat
0.967
0.967
0.967
0.950
0.917
0,883
0.933
Medicallmages
0.920
0.820
0.793
0.750
0.754
0.747
0.632
0.737
MiddlePtalans Cotrreealaaaatttttnnooooooee
0.859
0.859
0.838
0.825
0.818
0.818
0.753
0.698
0.662
0.662
0.636
0.656
0.643
0.630
0.617
0.500
MiddlePhalaxTTWX
0.636
0.617
0.591
0.591
0.571
0.610
0.506
0.506
MoteStraio
0.854
0.871
0,865
0,851
0,825
0.843
0.768
0.835
0,941
0.941
0.930
0.878
0.898
0.898
0.471
0.790
NonInvasive DenlavasiveFaullECCGG GGThooa
0.943
0.944
0.940
0.919
0.912
0.913
0.832
0.865
OliveOil
0.933
0.933
0.900
0.867
0.833
0.800
0.800
0.833
OSULeaf
0.760
0,760
0.876
0.760
0.723
0.723
0.545
0.591
ThalagessQutiinssCores
0.826
0.826
0.823
0.784
0.787
0.804
0,773
0.728
Phoneme
0.272
0.281
0.312
0.276
0. 180
0.242
0.139
0.228
Planc
1,000
1.000
1,000
0.990
1,000
1,000
0.933
1,000
Prodiaadtaalannouliiii Outineeonettt
0.924
0.927
0.900
0.859
0.866
0.873
0.770
0.784
ProximalPhalanx OutineAgeGroup
0.883
0.883
0.844
0.844
0.854
0.839
0.805
ProximalPhalanxTTV
0.849
0.844
0.824
0.771
0,810
0.800
0.780
0.761
RefiigerationDevios
0.624
0.624
0,589
0.515
0.565
0.563
0
0.464
Screen/Type
0.510
0.493
0.411
0.416
0. 0.509
0.419
0.499
0.397
ShapeletSum
0.856
0.856
1.000
0.672
0,589
0.683
0.489
0.650
ShapesAll
0.855
0.852
0,905
0.848
0. 0.788
0.773
0.733
0.768
SmallKichhenApppimmee
0.773
0.773
0.733
0.677
0.725
0.691
0.592
0.643
Sony aTBORaaauSSrrracee
0.921
0.927
0.903
0.902
8.4
0.899
0.724
0.725
SonyAlBORobooSurtaa
0.953
0.953
0,890
0.889
0.834
0.907
0.745
0.831
StarLightCurves
0.973
0.973
0.971
0.964
0.968
0.967
0.949
0.907
Strawberry
0.978
0.978
0.965
0.954
0.951
0.965
09166
0.941
SwedishLeaf
0.954
0.950
0.942
0.914
0.880
0.923
0.738
0.792
Symbols
0.979
0.979
0.976
0.963
0.
0.916
0.786
0.950
SymtheticControl
1.000
1,000
0.997
0.987
1,000
0.990
0.490
0.993
ToeSegmentationl
0.930
0.934
0.947
0.939
0.864
0.930
0.807
0.772
ToeSegmentatin22
0.923
0.915
0.915
0.900
0.831
0.877
0.615
0.838
Trace
1,000
1.000
1,000
0.990
1,000
1.000
1.000
1.000
TwoLeadECG
0.999
0.998
0.987
0.999
0.993
0.976
0.871
0.905
TwoPateers
1.000
1.000
1.000
0.999
1.000
0.999
0-466
1.000
UWaveGestureLibrary
0.819
0.810
0.785
0.781
0.733
0.569
0.728
LeaaeeGeettreell ihrary
0.745
0.736
0.729
0.710
0.697
0.641
0.348
0.634
0.768
0.768
0.770
0.757
0.721
0.690
0.655
0.658
U WaweeestuuellbrrrrAlll
0.966
0.967
0.934
0.896
0.903
0.692
0.475
0.892
21

---

Under review as a conference paper at ICLR 2022
InfoTS,
InfoTS
TS2Vee
T-Loss
TNC
TS-TOC
TST
DTW
Wafer
0.999
0.998
0.998
0.992
0.994
0.994
0.91
0.980
Wine
0.963
0.963
0.889
0.815
0.759
0.778
0.500
0.574
WordSmnoymms
0.691
0.531
0.649
Worms
0.766
0.753
0.701
0.727
0.623
0.753
0.455
0.584
Worms TwoClass
0.818
0.857
0.805
0.792
0.727
0.753
0.584
0.625
Yoga
0.937
0.869
0.887
0.837
0.812
0.791
0.830
0.837
ACSFI
0.850
0.850
0.910
0.900
0.730
0.730
0.760
0.640
"GGestueeWimmtee
0.560
0.630
0.777
0.763
0. 0.703
0.697
0.259
0.716
AllGesturWiimoee
0.623
0.686
0.793
0.726
0.699
0.741
0.423
0.729
AllGesture Wiimote7.
0.633
0.629
0.770
0.689
0.643
BME
1,000
1.0000
0.993
0.993
0.973
0.933
0,760
0.900
Chinstown
0.985
0,988
0.968
0.951
0.977
0.983
0
0.957
Crop
0.766
0.766
0.756
0.722
0.738
0.742
0.710
0.665
ROGHonizonaaliggal
0.577
0.572
0.544
0.605
0 0.442
0.401
0.373
0.503
EOGVerticaSSggall
0.459
0,459
0.503
0.434
0.
0.376
0.298
0.448
ElbanolLevel
0.710
0.712
0.484
0.382
0.424
0.486
0.260
0.276
0.998
0.996
0.986
0.956
0.989
0.922
0.899
FreszeSmaal/Taiin
0
0.988
0.894
0.933
0,982
0.979
0,920
0.753
Fungi
0.866
0,946
0.962
1.000
0.527
0.753
0.366
0.839
GestureMiAAirDI
0.592
0,592
0,631
0.608
0.431
0.369
0.208
0.569
GestureMidAir22
0.459
0.492
0.515
0.546
0. 362
0.254
0.138
0.608
GesturMMidAirDS
0.323
0.315
0.346
0.285
0.292
0.177
0,154
0.323
GesturePebbie221
0.895
0.802
0.930
0.919
0.378
0.395
0.500
0.791
GesturPebblk222
0.905
0.842
0.873
0.899
0.316
0,430
0,380
0.671
GunPointAgeSpan
0.997
1.000
0.994
0.994
0.984
0.994
0,991
0.918
GunPointMale VersusFeemale
1,000
1.000
1,000
0.997
0.994
0.997
L,00
0.997
CenPainOOliVeesuss Young
1.000
1,000
1,000
1,000
1,000
1,000
1,000
0.838
HouseTwenty
0.941
0. 0.924
0.941
0.933
0.782
0.790
0.815
0.924
InsectEGGccgula Train
1,000
PmectiPGSmalllTine
1.000
1.000
1,000
1.000
1,000
1,000
1.000
0.735
Melboun::Pdestrian
0.964
0,962
0.959
0.944
0.942
0.949
0.741
0.791
MixedStupeaReguie Train
0.940
0.935
0.922
0,905
0.911
0.855
0.879
0.842
0.892
0.887
0.881
0.860
0.813
0.735
0.828
0.780
Pickup Pickup Wiimote7
0.820
0.820
0.820
0.740
0.620
0.600
0.240
0.660
Pig Prrwysyrrrs
0.433
0.432
0,683
0.510
0.413
0.380
0.120
0.106
PigAriPresssure
PVgCVP
0.654
0.653
0.870
0.788
0.649
0.615
0.596
0.154
PLAID
0.356
0.355
0.561
0.555
0.495
0.445
0.419
0.840
PowerCons
0.995
1,000
0.972
0.900
0.933
0.961
0,911
0.878
Rock
0.760
0.760
0.700
0.580
0.580
0.600
0.680
0.600
SemnHandGGendehh22
0.939
0.944
0.963
0.890
0.
0.837
0.725
0.802
SemgHaodMooeeeetChh
0.833
0.836
0.893
0.789
0.593
0.613
0.420
0.584
SemnthandSSahjecthh2
0.945
ShakeGestureWimnne
0.920
0.920
0.940
0.920
0.820
0.860
0,760
0.860
SmoothSubspace
1,000
1.000
0.993
0.960
0.913
0.953
0.827
0.827
UMD
1.000
1.000
1,000
0.993
0.993
0.986
0.100
0.993
Dodgerl Dop
0.675
0.675
0.562
0.200
0.500
DodgerLLopGame
0.971
0.942
0.841
0.696
0.877
DodgerLoopWekkend
0.986
0.986
0.964
0.732
0.949
AVG
0.841
0.839
0.836
0.806
0.761
0.757
0.639
0.729
Rank
1.773
1.969
2.328
3.648
4.515
4.391
6.125
5.133
Table 9: Full results of univariate time series classification on 128 UCR datasets.
22

---

Under review as a conference paper at ICLR 2022
Datasct
InfoTS,
InfoTS
TS2Vee
T-Loos
TNC
TS-TOC
TST
DTW
ArticulayyoonReeennn WordReecngniionn
0.993
0.987
0.987
0.943
0.973
0.953
0.977
0.987
AtriafFibellation
0.267
0,200
0.200
0.133
0.133
0.267
0.067
0.200
MusioMotioos
0.975
Character Trajectories
0.987
0.974
0.998
0.993
0.967
0.985
0.975
0.989
Cricket
1.000
0,986
0.972
0.972
0.958
0.917
1,000
1.000
DucKeeeGe
0.600
0.540
0.680
0.650
0.460
0.380
0.620
0.600
WigenWorms
0.748
0.733
0.847
0.840
0.840
0.779
0.748
0.618
Epilepsy
0.993
0.971
0,964
0.971
0,957
0.957
0.949
0.964
ERing
0.911
0,911
0.874
0.133
0.852
0.904
0.874
0.133
RihanolConcentration
PaceDotection
0.525
0.534
0,501
0.513
0,536
0.544
0.534
0.529
FingerMovements
0.620
0,580
0,480
0,580
0,470
0.460
0.560
0.530
HannMMrveeenDDeectiion
0.514
0.392
0.338
0.351
0.324
0.243
0.243
0.231
Handwriting
0.354
0.452
0.515
0.451
0.249
0.498
0.225
0.286
Heartbeat
0.771
0.722
0.683
0.741
0,746
0.751
0.746
0.717
Nowells
0.986
0.984
0,984
0,989
0,978
0.930
0.978
0.949
Libras
0.889
0.867
0.883
0.817
0.656
0.870
LSST
0.593
0.591
0,537
0.500
0,595
0.474
0.408
0.551
Motorimagery
0.610
0.630
0.510
0,580
0.500
0.610
0.500
0.500
NATOPS
0.939
0.933
0.928
0.917
0.822
0.850
0.883
PEMS-SF
0.757
0.751
0.682
0.676
0.699
0.734
0.740
0.711
PonDigits
0.989
0,990
0,981
0.560
0.977
Pbonemeeppcctra
0.233
0,249
0.235
0.222
0.207
0.252
0.085
0.151
RacketSports
0.796
0.855
0.855
0.816
0.809
0.803
SellKegulatioSSCP
0.887
0.874
0.812
0.843
0.799
0.823
0.754
0.775
SellRegulationSCPP
0.572
0.578
0.578
0.539
0.550
0.533
0.550
0.539
SpakenArraheDDigis
0.927
0.947
0,988
0.905
0.934
0.970
0.923
0.963
StandWalkJupp
0.467
0.467
0.467
0.333
0,400
0.333
0.267
0.200
UWaveGesuueLibbaryy
0.884
0.875
0.903
Wingbeatt
0.465
0.465
0.466
0.156
0.469
0.264
0.105
Avg. ACC
0.724
0.711
0.704
0.658
0.670
0.668
0.617
0.629
Avg. Rank
2.133
2.633
2.933
3.733
4,233
4.033
4,867
4,233
Table 10: Full results of multivariate time series clussilication
UBA dalasets.
23


Reviews:
[{"review_id": "wdModhMBe8M", "reviewer": "Reviewer_7xBr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Detailed comments are listed below.\n\n1. About high fidelity: This criterion requires label information, where in the approach, both ground-truth labels and one-hot encoding pseudo labels have been exploited. Property 1 and Property 2 can be readily proved with the strong assumption of one-hot encoding pseudo labels. That is, the total number of classes is equal to the number of time series samples. While the assumption is hard to implement for practical use, the proposed relaxation reduces the number of (pseudo) classes to the batch size, and consequently weakly supports the validity of Property 1 and Property 2. What are the effects of this relaxation to the proposed approach, or more specifically, to the concept of high fidelity? \n\n2. About high variety: The technical correctness of this part needs to be justified. From (3), seeking an augmentation transform of high variety can be achieved by minimizing $\\mathrm{MI}(v; x)$, the mutual information between the augmented instance $v$ and the input $x$. The subsequent derivation seems to imply that minimizing the negative InfoNCE (as in (5)) can lead to minimize the above mutual information. This is questionable in that minimizing the negative InfoNCE is simply minimizing a lower bound of $\\mathrm{MI}(v;x)$; however, it does not ensure that $\\mathrm{MI}(v;x) $ will also be minimized.\n\n3. About the meta-contrastive learning in Section 2.3: While the local-wise contrastive loss in (7) follows Tonekaboni et al. (2021), the positive neighborhood and the negative neighborhood should be described in more detail. Also, the subscript $i$ in $\\mathcal{N}_i$ is confusing and undefined. The fusion scheme and (9) learn to yield a combined transform and appear to be heuristic.  The authors are expected to justify the connection of the proposed approach with meta learning. What are the justifications of naming the architecture as a meta-learner network?\n\n4. The experimental results are not convincing. From the ablation study in Table 3, the results suggest that using the sophisticated fusion scheme (Section 2.3.2) or random augmentations perform almost equally well, while using either “Fidelity” or “Variety” alone does not significantly degrade the MSE outcome at all.  In Table 2, as the model InfoTS_s indeed uses label information in training (to select suitable augmentations), the discussion (in the last paragraph of page 8) on outperforming other baselines may not be fair. \n", "summary_of_the_paper": "This paper describes an information-aware approach to representation learning for time series. The formulation focuses on how to obtain effective data augmentations and addresses the underlying problem from information-theoretic viewpoints, leading to the two optimization criteria, namely, high fidelity and high variety. The experimental results on several time series datasets for forecasting and classification show improvements over the methods in comparison. Detailed comments are listed below.", "summary_of_the_review": "Main concerns about the paper are its technical correctness and the experimental results, including the ablation analysis. This work seems to be not ready for publication yet.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "iU1W2OwuQSn", "reviewer": "Reviewer_ikmv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Concerns\n- page 4, right after eq (3): \"We dismiss the first part since the unconstrained entropy of v can be dominated by meaningless\nnoise.\" Why is it reasonable to drop H(v)? This is a pretty big assumption you make early on, which is never addressed in the paper.\n- Properties 1 and 2 are fairly trivial statements, which I don't think need proof.\n- I think this paper is best described as an online hyperparameter tuning method rather than a meta-learning method. While one can argue that hyperparameter tuning is an instance of meta-learning, I felt that this naming is misleading in this context.\n- All experiments: seeing error bars would be helpful because the differences between methods are fairly small in some cases.\n- Table 3: these results are so close that I think many of them are not statistically significant. In particular, removing components from the meta objective seems to have minimal effect on final performance.\n- I am not fully convinced that the proposed paper brings enough benefits to justify its complexity. For example, in Figure 8 and Table 5, the model mostly up-weights the subsequence policy, and the final performance is not very different from only using subsequence.\n\nMinor comments\n- Minor grammatical inaccuracies and typos throughout the paper. e.g., \"the information theory,\" \"important weight,\" \"combing candidate,\" to name a few.\n- page 3, \"High Fidelity\" paragraph: you call the augmented sequence v an augmentation. This is potentially confusing because augmentations typically refer to the mapping x->v, which you call g later.\n- page 3, \"High Fidelity\" paragraph: The information bottleneck paper of Tishby et al. is not a suitable reference for the general phrase \"information theory.\"\n- page 3, last paragraph: you state that v is a probabilistic function of x and noise variable epsilon. Based on later sentences, it seems that g is a deterministic function of x and epsilon.\n- Near eq (9), it would have been easier to understand how the method works if you described a few examples of transformation t_i you consider.\n- Table 3: inconsistent bolding. L_y=24+w/o Local+MAE, L_y=48+w/o Variety+(MSE, MAE) should be bolded. ", "summary_of_the_paper": "This paper proposes InfoTS, a method for learning augmentations that improve contrastive learning of time series data. The core contribution is a learnable augmentation strategy that uses Concrete/Gumbel-Softmax distributions. The paper shows empirical results on several time-series forecasting and classification benchmarks.", "summary_of_the_review": "While the paper studies an interesting problem, the reported performance gains are marginal, especially given the complexity of the model. I lean towards rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_6fca2LP2sO", "reviewer": "Reviewer_ZzdQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe authors propose an interesting solution to the challenge of seeking data augmentations on time-series data and provide a bunch of experiments on various datasets and with different benchmark models. The paper is well-written.\n\nWeaknesses:\nSome of the analyses and ablations could be more complete, in order to truly investigate whether the conclusions are correct.\n\nMore specific feedback:\n- It is unclear to me what the definition of the pseudo-labels is. Does it mean, if we have, e.g., 10k data points, that the pseudo label is a one-hot encoding with 10k elements? So each data points has its own 'class' label? A proper definition is lacking.\n- p4: The authors mention that direct optimization of Eq. 2 is inefficient and unscalable. But I don't really understand why, as it is in fact just a classification model like any other supervised classifier. So why is it impossible to use the label for each datapoint? And then second, how is it justified that the label of the full batch y_B does contain information about all data windows in the batch? What is it's definition? This addition to the model feels a bit strange, as it is also not further mentioned in the rest of the paper. \n- eq. 10: Are the p_i probabilities trainable parameters? And how do they relate to the importance scores mentioned in appendix D.3? Are the importance scores just unnormalized probabilities? How is the temperature set during training?\n- 3.1: This paragraph is a summation of a lot of models, but it does actually not explain the models that are finally used as benchmark models. If the reader is unfamiliar with any of the benchmark models, the paper does not provide any information to the reader about their difference compared to the proposed model. So I would extend this section with a (semi-technical) explanation of the benchmark models and how they differ in their main technique. For example, which of the proposed benchmarks is actually also a contrastive learning model, and which of the benchmarks is supervised? This information is useful to better interpret the provided results. \n- Contrastive Predictive Coding (van Oord et al., 2018) is another well-known contrastive learning method that is often used for time-series, but the authors did not mention it anywhere in the paper. Besides mentioning it in related work, it would be a very useful benchmark as well, possibly combined with the local and global loss as the authors proposed.\n- p7, last line: The authors mention that TS2Vec achies 2nd best perofrmance because it adopts subsequence by random cropping. How do the authors know this is the reason? No causal conclusions can be drawn from the presented Tabels.\n- Same remark for p8, where the authors argue that InfoTS can adaptively select the most suitable augmentations. At this point of reading, there is no proof of that yet. Later in the ablation studies the authors do however check for this, but the conclusion from Table 3 is mainly that all factors contributed (at least a bit), where the Fidelity and Variety criterion seemed to have contributed the least. So it's not super fair to state that the model's superior performance is (only) thanks to adaptive augmentation selection. In fact, from Fig. 7 it even seems that the model just learns to select one best augmentation, rather than a best combination of all of them. \n- Fig. 3: Can the authors also provide the plot for the ablation models that are not trained with the variety and fidelity criteria? Now it's unsure whether these criteria contributed to this positive relation, or whether this relation already exists by default in such models.  Also, how does this relation look like in the forecasting task?\n\nMinor things/ typos:\n- end p3:  \"parameterized\" instead of paramterized\n- I would refer to Fig. 2 already earlier, for example already both in the High fidelity and high variety sections. It helps the understanding of the reader.\n- p4: The authors mention that the nr of labels is equal to the number of instances in dataset X in the unsupervised case, but this is actually for supervised training right? In the unsupervised case there are no labels. Or do the authors refer to the pseudolabels then? But even then, in the supervised case this remark also holds. \n- p5: Is the batch X_b the same as the mini-batch used during training and mentioned later in the paper? Or are batch and mini-batch two \ndifferent entities in this work?\n- p5: \"non-neighboring samples\", is a sample here a subsequence, just like s?\n- eq. 10: The work from Jang et al., (\"Categorical reparameterization with gumbel-softmax\") concurrently invented the concrete/Gumbel-softmax distribution as the work of Madisson et al., so these works are typically cited together. \n- LSTnet is compared in table 1, but not cited in section 4.1, and StemGNN is cited but not added to the table. \n- 4.4: \"The advantage of adaptive selection\": So if I understand correctly with adaptive selection the authors refer to the concrete sample from a trained categorical over possible augmentations?\n- Appendix C.1: What's the difference between the cutout and the subsequence augmentation?\n- Appendix C.1: If time warping is applied, the number of samples per window is different right? So how do the authors deal with this? Is resampling applied maybe?\n- Appendix D.1: To which experiment does figure 5 relate? And are all runs run with the same randomized seed? \n- Appendix D.3: The MAE of Subsequence for L_y = 168 and 336 in Table 5 are missing a leading zero.", "summary_of_the_paper": "The authors propose a contrastive learning framework for time series data, where data augmentations are adaptively being selected, given a fidelity and variety criterion. Additionally to these two criteria, a contrastive learning objective is applied both on local and global level. The model is tested on a time-series forecasting and classification task on multiple datasets.\n\n", "summary_of_the_review": "The authors propose an interesting solution to the challenge of seeking data augmentations on time-series data. Downstream task performance seems promising compared to benchmark models. In general the paper is well-written, but some clarifying questions remain (see my earlier remarks). Also, some of the (ablation) analyses can be made more complete. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "7hlDg0gwNM", "reviewer": "Reviewer_61sZ", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a new data augmentation approach based on information theory (mutual information) and meta learning to identify optimal data augmentation approach. The rationale behind the method has been extensively described both in the paper and the supplementary material including proofs and additional ablation studies.\nThe paper is experimentally very strong with a comprehensive experiment section and results across various ablation studies.\nI do not see any weaknesses per se or limitations.\nPerhaps I missed it somewhere but is there any information on run times ans complexity?", "summary_of_the_paper": "This is a very good and solid paper that has both empirical elements and theoretical basis. As mentioned in the paper data augmentation in time series is a notoriously difficult problem for the simple reason that it can distort the time series completely - which in contrast to images - humans cannot verify this as it can happen with images. So the contribution of this paper is a new data augmentation approach based on information theory, a meta learning approach and an approach to select optimal data augmentation for contrastive learning.", "summary_of_the_review": "This is a very good submission that touches upon a kind of neglected and less fancy area of data augmentation for time series - authors propose a very neat solution that improves upon the state of the art and demonstrates good performance across datasets and various ablation studies.\nI do not think that being too picky on things that do not matter for the sake of finding some minor negative things that do not really affect the scientific merit of the paper.\nHaving said that, the although am working in contrastive learning and self supervised learning, time series data augmentation is not my cup of tea.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method which selects feasible data augmentations suitable for contrastive time series representation learning. The topic in this paper is timely and interesting. One of 4 reviewers did not complete the review, not responding to a few reminders. So, one emergency reviewer, who is an expert in meta-learning was added. While there is one review that strongly supports this work, two reviews remained unsupportive after the discussion period ended. I appreciate the authors for making efforts in responding to reviewers’ comments. However, after the discussion period, most of reviewers had concerns in this work, pointing out that the technical correctness needs further justification and experiments should be improved.  While the idea is interesting, the paper is not ready for the publication at the current stage. I encourage to resubmit the paper after addressing these concerns.
Decision: Reject