Parametric Augmentation for Time Series Contrastive Learning
Authors: Xu Zheng
Tianchun Wang
Wei Cheng
Aitian Ma
Haifeng Chen
Mo Sha
Dongsheng Luo
Venue: ICLR 2024 poster (2024)
Keywords: contrastive learning
neural networks
stream data analysis
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Modern techniques like contrastive learning have been effectively used in many areas, including computer vision, natural language processing, and graph-structured data. Creating positive examples that assist the model in learning robust and discriminative representations is a crucial stage in contrastive learning approaches. Usually, preset human intuition directs the selection of relevant data augmentations. Due to patterns that are easily recognized by humans, this rule of thumb works well in the vision and language domains. However, it is impractical to visually inspect the temporal structures in time series. The diversity of time series augmentations at both the dataset and instance levels makes it difficult to choose meaningful augmentations on the fly. Thus, although prevalent, contrastive learning with data augmentation has been less studied in the time series domain. In this study, we address this gap by analyzing time series data augmentation using information theory and summarizing the most commonly adopted augmentations in a unified format. We then propose a parametric augmentation method, AutoTCL, which can be adaptively employed to support time series representation learning. The proposed approach is encoder-agnostic, allowing it to be seamlessly integrated with different backbone encoders. Experiments on univariate forecasting tasks demonstrate the highly competitive results of our method, with an average 6.5\%  reduction in MSE and  4.7\% in MAE over the leading baselines. In classification tasks, AutoTCL achieves a $1.2\%$ increase in average accuracy.

Published as a conference paper at ICLR 2024
PARAMETRIC AUGMENTATION FOR TIME SERIES CON- TRASTIVE LEARNING
Xu Zheng', Tianchun Wang", Wei Cheng', Aitian Ma', Haifeng Chen', Mo Sha' Dongsheng Luo
"School of Computing and Information Sciences, Florida International University, US "College Information Sciences and Technology, The Pennsylvania State University, US *EEC Laboratories America, US
[xzhen01 9, a aima, msha, d1 uo) } @ @fiu..c edu 3kw53560psu edu, (weicheng, haifeng) @nec labs. com
ABSTRACT
Modern techniques like contrastive learning have been effectively used in many ar- eas, including computer vision, natural language processing, and graph-structure data. Creating positive examples that assist the model in learning robust and dis- criminative representations is crucial stage in contrastive learning approaches. Usually, preset human intuition directs the selection of relevant data augmenta- tions. Due patterns that are easily recognized by humans, this rule of thumb works well in the vision and language domains. However, it is impractical to visually inspect the temporal structures in time series. The diversity of time series augmentations at both the dataset and instance levels makes it difficult to choose meaningful augmentations on the fly. In this study, we address this gap by analyz- ing time series data augmentation using information theory and summarizing the most commonly adopted augmentations in a unified format. We then propose a contrastive learning framework with parametric augmentation, AutoTCL, which can be adaptively employed to support time series representation Icarning. The proposed approach is encoder-agnostic, allowing it to be seamlessly integrated with different backbone encoders. Experiments on univariate forecasting tasks demonstrate the highly competitive results of our method, with an average 6.5% reduction in MSE and 4.7% in MAE over the leading baselines. In classification tasks, AutoTCL achieves 1.2% increase in average accuracy. The source code is available at httts://ghhhbbboomm///aaaaagaaau
INTRODUCTION
Time series data is complex and gigh-dimensional, making it more difficult to gather the label than images or languages. This property hinders the deployment of powerful deep learning methods, which typically require a large amount of labeled data for training(Fldelle et al.) (021). Self-supervised learning is a promising solution due to its capacity to learn from unlabelled data. Self-supervised learning methods learn died-dimennionn embedding of the time series data that preserves its inherent features with better transfcrabilityy and generalization capacity. representative framework, contrastive learning, has been successful in representation learning for various types of data including vision, language, and graphs(Chen et al.] 22000; Xie et al.] 20200] You et al.) (020). These methods train an encoder to map instances to embedding space where similar instances (positive pairs) are easily distinguished from dissimilar ones (negative pairs). As a result, model predictions are unaffected by minor noise introduced into the inputs or hidden states. As key component, data augmentation such as jittering, scaling, permutation, and subsequence extraction (Fan et al. 2020 Wen et 202T is usually adopted to produce positive pairs.
Recently, some efforts have been made to develop contrastive learning methods for time series data (Eldele al. 2021 Franceschi et al.) 2019 Fan et al. 2020 Tonekaboni et al. (02T). However,
Corresponding author.

---

Published as a conference paper at ICLR 2024
due to the diversity and variability of real-world time series data, is challenging to apply general augmentation technique to all datasets. As a result, current approaches to contrastive learning for time series data frequently need particular data augmentation techniques that are guided by domain knowledge and necessitate trial and error to identify the most appropriate augmentations. Attempts have recently been made to study the theory behind adaptive augmentation selection for contrastive learning (Tian et al. according to InfoMin (Tian et al. 2020), produce -reeeeienn views with less shared information.
2020 Suresh ct al. 2021 Xu et al. (021). Good augmentations,
They discover the best perspectives through adversarial training in unsupervised or self-supervise environments by adding an invertible flow-based generative model. The InfoMin principle performs well in the vision area and has been successfully applied to graph-structure data (Xu et al., 2021 Suresh et al., 2021 Yin et al. 2022 You al. 2022). However, self-supervised environment, most existing studies soften the label-preserving property and place more significant emphasis on enhancing diversity by reducing the exchange of information between different views. They frequently use stronger transformers augmentations and undermine the semantics, which is inapplicable to time series data.
To accommodate various augmentation tactics for time series contrastive learning, investigate the data augmentation for time series from the information theory perspective and provide a theoretically sound definition of good augmentations based on input factorization. We further present contrastive learning framework with parametric augmentation, AutoTCL. to adaptively augment data for con- trastive time series learning based on the proposed factorization technique, which can prevent ad-hoc decisions or laborious trial-and-error tuning. Specifically, we utilize parametric neural network to learn to factorize an instance into two parts: the informative part and the task-irrelevant part. The informative component is then applied to a lossless transform function which keeps the instance's 's semantics. The adaptive transformation produces prior mask for the input instance to generate workable positive samples. We demonstrate how the most effective time series data augmentation methods can be viewed as specialized forms of the suggested mask-based transformation. By includ- ing another random variable with adequate variance, the diversity of the augmented view is further increased. In order to learn representations through contrast, augmented pairs are then fed into a time series encoder along with randomly chosen negative pairings. Parameters in the factorization and transform functions are optimized in tandem with contrastive learning loss. Our main contributions summarized as follows.
We introduce a novel factorization based framework to guide data augmentations for contrastive self-supervised learning without prefabricated knowledge.
To automatically learn workable augmentations for time series data, we provide straightforward yet effective instantiation that can handle a variety of frequently applied augmentations.
With comprehensive experimental studies, we empirically verify the advantage of the proposed method on benchmark time series forecasting datasets. We achieve highly competitive performances with a 6.5% reduction in MSE, 4.7% in MAE on univariate forecasting, a 2.9% reduction in MSE, and 1.2% in MAE on multivariate forecasting. In classification tasks, our method achieves a 1.2% increase in average accuracy.
RELATED WORK
Contrastive learning for time series. Contrastive learning has been widely used in representation
learning, achieving superior results across various domains (Chen et al. 2020 Xie et al. 2020
You et al. 2020). Recently, there have been efforts to apply contrastive learning to the time series domain Khaertdinov et al. [2021] Oord et al.) 2018 Franceschi et al.) 2019 Fan al. 2020 Eldele
et al. 2021: Tonckaboni et al.] 2021 Yue et al, 2022] Yang & Hong 2022). In (Franceschi et al.
2019). Franceschi et al. utilize subsequences to generate positive and negative pairs. TNC uses a debiased contrastive objective to make sure that in the representation space, signals from the local neighborhood are distinct from those that are not neighbors (Tonekaboni et al. 2021). TS2Vec uses hierarchical contrastive learning to acquire a representation for each time stamp Yue et al. 2022). TF-C utilizes the distance between time and frequency components as the self supervised signal for representation learning. Each component is independently optimized by contrastive estimation (Zhang ctt al. 2022). In (Nonnenmacher et al.) 2022), the authors introduce an approach that incorporates expert knowledge into time-series representation learning using expert features, surpassing existing
2

---

Published as a conference paper at ICLR 2024
methods in unsupervised and semi-superrised learning on real-world datasets. CLUDA (Ozyurt et al. 2023), a novel framework for unsupervised domain adaptation of time series data, utilizes contrastive learning to learn contextual representations that preserve label information, achieving state-of-the-an performance in time series unsupervised domain adaptation.
Adaptive data augmentation. Data augmentation is a crucial aspect of contrastive learning. Previous studies have shown that the choice of optimal augmentation methods depends on the specific task and dataset being used (Chen et al. 2020 Fan et al. (020). Some studies have explored the adaptive
selection of augmentation methods in the visual domain Tamkin et al. (202 )): Cubuk et al. (2019);
Hataya et al. (2020): Li et al.
(2020); Tian et al. (2020); Rommel et al. (2022); Aboussalah et al.
(2023): : Ho et al. (2019). For example, AutoAugment (Cubuk et al., (219)) uses reinforcement learning method to search for the best combination of policies. Later, CADDA investigates a gradient-
based class-wise method support larger search spaces for EGG signals (Rommel et al. 2022).
DACL adopts Comain-agnoste approach that does not rely on domain-specitty data augmentation
techniques (Verma al.] (02TT. MetAug (Li et al. 2022) and Hallucinator (Wu et al. (023) aim to
generate augmentations in the latent space. In the contrastive learning frameworks, the InfoMin theory
is applied to guide the selection of good views for contrastive learning in the vision domain (Tian)
al. 2020). it further proposes flow-based generative model to transfer images from natural color spaces into novel color spaces for data augmentation.
However, given the complexity of time series data, directly applying the InfoMin framework may not be suitable. Different from previous works, our focus is on the time series domain and we propose an end-to-end differentiable method to automatically learn the optimal augmentations for each time series instance.
Time series forecasting. Forecasting is an essential component of time series analysis, and various deep learning architectures have been employed in the literature, such as MLP (Ekambaram et al.)
2023), Recurrent Neural Networks (RNNs) (Salinas et al. 2020 Oreshkin et al. (020), Convolutional
Neural Networks (CNNs) (Bai al. 2018 Zhang al. 2023b), Transformers (Li et al. 2019 Zhou
it al. 2021 Lin et al. 2023] Yu et al. [023] Nie et al. [203] Zhang et al., 2023a), and Graph Neural Networks (GNNs) for this task (Cao et al.) 2020). In contrast to these works, the aim of our research is to learn general representations for time series data that can be applied not only to forecasting but also to other tasks, such as classification. Additionally, the proposed framework designed to be compatible with multiple types of architectures encoders.
METHODOLOGY
In this section, we first describe the notations used in this paper. Then, we try to answer the following research questions. (I) What are the good views for contrastive learning the self-supervised setting? (2) How to obtain good views for each time series instance for contrastive learning?
3.1 NOTATIONS
We use a T matrix to represent time series instance x, where T is the length of its sequence and F is the dimension of features. With F > 1, is a multivariate time series instance. Otherwise, with F = is a single variate instance. Self-supervised contrastive learning aims learn an encoder that maps from TRTx to vector space RD, where D is the dimension of embedding vectors. In the paper, to distinguish random variables and instances, we use the Sans-serif style lowercase letters, such x, to represent random time series variables, and italic lowercase letters, such as for real instances. Important notations are summarized in Appendix.
3.2 WHAT MAKES GOOD VIEWS FOR CONTRASTIVE SELF-SUPERVISED LEARNING?
In the literature, well-accepted intuitive principle for view generation is that good views preserve the semantics and provide sufficient variances (Tian et al.) 2020; Yin et al. 2002 Suresh et al. 2021). In the supervised setting, where training labels are available, the semantics of instance is usually approximated with the label. On the other hand, semantics-preseree the more popular superved learning. Moreover, while the semantics of images and natural
is much less explored in
3

---

Published as a conference paper at ICLR 2024
language sentences can be manually verified, the underlying semanties of time series data not easily recognizable to humans. This makes it challenging, if not impossible, to apply strong yet faithful data augmentations to such data. To avoid the degenerate solutions caused by dismissing the crmantics-ppeservvinn InfoMin utilizes an invertible flow-based function, denoted by g. to generate a view v for input (Tian et al., 2020). Such that can be restored by x = 9 (v) However, from the information theory perspective, invertible functions fail to include extra variance to the original variable. Formally, we have the following property.
Property 1. If view v is generated x with an invertible function v - gx).. Then H()) H(x) v), where H Hx)), HV) are entropy of variables x and v, respectively; MI(v:x) is mutual information between and x.
The detailed proof can be found in the Appendix. This property shows that the entropy of the augmented view, (v), is no larger than that of original data, H (x), indicating that the existing data augmentation methods don't bring new information for input instances, which limits their expressive power for time series contrastive learning. To address the challenge and facilitate powerful self- supervised learning in the time series domain, we propose a novel factorized augmentation technique. Specifically, given instance we assume that x can be factorized into two parts, informative x² and task irreeeeet part Az. Formally,
I Az.
(1)
As the informative part, x² encodes the semantics of the original x. Motivated by the intuitive principle, we formally define good views for contrastive learning as follows.
Definition 1 (Good View). Given a random variable x with its semantics x", a good view v for contrastive learning task-irrelevant noise, satisfying ((vv) ) 2 HAx)), and 17 is an augmentation function that satisfies that g(x*) - v is one-to- ma-t mapping.
be achieved by v = nx(g(x)), Av), where g is an inverible function, A is
Intuitively, a good view, based on our definition, maintains the useful information in the original variable and at the same time, includes a larger variance boost the robustness of encoder training. We theoretically show that the defined good view has the following properties.
Property 2 (Task Agnostic Label Preserving). If a variable is good view of x, and the downstream task label (although not visible to training) is independent to noise in x, the mutual information between and y is equivalent that belween raw imput and y, i.e., MI(v; Ml(x;
Property 3 (Containing More Information). A good view v contains more information comparing to the raw input x, i.e., (v) 2 (x).
Detailed proof's are given in the appendix. These properties show that in the self-supervised setting, adopting good view for contrastive learning theoretically guarantees that we will not decrease the fidelity, regardless of the downstream tasks. Simultaneously, the good view is flexible to the choice of Av, meaning that strong augmentations may be utilized to incorporate enough diversity for training.
3.3
HOW TO ACHIEVE GOOD VIEWS?
The theoretical analysis suggests a factorized augmentation to preserve agggssgi labels and improve the diversity of views. In this part. we introduce a practical instantiation to obtain good views based on parametric augmentations as demonstrated in Fig. [ First, a factorization function h: TxF + {0, 1]7x1 is introduced to discover where are the informative parts in input. Formally]
h = h(x),
xt
h O
Ar --=--a x²,
(2)
where h is the factorization mask, x" is the informative component, and Az is the noise component. @ is generalized Hadamard product operation performed between two (eatorssm When the inputs both are vectors, denoted as and m, the expression v C m refers to the element-wise product. In the case where the first operand is a vector v E R-V and the second operand a matrix M E N the procedure involves duplicating v to form a matrix V E R X.M and then
A more formal notation is 1 (x) as h is dependent on æ. In this section, for ease of notation, we use h to denote h in contexts where x fixed or evident from the surrounding discussion.
4

---

Published as a conference paper at ICLR 2024
Original instance x
Informative part V
Aug. view V
Input layer
Av
Hidden layer
Aug. network
Parameters shared
Factor, head & Trans, head
Contrast
Encuder network
Zp
Encoder network
Embedding
The framework of our AutoTCL. The augmentation network extracts the informative part from the original instance and losslessly transforms it to v". The encoder network is optimized with the contrastive objective.
Figure/ 1:
applying the standard Hadamard product V and M. both inputs matrices of the same shape. the symbol @ signifies the standard Hadamard product. For the invertible transformation function applied x", present mask-based instantiation. More sophisticated mechanisms, such as normalizing flows (Kobyzev et al., |020 Wang et al. (023), can also be used as plug-and-ple components. Specifically, we introduce a non-zero mask g E 1o to form the transformation, such that 8* = g @ original x can be restored by x² which might differ even within a dataset, it is impractical to have universal mask that applies to all instances. For instance, the cutout and jitter transform might not work well for the same time series data. To ensure each instance has suitable transform adaptively, a parametric mask generator, denoted by g : RTxF - Ro proposed generate the non-zero mask by learning for lossless transformation through a data-driven approach. Formally, g = g(x) Then a good view u for contrastive learning be represented as follows by integrating the factorization function, the mask generator, and introducing random noise for perturbation (Av).
easy show that such non-zero mask transformation is lossless as the
C v". Due to the varied nature of time series instances,
v n7("", Av) 17 Ox" Av)
= m(g(z) C O(z) Av).
Relationship to existing augmentations for time series. Various types of data augmentation techniques have been applied to enhance the performance of deep learning time series data (Wen
et al. 2021 Yue et al. 2022] Fan et al., 2020). including time domain, frequency domain, and their hybrids. Our instantiation can be considered a general augmentation framework in the time domain. Most existing augmentation operations in this category, such as cropping, flipping, scaling, and jittering can be unified in our framework. For example, by setting 17(1", Av) v². cropping. which deletes subsequence, can be achieved by letting = 1 and the cropping time steps in h be O; scaling, which multiplies the input time series by scaling factor, either constant or being sampled from Gaussian distribution, can also be obtained by setting h 1 and g being the scaling factor.
Practical instantiation with augmentation neural network. According to the Universal Ap- proximation Theorem (Chapter 6 in (Goodfellow al. 2016)), we implement g(x) and h(x) with neural networks, respectively. We first utilize the same input layer and stacked dilated CNN module (Franceschi al. 2019: Yue et al. 2022) for both g(x) and h((x), respectively. Then, we include two projector heads, a factorization head for h(x)), and an transformation head for g(x). The architecture of the overall augmentation network is shown in Fig. [ To ensure the binary output of the factorization function h(x), we introduce stochastic mechanism following the factorization head. Specifically, we assume that each element in the output h, denoted by h; is drawn from a Bernoulli distribution parameterized by n, which is calculated by the factorization head.
To enable efficient optimization with gradient- gradient-based methods, we approximate the discrete Bernuulli processes with hard binary concrete distributions (Louizos et al. 2018). Specifically, we first draw h, from binary concrete distribution with n; indicating the location (Maddison al. 2017 Jang et al.
5

---

Published as a conference paper at ICLR 2024
2017 Formally,
= ollog < clg(1 -) + log T )/T),
1 1- T
where e ~ Uniform(O, 1) is an independent variable, o(.) is the sigmoid function, and r is the temperature. The output of the binary concrete distribution is in the range of (0,1). To further
facilitate the binary selection, we stretch hy the range (r, (). with 7 < Ond Ç > 1. Then, the final masking element hy is obtained by clipping values to the range [0, 1]. Formally,
h; = min ma + 0)).
(5)
"no",, Av) as random timestamp masking. To increase the variance of augmented views, inspired by Dropout (Srivastava et al. 2014) and TS2 (Yue et al. 2022), we implement the function n(o",, Ae) by randomly masking the hidden representation. Specifically, given latent vector of a view v, after the first hidden layer, we randomly mask it along the time dimension with binary vector. Each element sampled independently from Bernoulli distribution, (0r5(.
3.4 TRAINING ALGORITHM
There are two parametric neural networks to be optimized in the proposed framework, i.e., the encoder and the augmentation networks. The augmentation network aims to generate good views with high diversities, and the encoder is trained with a contrastive learning objective. Our method is used as plug and play component and can be used in a wide range of contrastive learning frameworks, such TS2 (Yue et al. 2022) and CoST (Woo et al. (022)). In this section, first introduce new objective to train the augmentation network followed by the alternating training of encoder and augmentation networks.
Training the augmentation network with the principle of relevant information. Existing opti- mization techniques for the learnable augmentation and the encoder generally follow the principle of Information Bottleneck (IB) (Tishyy al., 20000), which aims achieve sufficient and minimal representations (Tian et al. 2020 Yin et al. 2022; Suresh et al.) 202T). However, IB relies on the class labels from the downstream task, making it unsuitable for self self-supervised training where there are few or no labels. Instead of following previous works that adversarially train the encoder and augmentation networks, which may fail to preserve the semantics in time series data, we train the augmentation network based on the Principle of Relevant Information (PRI) (Principe, 2010). Unlike supervised IB which relies on another variable as well as their joint distributions, PRI only exploits the self-organization of a random variable, making it fully unsupervised. Specifically, with PRI training the augmentation network, we aim to achieve a reduced statistical representation by minimizing the following function.
(6) where is the trade-off hyper- syper-parameter, HVv") is the entropy of representation variable v", and D(PxIIP,)) is the divergence between distributions of the original variable x and transformed variable v". The minimization of Hvv") aims to reduce uncertainty and obtain statistical regularity in v* and the second term is for preserving the descriptive power of v about x.
BH(V*) + D(P2I(P,):
Given an instance x, the transformed informative part 2" is obtained by applying binary factorization mask h E {0. 1 on and then invertible transformation function, thus, minimizing the first term in Eq. (6). Hvv*), can be achieved by minimizing the number of non-zero elements in the factorization mask, i.e. ||h||o. According to the calculation of h in Eq. (5). we have
|1/lo = - (1- log
->(1- me))
STT
To preserve the descriptive power of v about we follow existing works and estimate the second term [[Px (P-)) with the maximum mean diserepancy, MMD Pv-) (Gretton et al., 2012). In practice, given mini-batch of samples, X, and its associated view set V²', we first compute the embeddings by passing all instances from X through the function f. The same procedure applies to V*, Then, the loss function to train the augmentation network is shown as follows,
1
LPRI
Σ PEX
(z)
f(x)
Σ f(z*)|2. V'EV.
(8)
X
X
V
ZEX

---

Published as a conference paper at ICLR 2024
Regularization of temporal consistency. As shown in previous studies (Luo et al. 2023 informative signals tend to be continuous. Thus, we include regularization of temporal consistency when generating the factorization mask. Specifically, given batch X. for each instance z X, randomly select a time point a as the anchor. Then we randomly select a time point p from the left or
right position of create positive pair pa. Their mask values h((") and h should be similar, compared to another point n that is far away from a, whose mask value is denoted by har Formally, have the following triplet loss.
L
Σ ZEX
(1n5 - n1 - 11 1(22)
(9)
With a trade-off parameter A, the final augmentation network loss could be formulated as:
Laug = LRRI + ALa
(10)
Alternative Training, To train encoder and augmentation networks, we follow GAN (Goodfellow jet al., 2020) to use an alternating training schedule that trains the encoder network M times and then trains the augmentation network one time. M is hyper-parameter determined by grid search.
4 EXPERIMENTS
We compare ACthTCLwith extensive basclines on both forecasting and classification tasks. We also conduct ablation studies to show insights into each component in AutoTCL. Detailed experimental setups, full experimental results, and extensive experiments presented in the Appendix.
4.1 TIME SERIES FORECASTING
Datasets and baselines. Six benchmark datasets, ETThI, ETTh2, ETTml, (Zhou et al. 202 Electricity (Dua & Graff] 2017). Weather and Lora dataset are adopted for time series forecasting in both univariate and multivariate settings. Lora dataset new introduced real-world dataset that captures the wireless signal data using the LoRa devices It contains 74 days of data with timestamps. The proposed AutoTCL model compared to representative state-of-the-arr methods such as TS2Vec (Yue et al., (022)) Informer (Zhou et al.]
(202T). StemGNN (Cao et al.] 2020)). TCN (Bai et al.) 2018). Log Trans (Li et al., 2009)), LSTnet (Lai et al. 2018), CoST (Woo et al. 2022). TNC(Tonekaboni et al. 2021). TS-TCC (Eldele et al.) 2021). InfoTS (Luo al. 2023) and
N-BEATS (Oreshkin et al.) 2020). with N-BEATS being exclusive to univariate forecasting and StemGNN to multivariate.
Setup. We follow CoST (Woo et al. 2022) network architecture. A multi-layer dilated CNN module used for the backbone and we remove the seasonal feature disentangler module. The Augmentation network has the same feature extract architecture and two projectors shown in Fig In addition, the proposed AutoTCL is general contrastive learning framework that can also be combined with more recent methods, such as BTSF (Yang & Hong 2022). TF-C (Zhang et al., 2022 and LsST (Wang et al., (022) to further improve accuracy performances. We leave this as our future work. Time series forecasting aims predict future time stamps, using the last L observations. Following the method presented in (uue et al., 2022), linear model regularized with 1.2 norm penalty, trained to make predictions. Specifically, After pretraining by contrastive learning, the encoder network will be frozen in the following fine-tuning, A linear model is used to map representations to results. The linear model is trained by Linear least squares with 12 regularization in the package sk-learn Pedregosa et al. (20T)). We use the default setting during training. This part is kept the same for other competing methods for a fair comparison. In the univariate case, the model's 's output has dimension of Ly. while in the multivariate case, it has dimension of Ly F. The evaluation is based on standard regression metrics, Mean Squared Error (MSE), and Mean Absolute Error (MAE). To comprehensively evaluate the performances, consider different prediction lengths, Lyy
htttps//www.ncei.moet www.eeeiiioaaggggooo..o www.a//wwwwaaaa//aaaaaaaaaaaaoaa hhtps//horr-llliancc.org

---

Published as a conference paper at ICLR 2024
Results. For each dataset, we calculate the average forecasting performances in both univariate and multivariate settings. The results are shown in Table Jand Table 2 respectively. The detailed results of univariate and multivariate time series forecasting can be found in Table [0 and Table [T] in the Appendix. From these tables, we have several observations. First, in general, contrastive learning methods, including AutoTCII, TS2vec, COST, and InfoTSS achieve better performances compared to traditional baselines, indicating the effectiveness of contrastive learning for learning time series representations. Second, the consistent improvement of our method over CoST indicates that universal data augmentations may not be the most informative for generating positive pairs in various datasets. Specifically, Compared to CoST, AutoTTL decreases both MAE and MSE in all datasets in the univariate setting. On average, AutoTCL decreases the average MSE by 6.5% and the average MAE by 4.8% in the univariate setting. This is because ACL can adaptively learn the most suitable augmentations in data-driven manner, preserving semantics and ensuring sufficient variance. Encoders trained with these informative augmentations lead to representations with higher quality. In the multivariate setting, AutoTCL outperforms CoST in 7 cases. On average, it decreases the average MSE by 2.9% and the average MAE by 1.2%
Table 1: Univariate time series forecasting results.
AmtTCCI
TS2Voe
Informer
LogTrans
N-BEATS
TCN
CaST
TNC
TS-TCC
InRvTSS
Dasaset
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAB
BTTh,
0.076
0.307
0.110
0.252
0.186
0.347
0.196
0.365
0.218
0.375
0.263
0.431
0.091
0.228
0.150
0.303
0.168
0.316
0.091
0.227
0.362
0.168
0.322
0.298
0.478
0.149
0.299
0.191
0.158
0.299
0.050
Blec.
0.366
0.393
0.370
0444
0.388
0744
0.538
0.277
0.482
0.525
0/423
0.375
0.353
0.578
0.359
0.51]
0.603
0.369
0.34
WTH
0.100
0.287
0.181
0.308
0.243
0.370
0.280
0.411
0.256
0.374
0.166
0.291
0.183
0.307
0.175
0.303
0.302
0,442
0.176
0,301
Lora
0.177
0.273
0.356
0.385
1.574
0.999
0.656
0.550
0,311
0.319
T,160
0.927
0.186
0.282
0.620
0.565
0.490
0.591
0.333
0.325
0.157
0.258
0.307
0.301
0.486
0.477
0.382
0.441
0,370
0,388
0.419
0.465
0.168
0.271
0.256
0.340
0.315
0.441
0.188
0..74
Table 2: Multivariate time series forecasting results.
AutTCC
TS2Vec
Informer
LogTrans
SummGNN
TCN
CuST
TNC
TS-ICC
InfoTS
Dacaset
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ETTh:
0.656
0.590
0.758
0.616
0,907
0.739
1,013
0.890
0,738
0.632
1,021
0.816
0.650
0.585
0.901
0.702
0.748
0.635
0.784
1.622
0.651
6T1m
0,441
0.729
0.568
0.521
Blec.
0.135
0.72
0.319
0.397
0.095
0.488
0.351
0.412
0.501
0.489
0.332
0.404
0.165
0.268
0.387
0.466
0.511
0.502
0.289
0.376
WTH
0.23
0.457
0451
0.474
0.574
0.552
0.645
0.617
0.353
0.593
0.440
0.461
0.400
0.464
0.441
0.466
0.483
0.535
0.455
0.422
Lora
0.346
03372
0.356
0.384
0.743
0,586
0,766
0,520
0.258
0.492
1013
0.814
0.350
0,378
0,590
0.518
0.490
0.591
0.345
0.368
0.545
0.499
0.697
0.571
0.990
0,708
1,138
0.798
0.753
0.651
L057
0.781
0.561
0.505
0.837
0.637
0.838
0.675
0.665
0.556
4.2 TIME SERIES CLASSIFICATION
Datasets and baselines. For the classification task, we evaluate our method on the UEA dataset (Dau al., 2019), which contains 30 multivariate time series datasets. We compare our method with
8 state-of-the-eee baselines, including TS2 (Yue et al. 2022). T-Loss (Franceschi et al.) 2019).
TNC (Tonekaboni et al. 2021). TS-TCC (Eldele et al. 202 1. TST (Zerveas et al. 2021). DTW (Chen
et al. 2013). TF-C (Zhang et al. 2022) and InfoTS (Luo et al. 2023).
Setup. We use Vee Yuue et al.) 2022) ) network architecture. In the training stage, we use the
same strategy as the forecasting tasks which could be found in Appendix. We follow the previous setting (Yue et al. (022)) that the evaluation is conducted in standard supervised manner. A radial basis function kernel SVM classifier is trained on the training set and then makes predictions on test data. We report two metrics in the results, accuracy(ACC) and ranK(R
The results on the 30 UEA datasets are summarized in Table 3 The detailed results can be found in Table 12 in the Appendix. Overall, AutoTCL substantially outperforms baselines with an average rank value 2.3. As shown in Table 12] our method achieves the best results in 16 out of 30 datasets. In addition, it improves the classification accuracy by 1.6%, on average, over the second-bes baseline, InfoTS. The comprehensive comparison indicates the effectiveness of the proposed method.
Results.
4.3 ABLATION STUDY AND MODEL ANALYSTS.
In this set of experiments, we conduct ablation studies to investigate the effectiveness of each component in the proposed method. To present deep insights into automatic data augmentation and
8

---

Published as a conference paper at ICLR 2024
Table 3: Classification result of the UEA dataset
Datuset
AutoTCL
TS2
T-Loss
TNC
TS-TCC
TST
DTW
TF-C
InfoTS
Avg. ACC
0.742
0.704
0.658
0.670
0.668
0.617
0.629
0.298
0,730
Avg. RANK
2.300
3,700
4.667
5,433
5.133
6.133
5,400
8.200
2.367
factorization, we compare AutoTCL with multiple groups of variants. (1)W/o h(x), W/o g(z), and W/o Ar are ablation studies about the effectiveness of each part of AutoTCI.. In our experiments, W/o hxx) means the whole input instance would be regarded as the informative part. W/o g(x) represents the transformation head g(x) would be replaced by all vectors and no noise will be added in W/o Ae setting. (2) Cutout and Jitter are two commonly adopted data augmentation techniques for time series contrastive learning. We replace the augmentation network in AutoTCL. with these two static transformations as variants. (3) Adversarial training is routinely adopted in the literature to learn views for contrastive learning. For this variant, we adversarially train the augmentation network by minimizing the mutual information between views and original instances, approximated by the InfoNCE Tian et al. 2020). (4), Random Aug. randomly select augmentation operations from Cutout and Jitter with different parameters. The parameter of cutout ranges from 0.3 to 0.8. The mean of Jitter is set to O, and the standard deviation ranges from 0.3 to 1.0. We report the averaged performances in Table 4 the full results are shown in Table Appendix.
We have several observations in Table 4 First, by removing the factorization head, W/o (xx increase the MSE by 10.19% and MAE by 4.65% respectively, verifying the effectiveness of the factorization. The comparison between ACtonTLLand W/o ( indicates the importance of invertible view generation. Specifically, W/o g(x)) increases the MSE by 37.6% and MAE by 9.3%; The difference between Autan W/o Av indicates the importance of diversity in data augmentation. Second, the comparison between W/o Aug and Cutout shows that universal and non-parametric augmentation techniques may harm the performances of time series contrastive learning. On average, Cutout performs even worse than W/o Aug. This observation is consistent with the conclusion drawn in TS2 (Yue et al. 2022). By adaptive learning suitable augmentations, our methods can consistently and significantly outperform these baselines. Third, with the augmentation network trained in adversarial manner, the variant, Adversarial improves the performances, indicating the necessity of adaptive augmentation for time series data. However, overlooking the semantic preservation may generate trivial augmented views, hindering the performance of downstream contrastive learning. On the other hand, our method achieves the best performances in most cases, especially for forecasting long periods, which verifies the advantage of our training algorithm. The comparison between AutoTCL and Random Aug, further indicates the advantage of parametric augmentation.
Table Ablation studies and model analysis
AutoTCI.
Wil A(x)
Wio g(s)
Wio Av
W/o Aug
Cutout
Jiner
Adversuiial
Random Aug.
Dataset
MSB
MAE
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAB
MSB
MAE
MSE
MAE
MSE
MAB
ETTh:
0.076
0.207
0,077
0.308
0.078
0.300
0.086
0.219
0.095
0.331
0.058
0.231
0.086
0.219
0.080
0.324
0.112
0.354
ETTh;
0.158
0.299
0.168
0.305
0.178
0.312
0.176
0,311
0.170
0.309
0.160
0.306
0.173
0.317
0.187
0.319
0.158
0.321
0.154
0,052
0.050
0.159
0.165
0.053
0.055
0.164
0.170
0,052
0.065
0.387
Elee.
0.368
0.571
0.365
0.345
0.347
0.367
0.344
0.376
0.358
WTH
0.287
0.172
0.166
0.295
0.293
0.183
0.167
0.204
0.384
0.166
0.184
0.310
Lom
0.177
0.273
0.237
0.309
0.069
0.385
0.304
0.361
0.711
0.112
0.783
0.442
0.285
0.366
0.645
0.373
0.191
0.299
Ave
0.157
0.258
0.173
0.270
0.216
0.283
0.185
0.280
0.350
0.294
0.266
0.394
0.184
0.281
0.312
0.284
0.176
0.286
5 CONCLUSION AND FUTURE WORK
We present a novel factorization-bar augmentation framework for time series representation learning in the self-supervised setting. Theoretical analysis from the information theory perspective shows that the proposed framework is more flexible to persevere semantics and includes sufficient variances to augment views. On top of that, we provide a simple and effective instantiation and an efficient training algorithm. With time series forecasting as the downstream task, we compare the proposed method, AutoTCL, with representative methods and verify its effectiveness. In addition, AutoTCL exploits the informative part of time series data, which might help users better understand the time series data. In the future, we plan to investigate the usage of parametric augmentation in other contrastive learning frameworks and extend the technique to other domains.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This project was partially supported by NSF grants ASS-2331988 and CNS-2150010. The views and conclusions contained this paper are those of the authors and should be interpreted as representing any funding agencies.
REFERENCES
Amine Mohamed Aboussalah, Minjae Kwon, Raj G Patel, Cheng Chi, and Chi-Guhn Lee. Re- cursive time series data augmentation. In The Eleventh International Conference on Learning Representutions, 2023. URL pttttps://operrvvieee www.rreeeeeeeeeeeeeeeeeee
net/
124s 1244
Shaojie Bai, Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and 2018.
recurrent networks for sequence modeling. arXiv preprint
Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, et al. Spectral temporal graph neural network for multivariate time-series forecasting, Advances in neural information processing systems, 33:17766-17778 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pp. 1597-1607, 2020.
Yanping Chen, Bing Hu, Eamonn Keogh, and Gustavo EAPA Batista. Dtw-d: time series semi- supervised learning from single example. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 383-391, 2013,
Ekin D Cubuk, Barret Zoph. Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment: Learning augmentation strategies from data. In CVPR, pp. 113-123, 2019.
Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The ucr time series archive.
IEEE/CAA Journal of Automatica Sinica, (00):129313000 2019. doi: 10. 10.1100/AA..221..191177
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL attp:
ics. uci. wcw.edu// m1
Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. Tsmixer: Lightweight mip-mixer model for multivariate time series forecasting. In Ambuj Singh, Yizhou Sun, Leman Akoglu, Dimitrios Gunopulos, Xifeng Yan, Ravi Kumar, Fatma Ozcan, and Jieping Ye (eds.), Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6-10, 2023, pp. 459 469. ACM, 2023. URL https: /tops://doo.. /oi.. org 0tt.:/10.111//550333333 35503355 3599533
doi: 10.1145/3580305.559900
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In Zui-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI 2021, Virtual Event Montreal, Canada, 19-27 August 2021, pp. 2352-2359, ijcai.org, 2021.
doi: 10. 2144966//jaai22221/3333 URL https::///a/to. http:///bb...or.10.2299555//////
Haoyi Fan, Fengbin Zhang, and Yue Gao. Self-supervised time series representation learning by inter-intra relational reasoning. arXiv preprint arXiv:2011.13544 2020.
Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. Advances in neural information processing systems, 32, 2019.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
//www. www.:deeeplartiee eean i eeeeeaannnnooo ngbook. org
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.
10

---

Published as a conference paper at ICLR 2024
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, (311):73--73, 2012.
Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, and Hideki Nakayama. Faster autoaugment:
Learning augmentation strategies using backpropagation. In ECCV, pp. 1-16. Springer, 2020.
Daniel Ho, Eric Liang, Xi Chen, Ion Stoica, and Pieter Abbeel. Population based augmentation: Efficient learning of augmentation policy schedules. In International conference on machine learning, pp. 2731-2741. PMLR, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In Internationaal Conference Learning Representations, 2017. URL /tLLsppte////eeeeeviee net met//ooumm fow.mmidddeeeyyee-
Bulat Khaertdinov, Esam Ghaleb, and Stylianos Asteriadis. Contrastive self-supervised learning for sensor-based human activity recognition. 2021 IEEE International Joint Conference on Biometrics (IJCB), pp. 1-8. IEEE, 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Ivan Kobyzev, Simon JD Prince, and Mareus Brubaker. Normalizing flows: An introduction and review of current methods. IEEE transactions pattern analysis and machine intelligence, 43 (11):3964-397) 2020.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In SIGIR, pp. 95-104, 2018.
Jiangmeng Li, Wenwen Qiang, Changwen Zheng, Bing Su, and Hui Xiong. Metaug: Contrastive learning via meta feature augmentation. International Conference on Machine Learning, pp- 12964-12978. PMLR, 2022.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In NeurIPS, pp. 5243-5253, 2019.
Yonggang Li, Guosheng Hu, Yongtao Wang, Timothy Hospedales, Neil M. Robertson, and Yongxin Yang. Differentiable automatic data augmentation. In Computer Vision - ECCV 2020: 16th Euro- pean Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XXII, pp. 580-595, Berlin, Heidelberg, 2020. Springer- Verlag. ISBN 777----330-5844---- doi: 10. 0007779788--030-55542---
35. URL https: org/ 1007 0tt.:://.......11101000---0---20 585--------5--66----
Shengsheng Lin, Weiwei Lin, Wentai Wu, Songbo Wang, and Yongxiang Wang. Petformer: Long-term time series forecasting via placeholder-enhanced transformer. arXiv preprint arXiv:2308.0479) 2023.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_ regularization. In International Conference on Learning Representations, 2018.
Dongsheng Luo, Wei Cheng, Yingheng Wang, Dongkuan Xu, Jingchao Ni, Wenchao Yu, Xuchao Zhang, Yanchi Liu, Yuncong Chen, Haifeng Chen, et al. Time series contrastive learning with information- awart augmentations. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4534 4542, 2023.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation discrete random variables. In International Conference on Learning Representations, 2017. URL/:ttss///ppeeeeeee...ee wtt.:/eeee/ooommmdoe1111111e.e
Yuqi Nie, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-temn forecasting with transformers. In The Eleventh International Confer- ence on Learning Representations, 2023. URL
net/ hft.roorrrmii
JbdcOvTOco1]

---

Published as a conference paper at ICLR 2024
Manuel T Nonnenmacher, Lukas Oldenburg, Ingo Steinwart, and David Reeb. Utilizing expert features for contrastive learning of time-series representations. In International Conference on Machine Learning, pp. 16999. PMLR, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprini arXiv: 2018.
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-heats:
Neural basis ex- pansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2020,
Yilmazcan Ozyurt, Stefan Feuerriegel, and Ce Zhang. Contrastive learning for unsupervised domain adaptation of time series. In The Eleventh International Conference on Learning Representations, 2023.
net/ forum? wwt.:///rorrmmmeeeeeee////so/sa. d=XXKKYRSSGGMM
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825- 2830, 2011.
Jose C Principe. Information theoretic learning: Renyi's entropy and kernel perspectives. Springer Science & Business Media, 2010.
Cédric Rommel, Thomas Moreau, Joseph Paillard, and Alexandre Gramfort. Cadda: Class-wise automatic differentiable data augmentation for eeg signals. In ICLR 2022-International Conference on Learning Reprexentations, 2022.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):
1181-1191, 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199) 2014.
Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville. Adversarial graph augmentation to improve graph contrastive learning. Advances in Neural Information Processing Systems, 34: 14200-91-5 2021.
Alex Tamkin, Mike Wu, and Noah Goodman. Viewmaker networks: Learning views for unsupervised representation learning. In International Conference Learning Representations, 2021. URL
htttps://gpenreeview net. forum? www.oumm..deeooowww..........
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827-6839, 2020.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004055, 2000.
Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. In International Conference on Learning Representations, 2021. [ttt:ttttt//peereeeee
net/ wwtp:e/ooouurrdddaageeeeeeeee
Vikas Verma, Thang Luong, Kenji Kawaguchi, Hieu Pham, and Quoe Le. Towards domain-agnostic contrastive learning. In International Conference Machine Learning, pp. 10530-10541, PMLR, 2021.
Tianchun Wang, Farzanch Mirzazadeh, Xiang Zhang, and Jie Chen. Gc-flow: A graph-based flow network for effective clustering. arXiv preprint arXiv:2305.17284 2023.
Zhiyuan Wang. Xovee Xu, Goce Trajcevski, Weifeng Zhang, Ting Zhong, and Fan Zhou. Learning latent seasonal-trens representations for time series forecasting. In Advances in Neural Information Processing Systems, 2022.
12

---

Published as a conference paper at ICLR 2024
Qingsong Wen, Liang Sun, Xiaomin Song, Jingkun Gao, Xue Wang, and Huan Xu. Time series data augmentation for deep learning: A survey. In AAAI, 2021.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. CoST: Contrastive
learning of disentangled seasonal-trend representations for time series forecasting. In International
Conference on Learning Representations, 2022. URL BtLLhtts::// http://///ppnreeee......o net.:/wwwwwweeeeee............ee forum?
id=P112Y3omXV2
Jing Wu, Jennifer Hobbs, and Naira Hovakimyan. Hallucination improves the performance of unsupervised visual representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16132-16143, 2023.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in neural information processing systems, 33:6256-683 6268, 2020.
Dongkuan Xu, Wei Cheng, Dongsheng Luo, Haifeng Chen, and Xiang Zhang. Infogel: Information- aware graph contrastive learning. Advances in Neural Information Processing Systems, 34: 30414-30425 2021.
Ling Yang and Shenda Hong. Unsupervised time-series representation learning with iterative bilinear temporal-spectral fusion. In International Conference on Machine Learning, pp. 25038-20222 25055. PMLR, 2022.
Yihang Yin, Qingzhong Wang, Siyu Huang, Haoyi Xiong, and Xiang Zhang. Autogel: Automated graph contrastive learning via learnable view generators. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 8892-8900, 2022.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. NeurlPS, pp. 5812-5823, 2020.
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. Bringing your own view: Graph contrastive learning without prefabricated data augmentations. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pp. 1300-1309, 2022.
Chengqing Yu, Fei Wang, Zezhi Shao, Tao Sun, Lin Wu, and Yongjun Xu. Dsformer: A dou- ble sampling transformer for multivariate time series long-term prediction. arXiv preprint Stti:22308.02274 2023.
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 8980-8987, 2022.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. In SIGKDD, pp. 2114-2124, 2021.
Wenrui Zhang. Ling Yang. Shijia Geng, and Shenda Hong. Self-supervised time series representation learning via cross reconstruction transformer. IEEE Transactions on Neural Networks and Learning Systems, 2023a.
Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequeet frequency consistency. Advances in Neural Information Processing Systems, 05:3988-4003, 2022.
Zhiqiang Zhang, Yuxuan Chen, Dandan Zhang, Yining Qian, and Hongbing Wang. Ctfnet: Long- sequence time-series forecasting based on convolution and time-frequency analysis. IEEE Trans- actions on Neural Networks and Learning Systems, 2023b.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI, 2021.
13

---

Published as a conference paper at ICLR 2024
APPENDIX: PARAMETRIC AUGMENTATION FOR TIME SERIES CONTRASTIVE LEARNING
A
NOTATIONS
Important notations are summarized in Table
Table 5: Notations and their meanings.
Symbol
Meaning
I,x
Time series
r",
x
Informative part of
Ar,
Ax
Task-irrelevant part of x Augmented view Informative part of u Task-irrelevat noise Downstream task label
V,
V
V
Av. U,
Av
y.y
T
Length of time series Number of features of time series
F
Dimensions of hidden representations
Encoder function
Transformation function
Factorization function
Augmentation function
H()
Entropy
MI(A))
Mutual information Factorization mask Transformation mask i-th element of h
h
g
14
h4
Intermediate result from concrete distribution Parameter in Bern. distribution Random variable from uniform distribution Temperature in Concrete distribution
m
Ç
Hyper-parameters in hard concrete distribution
A batch/set time series instances
p,
Anchor, positive and negative time stamp
8,
Trade-off
M
Hyper-parameter for training
B DETAILED PROOFS
Property 1. view is generated from with invertible function g(x). Then H(v) = H(x) MI(x,v), where Hx)), (v)) are entropy of variables x and v, respectively; MI(v; x) is the mutual information between and x.
Proof. Since g is invertible function and g(x), have one-to-one mapping between variables v and x. Thus, g(x) for each pair of x and v. We have TPxx x] = Plv = v). From the definition of Shannon entropy, have
(x)) Σ logp(() -
Σ Plx 2 lPg x]]
Σ P(y w) log v) = =-E-pte Ep(e log)(( logp(v)
H()..
14

---

Published as a conference paper at ICLR 2024
From the definition of conditional entropy, we have
H(xlv)= = Σ p(u, x))
log
p(u)
p(u)
(xlvv) = Σ p(v) log
= O.
p(v)
The above results in the mutual information between v and x, given by
MI(v:x) = H(x) (xlv)) = H(v).
Property 2 (Task agnostic label preserving). If a variable a good view ofx, and the downstream task label (although not visible to training) is independent to noise in x, the mutual information between and y is equivalent that between raw input and y, i.e., MI(v.y) Ml
Proof. From the definition of the good view, we have
x = + Ax v= ==g(xx)) Av).
We first analyze the relationship between MI(x, y) and MI(X", yy)
MI(x. y) = Hy) - (ylxx)
= Hy + Σ p(z, v) log p(x, v)
p(x)
2.y
p(#", Au, v)
=H(y)= + Σ p(#", Ax,y) log
p(x", A.)
Hy) + Σ p( ,r, v) log pl ((Ax, y x+)
p( xx
AA,9
With the safe independence assumption, we have
p(Ax, u|2²)
Thus, show that
Ml(x,) H(y) + Σ p(x", Az. y) log p(x", v)
plx*)
x* A...y
= HH(y) + Σ p(a",vl log p(x"
plz*)
z'.y
Hy) - Hlylx")
= +II(X",, .y).
Letting v" =x(",, from the Property we have
MI(x", y) = MI(v",
Since Av is a random noise and is independent to the label y, similarly, we have
MI(v,y) MI(n(v", Au),y) MI((V, Av),y)
MI((",))
Combining them together results
MI(v.y) = MI(x.y).
15

---

Published as a conference paper at ICLR 2024
Property 3. (Containing more information). A good view contains more information comparing the raw input x, i.e,, H(v) 2 (x).
Proof. Since Av denotes the included random noise, we assume that its generation is independent of the augmented view v*. Thus we have
MI v*) = O.
(11)
Further, with our decomposing model, we can rewrite the entropy of as the joint entropy of x and Ax. Formally, we have
H(x) = [x"x, Ax) H(x)) H(Ax) MI(Axxx").
Then (x*) H (vv') holds (Property 1). From the definition of the good view, we have H/ ( 2 H (Ax). Thus, we have
(()) = (x") + H(x)) MI(Ax,
<H(v") + (Av)
= =H(v") + MI(A, v*)
=H(Av, =H(r((,,,)))
=H(v).
Derivation of Eq. (7) As described in Section 3.4 the factorization mask h is generated with a hard concrete distribution. Thus, the number of non-zero entries in h can be reformulated with
|h/l PL. (0))
Phe
where Ih,, (0) the cumulative distribution function (CDF) of h, (before clipping). We let S(.) be an affine function of the stretch process in Eq. (5), such that
he s(h.) = h. - 7) + +,
where 7 E (-o0, and Ç E (I, oo). As derived in (Maddison et al. 2017). the density of h2 is
Pn. (h+) Tach;" -r- (arh; + (1 h4) -)2
where 04 log 1-m:" The CDF of h4 reads
Pr, (he) a(log he log(1 h2))T 7 ore)
Thus, the probability density function of h4 is
Ph. (ha) Phi (S (S- (he))
s s-1(
0h:
(C (5- - - h2) -
(ar(he 7) 1 + h:r) -7)2
The CDF of h, is given by
Ph. (h+) = Př, (S-1(h)))
(log((h 7) log(s - h2))T - at).
When setting hr have the
Ph. (0) a(T log -7 -v))
16

---

Published as a conference paper at ICLR 2024
C IMPLEMENTATION DETAILS
C.I TRAINING THE ENCODER WITH LOCAL AND GLOBAL CONTRASTS.
Similar to the augmentation network, our method can work with different architectures. We formulate the feature extraction encoder as f R7xF R", where D is the dimensionality of output embeddings. Following existing work (Luo et al. 2023). we use both global and local contrastive losses.
Global contrast aims to improve the inter-instance robustness for representation learning. Given batch of time series instances X. for each instance I E X, we generate an augmented view v. Such a pair of the original instance and the corresponding view pairs of instances and views are treated as negative pairs. Formally, (x. o') is a negative pair, where is an augmented view æ' and x' A z. Following (Chen et al. 2020), we use the InfoNCE as the global-wise contrastive loss to train the encoder network. Formally, we have
then used as a positive pair. Other
exp(sim(Zz: z.))
Lg
log
(12)
Ev'ex exp(sim( (2,, Znr)
where 2x f(x), Zu f(v), Fos f(z)),ze f(v') are representations of u,v, xf and v',
respectively.
Local contrast is designed to enhance the encoder network to capture the intra-instance relationship. Given an augmented view v, we first segment it into set of subsequences S, where each subsequence s E S has length L. Following (Tonekaboni et al. (222),, two close subsequences (s. p) are considered positive pair, and the ones with large distance lead to negative pair. Formally, the loss of local contrast is:
exp(sim(22s zp))
C1
log exp(sim(z., zp)) + ΣJEN, exp(sim 2j))
(13)
where N₂ the set of negative pairs for a subsequence 8. 2, = [()), 2p = (p)) 2, = s(n) are representations of 8, and n generated by function f, respectively. Considering all instances in batch, we have
Li
G
(14)
X
With both local and global contrasts, we have our contrastive loss as follows.
Leo Ly + aLi.
(15)
where a is the hyper-parameter to achieve the trade-off between global and local losses.
C.2 TRAINING ALGORITHM
In the training stage, AutoTCL optimizes the augmentation network and encoder network simul- taneously. Similar to GAN (Goodfellow al., 2016), these networks were randomly initialized. Different from GAN, AutoTCL is less affected by the problem of gradient explosion and mode collapse, because our encoder network aims embed the information part from different views rather than distinguish them. Although our argumentation network tries to reduce the distribution between original instances and arguments, AutoTCL augmentations preserve the information part by using a reversible mapping function, which alleviates the mode collapse problem. Our training algorithm is shown in Alg.
D EXPERIMENTAL SETTINGS
All experiments conducted Linux machine with NVIDIA A100 GPUs, each with 40GB of memory. The software environment is CUDA 11.6 and Driver Version 520.61.05. We used Python 3.9.13 and Pytorch 1.1 construct project.
17

---

Published as a conference paper at ICLR 2024
Algorithm 1 AutoTCL training algorithm
Require: augmentation network faug. encoder network fenc, epochs E, hyperparameter M,
epoch 6-
while epoch E do
for in dataset do
Za - [mg[(
- Jane
Za - fene
if epoch%M == 0 then
Compute loss with using Eq. (10)
Update parameters in fang with backpropagation
end if'
Compute loss with using Eq. 15
Update parameters in fene with backpropagation
end for
epoch <- epoch + end while
D.1 BASELINE SETTINGS
In forecasting tasks, conducted baseline methods six benchmark datasets by following the experiment setting of TS2 Vec Yue et al. 2022) for most baseline methods, such as Informer (Zhou
al. 2021). (Tonekaboni
al. 202 StemGNN (Cao et al., 2020). TCN (Bai al. 2018).
N-BEATS (Oreshkin et al. 2020). etc. For Vec Yue et al. (022), CoST (Woo et al. (022), we followed its code default setting for Lora and Weather datasets. The representation dimension was 320 and the learning rate and batch size were 0.001 and 8. For InfoTS (Luo et al.) 2023). We used the default setting conduct experiments. As for TS-TCC (Eldele et al. 2021) forecasting tasks, we used the Epilepsy config as the default config and modified the network model to make the input and output channels remain the same. Due to its pooling layers, the network would require times the lengths of inputs of other baselines which is unfair for forecasting tasks. In the experiments, we used another interpolate layer to make the length of input data and prediction data the same. In classification tasks, similar to the forecasting task, we followed the experiment setting of TS2Vect Yue jt al.) 2022). In TF-C (Zhang et al., 2022) classification experiments, we use its HAR config as the default setting. Similar to TS-TCC, modifie the network so that the transformer encoder could fit the input length and the pre-train dataset is the same as the fine tune dataset.
D.2 HYPERPARAMETERS
In our experiments, we used grid search to obtain the best performance. We used the same strategy in forecasting and classification tasks that each dataset had its own group of hyperparameters. We provided all of the hyperparameters as well as their configurations in the following:
Optimizer: Two Adam optimizers (Kingma & Ba 2014) were used for the augmentation network and feature extraction network with learning rate and other hyperparameters were setting with default decay rates setting to 0,001 and (0.9,0.999) respectively.
The depth of the multi-layer dilated CNN module and the hidden dimension were designed to be able to change, which were searched in {6, 7, 7,8,9. 9, 10) and (256, 128, 64, 32, 16, 8). training, we used designed dropout rate avoid overfitting, which was tuned in [0.01, 1].
Encoder architecture:
Augmentation architecture: Same as encoder, the depth of multi-layer dilated CNN module and hidden dimension hyperparameters, searched in {1, 3, and (256, 128, 64, 32, 16, 8) and as mention in equation Eq. 5 is another hyperparameter, tuned in [0.0005, 0.5).
Trade-off hyperparameters: /3 in Eq. and in Eq. 10 are tuned in [0, 0.3].
Alternating training hyperparameters: M in Sec. (3.4 is tuned in 21, 16, 32).
18

---

Published as a conference paper at ICLR 2024
D.3 EXTRA EXPERIMENTS
D.3.1 PERFORMANCE WITH DIFFERENT BACKBONES
As general framework for time series contrastive learning, AutoTCL can be used plug-and-play component to boost performance. To further verify the generalization capacity of Auto Table 6 shows the full results of ablation studies and model analysis with CoST as the backbone. In addition, we adopt Ts2vec (uue et al., 2022) as the backbone and show the results in Table 7 We can draw similar conclusions that by adaptively selecting the optimal augmentations with the principle of relevant information, AutoTTCI. can outperform the vanilla TS2vec and other baselines.
Table 6: Ablation studies using CoST backbone
AmtoTCL
h(z)
Woo g(x)
Wio
Aug
Cutcut
Jiter
Adversarial
Random Aug.
Dataset
MSF
MAR
MSE
MAP
MSR:
MAR
MSF
MAF
MSH
MAF
MSF
MAF
MSF
MAE
MSR
MAF
MSE
MAF
24
0.037
0.148
0.037
0.148
0.037
0.148
0.038
0.149
0.037
0.148
0.037
0.147
0.038
0.147
0.039
0.149
0.056
0.178
0.054
0.176
0.176
0.054
0.177
0.180
0.053
0.175
0.176
0.056
0.076
0.208
KTTh
168
0.0778
0.210
0.079
0.210
0.080
0.211
0.083
0.217
0.100
0.237
0,078
0.210
0.081
0.212
0.090
0.227
0.132
0.278
336
0.093
0.231
0.093
0.251
0.094
0.232
0.096
0.234
0.108
0.251
0
0.230
0.095
0.235
0.106
0.250
0.137
0.289
0.120
0.272
0.274
0.124
0.277
0.157
0.317
0.175
0.340
0.179
0.345
0.325
0.152
0.313
0.157
0.318
24
0.079
0.206
0.077
0,204
0.076
0.205
0.079
0.209
0.078
0.208
0.076
0.204
0.092
0.217
0.078
0.207
0.090
0.229
48
0.117
0.255
0.124
0.158
0.113
0,256
0.118
0.259
0.127
0 0.29
0.110
0.253
0.135
0.272
0.124
0.265
0.125
0.272
ETT|2
0.176
0.191
0.329
0.212
0.358
0.220
0.191
0.340
0.356
0.227
0,361
0.331
336
0.193
0.344
0.201
0,350
0.243
0.371
0.204
0.349
0,200
0.357
0.201
0.355
0.212
0.366
0.253
0.375
0.207
0.368
720
0.223
0.373
0.246
0.384
0.246
0,380
0.238
0.379
0.221
0.374
0.220
0.376
0.217
0.374
0.251
0.385
0.245
0.408
0.016
0.091
0,087
0.015
0,090
0.015
0.089
0.013
0,017
0.092
0.015
0.091
0.015
0,092
0,020
48
0.026
0.120
0.025
0.119
0.025
0.117
0.027
0.122
0.024
0.116
0,028
0,123
0.027
0.124
0.028
0.125
0.032
0.138
ETTM:
96
0,036
0.145
0
0.146
0.038
0,146
0.099
0.150
0.036
0.144
0.099
0,150
0.043
0.158
0.040
0.151
0.045
0.165
0.063
0,191
0.205
0.072
0,204
0,072
0.205
0.080
0.216
0.078
0.211
0.218
0.075
0,205
0,093
0.238
672
0.090
0.225
0.108
0,250
0.098
0.239
0.104
0.248
0.114
0.248
0.104
0.246
0.112
0.260
0.100
0.240
0.134
0.285
24
0,240
0.266
0.244
0.266
0.241
0.267
0,242
0,264
0.243
0.272
0.241
0.264
0.240
0.264
0.242
0.265
0.251
0.279
0.285
0..994
0.291
0.295
0.285
0.295
0.294
0.290
3.000
0.286
0.291
0.284
0.242
0.287
0.309
Eloe.
168
0.392
0.371
0,400
0,371
0.392
0,366
0.394
0.367
0.398
0.372
0.394
0.365
0.395
0.362
0.393
0,364
0,413
0.383
336
0,542
0.461
0.547
0.465
0.541
0.464
0.542
0.461
0.541
0.470
0.545
0.460
0.545
0.457
0.539
0.457
0.541
0.460
0,093
0.211
0.220
0.209
0.207
0.066
0,092
0.095
0.092
0.209
48
0.131
0.256
0.139
0,266
0.130
0.255
0.127
0.250
0.141
0.266
0.129
0.252
0.134
0,260
0.131
0.257
0.142
0.266
WTH
168
0,182
0.311
0.194
0.324
0.185
0.314
0,184
0.316
0.208
0.336
0.186
0.315
0.195
0.327
085
0.315
0.207
0.335
336
0.195
0.325
0,210
0.342
0.208
0,342
0,206
0.341
0.231
0.357
0.209
0.339
0.215
0.349
0.203
0.535
0.225
0.355
0.198
0.330
0.218
0.355
0.217
0.353
0.211
0.349
0.240
0.369
0.220
0.352
0.370
0.218
0.352
0.244
0.371
24
0.052
0.141
0.000
0.152
0.138
0.219
0.128
0,313
0.078
0.171
0.067
0.170
0.158
0.225
0.057
0.154
0.070
0.185
0.080
0,181
0.092
0.195
0.117
0,215
0.181
0.264
0.127
0.233
0.112
0,218
0.185
0.257
0.084
0.189
0.096
0.218
Lora
0.155
0.263
0.246
0.317
0.196
0.308
0.334
0.481
0.393
0.676
0.433
0.311
0.359
0.235
0.169
0.290
336
0.229
0.335
0.302
0.372
0.395
0.444
0.363
0.435
0.941
0.532
1 1
0.619
0.378
0.429
0.535
0.475
0,245
0.353
720
0.370
0.445
0.453
0.509
1.60
0.729
0.617
0.561
1.926
0.739
1,655
0.771
0.395
0.461
1.315
0,722
0.375
0.450
Avg.
0.157
0.258
0.173
0.270
0.216
0.282
0.185
0.280
0.260
0.294
0.266
0.394
0.184
0.281
0.212
0.284
0.176
0.286
D.3.2
VISUALIZATION OF AUGMENTATION
The intuitive understanding of Property 3 is that if the map between x and v is one-to-many, the generated view will contain more information compared to the raw input x. In other words, good augmentation should preserve the underlying semantics that two different x cannot map the same v. In order to further explore the effectiveness of Auto in generating diverse and mmmantic-preserviie views, we used T-SNE to visualize the embeddings of different augmented views in Figure 2 We chose an instance, denoted by x, from dataset ETTh; and compare different augmentation methods, including Cutout, Jitter, and Adversarial. To avoid the special case, we reported 10 augmented views for TCL. We also include another x' instance as a reference. As shown in Figure 2 the instances augmented by AutoTCI. include more diversity compared with Jitter and Cutout. Moreover, the augmentation generated by the Adversarial is closer to x or x, indicating that it fails to preserve the underlying semantics.
D.3.3 VISUALIZATION OF CONVERGENCE
To show the convergence of our method, we plotted the curves of Eq, 10) and Eq. (1 on different datasets. As shown in Figure 3 our method converged easily in both the argumentation network and the embedding network. In Figure ((a) and 3(d observed that after the argumentation network converged to a certain level, the encoding network still benefited from that. In Figure 3(b) 3(c] and
3(e) they have the same patterns that the augmentation loss arrived the convergence level almost the
19

---

Published as a conference paper at ICLR 2024
Table 7: Ablation studies using Ve2 backbone
AutoTCI.
Wio M(x)
gfag
Wio
AV
Wio
Aug
Carous
Jitter
Adversriaa
Random Aug.
Danaset
MSF
MAE
MSE
MAE
MSB
MAR
MSE
MAE
VSE
MAE
MSB
MAR
MSE
MAR
MSE
MAE
MSE
MAF
0.0039
0.146
0.039
0.148
0.047
0.165
0.047
0.164
0.039
0.149
0.043
0.155
0.040
0.150
0.041
0.151
0.047
166
48
01055
0.150
0.060
0.185
0.075
0.212
0.073
0.205
0.963
0.190
0.063
0.187
0.063
0.190
0.062
0.186
0.069
0.203
0.106
0.245
0.259
0.264
0.255
0.280
336
0.121
0.266
0.139
0.289
0.159
0.317
0.148
0.303
0.141
0.291
0.133
0.285
0.130
0.277
0.132
0.280
0.140
0.300
720
0.154
0.314
0.181
0.347
0.198
0.365
0.180
0.344
0.193
0.359
0.167
0.331
0.164
0.323
0.167
0.330
0.171
0.342
0.106
0,095
0.252
0.246
48
0.131
0.284
0.134
0.285
0.129
0.279
0.136
0.289
0.140
0.290
0.133
0.287
0.135
0.287
0.137
0.288
0.129
0.281
ETTha
168
01182
0.343
0.185
0.344
0.212
0.365
0.211
0.366
0,203
0.360
0.198
0.355
0.194
0.353
0.195
0.353
0.185
0.344
336
0.190
0.351
0.191
0.351
0.205
0.362
0.209
0.366
0.206
0.367
0.204
0.363
0.201
0,362
0.199
0.360
0.201
0.362
0.214
0.570
0.204
0.368
0.203
0.366
0.200
0.364
0.205
0.205
0.367
0.200
0.364
0.194
0.359
0.234
0.329
24
0.014
0.085
0.018
0.098
0.015
0.089
0.014
0.087
0.014
0.087
0.015
0.089
0.014
0.087
0.014
0.085
0.021
0.109
0.026
0.117
0.027
0.121
0.028
0.123
0.026
0.0
0.027
0.121
0.027
0.121
0.027
0,121
0,026
0.117
0.036
0.144
HTTm;
0.038
0.147
0.83
0.149
0.099
0.147
0.041
0.183
0:153
0.041
0.153
0.038
0,147
0,38
0.147
0,050
0.171
288
0.081
0.216
0.083
0.219
0.062
0.216
0.084
0.222
0.084
0.222
0.084
0.222
0
0.215
0.081
0.216
0.106
0.251
672
0.119
0.363
0.123
0.269
0.122
0,266
0.124
0.271
0.124
0.270
0.121
0.267
0.120
0.265
0.119
0.363
0.176
0.529
0..447
0.369
0.249
0.271
0.248
0,269
01.447
0.770
0,250
0.271
0.248
0.270
10.249
0.273
0,250
0.770
0,251
0.277
48
0,297
0.M01
0.902
0. 0.306
0.297
0.301
0.297
0.303
0.200
3020
0.296
0.302
0.297
0.307
0.25%
M12
0.393
0.303
Eloc.
168
0,408
0.380
0.413
0.381
0,408
0,380
0.410
0.580
0,408
0,377
0,408
0.383
0.410
0,584
0,408
0.377
0,405
0.372
0.541
0.472
0.547
0.542
0.471
0.470
0.542
0.471
0.458
24
0.093
0.212
0.096
0,214
0.0944
0.210
0,056
0.214
0.094
0.211
0.099
0.215
D.096
0.213
0.056
0.213
0.104
0.222
48
0.133
0.258
0.134
0.259
0.130
0.253
0.134
0.258
0.134
0.257
0.132
0.256
0.135
0.259
0.133
0.251
0,142
0.268
0.188
0.316
0.322
0.192
0.324
0.317
0.192
0.321
0.322
0.195
0.323
336
0.201
0.333
0.208
0.341
0.202
0.335
0.2008
0.341
0.216
0. 0.347
0.208
0.338
0.211
0.342
0.212
0.344
0.209
0.340
720
0.204
0.339
0.210
0.347
0.204
0.339
0.210
0.347
0.225
0.357
0.211
0.343
0.220
0.353
0.211
0.352
0.211
0.344
0,053
0.156
0.169
0.061
0.156
0:219
0.158
0.080
0.173
0.157
0.177
48
0.082
0.188
0.087
0.193
0.136
0.229
0.086
0.193
0.296
0.285
0.086
0.192
0.155
0.238
0.088
0.195
0.115
0.222
Lora
168
0.161
0.278
0,210
0.306
0.210
0.308
0.344
0.360
0.341
0.371
0.188
0.294
0.229
0,318
0,219
0.310
0.219
0.321
0.231
0.299
0,591
0.420
0.281
0.438
0.401
720
0.375
0.451
0.369
0.447
0.400
0.473
0.370
0.428
0.484
0.482
0.367
0.446
0.443
0.493
0.364
0.454
0.465
0.512
Avg.
11165
0.271
0.179
0.280
0.178
0,284
0.180
0.283
0.199
0.291
0.175
0.279
0.176
0.284
0.179
0.279
0.185
0.292
Orignal Another Instance
1120
Bther Adversorisl AUBOTCL
50-
-100
-00
Jiter
220
2
-50-
Adversarial
AutoTEL
240
-35
-25
-30
-30
-80
-00
-20
(a) Samples
(b) Samples b
Figure 2: T-SNE visualization of different augmentation instances. In samples and b, AutoTCL- generated samples are closer to the original instance x than other instances x' with large variety
same as the contrastive loss. While the situation was different in Figure 3(t the beginning the augmentation network benefited from encoding loss, then two losses converged gradually.
D.3.4 PARAMETER SENSITIVITY STUDIES
In the proposed AutoTCL, we have three hyper-pann paper-parameter, in Eq. 15 Eq. (10). to get the trade-off in training the augmentation network and the encoder network. this part, we chose different values for these three variables in the range from 0.0001 to 0.3 and reported MSE and MAE scores in the ETTh1 dataset. The results of this part could be found in Figure 4 The sensitivity studies result of three results could be observed that our method is able to achieve comparative performances with a wide range of choices for these three hyperparameters, indicating the robustness of our method. The / in
/3 in Eq. 8 and y in
are shown in Figure 4 From this figure, some
20

---

Published as a conference paper at ICLR 2024
30
augmentation
augmentation contrastive loss
25
contrastive
3.0
2.0
1.5
1.0
0.5
0.0
100
250
100
100
600
700
400
000
1000
170h
(a) ETTh
(b) ETTh2
augmentation contrastive
3.0
augmentation loss contrastive loss
1.
500
1000
200
300
650
(c) ETTm:
(d) Elec.
augmentation loss
contrastive loss
augmentation loss
contrastive loss
WTH
Lora
Figure 3: The augmentation loss, Eq. (10) and contrastive loss, Eq. (15). in the training process
21

---

Published as a conference paper at ICLR 2024
Eq. 8 and in Eq. (TO) have the opposite effect the weight goes up. Second, observe that
small values, such as 0.001, give good performances on ETTh! datasets as well as others.
0.081
B
0.080
MSE
0.079
0.078
0.077
1E-4
3E-4
1E-3
3E-3
1E-2
3E-2
1E-1
3E-1
(a) MSE
0.214
B
0.213
0.212
MAE
0.211
0.210
0.209
0.208
1E-4
3E-4
1E-3
3E-3
1E-2
3E-2
1E-1
3E-1
(b) MAE
Figure 4: Parameter sensitivity studies ETTh;-
D.3.5
CASE STUDY
To further explore the augmentation of AutoTCL, we have done the case study in this section. We selected four instances to show the effectiveness of our method in Table As shown in Table we used the CricketX dataset as input instances and got the informative part by using the augmentation network to get masks, the result of l(z)). With the regularization loss help in Eq. ( our method could have a continuous mask that makes the informative part more consistent. From the results, the informative parts detected by h() appear retain the prominent peaks and troughs of the original sequence, which are typically considered significant by domain experts. In time series analysis, such prominent features often correspond critical events changes in the underlying system's behavior.
D.3.6 EXPERIMENTS ON OTHER DOMAIN DATASET
In order to further verify the adaptability of our method, we conducted experiments on Traffic dataset. This dataset comprises hourly information sourced from the California Department of Transportation.
https: pettps:/peet //mmss../////..///// dot. ca /
22

---

Published as a conference paper at ICLR 2024
This dataset delineates road occupancy rates observed by various sensors deployed the freeways in the San Francisco Bay area, Following the default setting, we adopt CoST as the backbone and conduct forecasting in both univariate and multivariate forecasting settings. We also include another SOTA method, TS2Vec as comparison. The results are shown in Table 8 We observe that, in both univariate and multivariate forecasting settings, AutoTCL achieves the best results in all prediction lengths. On average, AutoTCL decreases MSE by 9.4%, MAE by 5.7% in the univariate forecasting setting and MSE by 5.0%, MAE by 9.3% in the multivariate forecasting setting comparing the baseline.
Table 8: Forecasting results Traffic dataset.
AutoTCL
TS2Vec
CoST
Settings
Ty
MSE
MAE
MSE
MAE
MSE
MAE
96
0.253
0.353
0.431
0.484
0.284
0.379
192
0.271
0.373
0.437
0.489
0.302
0.398
Univariate
336
0.312
0.414
0.453
0.500
0.340
0.435
720
0.357
0.447
0.464
0.508
0.390
0.474
Avg.
0.298
0.397
0.446
0.495
0.329
0.421
96
0.715
0.396
1.038
0.574
0.759
0.442
192
0.722
0.396
1.042
0.588
0.757
0.434
Multivariate
336
0.730
0.396
1.064
0.594
0.765
0.435
720
0.746
0.403
1.085
0.604
0.784
0.444
Avg.
0.728
0.398
1.057
0.590
0.766
0.439
D.4 FULL EXPERIMENTS
Univariate forecasting. Full experiment results of univariate time series forecasting results can be found in Table TO In these experiments, AutoTCL achieved minimum error in most cases. Compared to the state-of-the-p CoST method, AutoTCL reduced the average MSE error by 6.5% and the average MAE by 4.8%.
Multivariate forecasting. We provided our full experiment results of multivariate time series forecasting results in TT In multivariate forecasting tasks, our method achieved fewer best results than univariate forecasting. AutoTCL reduced the average MSE error by 2.9% and the average MAE by 1.2% than CoST. In the column of stemGNN, because of the error out-of-memoon we can't t report part results.
Classification. In Table 12 the full results of 30 class datasets are provided. AutoTCL is the most powerful method than other baselines with the highest average accuracy rate and ranks. Some results not available due to out-of oft-or-memory errors.
23

---

Published as a conference paper at ICLR 2024
Table 9: Case study of parametric augmentation. The inputs are from the CricketX dataset, which is a univariate time series dataset. We demonstrate the informative parts in two settings, w/ and w/o (7 0) regularization loss. Input
h(
MuS
0.2
0.0
0.0
0.2
Amma
0.0
Awm
0.2
24

---

Published as a conference paper at ICLR 2024
Table 10: Univariate time series forecasting results.
ToI
TS2Vec
Informer
LogTrans
N-BEATS
TCN
CoST
TNC
TO-TOC
InfoTS
Dateset
MAE
MSE
MAE
MSE
MAR
MSE
MAE
MSE
MAE
MSE
MSE
MAE
MSE
0.148
0.039
0:152
0.259
0.015
0.210
0.040
0:152
0.237
0.149
0.176
0.0K2
0:191
0.158
0.367
0.328
0.210
0.227
0.402
0.000
0186
0.139
0.279
0.05
0.179
ETTT:
168
0.07%
0.210
0.134
0.282
0.183
0.346
0.207
0.375
0.232
0.391
0.316
0.433
0.097
0.236
0.171
0.379
0.253
0.408
0.100
0.230
136
11,093
0.231
0.154
0.310
0.222
0.387
0.230
0.308
0.232
0.388
0.306
0.4%5
0.112
0.25%
0.192
0.357
0.155
0,318
0.117
0.264
720
0.120
0.272
0.163
0.327
0.269
0.435
0.273
0.463
0.322
0.190
0.390
0.557
0.148
0.306
0.235
0.408
0,190
0.337
0.141
0.302
24
0.019
0.206
0,090
0.229
0,093
0.340
0.102
0.255
0 198
0.345
0.103
0.349
0.079
0.207
0.097
0.238
0.259
0.391
0.081
0.215
0.117
0.255
0.124
0.273
0.155
0.314
0.169
0.348
0.234
0.386
0.142
0.200
0.118
0.259
0.131
0.281
0.260
0.605
0.115
0.261
ETTb,
0.319
0.208
0.300
0.232
0.389
0.246
0.422
0.331
0.376
0.189
0.339
0.197
0.291
0,420
0.171
0.195
0.364
0.213
0.309
0.263
0.267
0.437
0.508
0.296
0.430
0.206
0.360
0.207
0.336
0.453
0.183
0,241
720
0.373
0.214
0.574
0.277
0.303
0.493
0.437
0.517
0.325
0.46
0.314
0.371
0.207
0.362
0.472
0.194
0.357
0.016
0.091
0,015
0,092
0,030
0.137
0,065
0.202
0.054
0.184
0.011
0.157
0.155
0.08K
0.019
0.103
0.089
0.228
0,014
0,087
0.026
0.120
0.022
0.126
0.009
0.203
0.078
0.220
190
0.361
0.101
0.257
0.025
0.117
0.056
0.142
0.134
0.250
0.117
ETTm;
96
0.0136
0.145
0,044
0.161
0.194
0.572
0.199
0.356
0. 183
0.151
0.142
03111
0.038
0.147
0.054
0.178
0.159
0.305
0/136
0.142
288
0.0663
0.191
0.103
0.246
0.401
0.554
0,11
0.572
0 186
0.362
0.318
0472
0.077
0.209
0.098
0.244
0.204
0.327
0.771
0.200
672
0.090
0.225
0.156
0.307
0.512
0.544
0.598
0.702
0.197
0.368
0.397
0.547
0.113
0.257
0.136
0.290
0.206
0.354
0.102
0,240
0.262
0.250
0.288
0.279
0.343
0.287
0.292
0.319
0.324
0.346
0.414
0.34
0.292
0.300
0.300
0.453
0.600
0,301
Elec.
0.394
0.365
0.427
0.394
0.544
0,434
0,599
0,612
0.893
0.009
0.622
0.405
0.375
0.412
0.384
0:575
0.616
0,402
0,267
336
0.543
0,460
0,565
0,474
0.713
0.512
1,079
0.639
1.035
0.069
0.855
0.006
0.560
0.475
0.548
0.466
0.637
0,633
0.533
0.453
24
0.093
0.211
0.096
0,215
0.117
0.251
0.136
0.279
0 0.176
0.264
0.
0.217
0.096
0.213
0.102
0.221
0.221
0.386
0.101
0.223
48
0.131
0.256
0.140
0.264
0.178
0.318
0.206
0.356
0.198
0.119
0.143
0.269
0.138
0.262
0.139
0.264
0.255
0.406
0.14]
0.266
WTH
168
0.182
0.311
0.207
0.335
012666
0.398
0.309
(1439
0.309
0.420
0.
0.319
0.007
0.394
0.198
0.328
0.389
0.458
0.199
0.328
336
0.195
0.325
0.251
0.360
0.297
0.416
0.359
0.484
0.369
0.460
0.192
0.320
0.230
0.356
0.215
0.347
0.372
0.49]
0.220
0.351
0.253
0.365
0.350
0,499
0.32V
0.342
0.322
0.141
0.212
0.268
0.917
0.730
0.264
0.371
0.072
0.170
0.811
0.899
0.533
0.144
0.206
0.365
0.514
0.149
0.080
0.181
0.267
0.516
1.067
0,786
0,364
0.44
0111
0.981
0.888
0.082
0.184
0.286
0.349
0.426
0.562
0.090
0.192
Lora
168
0.155
0.263
0.355
0.389
1,745
1,067
0,422
0.465
0.286
0.350
1.266
0.96
0.166
0.274
0.523
0.589
0.181
0.587
0.156
0.267
336
0.229
0.335
0,425
0.441
1.661
1,050
0.950
0.083
0.405
0.429
1.273
0.945
0.522
0.355
0.772
0.724
0.588
0.645
0.313
0.3%6
720
0.370
0.445
0,525
0,509
2,482
1,370
1,248
01307
0.679
0.573
1,290
0.950
0.379
0.451
1,313
0.929
0,592
0,649
1,047
0,635
Avg.
0.157
0,258
0.207
0.501
0,486
0,477
0.360
0,441
0.320
0.388
0.419
0.465
01168
0.271
0,256
0,340
0,315
0.441
0,168
0.274
Table 11: Multivariate time series forecasting results.
AutOTCL.
TSZVec
Infommer
Log Trans
StemGNN
TCN
COST
TNC
TS-TCC
InfeTS
fy.
MAE
MSE
MAE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0.439
0.599
0.577
0.589
0.686
0.571
0.767
0.708
0.592
0.520
0.447
0.427
0.629
0.555
0.685
0.625
0.766
0.757
0:748
0.618
0.713
0.617
0,437
0,464
0.749
0.619
0.644
0.579
0.607
0.553
BTTh
168
0.615
0.574
0.755
0.636
0.931
0.752
1.002
0.846
0.663
0.608
0,995
0.738
0.643
0.582
0.884
0.699
0.678
0.619
0.746
0.63N
336
0
0.671
0.907
0.717
1.128
0.873
1.362
0.952
0.927
0.730
1.175
0.800
0.813
0.679
1.000
0.768
0.667
0.754
0.904
0.722
720
1.008
0.789
1.018
0.790
1.215
0.896
1.397
1.29]
1/453
1,311
0.970
0.711
1.157
0.830
0.935
0.715
1.098
0.811
24
0.337
0.433
0.398
0.461
0.720
0.665
0.828
0.750
1.292
0.883
1.365
0.888
0.447
0.502
0.612
0.595
0.782
0.666
0.783
0.462
0.573
0.576
0.578
0.573
1.457
1.001
1,806
1,034
1,099
0.847
1.395
0,960
0.899
0.637
0.540
0.716
1.357
0.881
0.567
0.582
3.166
540
2.304
1.215
2.723
1,300
3,875
1.763
3,085
1.351
3.256
1,012
2,097
2.120
1.161
1.373
3.467
3,913
1,552
3,600
1,588
197
2.753
2.047
1.227
2.511
1,316
0.256
0.339
0.443
0.436
0.323
0.369
0.119
0,412
0.620
0.570
0.324
0.374
0.246
0.329
0.522
0.422
0.403
0.455
0.291
0.108
01350
(139%
0.592
0.515
0.494
0.505
0.507
0.585
0.744
0.628
0.477
0.450
0.331
0.386
0.095
0.567
0.618
0.522
0.503
0.475
BTTm
96
0.376
0.422
0.622
0.549
0.678
0.615
0.768
0.792
0.2099
0.624
11636
0.601
0.378
0.419
0.731
0.595
0.007
0.522
0.537
0.305
788
0.464
0.484
0.709
0.609
1056
0.786
1.462
1,320
0.843
0.683
1.270
1.351
0.473
0.486
0.818
0.649
0.732
0.688
0653
0.579
672
0.608
0.566
0.786
0.655
1.192
0.926
1.669
1,461
1.381
1,467
0.620
0.511
0.932
0.712
0.708
0.601
0.757
0.642
0.250
0.388
0.505
0.354
0.23
0,255
0.350
0.261
0.388
0.431
0.316
0.389
0.413
0.455
0.317
0.153
0,258
0.438
0.279
0.368
Bloc.
0.170
0.275
0.332
0.407
0.515
0.509
0.426
0.466
0.506
0.518
0.35%
0,423
1,175
0.275
0.402
0.46
0.575
0616
0,302
0.385
336
0.199
0.292
0.349
0.420
0.759
0.625
0.365
0,417
0.647
0.596
0.349
0,416
0.196
0.296
0.417
0.466
0.637
0.633
0.320
0.399
0.302
0.364
0.307
161
0.335
0.381
0.435
0.477
0.263
0.507
0.321
0.367
0.208
0.360
0.330
0.373
0.356
0.463
0.116
0.369
48
0.361
0.412
0.374
0.118
0.395
0.459
0.426
0.195
0.337
0.573
0.386
0.423
0.359
0.111
0.380
0.421
0.429
0.500
0.281
0.420
WTH
168
0.455
0.484
0.491
0.506
0.608
0.567
0.727
D.671
0,397
0.652
0.291
0.501
1.464
0
0.479
0.495
0.511
0.550
0.4%
0.501
0.487
0.505
0.525
0.754
0.620
0.394
0.639
0.501
0.517
0.505
0.514
0.527
0.556
0.731
0.885
0.775
0.49%
0.554
0.543
0.198
0.252
02122
0.267
0.376
0.345
0.456
0.161
0.373
0.854
012002
0.259
0.264
0.302
0.365
0.514
0.998
0.245
0.301
0.366
0.116
0.478
0.420
0.663
0.467
0.204
0.490
0.851
0.774
0.25
0,007
0519
0.345
0.476
0.563
0.254
0.297
Lora
168
0.346
0.277
0.354
0.389
0.734
0.597
0.682
0,510
0.270
0.536
1.118
0.839
0,300
0283
0474
0.477
0.481
0.587
0.45
0.374
336
0.414
0.428
04455
0.441
0.985
0.738
1.068
0.608
0.395
0.618
1.11]
0.836
D1417
0442
0625
0.588
0.588
0.645
D412
0.427
720
0.512
0.502
0.522
0.509
1.181
0.831
0.959
0.622
1.131
0.444
0.524
0.07
1226
0.376
0.292
0.649
0.514
0.501
Avg.
0.545
0.499
0.697
0.571
0.990
0,708
1.138
0.798
0.753
0.651
1.057
0,781
0.561
0.505
0.87
0.637
0.838
0.675
0.663
0.556
25

---

Published as a conference paper at ICLR 2024
Table 12: Classification result of the UEA dataset
Dataset
AutoTCL
TS2Vec
T-Loss
TNC
TS-TCC
DTW
TF-C
InfoTS
Articulary WordRecognition
0.987
0.943
0.973
0.987
0.467
AtrillFibellltioon
0.467
0.200
0.133
0.133
0.267
0.067
0.200
0.040
0.267
BasicMotions
1.000
0.975
1.000
0.975
L,00
0.975
0.975
0.475
1.000
Trajectriress
0.995
0.967
0.090
0.987
Cricket
1.000
0.972
0.972
0.958
0.917
1.000
1,000
0.125
1.000
DuckDecGGeese
0.700
0.680
0.650
0.460
0,380
0.620
0.600
0.340
0.600
Rigen Worms
0.847
Epilepsy
0,978
0.964
0.971
0.957
0.957
0.949
0.964
0.217
0.993
ERing
0.944
0.874
0.133
0.852
0.904
0.874
0.133
0.167
0,953
EdhanolConcentration
0.354
0,308
0.205
0.297
0.285
0,262
0.323
0.247
0.323
FaceDetection
0,581
0.501
0,513
0.536
0.544
0,534
0,529
0,502
0.525
HingeMovements
0.640
0.480
0.580
0.470
0.460
0.560
0.530
0.510
0.620
HandMoveemnnDiieetion
0.432
0.338
0.351
0.324
0.231
0.405
0.514
Handwriting
0,384
0.515
0.451
0.249
0.498
0.225
0.286
0.051
0.554
Heartheat
0.785
0.6K3
0.741
0.746
0.751
0.746
0.717
0.737
0.771
Jupanese Vowels
0,984
0,984
0,989
0.978
0.930
0.978
0,949
0.135
0.986
Librras
0,833
0.867
11883
0.817
0.822
0.656
0.870
0.067
0.889
LSST
0.554
0.537
0.509
0.595
0.474
0.408
0.551
0.314
0.593
MotorImagery
0.570
0,510
0,580
0.500
0.610
0,500
0.500
0.500
0.610
NATOPS
0.944
0.928
0.917
0.911
0.822
0.850
0.883
0.533
0.939
PEMS-SF
0.838
0.682
0.676
0.699
0.734
0.740
0.711
0.312
0.757
PenDigits
0.984
0.989
0.981
0.979
0.974
0.560
0.977
0.236
0.989
PhonemeSpectra
0,218
0.233
0,222
0.207
0.252
0.085
0.151
0.026
0.233
RacketSports
0.914
0.855
0.855
0.776
0.816
0.803
0.480
0.829
SelFRegulationSCPI
0.891
0.812
0.843
0.799
0.823
0.754
0.775
0.502
0.887
Self SelfRegulationSCC2
0.578
0,578
0,539
0.550
0.533
0.550
0.539
0.500
0.527
SpokenArabiciigits
0.925
0.932
0.905
0.934
0.970
0.963
0.100
0.988
StandWalJJump
0,533
0.467
0.333
0.400
0.333
0.267
0.200
0.333
0.467
UWaveGestureLibrar
0.893
0.884
0.875
0.759
0.753
0.575
0.903
0.125
0.906
Wingheeingeeat
0.466
0.156
0.469
0.264
0.108
0.472
Avg.
0.704
0.658
0.670
0.298
Avg. RANK
2.300
3,700
4.667
5.433
5.133
6.133
5,400
8,200
2.367
26


Reviews:
[{"review_id": "zEMu1JveJL", "reviewer": "Reviewer_3kKY", "summary": "The paper starts by making the following observation: time series data is complex, high-dimensional, and harder to label than images or languages.\nDeep learning requires lots of labeled data, but self-supervised learning offers a way to learn from unlabeled data. Applying general augmentation to diverse time series data is challenging; specific augmentations guided by domain knowledge are often needed.\n\nContributions:\n- The paper introduces a factorization-based framework, AutoTCL, for adaptive data augmentation in time series contrastive learning. AutoTCL uses a neural network to factorize time series instances, preserving semantics, and optimizing against a contrastive loss.\n- Empirical studies show the method outperforms benchmarks.", "questions": "- Could the authors please provide experimental results on the other usual datasets, Illness / Exchange rate / Traffic? \n\n- Could the authors please provide error bars/statistical tests for at least a subset of the experiments in tables 6/7? It is difficult to estimate whether the findings are statistically significant.\n\n- In table 4, Lora is the only dataset for which CoST results from prior tables diverge strongly from w/o Aug. We would expect those two results to be quite close. Could the authors expand upon that point?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has the following strenghts:\n- It is clear, the writing is good.\n- The empirical analysis seems sound. From table 4, there are benefits to the approach compared to relevant baselines and ablated versions of the model.\n- The idea is proposed in a principled, justified way.", "weaknesses": "The paper's weaknesses are:\n- Tables 6 and 7 along with the tables from the main paper seem to showcase marginal improvements over CoST, which was the architectural basis of their approach. In particular it is hard to determine if the difference is statistically significant.\n\n- Most of the comparison in table 6/7 is less relevant than the ablation study. The reason is the following: the authors are comparing different architectures. Most of these perform less well than CoST, and they are building on top of CoST. Hence these results are quite expected and should in my opinion not be presented first.\n\n- The authors propose a new augmentation scheme but focus only on CoST to showcase the performance of their approach. This means it is hard to determine whether their technique's benefits generalize to other settings.\n\n- Many of the comparisons hinge on the average of performance on other datasets. However I have some concerns about Lora values being strongly different from other approaches (my 3rd question in the next section). This and the unweighted average of results could bias the conclusions somewhat.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bSAvAD9WC4", "reviewer": "Reviewer_3ku1", "summary": "In this paper, the authors introduced a comprehensive augmentation framework for contrastive self-supervised learning. The framework effectively tackles the challenge of time series data augmentation by leveraging principles from information theory. Additionally, the authors conducted experiments to validate the performance of the proposed framework.", "questions": "none", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed approach effectively deals with the data augmentation problem by unifying various methods into a comprehensive framework through the utilization of information theory.\n- Not only has the effectiveness of the framework been theoretically proven from an information theory perspective, but it has also been extensively validated through empirical experiments.", "weaknesses": "- There might be some minor inconsistencies or gaps in the proof that require further attention to ensure its rigor. Such as, in the proof of Property 1. An invertible mapping is not necessarily a one-to-one mapping, which depending on the domain. Of course, this does not affect the subsequent proof.\n- Some errors on formatting：page 6 “as random timestamp masking“, it seems unnecessary to bold it.\n- Font size of some tables is too low.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SdlAZFzU2s", "reviewer": "Reviewer_4aGN", "summary": "The study employs a parametric neural network to decompose time series into two components: the informative segment and the task-irrelevant segment. Through a data-driven approach, the method adaptively transforms input instances by generating masks to produce viable positive samples, ultimately enhancing the unsupervised performance in time series analysis.", "questions": "Please see the questions in the Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The study is well-grounded in theory concerning its motivation and summarizes the theoretical conditions that the \" GOOD VIEWS\" of contrastive learning should meet.\n\n2. The proposed method exhibits outstanding performance in experimental results.", "weaknesses": "The paper emphasizes the use of a parametric module to decompose time series into an informative part and a task-irrelevant part and perform parameter transformation on the informative part to obtain an enhanced view. However, it is not sufficiently clear. \n\n1. What role does g play specifically? Especially in the analysis of the ablation experiments, I only observed differences in the results; \n\n2. It's ambiguous why h is able to focus on the informative part of the sequence. The author should provide several case studies to prove that h can indeed play a role in separating the information part and noise part of the time series. \n\n3. The two networks h and g use the same structure. How to ensure that they indeed perform the two different functions claimed by the author?\n\n4. How is Δv set in the experiments?\n\n5. The paper does not verify whether the positive view generated by the proposed method contains more information than the original sequence. Although the paper showcases the augmented instances through visualization in Figure 5, as the authors mentioned in the abstract, \"it is impractical to visually inspect the temporal structures in time series.\" Providing explanations for the generated positive samples based on Property 3 or other relevant data would be beneficial.\n\n6. From the visualization in Figure 5, it appears that the output mask of g is a binary mask, which does not match the description in the article. The article interprets g as a non-zero transformation mask. If the output of g is also a binary mask, does it mean that the optimal transformation method for sample enhancement that the model learns is simply to mask some observations of the original sequence?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hy2B9tvW7t", "reviewer": "Reviewer_CECe", "summary": "The paper proposes a parametric augmentation method to automate the selection of augmentations in contrastive time-series representation learning. The method is evaluated on multiple datasets for the tasks of forecasting and classification.", "questions": "Why is the evaluation for time-series classification limited to the UEA dataset? There are several other datasets available that cover modalities like EEG, ECG, and IMU. This choice is not very convincing. \n\nAlso, considering the success of masked autoencoders that do not require anchor/positive generation, what is the usefulness of such an approach? In most cases, the performance is close to that of InfoTS, and further hyperparameter tuning of InfoTS is likely to bridge the performance gap. I see very limited utility of the proposed method.\n\nThere is no silver bullet, so what are the limitations of the proposed method? The paper also mentions the incorporation of frequency domain augmentations as future work, but this has already been extensively explored in the literature.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper addresses an important topic in contrastive self-supervised learning: automating the selection of augmentations.", "weaknesses": "My main concern with the paper is that similar techniques were proposed earlier, such as Rommel et al., 2022. However, the paper does not provide any explanation beyond merely mentioning it in the related work section. It fails to clarify why the proposed method is different, necessary, or how it compares against Rommel et al., 2022. Additionally, there is no baseline for when augmentations are selected at random, for example, using RandAugment. Several other important prior works are missing as well, such as transformation prediction in HAR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6gdkkCPVUV", "reviewer": "Reviewer_NeuG", "summary": "The study addresses the challenge of data augmentation in time series contrastive learning, where traditional human-guided augmentations might not be directly applicable. The authors analyze time series data augmentation using an information theory perspective and introduce a new parametric augmentation method, named AutoTCL. This method adaptively creates augmentations for time series contrastive learning without relying on pre-defined knowledge. The proposed approach is encoder-independent, making it compatible with various backbone encoders. Experimental results indicate that AutoTCL outperforms leading baselines in both univariate and multivariate forecasting tasks, as well as in classification tasks.", "questions": "The main questions are listed in Weaknesses. I'd raise my score if they were appropriately addressed.\n\nThe performance of Multivariate time series forecasting results seems to be sub-optimal compared with univariate time. Could the authors provide some insights?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. A Well-Written Paper: the motivation, method, and experiments are illustrated clearly,\n\nS2. Novel Approach: The introduction of AutoTCL, a parametric augmentation method, offers an adaptive solution to time series contrastive learning, filling a significant gap in existing research.\n\nS3. Novel and Automatic Augmentation Learning: Unlike traditional methods that rely on human intuition or domain knowledge, AutoTCL can automatically learn effective augmentations for time series data, reducing the need for manual tuning or trial-and-error approaches.\n\nS4. Theoretical Foundation: The paper analyzes time series data augmentation from an information theory perspective, providing a good theoretical underpinning for its methodology.", "weaknesses": "W1. Support of Experiments:\nMy main concern is that the authors of this paper claim AutoTCL is a general augmentation method. While the method has been empirically proven effective in crossing different datasets, it would be very critical to test the methods on other contrastive learning frameworks.\n\n\nW2. Discussion of Related Work:\nThe authors have given sufficient discussion about augmentation techniques in the original data space. Recently, another branch of augmentations has stemmed from the feature space. I'd also suggested authors to discussion them for a more comprehensive review. [1,2,3] \n\n[1] Towards domain-agnostic contrastive learning\n\n[2] Metaug: Contrastive learning via meta feature augmentation\n\n[3] Hallucination Improves the Performance of Unsupervised Visual Representation Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)