Soft Contrastive Learning for Time Series
Authors: Seunghan Lee
Taeyoung Park
Kibok Lee
Venue: ICLR 2024 spotlight (2024)
Keywords: Soft Contrastive Learning
Time Series Analysis
Self-supervised Learning
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: This paper proposes SoftCLT, a soft contrastive learning framework for time series.

Abstract: Contrastive learning has shown to be effective to learn representations from time series in a self-supervised way.
However, contrasting similar time series instances or values from adjacent timestamps within a time series leads to ignore their inherent correlations, which results in deteriorating the quality of learned representations.
To address this issue, we propose \textit{SoftCLT}, a simple yet effective soft contrastive learning strategy for time series.
This is achieved by introducing instance-wise and temporal contrastive loss with soft assignments ranging from zero to one.
Specifically, we define soft assignments for 1) instance-wise contrastive loss by distance between time series on the data space, warping and 2) temporal contrastive loss by the difference of timestamps.
SoftCLT is a plug-and-play method for time series contrastive learning that improves the quality of learned representations without bells and whistles.
In experiments, we demonstrate that SoftCLT consistently improves the performance in various downstream tasks including classification, semi-supervised learning, transfer learning, and anomaly detection, showing state-of-the-art performance.
Code is available at this repository: https://github.com/seunghan96/softclt.

Published as a conference paper at ICLR 2024
SOFT CONTRASTIVE LEARNING FOR TIME SERIES
Seunghan Lee, Tacyoung Park, Kibok Lee
Department of Statistics and Data Science, Yonsei University {seunghann6113 tpark, kibok)eyonsee yyonseii.c.....................
ABSTRACT
Contrastive learning has shown to be effective to learn representations from time series in a self-supervised way. However, contrasting similar time series instances or values from adjacent timestamps within a time series leads to ignore their inherent correlations, which results in deteriorating the quality of learned representations. To address this issue, we propose SoftCTT, a simple yet effective soft contrastive learning strategy for time series. This is achieved by introducing instance-win and temporal contrastive loss with soft assignments ranging from zero to one. Specifically, we define soft assignments for 1) instance-wise contrastive loss by the distance between time series on the data space, and 2) temporal contrastive loss by the difference of timestamps. SoftCLT is a plug-and and-play method for time series contrastive learning that improves the quality of learned representations without bells and whistles. In experiments, improves the performance in various downstream tasks including classification, semi-supervised learning, transfer learning, and anomaly detection, showing state- of-the-art performance. Code is available at this repository: https: / gighub.. com/ http///unghann6//eeee sofftclt..
demonstrate that SoftCLT consistently
INTRODUCTION
Time series (TS) data are ubiquitous in many fields, including finance, energy, healthcare, and transportation (Ding et al., 2020; Lago et al., 2018; Solares et al., 2020; Cai et al., 2020). However, annotating TS data can be challenging as it often requires significant domain expertise and time. To overcome the limitation and utilize unlabeled data without annotations, self-supervised learning has emerged as promising representation learning approach not only in natural language process- ing (Devlin et al., 2018; Gao et al., 2021) and computer vision (Chen et al., 2020; Dosovitskiy et al., 2021), but also in TS analysis (Franceschi et al., 2019; Yue et al., 2022). In particular, contrastive learning (CL) has demonstrated remarkable performance across different domains (Chen et al., 2020; Gao et al., 2021; Yue et al., 2022). As it is challenging to determine similarities of instances in self-supervised learning, recent CL works apply data augmentation to generate two views per data and take views from the same instance as positive pairs and the others as negatives (Chen al., 2020). However, we argue that the standard CL objective might be harmful for TS representation learning. because inherent correlations in similar TS instances and values nearby timestamps within TS, which could be strong self-supervision, are ignored in CL. For example, distance metrics such as dynamic time warping (DTW) have been widely used for measuring the similarities of TS data, and contrasting TS data might lose such information. Also, values with close timestamps usually similar in natural TS data, so contrasting all values with different timestamps with the same degree of penalty as in previous CL methods (Eldele et al., 2021; Yue et al., 2022) might not be optimal. Motivated by this, we explore the following research question: how can we lake account of the similarities of time series data for better contrastive representation learning? To this end, we propose Soft Contrastive Learning for Time series (SoftCLT). Specifically, we propose to consider the InfoNCE loss (Oord et al., 2018) not only for the positive pairs but also all other pairs and compute their weighted summation in both instance-wist CL. and temporal CL, where instance- wise CL contrasts the representations of TS instances, while temporal CL contrasts the representations of timestamps within a single TS, shown in Figure 1. We propose to assign soft assignments based on the distance between TS for the instance- wise CL, and the difference timestamps for the temporal CL. This formulation can be seen as generalization of the standard contrastive loss, as the proposed loss becomes the contrastive loss if we replace soft assignments with hard assignments of either zero for negative or one for positive.

---

Published as a conference paper at ICLR 2024
Tlar (NewiPS
SaltTre (waw new.
TNC (IT1 (61)
T5. 5D (UCNN
75 TDC (ICM 2001)
TK3W: (AAM] 3015
Mideglly Lattee 5/2ty
.UST (00.
Teeet (636 30hly
TTCC (NeeitS (01))
CA-TEC (TRAMI NE2))
Settiz (0ay)
/
Table Comparison table of contrastive learning methods in time series.
We conduct extensive experiments in various tasks, including TS classification, semi-supervised classification, transfer learning, and anomaly detection tasks to prove the effectiveness of the proposed method. Experimental results validate that our method improves the performance of previous CL methods, achieving state-of-the-arr (SOTA) performance contributions of this paper are summarized as follows:
range of downstream tasks. The main
We propose SoftCLT, simple yet effective soft contrastive learning strategy for TS. Specifically, we propose soft contrastive losses for instance and temporal dimensions, respectively, to address limitations of previous CL methods for TS.
We provide extensive experimental results on various tasks for TS, showing that our method improves SOTA performance on range of downstream tasks. For example, SoftCT improves the average accuracy of 125 UCR datasets and 29 UEA datasets by 2.0% and 3.9%, respectively, compared to the SOTA unsupervised representation for classification tasks.
SoftCLT is easily applicable to other CL frameworks for TS by introducing soft assignments and its overhead is negligible, making it practical for use.
2 RELATED WORK
Self-supervised learning. In recent years, self-supervised learning has gained lots of attention for its ability to learn powerful representations from large amounts of unlabeled data. Self-supervised learning is done by training model to solve pretext task derived from certain aspect of data without supervision. As a self-supervised pretext task, next token prediction (Brown et al., 2020) and masked token prediction (Devlin et al., 2018) are commonly used in natural language processing, while solving jigsaw puzzles (Noroozi & Favaro, 2016) and rotation prediction (Gidaris & Komodakis, 2018) are proposed in computer vision. In partieular, contrastive learning (Hadsell et al., 2006) has shown to be an effective pretext task across domains, which maximizes similarities of positive pairs while minimizing similarities of negative pairs (Gao et al., 2021; Chen al., 2020; Yue et al., 2022).
Contrastive learning in time series. In the field of TS analysis, several designs for positive and negative pairs have been proposed for CL., taking into account the invariant properties of TS. Table 1 compares various CL methods in TS including ours in terms of several properties. T-Loss (Franceschi et al., 2019) samples random subseries from a TS and treats them as positive when they belong to its subseries, and negative if belong to subseries of other TS, Self-Time (Fan et al., 2020) captures inter-sample relation between TS by defining augmented sample of same TS as positive and negative otherwise, and captures intra-temporal relation within TS by solving classification task, where the class labels are defined using the temporal distance between the subseries. TNC (Tonekaboni et al., 2021) defines temporal neighborhood of windows using normal distribution and treats samples in neighborhood as positives. TS-SD (Shi et al., 2021) trains a model using triplet similarity discrimination task, where the goal is to identify which of two TS is more similar to a given TS, using DTW to define similarity. TS-TCC (Eldele et al., 2021) proposes temporal contrastive loss by making the augmentations predict each other's future, and CA-TCC (Eldele et al., 2023), which is the extension of TS- TCC to the semi-supervised setting, adopts the same loss. TS2 (Yue et al., 2022) splits TS into two subseries and defines hierarchical contrastive loss in both instance and temporal dimensions. Mixing-up (Wickstrøm al., 2022) generates new TS by mixing two TS, where the goal is to predict the mixing weights. CoST (Woo et al., 2022) utilizes both time domain and frequency domain contrastive losses to learn disentangled seasonal-trend representations of TS. TimeCLR (Yang et al.. 2022) introduces phase-shift and amplitude change augmentations, which are data augmentation methods based on DTW. TF-C (Zhang et al., 2022) learns both time- and frequency- based representations of TS and proposes a novel time-freyuency consistency architecture. In the medical domain, Subject-Awand CL (Cheng et al., 2020) proposes an instance-wise CL framework where the temporal information is entangled by architecture design, and CLOCS (Kiyasseh et al., 2021) 1) proposes to consider spatial dimension specifically available in their application, which is close to the channels in general TS. While previous CL methods for TS compute hard contrastive loss, where the similarities between all negative pairs are equally minimized, we introduce soft contrastive loss for TS.
2

---

Published as a conference paper at ICLR 2024
Instance-wise CL
Temporal CL
Hard CL
Soft CL
Hard CL
Soft CL
Reference
Positive
Soft Assignment
Nogative
74
Instance-wise contrast multiple at a single timestamp
Temporal CL
:contrast multiple timestamps within a single TS
Figure 1: Overall framework of SoftCLT. Unlike the conventional hard CL that gives either positive or negative assignments to sample pairs, SoftCLT gives soft assignments to both instance- wise and temporal relationships. Two views of the same sample are denoted as r and f, respectively.
Soft contrastive learning. CL is typically done by batch instance discrimination, where each instance is considered to be in distinct class. However, this approach can pose a risk of pushing similar samples farther apart in the embedding space. To address this issue, several methods have been proposed, including method that utilizes soft assignments of images (Thoma et al., 2020) based on feature distances and geometric proximity measures. NNCLR (Dwibedi et al., 2021) defines additional positives for each view by extracting top-k neighbors in the feature space. NCL (Yèche et al., 2021) finds neighbors using supervision from the medical domain knowledge and jointly optimizes two conflicting losses with trade-off: the neighbor alignment loss maximizing the similarity of neighbors as well as positive pairs, and the neighbor discriminative loss maximizing the similarity of positive pairs while minimizing the similarity of neighbors. SNCLR (Ge al., 2023), which extends NNCLR with soft assignments, employs an attention module to determine the correlations between the current and neighboring samples. CO2 (Wei et al., 2021) introduces consistency regularization to enforce relative distribution consistency between different positive views and all negatives, resulting in soft relationships between samples. ASCL (Feng & Patras, 2022) introduces soft inter-sample relations by transforming the original instance discrimination task into multi-instance soft discrimination task. Previous soft CL methods in non-TS domains compute soft assignments the embedding space, because similarities of instances the data space difficult to measure, particularly in computer vision (Chen et al., 2020). In contrast, we propose to compute soft assignments based the distance between TS instances the data space.
Masked modeling in time series. Other than CL., masked modeling has recently been studied as a pretext task for self-supervised Icarning in TS by masking out a portion of TS and predicting the missing values. While CL has demonstrated remarkable performance in high-level classification tasks, masked modeling has excelled in low-level forecasting tasks (Dong et al., 2023; Huang et al., 2022; Xie et al., 2022). TST (Zerveas et al., 2021) adopts the masked modeling paradigm to TS, where the goal is to reconstruct the masked timestamps. PatchTST (Nie et al., 2023) aims to predict the masked subseries-leved patches capture the local semantic information and reduce memory usage. SimMTM (Dong et al., 2023) reconstructs the original TS from multiple masked TS.
METHODOL.OGY
In this section, we propose SofICLT by introducing soft assignments to instance-wise and temporal contrastive losses capture both inter-sample and intra-temporal relationships, respectively. For instance- sriance-wiseeae CL, we use distance between TS on the data space to capture the inter-sample relations, and for temporal CL. we use the difference between timestamps to consider the temporal relation within a single TS. The overall framework of SoftCLT is illustrated in Figure 1.
3.1 PROBLEM DEFINITION
This paper addresses the task of learning nonlinear embedding function fo : x + r, given a batch of N time series X = {II,.... ). Our goal is to learn fe mapping time series x; E RT' 7g7'x to representation vector r; [r,,,..... ... Firr]" T where T is the sequence length, D is the input feature dimension, and M is the embedded feature dimension.
3.2 SOFT INSTANCE- WISSE CONTRASTIVE LEARNING
Contrasting all instances within batch might be harmful for TS representation learning because similar instances are learned to be far away from each other on the embedding space. Unlike other domains such as computer vision, the distance between TS data computed the data space
3

---

Published as a conference paper at ICLR 2024
is useful for measuring the similarity of them. For example, the pixel-by-pixx. distance of two different images is not related to their similarities in general, that of two TS data is useful to measure their similarities. With min-max normalized distance metric D(.-)), we define soft assignment for a pair of data indices (i, i') for the instance-wise contrastive loss using the sigmoid function (a) = 1/(1 + exp(-x))
wr wr(i, 2ax 0 (-71 D(a;, x2)),
where Ty is hyperparameter controlling the sharpness and is the upper bound in the range of [0, 1] to distinguish pairs of the same TS and pairs of different TS close to each other; when a = 1, we give the assignment of one to the pairs with the distance of zero as well the pairs of the same TS. Note that distances between TS are computed with the original TS rather than the augmented views, because the pairwise distance matrix can be precomputed offline or cached for efficiency.
For the choice of the distance metric D, conduct ablation study in Table 6d, comparing 1) cosine distance, 2) Euclidean distance, 3) dynamic time warping (DTW), and 4) time alignment measurement (TAM) (Folgado et al., 2018). Among them, we choose DTW as the distance metric throughout the experiments based on the result in Table 6d. While the computational complexity of DTW is ( for two TS of length which might be costly for large-scale datasets, can be precomputed offline or cached to facilitate efficient calculations, or its fast version such as FastDTW (Salvador & Chan, 2007) with the complexity of (T) can be used. We empirically confirmed that the output of DTW and FastDTW is almost the same, such that the CL results also match.
Let ri, T.+2N, and Fi, = Ti+N. be the embedding vectors from two augmentations of x; at timestamp for conciseness. Inspired by the fact that the contrastive loss can be interpreted as the cross-entropy loss (Lee et al., 2021), we define a softmax probability of the relative similarity out of all similarities considered when computing the loss as:
exp(rie o o.v,))
pi pi((i
بイ=1 exp(re, o ry.e)
where we use the dot product as the similarity measure o. Then, the soft instance-wise contrastive loss for x; at timestamp is defined as:
2N Σ j (GAGNN))))
elte)) log pr ((1, N).t)
wr(i.j mod N) - log pr ((t,f).
(3)
The first term in (i-t)) corresponds to the loss of the positive pair, and the second term corresponds to that of the other pairs weighted by soft assignments wy (i, Note that this loss can be seen as generalization of the hard instance-wise contrastive loss, which is the case when Vwr (i, d') =
3.3 SOFT TEMPORAL CONTRASTIVE LEARNING
Following the intuition that values in adjacent timestamps similar, propose to compute a soft assignment based on the difference between timestamps for temporal contrastive loss. Similar to the soft instance-wist contrastive loss, the assignment is close to when timestamps get closer and zero when they get farther away. We define a soft assignment for pair of timestamps (t, t') for the temporal contrastive loss as:
WT (t,t')= 2 2-0 1 (-TT It1),
where Tr is a hyperparameter controlling the sharpness. As the degree of closeness between times- tamps varies across datasets, we tune Tr to control the degree of soft assignments. Figure 2a illustrates an example of soft assignments with respect timestamp difference with different Ty-
Hierarchical loss. For temporal CL. we consider hierarchical contrasting on intermediate represen- tations in the network fo as done in prior CL methods for TS. Specifically, we adopt the hierarchical contrastive loss proposed in Te2 (Yue et al., 2022), where the losses are computed on intermedi- ate representations after each max max-pooling layer along the temporal axis and then aggregated. As shown in Figure 2b, similarities between adjacent time step decrease after pooling, we adjust Ty by multiplying n in Eq. 4, i.c., Ty depth, and Fy is the base hyperparameter.
= m łt where m is the kernel size of pooling layers, is the
4

---

Published as a conference paper at ICLR 2024
e+1
Dissimbrr (angn?)
The Twedars
0.00
0.00
0.00 0.05
0.30
5.05
0.50
6.00
1.0
0.00 0.03 0.01 0.04 6:89 5.24
0:58
0.54
6:24
0.09
0.04
0.01
6.02
0.00
1hem Sap
0.5
0.06 0.03 0.15 0.24 0.35
C34
0:79
0.75
6:34
0.50
0.24
0:23
0.00 0.00
Ty
0:00
0.75 0.00
C.2
0.90
5:2
0.75 5.75
Mox Pool
VVVV
Reference Point
kecnel size pooling
Similrr juendeed 7y>
(a) Soft assignments with different Ty.
(b) Hierarchical representations.
Figure 2: (a) shows examples of soft assignments for soft temporal CL, where a smaller тт results in smoother assignments. (b) is example of hierarchical representations, demonstrating that increasing layer depth results a larger semantic difference between adjacent time steps, so Ty should be increased to compensate for it.
Now, let ri,r = 27,427 and Fi = -ii+T be the embedding vectors from two augmentations of s; timestamp /. for conciseness. Similar Eq. 2, define softmax probability of the relative similarity out of all similarities considered when computing the loss as:
pr(i, (1,11) exp(Ti, o Ta,e)
(5)
Light exp(ri, oTiss)
Then, the soft temporal contrastive loss for æ; at timestamp t is defined as:
27 Σ
(.. = 1oggrr(i,((6 T
W1(,, mod T) Pr (1,8)).
(6)
(L.77
Similar to the soft instance-wise contrastive loss, this loss can be seen as a generalization of the hard temporal contrastive loss, which is the case when (wr(t, O.
The final loss for SoftCLT is the joint of the soft instance-wii and temporal contrastive losses:
2N 2T
1 efitt
+ (1 A) x)-e(.))
4NT
where is a hyperparameter controlling the contribution of each loss, set to 0.5 unless specified. The proposed loss has an interesting mathematical interpretation that it can be seen the scaled KL divergence the softmax probabilities from the normalized soft assignments, where the scale the sum of soft assignments. We provide more details in Appendix D.
EXPERIMENTS
We conduct extensive experiments to validate the proposed method and assess its performance in different tasks: (1) classification with univariate and multivariate TS, (2) semi-supervised classification by (i) self-supervised learning followed by fine-tuning and (ii) semi-supervised learning, (3) transfer learning in in-domain and cross-domain scenarios, and (4) anomaly detection normal and cold-start settings. We also conduct ablation studies to validate the effectiveness of SoftCLT as well as its design choices. Finally, we visualize pairwise distance matrices and t-SNE (Van der Maaten & Hinton, 2008) of temporal representations show the effect of SoftCLT over previous methods. We use the data augmentation strategies of the methods we apply our SoftCLT to: TS2Vec generates two views as TS segments with overlap, and TS-TCC/CA-TCC generate two views with weak and strong augmentations, using the jitter-and and-scale and permutation-ande strategies, respectively.
4.1 CLASSIFICATION
We conduct experiments on TS classification tasks with 125 UCR archive datasets (Dau et al., 2019) for univariate TS and 29 UEA archive datasets (Bagnall et al., 2018) for multivariate TS.
Some of the previous methods cannot handle missing observations, so three of the 128 datasets are omitted. One of the 30 datasets is omitted for fair comparison with of the previous methods.
5

---

Published as a conference paper at ICLR 2024
125 UCR datasets
29 UEA datasets
Method
Avg. Acc.(%)
Avg. Rank
Avg. Acc.(%)
Avg. Rank
TST 4.19
141 TORN
DTW-D
TNC
76.1
72.7
4.42
5.30
65.0
67.7
4,60
4.76
TWC
44
TRON
TST
64_1
6.19
63.5
5.26
TS-TCC
75.7
4.29
68.2
4.38
29 UEA Dataset
T-Loss
80.6
3.50
67.5
3.86
TS2Vec
83.0
2,80
71.2
3.28
The
1ST
Ours
85.0(+ 2.0)
1.49
75.1(+ 3.9)
1.86
15 700
Table 2: Accuracy and rank on UCR/UEA.
Figure 3: CD diagram on UCR/UEA.
18 of Tabded Das
Selfaagarsiee kuntin
Semt-sagerisel
Icuntit
S5L-EX00
CPC
SurCLK
TS2N0c
1S-ICC
Moas Teacher
LivikiMii
Sera
Fishiach
CA-TUC
HAIE
8000-440
35.5/ $2.5
91.0 9910
70.5
82-9-82.8
15.07100
16.9/75.4
22.6 138-
70.8/756
77.57
MLL
Iglippey
43.4735.1
05.3 78.4
01.97 18.5
655//56.1
03.37 59.1
98.5 99.5
96.71663
41.3192.0
94.//444
95.1 95.1785.
96.9/ 98.5
ThelA
08.87 25-2
15.97 58.7
10.4/ 85.4
87.8 8513
90.
81.3 81.5-811 161-2
71:2271.5
73277333
75.1/344
73.5/26.3
9L.: 9LL5
444180.
00.5
3097
85.4/95.4 95-6
01.0 /013
784
19.8 14.8
85.0/85.
343/34.1
018/815
56.7133
7339
7K.
78,3
80.6/ 71.6
83.5 00.5
85.6 16.
86.9 75.2
96.0 79.3
79.4-77.3
75007753
755/7778
71.6
058/ 77.
96.1 66.1/63
BlookDDekos
00.1-900
62.5/51.2
$7.6/48.6
/3.0/53.0
63.6 /353
64.6 663.2
45.974483
59.3149.4
51.3/48.
58.31 40.9
(0.9) 56.7
70.3/ 08.5
HAI
6377/19.
08.8 F587
75.87 10.9
$1./7910 91.0
63.1 7M3.
92.6 92.6
RK 90.1
57.5-5711
16.57 105.5
91.4/914
Ippippyy
M2:8-112
80.3/89.2
19.07 65.5
03.7 1983
143:2
96.3 1961
0139-914
94.0/9101
M3.7/9223
11.57 16.10
96.8/999 97.9
2386 7357
08.3-96.3
698- 18.9
91.37 91.1
92. 1425
99.1-969
08.2 98.2
K2.0 82.5
$401311
S15/512
0:9/911
95.5/ 95.3
Predll
78.7-09.88
0335180 96.2
65.07 MLY
16.7 16.0
7838/78.
96.1 1959
88.0 (880
M687 822
0622/513
02:71007
85.27 185
19.4 2014
029/44133
82.77
384/58.
7839 697
62. 41.8
19.3
629/45-3
62.4-418
63:1/436
66.37 823
70.7
632/136.1
62.6788
629754.7
624/344
45.1
68.1 63.8
228/821
718/6111
028/235
66.91993 29.3
10.5 68.
Table 3: Semi-supervised classification results. The table shows the results of fine-tuning self- and semi-supervised models, with 1% and 5% of labels. Best results across each dataset are in bold, while the second-best results are underlined. The accuracy and MF1 score are reported in order.
respectively. Specifically, we apply SoftCLT to TS2Vec (Yue et al., 2022), which has demonstrated SOTA performance on the above datasets. As baseline methods, we consider DTW-D (Chen et al., 2013), TNC (Tonekaboni et al., 2021). TST (Zerveas et al., 2021), TS-TCC (Eldele et al., 2021), T-Loss (Franceschi et al., 2019), and TS2Vec (Yue et al., 2022). The experimental protocol follows that of T-LLoss and TS2Vec, where the SVM classifier with the RBF kernel is trained on top of the instance-level representations obtained by max-pooling representations of all timestamps. Table 2 and the critical difference (CD) diagram based on the Wilcoxon- shown in Figure 3 demonstrate that the proposed method improves SOTA performance by a significant margin on both datasets in terms of accuracy and rank. In Figure 3, the best and second-best results for each dataset are in red and blue, respectively. We also connect methods with a bold line if their difference is not statistically significant in terms of the average rank with confidence level of 95%, which shows that the performance gain by the proposed method significant.
method (Ismail Fawaz et al., 2019)
4.2 SEMI-SUPERVISED CLASSIFICATION
We conduct experiments on semi-supervised classification tasks by adopting SoftCLT to TS-TCC (EI- dele et al.. 2021) and its extension CA-TCC (Eldele et al., 2023), which are the methods that incorporate CL into self- and semi-supervised learning, respectively. As baseline methods, con- sider SSL- (Sarkar & Etemad, 2020), CPC (Oord et al., 2018), SimCLR (Chen et al., 2020) and TS-TCC (Eldele al., 2021) for self supe-suue learning, and Mean-Teacher (Tarvainen & Valpola, 2017), DivideMix (Li et al., 2020), SemiTime (Fan et al., 2021), FixMatch (Sohn et al., 2020) and CA-TCC (Eldele al., 2023) for semi-supervised learning. Note that both TS-TCC and CA-TCC perform instance-wise and temporal contrasting, however, their temporal contrasting is achieved by predicting one view's 's future from another, which different from the conventional contrastive loss with positive and negative pairs. Therefore, we adopt our soft temporal contrastive loss as an additional loss to both methods. For evaluation, we utilize the same experimental settings and datasets of CA-TCC, which includes eight datasets (Anguita et al., 2013; Andrzejak et al., 2001; Dau et al., 2019), six of which are from the UCR archive. We consider two cemi-supervised learning scenarios, (1) self-supervised learning with unlabeled data followed by supervised fine-tuning with labeled data and (2) semi-supervised learning with both labeled and unlabeled data, following CA-TCC (Eldele et al., 2023). Table 3 presents the experimental results with both methods in scenarios with 1% and 5% labeled datasets, showing that applying SoftCLT achieves the best overall performance across most of the datasets in both scenarios.

---

Published as a conference paper at ICLR 2024
In-domsin trandfrr karning
Cros-lmmiin leaming
SkepkEEG
Epiepoy
SleepEBG FD-B
SleepäBBGi
Gesture
SleephEGG
EMG
ACC.
PRE
REC.
Fr
.CC.
PRC.
REC.
ACC.
PRC
RDC.
F:
ACC.
PRC.
RDC.
TS-SD
50.52
80.18
76.47
77.67
55,06
57.10
60.54
57.05
60:22
66.08
66.67
66.56
46.06
15:45
$3.33
21:11
TS2Vec
91.95
90.59
90.39
90.45
47,90
43:39
48.42
43.89
88.17
65.45
68.54
65.20
78.56
$0.40
67.65
67.66
MixingUp
50.21
40.11
50,00
64.51
67.89
71.46
76.13
72.73
88.33
67.19
69.33
64.97
30.24
10.59
25.83
15.41
CLOCS
95.07
93.01
92.06
58.73
44.33
40.14
53.06
53.54
CoST
58.40
88.20
72.34
76.88
47,00
38.79
38,42
34.79
68.33
65.30
68.33
66.42
53.65
49,07
42.10
35.27
LSST
56.46
90.77
66.35
70.67
43.90
47.71
45.17
64.17
70.36
64.17
58.26
66.36
79.34
63.33
72.35
94.95
91.49
72.02
76.42
75.72
TST
20111
44.51
46.40
41.58
45.50
69.17
66.01
15.45
33.33
SimMTM
05.42
TS-TCC
92.53
96.51
35
63.55
14
71
24
5M.51
63.10
Ouus
97.00
97.07
97.00
96.92
85,68
95.00
95.59
95,00
95.12
100
100
(a) Transfer learning under in- and cross-domain scenarios.
A- B
A-
A-
B-+
B-+C
B D
C- A
C-
B
C-+
D A
D-+B
D D
Arg
Supervised
34,38
44.94
34,57
52.93
63.67
99.82
12.93
84.02
83.54
53.15
99.56
62.43
63.8
TS-TCC
43.15
$1.50
42.74
47.98
70.36
99.30
38,89
98.31
99.38
51.91
99.96
20.31
67.82
Ours
76.83
74.35
78.34
53.37
75.11
53.26
85.59
56.29
53.30
93.55
70.93
75,03 (+7.215)
CA-TOC
44.75
52.09
45.63
46.26
71.33
100.0
52.71
99.85
99.84
46.48
100.0
77.01
Ours
76.85
77.16
79.99
53.26
86.36
100.0
53.23
99.67
99.01
53.56
100.0
84.93
80.34 (-1.681)
(b) Transfer learning without adaptation under self- and semi-supervised settings on the FD datasets. TS- TCC and CA-TCC are used as baselines for self- and semi-supervised learning, respectively.
Table 4: Results of transfer learning task on both in- and cross-domain settings.
4.3 TRANSFER LEARNING
We conduct experiments on transfer learning for classification in in-domain and cross-domain settings which are used in previous works (Zhang et al., 2022; Eldele et al., 2021; 2023; Dong et al., 2023), by adopting our SoftCLT to TS-TCC and CA-TCC. As baseline methods, we consider TS-SD (Shi et al., 2021), TS2Vec (Yue et al., 2022), Mixing-Up (Wickstrøm et al., 2022), CLOCS (Kiyasseh et al., 2021). CoST (Woo et al., 2022), LaST (Wang et al., 2022). TF-C (Zhang et al., 2022), TS-TCC (Eldele et al., 2021), TST (Zerveas et al., 2021) and SimMTM (Dong et al., 2023). In in-domain transfer learning, the model is pretrained on SleepEEG (Kemp et al., 2000) and fine-tuned on Epilepsy (Andrzejak et al., 2001), where they are both EEG datasets and hence considered to be in a similar domain. In cross-domain transfer learning, which involves pretraining on one dataset and fine-tuning on different datasets, the model pretrained on SleepEEG, and fine-tuned on three datasets from different domains, FD-B (Lessmeier et al., 2016), Gesture (Liu et al., 2009), and EMG (Goldberger et al., 2000). Also, we perform transfer learning without adaptation under self-and semi- supervised settings, where source and target datasets share the same set of classes but only 1 1% of labels are available for the source dataset, and no further training on the target dataset is allowed. Specifically, models are trained on one of the four conditions (A, in the Fault Diagnosis (FD) datasets (Lessmeier et al., 2016) and test on another. Table 4a shows the results of both in- and cross-domain transfer learning, and Table 4b shows the results of both self- and semi-supervised settings with FD datasets. Notably, SoftCLT applied to CA-TCC improves average accuracy of twelve transfer learning scenarios with FD datasets by 10.68%.
4.4 ANOMALY DETECTION
We conduct experiments on univariate TS anomaly detection (AD) task by adopting SoftCLT to Te2 (Yue et al., 2022) under two different settings: the normal setting splits each dataset into two halves according to the time order and use them for training and evaluation, respectively, and the cold-start setting pretrains models on the FordA dataset in the UCR archive and evaluates on each dataset. As baseline methods, we consider SPOT (Siffer et al., 2017), DSPOT (Siffer et al., 2017), DONUT (Xu et al., 2018). SR (Ren et al., 2019), for the normal setting, and FFT (Rasheed et al., 2009), Twitter-AD (Vallis et al., 2014), Luminol (LinkedIn, 2018) for the cold-start setting, and TS2Vec (Yue et al., 2022) for both. The anomaly score computed by the L1 distance of two representations encoded from masked and unmasked inputs following VS2 We evaluate the compared method on the Yahoo (Laptev et al., 2015) and KPI (Ren et al., 2019) datasets. We found that suppressing instance-wise CL leads to better AD performance on average, so we report Te2Vec and SoftCLT performances without instance- wise CL: more details can be found in the Appendix G. As shown in Table 5, SoftCLT outperforms the baselines in both settings in terms of the F1 score, precision, and recall. Specifically, SoftCLT applied to TS2Vec improves the Fl score approximately 2% in both datasets under both normal and cold-start settings.
7

---

Published as a conference paper at ICLR 2024
Yahio
KPI
Yahoo
KPI
Ft
Prec.
Rec.
Fj
Prec.
Rec.
F₁
Proc.
Rec.
F₁
Pree.
Rec.
SPOT
33.8
26.9
45.4
21.7
78.6
12.6
FFT
29.1
20.2
51.7
53.8
47.8
61.5
DSPOT
24.1
52.1
62.3
Twitter-AD
46.2
DONUT
2.6
1.3
82.5
34.7
37.1
32.6
Luminol
38.8
25.4
81.8
41.7
30.6
65.0
SR
5.63
45.1
74.7
62.2
64.7
59.8
SR
52.9
40.4
76.5
66.6
63.7
69.7
TS2Vec
72.3
69.3
75.7
67.6
91.1
51.7
Vo2
74.0
70.7
77.6
68.9
89.3
56.2
+ Ours
74.2
72.2
76.5
70.1
91.6
57.0
Ours
76.2
75.3
77.3
70.7
92.1
57.4
(a) Results of AD task on normal setting.
(b) Results of AD task cold-start setting.
We used the official code to replicate the results without the instance- wise contrastive loss.
Table 5: Anomaly detection results.
assignment
UCR
UEA
Tomporal CL
Instance-wiee wise CL.
Insi. CL
Temporal CL
Instance-wise
Tamporal
Aug. (oc.(%)
Ang. (%c)
Method
Avg. Asc(S)
Avg. (cc.55)
Metric
Hard
Soft
82.3
70.5
Neighhar
COS
84.7
83.9 (+1
(+2.5)
Linear
77.2
0.50
83.9
EUC
83.9
84.8
83.7 (+1.4)
73.8 (+3.3)
Goussian
83.5
0.75
83.4
DTW
83.9
85.0
85.0 (+2.7)
74.2 (+3.7)
Sigmoid
83,7
TAM
83.9
85,0
(a) Application of soft assignments.
(b) Assignment func. (c) Upper bound.
(d) Distance func.
Table 6: Ablation study results.
4.5 ABLATION STUDY
Effectiveness of SoftCLT. Table 6a shows the effect of soft assignments from the standard hard CL. Applying soft assignments to instance-wiss or temporal CL provides performance gain, and applying them both dimensions results in the best performance, improving the accuracy on the UCR and UEA datasets by 2.7% and 3.7%, respectively.
Design choices for soft temporal CL. Table 6b compares different choices of the soft assignment WT. Neighbor takes neighborhood within a window around the reference point as positive and the others as negative. Linear gives soft assignments linearly proportional to the time difference from the reference point, where the most distant one gets the value of zero. Gaussian gives soft assignments based on a Gaussian distribution with the mean of the reference point and the standard deviation as hyperparameter. Among them, Sigmoid in Eq. 4 shows the best performance as shown in Table 6b.
Upper bound for soft instance-wise CL. In the soft instance-wiss contrastive loss, a is introduced to avoid giving the same assignment to pairs of the same TS and pairs of the different TS with the distance of zero, where effect of tuning a. Based on the results, a 0.5 is the best, i.c., the similarity of the pairs of the same TS should be strictly larger than other pairs, but not by much.
makes both cases to have the same assignment. Table 6c studies the
Distance metrics for soft instance-wise CL. Table 6d compares different choices of the distance metric D in Eq. 1: cosine distance (COS), Euclidean distance (EUC), dynamic time warping (DTW), and time alignment measurement (TAM) (Folgado al., 2018) on 128 UCR datasets, where the baseline is Te2Vec and the hard or best soft temporal CL. is applied together. The result shows that the improvement by soft instance-wist wise CL robust to the choice of the distance metric. We use DTW throughout all other experiments because DTW is well-studied, commonly used in the literature and fast algorithms such as FastDTW are available.
4.6 ANALYSIS
Comparison with soft CL methods in computer vision. While soft CL methods have been proposed in other domains, they compute soft assignments on the embedding space because it is difficult to measure the similarities on the data space, particularly in computer vision. However, we argue that the similarities on the data space is indeed a strong self self-supervision, leading to better representation learning. To confirm this, we compare SofiCLT with soft CL methods proposed in other domains working on the embedding space: NNCLR (Dwibedi et al., 2021) and ASCL (Feng & Patras, 2022), on UCR datasets. For a fair comparison, we apply all compared methods to TS2Vec under the same setting. As shown in Table 7, different from the proposed method, NNCLR and ASCL deteriorate the performance of TS2Vec, implying that similarities measured on the data space is strong seff-supervision, while similarities measured on the learnable embedding space might not
8

---

Published as a conference paper at ICLR 2024
Method
Total
Length of time scrics
Gap (A-B)
Temporal CL
Seasonality
200 (A)
> 200 (B)
Soft
Low (103/128)
High (25/128)
TS2Vec
82.3
88.1
79.6
5.8
x
84.1
80.1
NNCLR
66.0
82.6
58.2
24.4
85.6
81.7
ASCL
76.5
86.6
71.8
14.8
Ours
85.0
89.8
81.9
7.9
Gain
+1.5
+ +1.6
Table 7: Comparison of soft CL methods.
Table 8: Effect of soft temporal CL by seasonality.
Layer Depth
Paact
Tpech
Class 1
Hard CL
Cloes
Cloos 3
D/2 MI2 #51 DIAA x2)
X21 X2)
Soft CL
(16) 1X2, K.) 2.1
022 42) 147
raw time-series Z; embedding of 1-th layer
z of CL
D(K, painwise distance matrix
z So
Figure 4: Pairwise distance matrices.
Figure 5: Visualization of temporal representations.
be useful in some domains. To further investigate the failure modes of the previous methods, we categorize datasets by the average TS length of 200 in Table 7, and observe that previous methods fail to capture the similarities of long TS data.
Robustness seasonality. An assumption behind the proposed soft temporal CL is that values in adjacent timestamps are similar, which may raise a concern that seasonality in TS might not be captured. To address this, we categorize UCR datasets based on seasonality by ADF test (Sims et al.. 1990) the significance level of p 0.05. As shown in Table 8. the performance gain by SoftCLT is consistent regardless of the seasonality. Our conjecture is that TS in the real world usually do not exhibit the perfect seasonality, indicated by the ADF test result, such that SoftCLT takes advantage of the non-seasonal portions. Meanwhile, previous works have tried to decompose trend and seasonality in TS for representation learning (Wang et al., 2022: Woo al., 2022). However, this may not be realistic for TS that are neither simultaneously auto-regressive nor stationary (Shen et al., 2022). In summary, we do not consider seasonality in TS directly. because is not only challenging to extract but we can still achieve good performance without considering it in practice.
Instance-wise relationships. To see whether instance-wise relationships are preserved in the encoder, we visualize the pairwise mnstance-wise distance matrices of representations the InsectEP. GRegular dataset from the UCR archive (Dau et al., 2019) extracted from each layer, where the brighter color indicates the lower distance between instances. The top and bottom panels of Figure show the changes in pairwise distance matrices of representations as depth progresses when adopting hard and soft CL, respectively. The results indicate that SoftCLT preserves the relationships between TS instances throughout encoding, while the standard hard CL fails to preserve them.
Temporal relationships. To assess the quality of temporal relationships captured by SoftCLT, we apply t-SNE (Van der Maaten & Hinton, 2008) to visualize the temporal representations, which representations of each timestamp in a single TS. Figure 5 compares t-SNE of the representations learned with hard and soft CL over different training epochs, with the points getting darker as time progresses. While hard CL finds coarse coarse-grained neighborhood relationships such that it fails to distinguish late timestamps dark red, soft CL. finds more fine-grained relationships.
5 CONCLUSION
In this paper, we present a soft contrastive learning framework for time series. In contrast previous methods that give hard assignments to sample pairs, our approach gives soft assignments based on the instance-wise and temporal relationships on the data space. We demonstrate the effectiveness of our method in range of tasks, leading to significant improvements in performance. We hope our work enlightens the effectiveness of self-supervision from the data space and motivates future works on contrastive representation learning in various domains to take account of it.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
The proposed soft contrastive learning algorithm for time series has potential to make significant impact the field of representation learning for time series data. The ability to apply this algorithm to various tasks and solve the general problem time series representation learning is promising. In particular, the algorithm can be applied transfer learning, which may be useful in scenarios with small datasets for downstream tasks. Furthermore, we expect that the idea of utilizing self-supervision from the data space for contrastive representation learning motivates future works in various domains.
However, as with any algorithm, there are ethical concerns to be considered. One potential ethical concern is potential for the algorithm to perpetuate biases that may exist the datasets used for pretraiming. For example, if the pretraining dataset is imbalanced with respect certain demographic attributes, this bias may transferred to fine-tuning, potentially leading to biased predictions. is essential evaluate and address potential biases in the pretraining dataset before using the algorithm in real-world scenarios.
To ensure responsible use of the algorithm, we will make the datasets and code publicly available. Public availability of datasets and code allows for transparency and reproducibility. allowing other researchers to evaluate and address potential biases and misuse.
ACKNOWLEDGEMENTS
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2020R1 22C1A00005949, 2022R A4A 1033384, RSS2003-000217755) the MSIT(Ministry of Science and ICT), Korea, under the ICAN(ICT Challenge and Advanced Network of HRD) support program (RS-2023-00259934) supervised by the [Intttte for Infor- mation & Communications Technology Planning & Evaluation), the Yonsei University Research Fund (2023-22-0071) 1), and the Son Jiho Research Grant of Yonsei University (2023-22-0006)
REFERENCES
Ralph G Andrzejak, Klaus Lehnertz, Florian Mormann, Christoph Rieke, Peter David, and Christian E Elger. Indications of nonlinear deterministic and finite-ding dimenssomnii structures in time series of brain electrical activity: Dependence on recording region and brain state. Physical Review E, 64 (6):061907, 2001.
Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra Perez, and Jorge Luis Reyes Ortiz. A public domain dataset for human activity recognition using smartphones. In ESANN, pp. 437-442, 2013.
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.0007. 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. In NeurIPS, 2020.
Ling Cai, Krzysztof Janowicz, Gengchen Mai, Bo Yan, and Rui Zhu. Traffic transformer: Capturing the continuity and periodicity of time series for traffic forecasting. Transactions in GIS, 24(3): 736-755, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
Yanping Chen, Bing Hu, Eamonn Keogh, and Gustavo EAPA Batista. Dtw-d: time series semi- supervised learning from single example. In SIGKDD, 2013.
Joseph Y Cheng, Hanlin Goh, Kaan Dogrusoz, Oncel Tuzel, and Erdrin Azemi. Subject-aware contrastive learning for biosignals. arXiv preprint arXiv:2007.0488), 2020.
10

---

Published as a conference paper at ICLR 2024
Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The ucr time series archive. IEEE/CAA Journal of Automatica Sinica, 6(6):1293-1300) 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2018.
Qianggang Ding, Sifan Wu, Hao Sun, Jiadong Guo, and Jian Guo. Hierarchical multi-scale gaussian transformer for stock movement prediction. In IJCAI, pp. 4640-4646, 2020.
Jiaxiang Dong, Haixu Wu, Haoran Zhang, Li Zhang, Jianmin Wang, and Mingsheng Long. Simmtm: A simple pre-training framework for masked time-series modeling. In NeurlPS, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x words: Transformers for image recognition at scale. In ICLR, 2021.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: neirhstt contrastive learning of visual representations, In ICCV, 2021.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee-Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting, In IJCAI, 2021.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee-Keong Keong Kwoh, Xiaoli Li, and Cuntai Guan. Self-supervised contrastive representation learning for semi-supervised time-series classification. TPAMI, 2023.
Haoyi Fan, Fengbin Zhang, and Yue Gao. Self-supervised time series representation learning by inter-intra relational reasoning. arXiv preprint arXiv:2011.13544 2020.
Haoyi Fan, Fengbin Zhang, Ruidong Wang, Xunhua Huang. and Zuoyong Li. Semi-supervised time series classification by temporal relation prediction. In ICASSP, 2021.
Chen Feng and Ioannis Patras. Adaptive soft contrastive learning. ICPR, 2022.
Duarte Folgado, Marília Barandas, Ricardo Matias, Rodrigo Martins, Miguel Carvalho, and Hugo Gamboa. Time alignment measurement for time series. Pattern Recognition, 81:268-279, 2018.
Eves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In NeurIPS, 2019.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. EMNLP, 2021.
Chongjian Ge, Jiangliu Wang, Zhan Tong, Shoufa Chen, Yibing Song, and Ping Luo. Soft neighbors are positive supporters in contrastive visual representation learning. arXiv preprint 1rXiv:2303.17142, 2023.
Spyros Gidaris and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Ary I. Goldherger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung- Peng. and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: Components of a new research resource for complex physiologic signals. Circulation, 101(2))e215-<220. 2000.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006.
Zhicheng Huang. Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen. and Jiashi Feng. Contrastive masked autoencoders are stronger vision learners. arXiv preprint arXiv:2207. 13532, 2022.
11

---

Published as a conference paper at ICLR 2024
Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre- Alain Muller. Deep learning for time series classification: a review. Data Mining and Knowledge Discovery.33(4):917 33(4):917-90) 2019.
Bob Kemp, Aeilko H Zwinderman, Bert Tuk, Hilbert AC Kamphuisen, and Josefien JL Oberye. Analysis of a sleep-dependent neuronal feedback loop: the slow-wave microcontinuity of the eeg. IEEE Transactions on Biomedical Engineering. 47(9):1185-113 2000.
Dani Kiyasseh, Tingting Zhu, and David Clifton. Cloes: Contrastive learning of cardiac signals across space, time, and patients. In ICML, pp.
PMLR, 2021.
Jesus Lago, Fjo De Ridder, and Bart De Schutter. Forecasting spot electricity prices: Deep learning approaches and empirical comparison of traditional algorithms. Applied Energy. 221:386-405, 2018.
Nikolay Laptev, Y. B., and Saeed Amizadeh. A benchmark dataset for time series anomaly detection. httts::/yahooreseardd tumblr. http//ost////// 1eeest/114994234688 --benchhmar dataset- for-time series- -nomaly, 2015. Accessed: April 19, 2023.
Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee. i-mix: A domain-agnostic strategy for contrastive representation learning. In ICLR, 2021.
Christian Lessmeier, James Kuria Kimotho, Detmar Zimmer, and Walter Sextro. Condition monitoring of bearing damage in clcctromechanical drive systems by using motor current signals of electric motors: A benchmark data set for data-driven classification. In PHM Society European Conference, volume 3. PHM Society, 2016.
Guanghui Li, Jiahua Shen, Chenglong Dai, Jia Wu, and Stefanie Becker. Shveege: Eeg clustering with improved cosine timilarity-transfommee shapley value. IEEE Transactions on Emerging Topics in Computational Intelligence, 2022.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In ICLR, 2020.
LinkedIn. Luminol: Anomaly detection and correlation library. https: https:://github...
2018
Jun Liu, Lin Zhong, Jehan Wickramasuriya, and Vijay Vasudevan. Uwave: accelerometer- personalized gesture recognition and its applications. Pervasive and Mobile Computing, 5(6): 657-675, 2009.
Yushan Nie, Nam H Nguyen, Pattarawat Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In ICLR, 2023.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 18Xiv:180.003744 2018
Faraz Rasheed, Peter Peng, Reda Alhajj, and Jon Rokne. Fourier transform based spatial outlier mining. In Intelligent Data Engineering and Automated Learning-IDEA. 2009: 10th International Conference, Burgos, Spain, September 23-26, 2009, Proceedings 10, pp. 317-324. Springer, 2009,
Hansheng Ren, Bixiong Xu, Yujing Wang, Chao Yi, Congrui Huang, Xiaoyu Kou, Tony Xing, Mao Yang. Jie Tong, and Qi Zhang. Time-series anomaly detection service at microsoft. In SIGKDD, 2019.
Stan Salvador and Philip Chan. Toward accurate dynamic time warping in linear time and space. Intelligent Data Analysis, 11(5):561-580, 2007.
Pritam Sarkar and Ali Etemad. Self-supervised learning for beg-based emotion recognition. In ICASSP, 2020.
12

---

Published as a conference paper at ICLR 2024
Li Shen, Yuning Wei, and Yangzhu Wang. Respecting time series properties makes deep time series forecasting perfect. arXiv preprint arXiv:2207.10%. 2022.
Pengxiang Shi, Wenwen Ye, and Zheng Qin. Self-supervised pre-training for time series classification. In IJCNN, 2021.
Alban Siffer, Pierre-Alain Fouque, Alexandre Termier, and Christine Largouet. Anomaly detection in streams with extreme value theory. In SIGKDD, 2017.
Christopher A Sims, James H Stock, and Mark W Watson. Inference in linear time series models with some unit roots. Econometrica: Journal of the Econometric Society, pp. 113-144, 1990.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang. Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. NeurlPS, 2020.
Jose Roberto Ayala Solares, Francesca Elisa Diletta Raimondi, Yajie Zhu, Fatemeh Rahimian, Dexter Canoy, Jenny Tran, Ana Catarina Pinho Gomes, Amir H Payberah, Mariagrazia Zottoli, Milad Nazarzadeh, al. Deep learning for electronic health records: A comparative review of multiple deep neural architectures. Journal of biomedical informatics, 101:103337, 2020.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NewrIPS, 2017.
Janine Thoma, Danda Pani Paudel, and Luc V Gool. Soft contrastive learning for visual localization. In NeurlPS, 2020.
Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. In ICLR, 2021.
Owen Vallis, Jordan Hochenbaum, and Arun Kejariwal. novel technique for long-term anomaly detection in the cloud. In 6th USENIX workshop on hot topics in cloud computing (HorCloud 14), 2014.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(11), 2008
Zhiyuan Wang, Xovee Xu, Weifeng Zhang, Goce Trajcevski, Ting Zhong, and Fan Zhou. Learning latent seasonal-trend representations for time series forecasting. In NeurlPS, 2022.
Chen Wei, Huiyu Wang, Wei Shen, and Alan Yuille. Co2: Consistent contrast for unsupervised visual representation learning. In /CLR, 2021.
Kristoffer Wickstrøm, Michael Kampffmeyer, Karl Øyvind Mikalsen, and Robert Jenssen. Mixing up contrastive learning: Self-supervised representation learning for time series. Pattern Recognition Letters, 155:54 61, 2022.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. In ICLR, 2022.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR, 2022.
Haowen Xu, Wenxiao Chen, Nengwen Zhao, Zeyan Li, Jiahao Bu, Zhihan Li, Ying Liu, Youjian Zhao, Dan Pei, Yang Feng, et al. Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications. In WWW, 2018.
Xinyu Yang. Zhenguo Zhang, and Rongyi Cui. Timeclr: A self-supervised contrastive learning framework for univariate time series representation. Knowledge Maased Systems, 245:10 2022.
Hugo Yèche, Gideon Dresdner, Francesco Locatello, Matthias Hüser, and Gunnar Rätsch. Neigh- bothood contrastive learning applied to online patient monitoring. In ICML, pp. 11964-11974. PMLR, 2021.
Zhanwei Yue, Yiqun Wang, Jinghua Duan, Tao Yang, Chen Huang, Yunhai Tong, and Bo Xu. Ts2vec: Towards universal representation of time series. In AAAI, 2022.
13

---

Published as a conference paper at ICLR 2024
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer- based framework for multivariate time series representation learning, In SIGKDD, 2021.
Xiaotian Zhang, Zeyu Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. In NeurIPS, 2022.
Haoyi Zhou, Shanghang Zhang. Jiegi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI, 2021.
14

---

Published as a conference paper at ICLR 2024
A DATASET DESCRIPTION
A.1 CLASSIFICATION
For time series classification, we use the UCR archive (Dau al., 2019) and UEA archive (Bagnall et al., 2018). The UCR archive contains 128 univariate datasets, while the UEA archive contains 30 multivariate datasets. Among them, some datasets cannot be handled by T-Loss (Franceschi et al., 2019), TS-TCC (Eldele et al., 2021), and TNC (Tonekaboni et al., 2021) due to missing observations, such as DodgerLoopDay, Lopgere and DodgerLooop reported result for the DTW-D (Chen et al., 2013) on the InsectWingheat dataset in the UEA archive. Hence, the comparison is conducted using the remaining 125 UCR datasets and 29 UEA datasets in the main paper. However, Ve2 works well on all UCR and UEA datasets, all 128 UR datasets and UEA datasets for ablation studies for our method on top of TS2
Additionally, there is no
experiment with
A.2 SEMI-SUPERVISED CLASSIFICATION
Table A.I describes the summary of the statistical information for eight datasets (Anguita et al., 2013; Andrzejak al., 2001; Dau al., 2019) used in semi-supervised classifiaction, including the number of training and testing samples, data length, the number of sensor channels, and the number of classes.
Dataset
Train
# Test
Length
# Channel
# Class
HAR
7,352
2,947
128
6
Epilepsy
9,200
2,300
178
2
Wafer
1,000
6,174
152
2
FordA
1,320
3,601
500
2
FordB
3,636
810
500
2
POC
1,800
858
80
2
StarLightCurves
1,000
8,236
1,024
3
ElectricDevices
8,926
7,711
96
7
Table A.1: Eight datasets used for semi-supervised classification
A.3 TRANSFER LEARNING
We evaluate our approach on various datasets, which cover a wide range of application scenarios, in- cluding neurological healthcare, human activity recognition, mechanical fault detection, and physical status monitoring. Table A.2 deseribes the datasets in-domain and cross-domain transfer learning.
Fault Diagnosis (FD) datasets were used for transfer learning under self- and semi-supervised settings.
The data statistics are described below.
Dataset
Samples
Channels
Classes
Length
Freq (H2)
Pre-training
SleepEEG
371.055
1
5
200
100
In do-annii
Epilepsy
20 11,420
178
174
Fine-tuning
FD-B
/ 13,559
1
5,120
64,000
Gesture
320 / 120 120
3
8
315
100
EMG
122 /1 /41
1
1,500
4,000
Table A.2: In the four application scenarios, utilize pre-training dataset and fine-tuning dataset, with the latter having a sample size denoted by "A/B/C," where each denotes the number of samples used for fine-tuning, validation, and testing, respectively. Our evaluation also focuses small datasets, with a very limited sample size of less than 320 samples in the fine-tuning dataset, ensuring that the tine-tunin set is balanced in terms of classes. This approach enables us test our model's effectiveness small datasets, which has practical significance.
(1) SleepEEG (Kemp et al., 2000) dataset contains EEG recordings of 153 whole-night sleep sessions from 82 healthy individuals. We segmented the EEG signals using a non-overlapping approach, following the same preprocessing method as (Zhang al., 2022), to obtain 371,055 univariate brainwaves, each sampled at 100 Hz and categorized into one of five sleep stages: Wake, Non-rapid eye movement (3 sub-states), and Rapid Eye Movement. When using SleepEEG dataset source
15

---

Published as a conference paper at ICLR 2024
dataset in transfer learning task, cosine similarity instead of DTW due to the property of EEG datasets (Li et al., 2022).
(2) Epilepsy (Andrzejak et al., 2001) dataset monitors brain activity using a single-channel EEG sensor on 500 subjects, with each subject being recorded for 23.6 seconds. The dataset is sampled at 178 Hz and contains 1 11,500 samples. We followed the same preprocessing method as (Zhang et al., 2022) and classified the first four classes (eyes open, eyes closed, EEG measured in the healthy brain region, and EEG measured in the tumor region) of each sample as positive, while the remaining classes (whether the subject has seizure episode) were classified as negative.
(3) FD-B (Lessmeier et al., 2016) dataset is collected from electromechanical drive systems and monitors the condition of rolling bearings to detect their failures based on monitoring conditions such as speed, load torque, and radial force. It consists of 13,640 samples, each recorded at 64k Hz and categorized into three classes: undamaged, inner damaged, and outer damaged.
(4) Gesture (Liu et al., 2009) dataset includes data on 8 hand gestures based on hand movement paths recorded by an accelerometer. The eight gestures are hand swiping left, right, up, and down, hand waving in a counterclockwise or clockwise circle, hand waving in a square, and waving a right arrow. The dataset contains 440 balanced classification labels, with each sample having eight different categories gestures.
(5) EMG (Goldberger et al., 2000) dataset consists of 163 single-channel EMG recordings from the tibialis anterior muscle of three healthy volunteers suffering from neuropathy and myopathy. Each sample associated with one of three classes, with each class representing a different patient. The dataset is sampled at 4K Hz.
(6) FD (Lessmeier et al., 2016) dataset was obtained by monitoring the sensor readings of a bearing machine while operated under four distinct working conditions. Each working condition can be regarded as separate domain since they exhibit unique features, such as variations in rotational speed and load torque. Within each domain, there are three categories: two fault classes, inner and outer fault, and one healthy class. The FD dataset has 8,184 training samples, 2,728 test samples, a data length 5,120, one channel, and three classes. Our main goal is to use this dataset to conduct transferability experiments under both self- and supi-supe settings and demonstrate the efficiency of our approach in transfer learning situations.
A.4 ANOMALY DETECTION
We employed Yahoo (Laptev et al., 2015) and KPI (Ren et al., 2019) for the anomaly detection task. Yahoo a benchmark dataset that contains 367 hourly sampled time series with annotated anomaly points. This dataset covers a wide range of anomaly types, including outliers and change-points KPI a competition dataset released by AIOPS Challenge 2019. It contains several minutely sampled real KPI curves from diverse internet companies.
B BASELINE METHODS
Classification:
The results of all baseline methods for the classification task (DTW-D (Chen et al.,
2013), TNC (Tonekaboni et al., 2021), TST (Zerveas et al., 2021), TS-TCC (Eldele et al., 2021), T-Loss (Franceschi et al., 2019), and TS2Vec (Yue et al., 2022)) are reported in Yue et al. (2022).
DTW-D (Chen et al., 2013): DTW-D (Dynamic Time Warping-Delta is a variant of DTW under semi-supervised learning settings.
TNC (Tonekaboni et al., 2021): TNC (Temporal Neighborhood Coding) defines temporal neighborhood of window using normal distribution, and defines samples in neighborhood and non-neighborhood as positives and negatives, respectively.
TST (Zerveas et al., 2021): TST (Time Series Transformer) adopts the masked modeling paradigm to time series domain, where the goal is to reconstruct the masked time stamps.
TS-TCC (Eldele et al., 2021): TS-TCC (Time-Series representation learning framework via Temporal and Contextual Contrasting) proposes a new temporal contrastive loss by making the augmentations predict each other's future.
16

---

Published as a conference paper at ICLR 2024
T-Loss (Franceschi et al., 2019): T-Loss is a triplet loss designed for time series. It samples a random subseries from a time series and treats them as positive when they belong to its subseries, and negative if belong to subseries of other time series.
TS2Vec (Yue et al., 2022): TS2Vec splits time series into several subseries and defines hierar- chical contrastive loss in both instance-wise and temporal dimensions.
Semi-supervised classification: The results of all baseline methods for semi-supervised classification
using self-suuervised methods (SSL-ECG (Sarkar & Etemad, 2020), CPC (Oord al., 2018), SimCLR (Chen et al., 2020), TS-TCC (Eldele et al., 2021)) and semi-supervised methods (Mean- Teacher (Tarvainen & Valpola, 2017), DivideMix (Li al., 2020), SemiTime (Fan et al., 2021), FixMatch (Sohn et al., 2020), CA-TCC (Eldele et al., 2023)) are reported in Eldele et al. (2023).
SSL-ECG (Sarkar& & Etemad, 2020): SSL-ECG (Self-supervised ECG Representation Learning for Emotion Recognition) proposes ECG-based emotion recognition using multi-task self- supervised learning
CPC (Oord et al., 2018): CPC (Contrastive Predictive Coding) combines predicting future observations (predictive coding) with a probabilistic contrastive loss.
SimCLR (Chen et al., 2020): SimCLR proposes simple framework for contrastive learning of visual representations, without requiring specialized architectures or a memory bank.
Mean-Teacher (Tarvainen & Valpola, 2017): Mean-Teache is an algorithm for semi-supervised algorithm, that averages model weights instead of predictions.
DivideMix (Li et al., 2020): DivideMix uses a mixture model to divide training data into labeled clean samples and unlabeled noisy samples, and trains a model on both sets in semi-supervised way.
SemiTime (Fan et al., 2021): SemiTime conducts supervised classification on labeled time series data and self-supervised prediction of temporal relations unlabeled time series data. It achieves this by sampling segments of past-future pairs from the same or different candidates and training the model to distinguish between positive and negative temporal relations between those segments.
FixMatch (Sohn et al., 2020): Fix.Match generates pseudo-labels using the model's predictions on weakly-augmenter unlabeled images, and retain the pseudo-label with a high-confidence prediction. Then, the model is trained to predict the pseudo-label when fed a -uggentdd version of the same image.
CA-TCC (Eldele et al., 2023): CA-TCC supevvupe Contrastive Representation Learning for Semi-supervised Time-Series Series Classification) is the extension of TS-TCC to the semi- supervised settings, and adopts the same contrastive loss as TS-TCC.
Transfer learning: The results of baseline methods for transfer learning in both in-domain and cross-domain settings (TS-SD (Shi al., 2021), TS2Vec (Yue et al., 2022), Mixing-Up Up (Wickstrøm et al., 2022), TF-C (Zhang et al., 2022). TS-TCC (Eldele et al., 2021), TST (Zerveas et al., 2021), SimMTM (Dong et al., 2023)) using SleepEEG dataset as the pre-training dataset, are reported in Dong et al. (2023), except for results of TS-SD which are reported in Zhang et al. (2022). The results of baseline methods for transfer learning in both self-supervised and semi-supervised settings (Supervised, TS-TCC (Eldele et al., 2021), CA-TCC (Eldele et al., 2023)), using FD dataset as the pre-training dataset, are reported Eldele et al. (2023).
TS-SD (Shi et al., 2021): TS-SD utilizes triplet similarity discrimination task train a model. The objective is to determine which of the two TS is more similar to given TS, with DTW employed as a means to define the similarity.
Mixing-Up Up (Wickstrom et al., 2022): Mixing-up generates new time series by mixing two time series, and predicts the mixing weights.
TF-C (Zhang et al., 2022): TF-C generates both time-based and frequency-based representations of time series and proposes a novel time-frequency consistency architecture.
SimMTM (Dong et al., 2023): SimMTM adopts the masked modeling paradigm to time series domain, where the goal is to reconstruct the original time series from multiple masked series.
17

---

Published as a conference paper at ICLR 2024
Anomaly detection; The results of all baseline methods for the anomaly detection task (SPOT (Sif- fer et al., 2017), DSPOT (Siffer et al., 2017), DONUT (Xu et al., 2018), SR (Ren et al., 2019), FFT (Rasheed et al., 2009), Twitter-DD (Vallis et al., 2014), Luminol (Linkedln, 2018), TS2Vec (Yue et al., 2022)) are reported in Yue et al. (2022).
SPOT (Siffer et al., 2017): SPOT is novel outlier detection approach for streaming univariate time series, based on Extreme Value Theory, which does not rely on pre-set thresholds, assumes no distribution, and only requires single parameter to control the number of false positives.
DONUT (Xu et al., 2018): DONUT is an unsupervised anomaly detection algorithm based on variational autoencoder.
SR (Ren et al., 2019): SR is a time-series anomaly detection algorithm that is based on the Spectral Residual (SR) model and Convolutional Neural Network (CNN), where the SR model is borrowed from visual saliency detection and combined with CNN to improve its performance.
FFT (Rasheed et al., 2009): FFT uses fast fourier transform to detect the areas with high frequency change.
Twitter-DD (Vallis et al., 2014): Twitter-DD automatically detects long-term anomalies in cloud data by identifying anomalies in application and system metries.
Luminol (LinkedIn, 2018): Luminol is a Python library for time series data analysis that provides two main functionalities anomaly detection and correlation and can be utilized to investigate the potential causes of anomalies.
C IMPLEMENTATION DETAILS
The table of hyperparameter settings that utilized can be found in Table C.I. We made of five hyperparameters: T/. Tr, X, batch size (bs), and learning rate (Ir). For semi-supervised classification and transfer learning, we set the weight decay to 3e-4, 81 0.9, and 83 = 0. The number of optimization iterations for classification and anomaly detection tasks is set to 200 for datasets with size less than 100,000; otherwise, it is set to 600. Additionally, the training epochs for semi-superviae classification are set to 80, while for transfer learning, it is set to 40.
Since we utilized soft contrastive loss as an auxiliary loss for TS-TCC and CA-TCC, which are the methods involved in solving semi-supervised classification and transfer learning tasks, introduced an additional hyperparameter X to control the contribution of the auxiliary loss to the final loss.
Classification Forecasting
Semi-supervised classification Transfer learning
Anomaly detection
T7
[1. 2. 3, 4, 5. 10, 10,
[10. 20, 30, 40. 50)
FT
[0.5, 1.0, 1.5, 2.5)
[1.5, 2.0, 2.5]
A
0.5
[0.3. 0.5]
0.5
Aguz
[0.1, 0.3, 0.5]
bs 0.001
16
(yahoo) / (kpi)
Ir
0.0003
0.001
Table C.I: Hyperparameter settings for various tasks
D PROBABILISTIC INTERPRETATION OF SOFT CONTRASTIVE LOSSES
Inspired by the fact that the contrastive loss can be interpreted as the cross-entropy loss with virtual labels defined per batch, or equivalently, the KI. divergence of the predicted softmax probability from the virtual label or hard assignment (Lee et al., 2021), we define a softmax probability of the relative similarity out all similarities considered when computing the loss, and interpret our soft contrastive losses weighted sum of the cross-entropy losses. In this section, we show that the proposed contrastive loss can also be seen as the scaled KL divergence of the predicted softmax probabilities from the normalized soft assignments, where the scale is the of soft assignments. When hard assignment is applied, the loss becomes the standard contrastive loss, which is often called InfoNCE (Oord et al., 2018).
18

---

Published as a conference paper at ICLR 2024
D.1 PROBABILISTIC INTERPRETATION SOFT INSTANCE- WISE CONTRASTIVE LOSS
To simplify indexing, we extend soft assignments to incorporate the positive sample and anchor itself:
if i'; if + and i = i' i'(mdd N);
w/((,0)
(D.1)
(i, mod N), otherwise:
and let qr(i, = w/y (i, i')/2y be its normalization, where 21 Σ j=1 w/y (i, is the partition function. Then, we can rewrite the proposed soft instance-wise contrastive loss as follows:
2N
e(
elc)) = - log + (),
wr(i,j mod N) log pr ().
2N (i,j) (oggr(((,,)))
Σ j=l
2N
w'y (i,j) loggr(((..))tt)
Zi
Z1
j=1
2N
log qr(i,
Zi
_ j=1
(i(i,))
qı (i.j) (..j)logg log 91 (i.
(D.2)
pr((i.j),t)
constant
Let Qr and Pr be the probability distributions of qr (i.j), and pr((i.)), respectively, Then, can rewrite the above loss as:
efteet Z₁ (K(()))) + const,
(D.3)
which is the scaled KL divergence of the predicted softmax probability from the soft assignments.
D.2 PROBABILISTIC INTERPRETATION SOFT TEMPORAL CONTRASTIVE Loss
To simplify indexing, we extend soft assignments to incorporate the positive sample and anchor itself:
0, 1, (1, l' mod T), otherwise;
i =t; if + 1' and / = t'(mod T);
(1,
(D.4)
and let (t, t') = (t, be its normalization, where 27
Σe w/,
2T
(t, s) is the partition
function. Then, we can rewrite the proposed soft temporal contrastive loss as follows:
2T
(1.0) = log (7)
Σ
wr(t, s mod N) Pr (f, s))
2T
Σ
(t, (t, 5) prg (t)
2T
w (t,
ZT
log (t)
ZT
2T
log (t,
Zr
Σ
qr (t. s)
pr(i, (t, qr (t, s) log (t,
(D.5)
constant
Let Qr and Pr. be the probability distributions of qr, s), and (r (t, s)). respectively. Then, we can rewrite the above loss as:
(101 - KT (L((KLQQPPP) Pr) + const,
(D.6)
which is the scaled KL divergence of the predicted softmax probability from the soft assignments. These answer to concern that targets fixed while the predicted softmax probabilities are relative to the samples in the batch: the formulation with fixed targets proportional to the formulation with relative targets, and their difference is only in the optimization speed by the scale Z, and ZT.
19

---

Published as a conference paper at ICLR 2024
E HIERARCHICAL SOFT TEMPORAL CONTRASTIVE LOSS
In our approach, we adopt the hierarchical contrastive loss proposed in Vec (Yue et al., 2022), where we apply max- pooling on the representations along the temporal axis and contrastive learning is performed at each level. However, as max pooling proceeds, semantic similarities between adjacent time step decrease, so the sharpness needs to be adjusted based on the hierarchy depth and kernel size.
Sharpness
Avg. Acc. (%)
TT
83.3
m TT
83.7
Table E.1: Effect of hierarchical Fr
To address this, we increase the value of sharpness in soft temporal contrastive loss as the depth of the network increases, thereby reflecting the hierarchy of the time series. That is, we use m Tr instead of Tr for the sharpness value soft temporal contrastive loss, where m is the kernel size pooling layers and k is the depth. For all datasets, we set m to 2, while the value of l: depends on the length of each specific dataset. We conducted an ablation study to assess the effect of using hicrarchical sharpness, by comparing the performance of using hierarchical sharpness (m* TT) against constant sharpness (Tr) using 128 datasets in UCR archive (Dau et al., 2019). To solely observe the effect of hierarchical temporal contrastive loss, employ the original hard instance- wise contrastive loss for this experiment. The results presented Table E. demonstrate that increasing TT as the depth of the network increases leads to improved performance.
F DESIGN FOR INSTANCE- WISE CONTRASTIVE Loss
In this study, we explore different options for the soft assignments used in the soft instance-wise contrastive loss: without kernel (w/o kernel), Laplacian kernel, and Gaussian kernel. For w/o kernel, we use w/ (i.j)
Method
Avg. Acc.(%)
w/o kernel
79.1
Gaussian
82.6
1 D(I;, where D is min-max normalized dis- tance metric. For the Laplacian kernel, we use wy (i,j) =
Laplacian Sigmoid 83.9
83.1
exp (xp(-24 -( and for the Gaussian kernel, we use
wy (i,j) = exp (D(R..2)))) where a is a hyperparam- eter. For all kernels, sample pairs with lower distance
Table F.1: Design for instance-wise CL
tends to have soft assignments closer to one. We conduct an ablation study by comparing the per- formance of using the above kernels to model the soft assignments using the UCR archive datasets, and the results are presented in Table F.1. For this experiment, we employ the original hard temporal contrastive loss to solely observe the effect of the functions used for the instance-wise contrastive loss.
G CONTRASTIVE LEARNING FOR ANOMALY DETECTION TASK
Table G.I indicates that employing only temporal contrastive loss, while excluding instance-wist contrastive loss, yields better performance in the majority hard CL and soft CL settings for anomaly detection tasks. This can be attributed to the nature of the anomaly detection task, which involves detecting anomalies within a time series, and is less concerned with other time series.
Yahoo
Yahoo
Fi
Prec:
Rec.
Fi
Prec.
Rec.
F;
Prec.
Rec.
Fj
Prec:
Rec.
TS2Ve:
w/o inst
w/ inst
71.8
72.4
69.3
67.6
76.5
75.7
67.6
68.3
90.9
90.9
54.6
53.7
TS2Vec
who inst
w/ inst
75.5
74.0
73.6
70.7
77.4
77.6
68.9
69.7
89.3
88.8
57.4
56.2
w/ inst
71.2
67.8
74.9
66.4
94.3
51.4
w/ inst
74.6
72.1
77.5
69.0
92.1
56.2
Ours
w/o
74,2
72.2
76.5
70.1
91.6
57.0
Ours
w/o inst
76.2
75.3
77.3
69.7
92.1
57.4
(a) Results of AD task on normal setting
(b) Results of AD task on cold-start setting
Table G. Results of anomaly detection task by the of instance-wise contrastive loss.
20

---

Published as a conference paper at ICLR 2024
H TIME SERIES FORECASTING
The tasks mentioned in the main paper, except for anomaly detection, can be classified as high-level tasks, which requires capturing instance-wiss representations. High-level tasks generally perform better with CL. methods than with masked modeling methods (Dong al., 2023; Huang et al., 2022; Xie et al., 2022). However, we can perform low-level tasks such as time series forecasting, when using encoder architectures that can obtain representations of each timestamp.
For TS forecasting, we apply SoftCLT to both TS2Vec and CoST (Woo et al., 2022). Capturing temporal information within time series is crucial for time series forecasting, so we use soft CL in two ways for Te2Vec: by adopting only temporal contrastive loss and by using both temporal and instance-wise contrastive loss. For the experiment, we use four datasets, ETTh1, ETTh2, ETTml, and electricity dataset (Zhou et al., 2021) for TS2Vec, and 1TTh ETTh2, ETTml, and Weather dataset (Zhou et al., 2021) for CoST, under both univariate and multivariate settings. Table H.1 describes the summary of the statistical information for the five datasets. As demonstrated in Table H.2 and Table H.3, our method results in performance gains compared to hard CL in both univariate and multivariate TS forecasting for both TS2Vec and CoST.
Datasets
Channels
Prediction Length
Samples
ETTh,, ETTh₂
7
8640 / 2880 2880
Electricity
321
(24,46,168,336,72)
15782 5261 / 5261
Weather
21
36792 5271 10540
ETTm;
7
92,46,996,88,672)
34560 / 11520 / 11520
Table H.1: Four datasets used for time series forecasting, organized in the format of train/valid/test.
Univariste forecasting
Molticuriate forrcasing
wi insiance CL
w/o insance Wise
insuunceee
Wise
TS2Nc
TS2Voc
TS2Voc
TS2Vcc
Dutaset
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAE
MSB
MAE
24
0.042
0.152
0.641
0.156
0,016
0.164
0,645
0,161
0.568
0.513
0.554
0.506
0.568
0.525
0.554
0,510
48
0.067
0.197
0,064
0.194
0.079
0.216
0.080
0.216
0.897
0.538
0.595
0,532
0.617
0.557
0.598
0,542
ETTh
336
168
0,174
0.154
0.304
0.332
0.144
0.162
0.293
0.318
0.172
0,153
0.302
0.328
0.144
0.160
0.291
0.314
0.937
0.742
0,622
0.726
0.737
0,880
0.712
0.634
0.786
1,024
0.664
0.777
0.766
0.867
0.647
0,702
720
0,209
0,376
0.179
0.345
0.192
0.357
0.178
0.34
1,088
0.800
1,056
0.798
1,055
0
1,046
0.795
Avg:
0.179
0.272
0.120
0.261
0.178
0.233
0.121
0.265
0.784
0.640
0.766
0.634
0.814
0.665
0.765
0.639
24
0,090
0.230
0.086
0.224
0,390
0.229
0,888
0.326
0.373
0.465
0.370
0,462
0.371
0.462
0,362
0.452
45
0.126
0.273
0.121
0.268
0.121
0.258
0,119
0.265
0.361
0.579
0.557
0,577
0,548
0,571
0,535
0.559
168
0.208
0,359
0.302
0.354
0.196
0.349
0.194
0,347
1.713
1.015
1.713
1.06
1,693
1,024
1,666
1,001
336
0.219
0.314
0.306
0,363
0.207
0,364
0,205
0,362
11155
L167
2,061
1,147
1.096
1.172
1.973
L.135
720
0.221
0,351
0.316
0,377
0.217
0.377
0,215
0.376
2.637
1.299
2.394
1.278
2.664
1.319
2.297
1.259
Aeg:
0.173
0.323
0.166
0.316
0.166
0.317
0.164
0.315
1.447
0.905
1.441
0.895
1.434
0.910
1.365
0.881
24
0.016
0
0.014
0,088
0.015
0,092
0,014
0
0.699
0.649
0.418
0.426
0.428
0.430
0.421
0.423
45
0.029
0.128
0,627
0.124
0.028
0.126
0,627
0.124
0.000
0.521
0.567
0.501
0.587
0.512
0.568
0
ETTM:
288
96
0.108
0.044
0.15%
0.246
0.093
0.841
0.155
0.232
0.113
0.448
0.25%
0.166
0.115
0.848
0.166
0.260
0.927
0.000
0.532
0.586
0.591
0.647
0.577
0.530
0/623
0.704
0.544
0.500
01.95
01669
0.524
0.580
672
0.155
0.29%
0.135
0,283
0.163
0.313
0,160
0,311
0.750
0.639
0.743
0.637
0,797
0.659
0.753
0.642
Aog.
0.059
0,185
0,062
0.176
0.073
0.191
0,673
0.190
0.617
0.545
0,993
0.534
0.628
0.549
0
0,534
24
0.259
0.268
0.252
0.375
0.226
0.317
0.315
0.298
0.309
0.323
0.364
0.376
0.350
0.386
0.323
0.368
0.301
0.308
0.540
0.415
0.338
0.133
Flustriity
336
168
0.426
0.567
0.397
0.454
0.418
0.560
0.479
0.46
0.589
0.524
0.425
0.571
0
0,494
0.335
0.352
0.424
0.411
0.334
0.424
0.411
0.364
0.380
0.432
0.443
0.377
0.362
0.430
0.441
720
0.800
0.650
0.858
0.645
0.882
0.700
0.879
0
0.378
0.442
0.378
0.442
0.403
0.459
0.40)
0.457
Avg:
Table H.2: Results of univariate and multivariate time series forecasting.
21

---

Published as a conference paper at ICLR 2024
Univariate forecasting
Multivariate forecasting
CoST
+ Ours
CoST
+ Ours
Dataset
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
24
0.040
0.152
0.040
0.152
0.386
0.429
0.377
0,422
48
0.064
0.186
0.061
0.186
0.437
0.464
0.428
0,455
168
0.097
0.236
0.093
0.230
0.643
0.582
0.626
0,571
ETTh1
336
0.112
0.258
0.110
0.254
0.812
0.679
0.773
0.660
720
0.158
0.316
0.155
0.314
0.970
0.771
0.891
0.744
Avg.
0.094
0.230
0.091
0.227
0.650
0.585
0.619
0,570
24
0.079
0.207
0.078
0.206
0,480
0.525
0.472
0,522
48
0.118
0.259
0.117
0.257
0.751
0,669
0.749
0.667
168
0.189
0.339
0.179
0,332
1.613
1.017
1.602
1.009
ETTh2
336
0.206
0.360
0.201
0.357
1.807
1,078
1.800
1.075
720
0.214
0.371
0.208
0.369
1.959
1.099
1.951
1 1.090
Avg.
0.161
0.307
0.156
0.304
1.322
0.876
1.315
0.872
24
0.015
0.088
0.014
0.087
0.246
0,329
0.243
0.326
48
0,025
0.117
0.024
0.116
0.331
0.386
0.332
0,386
96
0.038
0.147
0.036
0.145
0.378
0,419
0.376
0.418
ETTm,
288
0.077
0.209
0.075
0.204
0.472
0.486
0,474
0.484
672
0.113
0.257
0.105
0.245
0.620
0.574
0.613
0.567
Avg.
0.054
0.164
0.051
0.159
0.409
0.439
0.407
0.436
24
0.096
0.213
0.095
0.212
0.299
0.361
0.298
0.360
48
0.139
0.263
0.132
0.260
0.359
0.411
0.357
0.410
168
0.213
0.338
0.208
0.335
0.464
0.491
0.463
0.491
Weather
336
0.235
0.360
0.208
0.357
0.499
0.517
0.496
0,513
720
0,251
0.375
0,231
0.357
0,535
0.544
0.538
0,544
Avg.
0.189
0.310
0.181
0.307
0.432
0.465
0.430
0.463
Table H.3: Results of univariate and multivariate time series forecasting for CoST.
I INSTANCE- WISE VISUALIZATIONS
Hard CL Soft CL. To assess the quality of instance-wise relationships captured by SoftCLT, we apply t-SNE (Van der Maaten & Hinton, 2008) to visualize the instance-wist representations, which are representations of whole time series obtained by max- pooling the representations of all time stamps, to both hard and soft CL. For this experiment, we apply our method to TS2Vec (Yue et al., 2022) with the TWaveGestureLibraryy dataset from UCR archive (Dau al., 2019). The results shown in Figure 1.1 demonstrate that soft CL finds more fine-grained neighborhood relationships and distinguishes them better than hard CL.
Embedding space vs. Input space. To assess the relationship between the shape of time series and their positions in the embedding dimension, we employ t-SNE (Van der Maaten & Hinton, 2008) to embed instance-wise representations of time series using the InsectEPGRReullrr UCR archive (Dau et al., 2019). Figure 1.2 illustrates the results, with the left panel displaying the points in the embedding space and the right panel presenting line plots of the original TS. The colors of the points and lines assigned based on the distances with their neighbors in the embedding space. From this figure, we observe that TS with the same color not only exhibit similar shapes, but also the points in the embedding space TS shift towards the upper left. This demonstrates that our method effectively captures detailed neighborhood relationships while maintaining alignment between the distances in the embedding space and the original input space.
dataset from
towards the upper right, the line plots of the original
22

---

Published as a conference paper at ICLR 2024
Hard
Soft CL
2000-
Figure I.1: Hard CL vs. Soft CL
Figure I.2: Instance-wise visualizations
TRANSFER LEARNING UNDER SEMI-SUPERVISED SETTINGS
In this study, we perform transfer learning in the semi-supervised settings using SleepEEG (Kemp et al., 2000) and EMG (Goldberger et al., 2000) datasets as the source and target datasets, respectively. Specifically, we apply our SofCCLT TS-TCC under semi-supervised settings where we perform fine-tuning using partially labeled datasets. Figure J.1 presents the results, which indicate that by using only 10% of labeled data with the soft CL framework (red line), are able to achieve an accuracy 92.69%, which is approximately 15% higher than the accuracy obtained from the hard CL framework (blue line) under fully supervised settings. Furthermore, using only 50% of the labeled dataset allowed achieve 100% accuracy, whereas the state-of-the-art performance using fully labeled datasets is 97.56%.
Transfer Learning Seni-supervised settings
TLTTE
Label Rot
Figure J.1: results
K EFFECT OF DISTANCE METRICS BY TIME SERIES WITH VARYING LENGTH
We compare the average accuracy of 128 UCR datasets, where 11 datasets have varying time-length, and the other 117 datasets have the same time-length. As shown in Table K.1, DTW and TAM, both capable of comparing time series of different lengths using time warping, demonstrate better performance.
UCR datasets (Avg. Acc.(%))
Temporal CL
Hard
Soft
Non-Varying (117/128)
Varying (11/128)
Total (128/128)
Non- Varying (117/128)
Varying (11/128)
Total (128/128)
TS length
COS EUC
84.8
72.6
83.7
85.7
75.0
84.7
73.3
83.9
85.8
73.9
84.8
84.8
73.6
83.9
85.9
85.0
TAM
85.0
73.4
83.9
85.9
75.3
85.0
Table K.I: Effect of DTW on time series with varying/non-varying length.
L DESIGN CHOICES FOR SOFT TEMPORAL CONTRASTIVE LEARNING
Various design choices can be considered for assigning soft labels in soft temporal contrastive learning. In this paper, we explore four different choices for the experiment, all of which assign high values to adjacent timestamps. Figure L. illustrates these four different choices. For Neighbor, Gaussian, and Sigmoid, we conduct search for the optimal hyperparameter within the following range:
t-3
Neighborhood
Linear
0.2
0.6
0.8
0.4
0.2
Gaussian, Bigmoid
0.1
0.2
0.7
0.9
0.9
0.7
0.2
D.1
Reference
Figure L.1: Design for soft temporal CL
Neighbor: A certain range within the reference point, with 10%, 30%, 50% of the sequence length.
Gaussian: Standard deviation values of [0.5, 1.0, 1.5, 2.0, 2.5].
Sigmoid: 77 of [0.5, 1.0, 1.5, 2.0, 2.5].
23

---

Published as a conference paper at ICLR 2024
M SOFT CONTRASTIVE LEARNING WITH NON-STATIONARY TIME SERIES
As our proposed soft instance-wise CL method generates labels by considering the distances between the original TS, global information from the entire TS is encapsulated within the representation of a single time step of that TS, which might enables to take account of non-stationarity. such as seasonality or distribution shifts present in the TS.
Time series with seasonality. Figure M.1 displays a single TS of Adiac data from the UCR archive (Dau al., 2019) and its visualization of temporal representations obtained from Ve2Vec (Hard CL) and Ve2 applied with our method (Soft CL). Note that obvious seasonal pattern are observed from the left panel of the figure. Each point in the right panel represents representation of single timestamp. The figure indicates that while hard CL fails to capture the seasonal patterns in the TS (as similar values, regardless of which phase they are in, are located closely), our proposed soft CL can grasp the global pattern, enabling it to capture the seasonal patterns (as similar values but with different seasonal phases are located differently).
(2)
(2)
(6)
(method: TS2Vec
id
t-SNE
Indox
phases:: (d)
Hard CL
Soft CL
Figure M.1: Temporal visualization TS with seasonality.
Time series with distribution shift. Figure M.2 displays single TS of EMD data from the UCR archive (Dau et al., 2019) and its visualization of temporal representations obtained from TS2Vec (Hard CL) and TS2Vec applied with our method (Soft CL). Note that distribution shifts exist in this data, and six different phases are observed from the left panel of the figure. Each point in the right panel represents representation of single timestamp. The figure indicates that while hard CL fails to capture the sudden change in the TS (as all points are located gradually regardless of the sudden change in the TS), our proposed soft CL. can detect such distribution shifts (as points before/after certain change are clustered in groups).
method: TS2Vec
(b)
10-
(e)
In
t-SNE
(di
(4)
phases: (a) (1)
Hard CL
Soft CL
Figure M.2: Temporal visualization of TS with distribution shift.
24

---

Published as a conference paper at ICLR 2024
N APPLYING SOFT CONTRASTIVE LEARNING TO TNC
In this section, we apply our SoftCLT temporal neighborhood coding (TNC) (Tonekaboni et al., 2021), which employs temporal CL by leveraging the local smoothness of a signal's generative process to define neighborhoods, or positive pairs, time with stationary properties. We apply our method to TNC by assigning soft temporal assignments based on the difference between the centroids of two time windows. Note that TNC does not include instance-wise CL, so we only apply soft temporal CL. To evaluate the effectiveness of this application, we conduct experiments using two datasets: simulation dataset constructed in TNC and HAR 3, The results are shown Table N.I, demonstrating that SoftCLT applied TNC improves performance on both datasets in terms of accuracy (Acc.) and AUPRC.
Simulation
HAR
A%c
AUPRC
Acc.(%)
AUPRC
T-Loss
76.66
0.78
63.60
0.71
CPC
70.26
0.69
86.43
0.93
TNC
97.52
0.996
88.21
0.940
TNC Ours
97.95
0.998
89.14
0.952
Table N.1: Effect of soft temporal CL on TNC.
hhttps://archive, www.:u.....eeeeee.oo /ttt.ccecttattsse/////////////// aatt i tecogn tion= maingeesmatspsnnes
25


Reviews:
[{"review_id": "eIozRiQg6V", "reviewer": "Reviewer_bKG2", "summary": "The authors address challenges in time series data annotation and the limitations of standard contrastive learning (CL) in representing TS data. Key contributions of the paper are:\n\n- Introduction of SoftCLT, a soft contrastive learning strategy tailored for time series data. Their framework can be adapted to other CL frameworks relatively easily.\n- The proposal of soft contrastive losses for both instance and temporal dimensions, addressing the shortcomings of existing CL methods for TS.\n- Comprehensive experimental evidence demonstrating that SoftCLT enhances state-of-the-art performance across multiple TS tasks.", "questions": "As noted in the Weaknesses section, I would like a detailed comparison with CoST and an evaluation in the regression setting. If the authors provide this, and the results warrant it, I will raise my score.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The submission has the following strenghts:\n\n- Ablation study is present and seems to demonstrate the usefulness of the proposed additions.\n- Compared to the selected baselines (emphasis on selected), the model performs well.\n- I appreciate that the authors have chosen to go for a more detailed analysis of the representation learning abilities of their model. By this I mean that rather than considering only task-performance, they also investigate aspects such as robustness to non-stationarity, and also semi-supervised learning. This is usually absent from related papers.\n- The ideas are well explained, the paper is clear.", "weaknesses": "The submission has the following weaknesses:\n- Problem with the comparisons. Entirely absent from the main paper is any comparison with CoST [1], or any more recent contrastive approach. While this is a single issue it is one I am quite concerned about. Similarly, a comparison to recent approaches in the regression setting of TS2Vec (nothing prevents that comparison, the TS2Vec code works seamlessly for both approaches). My reasoning is that the proposed idea is interesting, but also relatively simple. This is fine in general: simple ideas bring value in research as well. However, coupled with a lack of comparison to recent approaches, it is very difficult to ascertain the value of the contribution. \n\nCurrently this is enough for me to not recommend acceptance, but as noted in the questions section, I am willing to update my score should the authors adress this.\n\nReferences\n[1] CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MPT6GNPiay", "reviewer": "Reviewer_EHZC", "summary": "This study argues that when using time series data in contrastive learning, contrasting ( between positive and negative) instances or values located in proximity can lead to the neglect of their inherent correlation. Therefore, they introduce a continuous (referred as soft) weighting approach as an alternative to binary labeling, serving as a generalization of the standard contrastive loss, with the transformation occurring when replacing soft assignments with hard assignments of zero for negatives and one for positives. For soft assignment, the authors take into account two aspects: the similarity between two time series in data space and the proximity of two time series with respect to their timestamps.", "questions": "Can this case be elaborated a bit further:” when α = 1, we give the assignment of one to the pairs with the distance of zero as well as the pairs of the same TS” What if in the same TS we are experiencing two different patterns, shifts or different distribution\n\nIn equation 3, augmentation for I and i+N, how it  is performed? What if there is only a shift in the pattern in the instances, otherwise there are very similar how you address this in your computation, It would be great to include an illustration for this case to show you approach is robust to shift (or some noise) which is very common in real world applications.\n\n\nHow do you manage non stationary in the time series, where the immediate next point might be the start of a different distribution? How your similarity comparison handles it when the the proximity is assumed to have high similarity which is not necessarily true.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The papers is well written and clear. The figures presented help to clarify the main idea and how it is implemented. The idea is novel for the simplified setup that is considered. The experimental results cover 3 downstream tasks and comprehensively evaluate the assumed setup.", "weaknesses": "The paper addresses a simplified scenario in which issues related to noise, seasonality, and non-stationarity are not considered, as there is no apparent mechanism in the approach to address these prevalent challenges found in real-world time series data.\nRegarding robustness in the presence of noise and non-stationarity there  is no specific discussion or empirical evaluation.  Regarding seasonality, the authors mentioned \"Our conjecture is that TS in the real world usually do not exhibit the perfect seasonality, as indicated by the ADF test result, such that SoftCLT takes advantage of the non-seasonal portions.\" While perfect seasonality may be absent in some datasets and may vary in intensity across different datasets, I believe completely disregarding it is not a practical approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y0TM9bHSKr", "reviewer": "Reviewer_mvHV", "summary": "This study introduce a new method of performing constrastive learning, a soften version of normal positive-negative strategy. These soft assignments are determined by the distance between time series in the data space for instance-wise contrastive loss and the difference in timestamps for temporal contrastive loss.", "questions": "The authors please address or provide answers to any questions from the weaknesses listed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Contribution:\n\n- The idea of soft constrastive learning is straight-forward and natural. The underlying functions are widely-adopted and straightforward to implement.\n- The experiments are extensive and cover many time-series tasks (classification, anomaly detection) as well as scenario (self/semi supervised and supervised learning). The comparison with soft-CL techniques from other domains and ablation study make the whole experimental section be quite well-rounded.\n\nRepresentation:\n\n- Intuitive and direct illustrations via Figures (e.g. Fig.1, 2)", "weaknesses": "- Contribution:\n    - For instance-wise CL:\n        - the use of DTW might be a potential bottleneck in case of dealing with lengthy time-series. While the authors suggest the use of FastDTW, the complexity regarding the memory might be increased, and also the potential reduce in approximation (in case the warping path between two time series instances is highly nonlinear). In other words, the choices of DTW or FastDTW are hurting the pipeline in some ways.\n        - the calculation of weight based on the distance in the data space. However, this make the weighting process be dependent on the scale of input data. Together with the wrapper of Sigmoid function, it might be saturated upon too large or too small input. This effect might make the weights not representative to use in instance-wise CL. While empirically, it illustrates the effective over in latent space, more effort need to be done to consider on which space one should rely on to calculate distance.\n    - For temporal-wise CL, the current weight assignment implicitly assume the data from neighbors’ timesteps should be weighted heavier than the data from far timesteps. However, that behavior might not always hold true, as illustrated in work of Tonekaboni (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aAfq18jlxA", "reviewer": "Reviewer_e9k3", "summary": "The authors propose a new contrastive learning method for time series. Specifically, they propose to remove the hard positive/negative assignment from the original NCE by a soft reweighting incorporating prior information about the temporal closeness or similarity of inputs. The authors evaluate their method on various time series-related tasks showing strong improvement compared to \"hard\" CL methods. They also provide ablation experiments concerning their new objective hyperparameters.", "questions": "I don't have any questions beyond the points raised in the above sections.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper is of high quality with the following strengths: \n\n- Overall very well-written paper with an easy-to-follow structure, in particular:\n    - Related work is structured and extensive (with the exception mentioned below in weaknesses).\n    - Method is clear and figures relevant making the method easy to understand.\n    - Experiments are very extensive and well described both in the manuscript and the supplementary materials.\n\n- I appreciate that despite introducing numerous components and hyperparameters ( temperature, distance metric, weight function, etc..), authors provide ablations to each of these components. \n- The authors discussed the additional computational complexity of their method and in particular DTW known to have a squared complexity. \n- The amount of experiments carried out is very large and diverse", "weaknesses": "There are, however, some weaknesses, in particular in terms of related work, which I detail below:\n\n\n**Related work**\nAs mentioned the related work is extensive but misses some seminal works regarding contrastive learning methods for time series tackling the challenges of inter/intra samples dependencies:\n- First, \"Subject-aware contrastive learning for biosignals\" by Cheng et al (2020) proposes to only use negative representations from the same time series to \"promote subject-invariant\" representation, what the authors would refer to as \"temporal CL\".\n- Second, \"CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients\" by Kiyasseh et al. (2021) proposes to on the contrary use representations from the same time series as positives, what the authors would refer to as \"instance-wise CL\". This is similar to TNC but with a neighborhood being defined as being from the same time series. \n- Finally, and more importantly, \"Neighborhood Contrastive Learning Applied to Online Patient Monitoring\" by Yeche et al. (2021), introduces the trade-off used by the authors between instance-wise and temporal-wise CL controlled by $\\lambda$. In particular, the objective proposed by Yeche et al., namely NCL, is similar to taking a hard assignment for instance-wise CL $w_I(i,j) = \\mathbb{1}_{[i = j]}$ and a (discontinuous) uniform one over a window for temporal CL. \n\nThus, I think it's really important that this work refers to these three works and in particular NCL from which the SoftCLT is an extension to continuous neighborhood definitions. It would be nice to have a comparison to it as well. \n\n\n**Clarity**\n\n- The authors refer multiple times to instance-wise and temporal CL before defining it properly in the method section. I think pointing the reader to this section or defining the terms in the introduction could improve clarity.\n\n- referring to a \"temperature\" parameter $\\tau_t$ and $\\tau_i$ can be quite misleading in the context of contrastive learning, where this term was coined by Chen et al. (2020), in the simCLR paper. (See my comment below on the choice of assignment function) Given the assignment function is some form of Laplacian kernel, referring to $l =\\frac{1}{\\tau}$ as a lengthscale parameter would be more coherent with literature and avoid confusion with temperature parameters from previous works on CL. \n\n**Method**\n- The authors define their assignment function around a sigmoid function which is defined over $\\mathbb{R}$ whereas its input $D$ lies in $\\mathbb{R}^+$. It seems to overcome this, they tweak around their sigmoid function to obtain a symmetric function $w(D) = \\frac{2}{1+e^{Dt}}$. Why not rely on existing literature instead and typically use a Laplacian kernel $w(D) = e^{-\\frac{D}{l}}$? \n- Exploring further different kernel and their impact on performance would have been a nice addition. In particular, using a generalized Gaussian kernel and looking at the impact of the shape parameter $\\beta$ would be nice as $\\beta=1$ is SoftCLT and  $\\beta=\\infty$ is NCL temporal CL. \n- Exploring further the impact of the trade-off between local (temporal) and global (instance) features learning ruled by $\\alpha$ would be a nice addition to ablations. \n**Conclusion**\n\n\nClarity and Method weakness are easily addressable. Regarding related work, despite the similarities with NCL, I still think the contribution to be significant given the novelty around the neighborhood/assignment function and the extent of the experiments on various tasks, justifying my choice of recommending acceptance. However,  I firmly believe the three works I mentioned should be correctly cited in particular the link to Yeche et al. (2021) work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)