T-Rep: Representation Learning for Time Series using Time-Embeddings
Authors: Archibald Felix Fraikin
Adrien Bennetot
Stephanie Allassonniere
Venue: ICLR 2024 poster (2024)
Keywords: Multivariate time series
Self-supervised
Time series representations
Temporal features
Time-Embeddings
Representation Learning
Missing data
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: T-Rep is a self-supervised method to learn time series representations at a time-step granularity that outperforms existing self-supervised algorithms in classification, forecasting and anomaly detection tasks

Abstract: Multivariate time series present challenges to standard machine learning techniques, as they are often unlabeled, high dimensional, noisy, and contain missing data. To address this, we propose T-Rep, a self-supervised method to learn time series representations at a timestep granularity. T-Rep learns vector embeddings of time alongside its feature extractor, to extract temporal features such as trend, periodicity, or distribution shifts from the signal. These time-embeddings are leveraged in pretext tasks, to incorporate smooth and fine-grained temporal dependencies in the representations, as well as reinforce robustness to missing data. We evaluate T-Rep on downstream classification, forecasting, and anomaly detection tasks. It is compared to existing self-supervised algorithms for time series, which it outperforms in all three tasks. We test T-Rep in missing data regimes, where it proves more resilient than its counterparts. Finally, we provide latent space visualisation experiments, highlighting the interpretability of the learned representations.

Published as a conference paper at ICLR 2024
T-REP: REPRESENTATION LEARNING FOR TIME SE- RIES USING
Archibald Fraikin
Adrien Bennetot
Let Care
Let Care
PariSanté Campus, Paris, France htt.bbalddfraikiiein...
PariSanté Campus, Paris, France adrien www.eeetteeeetteeerreeeeeeeeeeee
Stéphanie Allassonnière
Université Paris Cité, INRIA, Inserm, SU Centre de Recherche des Cordeliers, Paris stephanie. all aassnnieree@inriaa e
ABSTRACT
Multivariate time series present challenges to standard machine learning tech- niques, they are often unlabeled, high dimensional, noisy, and contain miss- ing data. To address this, we propose T-Rep, a self-supervised method to learn time series representations dings of time alongside its feature extractor, to extract temporal features such as trend, periodicity, distribution shifts from the signal. These time-embeddings are leveraged in pretext tasks, to incorporate smooth and fine-grained temporal de- pendencies in the representations, as well reinforce robustness missing data. We evaluate T-Rep downstream classification, forecasting, and anomaly detec- tion tasks. It is compared to existing self-supervised algorithms for time series, which it outperforms in all three tasks. We test T-Rep in missing data regimes, where it proves more resilient than its counterparts. Finally, we provide latent space visualisation experiments, highlighting the interpretability of the learned representations.
timestep granularity. T-Rep learns vector embed-
INTRODUCTION
Multivariate time series have become ubiquitous in domains such as medicine, climate science, or finance. Unfortunately, they are high-dimensional and complex objects with little data being labeled (Yang & Wu 2006). as is an expensive and time-consuning process. Leveraging unlabeled data to build unsupervised representations of multivariate time series has thus become challenge of great interest, as these embeddings can significantly improve performance in tasks like forecasting, classification, anomaly detection (Deldari al. 2002 Su al. 2019). This has motivated the development of self-supervised learning (SSL. models for time series, first focusing on constructing instance-level representations for classification and clustering (Tonekaboni al. 202 I Franceschi et al. 2019 Wu et al. 2018). More fine-grained representations were then developed to model time series at the timestep-level (uue et al. (022), which key in domains such healthcare or sensor systems. With fine mme-grained embeddings, one can capture subtle changes, periodic patterns, and irregularities that are essential for anomaly detection (Keogh et al., (006) as well as understanding and forecasting disease progression. These representations can also be more resilient than raw data in the face of inter-sample variability or missing data (Yue al. 2022), common issues in Human Activity Recognition (HAR) and medicine.
A central issue when learning representations of time series is the incorporation of time the latent space, especially for timestep-level embeddings. In SSL, the temporal structure is learned thanks to the pretext tasks. In current -fate-of-the-arr (SOTA) models, these tasks are contrastive (ooekabooi et al. 2021 Yue et al. 2022 Banville et al.) 2021), which poses important limitations (Zhang al. 2023 In contrastive techniques, the learning signal is binary: positive pairs should be similar, while
negative pairs should be very different (Chen et al. 2020 This makes obtaining continuous

---

Published as a conference paper at ICLR 2024
fine-grate finegrainndd notion of time in the embeddings unfeasible, these tasks only express whether two points should be similar, but not how close or similar they should be. Embedded trajectories are thus unlikely to accurately reflect the data's 'sss temporal structure.
Further, temporal contrastive tasks are incompatible with finite-state systems, where the signal tran- sitions between S states through time, regularly (periodic signal) or irregularly. Such tasks define positive pairs by proximity in time, and negative pairs by points that are distant time (Banville) al. 2021 Franceschi et al.) 2019), which can incur sampling bias issues. Points of negative pair might be far in time but close to a period apart (i.e. very similar) and points of a positíve pair might be close but very different (think of pulsatile signal for example). This incoherent information hinders learning and may result in a poor embedding structure. Finite-state systems are extremely common in real-world scenarios such as sensor systems, medical monitoring, or weather systems, making the treatment of these cycles crucial.
To address the above issues, we propose T-Rep, a self-supervvsed method for learning fine-grainel representations of (univariate and multivariate) time series. T-Rep improves the treatment of time in SSL thanks to the use of time-embeddings, which are integrated in the feature-extracting encoder and leveraged in the pretext tasks, helping the model learn detailed time-related features. We define as time-embedddig a vector embedding of time, obtained as the output of a learned function hgs which encodes temporal signal features such trend, periodicity, distribution shifts etc. Time- embeddings thus enhance our model's resilience to missing data, and improve its performance when faced with finite-state systems and non-stationarity. We evaluate T-Rep wide variety of datasets in classification, forecasting and anomaly detection (see section 5). notably on Sepsis (Reyna et al.) 2020a Goldberger al. 2000) a real-world dataset containing multivariate time series from 40, 336 patients in intensive care units (ICU), featuring noisy and missing data. Our major contributions are summarised as follows:
To the best of our knowledge, we propose the first self-supervised framework for time series to leverage time-embeddings in its pretext tasks. This helps the model learn fine-grained temporal dependencies, giving the latent space a more coherent temporal structure than existing methods. The use of timeeembeddings also encourages resilience to missing data, and produces more information-dens and interpretable embeddings.
We compare T-Rep SOTA self-supervised models for time series in classification, fore- casting and anomaly detection. It consistently outperforms all baselines whilst using a lower-dimensional latent space, and also shows stronger resilience to missing data than ex- isting methods. Further, our latent space visualisation experiments show that the learned embeddings are highly interpretable.
RELATED WORK
Representation Learning for time series The first techniques used in the field were encoder-
decoder based, trained to reconstruct the original time series. Such models include Autowarp (Abid
& Zou| 2018), TimeNet (Malhotra et al., 2017) and LSTM-SAE (Sagheer & Kotb 2019). which
all feature an RNN-based architecture. Variational Auto-Encoders (Kingma & Welling 2013)
inspired models have also been used, notably Interfusion (Li et al. 2021). SOM-VAE (Fortuin et al. 2018), and OmniAnomaly (Su al. 2019) which combines VAE with normalising flows. Encoder-only methods have been more popular recently, often based on contrastive approaches (Zhang et al. (023). The Contrastive Predictive Coding (CPC) (Oord et al. 2018 framework tries to maximise the mutual information between future latent states and a context vector, using the infoNCE loss. This approach has been adapted for anomaly detection (Deldari et al.) 2021)
and general representations (Eldele et al., 2021). TS-TCC (Eldele et al. 2021) augments CPC by applying weak and strong transformations to the raw signal. Also, augmentation--asee contrastive methods in computer vision (Chen et al.) 2020) have been adapted to time series by changing the augmentations (Kiyasseh et al. 2021 J. Domain-specific transformations were proposed for wearable sensors (Cheng al. 2020) and ECGs (Kiyasseh et al.) 2021), such as noise injection, cropping, warping, and jittering (Pöppelbaum et al. 2022). The issue with these methods is they make cuinsformationnivaaiinn
assumptions which may not be satisfied by the signal (Zhang al. 2023 Yue et al. 2022). TS2 (Yue et al. 2022) addresses this with contextual consistency.
2

---

Published as a conference paper at ICLR 2024
Time--mbeddiing in time series representations have only been used in transformer- architectures, which require a positional encoding module (Vaswani et al.) 2017). While some use the original fixed sinusoidal positional encoding [Haresamudram et al., 2020; Zhang et al.
2022), Zerveas et al. (2021) and [Tipirneni & Reddy (2022) chose to learn a time-enbedding using a linear layer and fully-connected layer respectively. Using learning more sophisticated time-embeddings is a largely unexplored avenue that seems promising for dealing with long-term trends and seasonality (periodicity) in sequential data (Zhang et al. [023] Wen al. (022). The most elaborate timeednbediing for time series is Time2 Vec (Kazemi et al., 2019), which was developed for supervised learning tasks and has not yet been exploited in a self self-supervised setting.
In existing self-supprvised models, the time-embedding is used by the encoder to provide positional information (Zerveas et al. 2021] Tipirneni & Reddy] 2022, Haresamudram et al.] 2020 Zhang et al. 2022). but is never exploited in pretext tasks. The best performing models use contrastive techniques to learn temporal dependencies, which only provide a binary signal (Zhang et al. 2023). T-Loss (Franceschi et al. (2199) follows the assumption that neighboring windows should be similar, and builds a triplet loss around this idea. TNC (Tonekaboni et al. 2021) extends this framework, dividing the signal into stationary windows to construct its positive and negative pairs. In Banville et al. 2021];; Yue et al. (2022); Franceschi et al. (2019), positive and negative pairs are delimited by threshold on the number of differing timesteps, which makes these methods unsuited to capturing periodic or irregularly recurring patterns in the data. Further, all these contrastive methods are quite coarse, making hard to learn inn--grainet temporal dependencies.
To summarise, existing methods have made tremendous progress in extracting spatial features from time series, but temporal feature learning is still limited. In particular, they are not suited to handling recurring patterns (periodic or irregular), and struggle to learn fine -rained temporal dependencies, because of the binary signal and sampling bias of contrastive tasks.
3 BACKGROUND
The aim of this work is to improve the treatment of time in representation learning for temporal data. These innovations are combined with state-of-thh-arr art methods for spatial feature-learming and model training, the contextual consistency and hierarchical loss frameworks (Yue et al. 2022..
3.1 PROBLEM DEFINITION
Given a dataset X = (x1, (x., x.) E NXTXC of N time series of length T with C' channels, the objective of self-supervised learning is to learn function fa. s.t. Vi E [0, z, fo(x)). Each 2, E 7 is a representation of x; of length T and with F channels, which should preserve as many features of the original data possible. fo(-) is learned by designing artificial supervised signals, called pretext tasks, from the unlabeled data X.
3.2 CONTEXTUAL CONSISTENCY
The objective of contextual consistency is to learn context-invariant representations of time series. The idea is to sample two overlapping segments x, and x2 of the time series x, to which random timestamp masking is applied, thus creating two different contexts (random choice of window and masked timesteps) for the overlapping window. Representations in the overlapping timesteps then encouraged to be similar, leading to context-invariant representations. Two contrastive tasks were introduced by Yue et al. (202)) alongside the contextual consistency framework to extract spatial and temporal features.
Instance-wist wise contrasting encourages representations of the same time series under different con- texts to be encoded similarly, and for different instances to be dissimilar (Yue et al., 2022). Let B be the batch-size, i the time series index and f timestep. Zi, and z denote the corresponding representation vectors under different contexts. The loss function given by:
exp (z.. 2(,1)
(i.t) inst
- -log D/= (exp (Zi 24,,) + 1 Lizs exp (Zi, 2z,1))
(1)
3

---

Published as a conference paper at ICLR 2024
Temporal contrasting encourages representations of time series under different contexts to be coded similarly when their respective timesteps match, and far apart when the timesteps differ (Yue et al. 2022). The loss function is given in Eq. 2 where f2 is the set of timesteps in the overlap
between the subseries:
exp (z,, zi,e)
(i,1) ttmmp
log
(2)
Ever eepp 24, k/,0) 1y22 exp (Zise mn,)))
3.3 HIERARCHICAL LOSS
The hierarchical loss framework applies and sums the model's loss function at different scales, starting from a per-timestep representation and applying maxpool operations to reduce the time- dimension between scales (Yue et al. 2022). This gives users control over the granularity of the representation used for downstream tasks, without sacrificing performance. It also makes the model more robust to missing data, as it makes use of long-range information in the surrounding represen-
tations to reconstruct missing timesteps (Yue et al. 2022
METHOD
4.1 ENCODER ARCHITECTURE
We present below our convolutional encoder, which contains 3 modules. The overall model structure illustrated in Figure
Spolial features
I....
nnsaace wise contrasting
Unner Projection Layer
Embodding
conditioned forocasting
+
Herarchical loss
Tamporal contrasting
Diitted 1D CNN Encocw (fs)
TE chergenne prediction
Temporal featuree
T-Rep architecture and workflow. The left part shows how the different modules interact (linear projection, time-embedding module, and encoder). The middle part shows the pretext tasks used to train the model, and the kind of features they extract. The right hand side shows the loss computation: a linear combination of individual pretext task losses is passed to the hierarchical loss algorithm (see Appendix A.2.I
Figure 1:
Linear Projection Layer The first layer projects individual points X C to vectors U, R. with fixed number of channels F. Random timestamp masking is applied to each u, independently after the linear projection (only during training), as part of the contextual consistency framework Yue et al. 2022).
Time- Embedding Module The time-embedding module h& is responsible for learning time-related features тe (trend, periodicity, distribution shifts etc.) directly from the time series sample indices The time-embedding function not fixed like a transformer's positional encoding module, is learned jointly with the rest of the encoder. This makes the time-embeddings flexible, they adapt to the data at hand. The choice of architecture for the time-embedding module can impact performance in downstream tasks, and is discussed in Appendix A.4 For general applications, using Time2Vec (Kazemi et al. 2019),, which captures trend and periodicity. To the best our knowledge, T-Rep is the first model to combine a time-embedding module and convolutional encoder in self-super learning for time series.
recommend
The time-embedding module must return vectors which define probability distribution (positive components that sum to 1). This is due to the use of statistical divergence measures a pretext task, which is detailed in section 4.2.T We find experimentally that the optimal way satisfy this
4

---

Published as a conference paper at ICLR 2024
constraint is by applying sigmoid activation to the final layer of the module, and then dividing each element by the vector sum:
(TT)& (ho(tt))
where contains K elements, a(.) is the sigmoid function and he is the time-embedding mod- ule parameterised by v. Time-embeddings Tr are concatenated with vectors U after the linear projection, and the vectors [ug, T.]² are fed to the encoder fa.
Temporal Convolution Network (TCN) Encoder The main body of the encoder, fo()), is struc- tured as a sequence of residual blocks, each containing two layers of ID dilated convolutions inter- leaved with GeLU activations. The convolution dilation parameter increases with the network depth, to first focus local features and then longer-term dependencies: d 2 where i is the block index. Dilated convolutions have proven to very effective both supervised and unsupervised learning for time series (Zhou et al. 2021 Tonekaboni et al., 2021: Bai et al. 2018).
4.2 PRETEXT TASKS
We present below two novel SSL pretext tasks which leverage time-embeddings, and are designed to complement each other. The first, "Time-embedding Divergence Prediction', describes how the information gained through time-embeddings should structure the latent space and be included in the time series representations. On the other hand, the wetee--ebeddnggoonntttioooo Forecasting' task focuses what information the time- embedddnd and representations should contain.
4.2.1 TIME-IMBEDDING DIVERGENCE PREDICTION
The first pretext task developed aims integrate the notion of time in the latent space structure. It consists in predicting a divergence measure between two tine-embeddings T and T', given the representations the corresponding time steps. The purpose of this task for distances in the latent space to correlate with temporal distances, resulting in smoother latent trajectories than with contrastive learning.
Let us define this regression task formally. Take batch X RBxxxC from which we sample x3,6 and x,, Vi,j E [0, B] and 1, [o, T] s.t. &f 1'. The task input the difference 2,, - 2/,1-- where Zi is the representation of x,, under the context and vij.e is the representation of x,, under the context c'. Taking representations under different contexts further encourages context-invariant representations, as detailed in section 3.2 The regression target is y D(T, and T' the respective time-embeddings the discrepancy between the time-enbedding distributions. We use the Jensen-Shannon divergence (JSD), smoothed and symmetric version of the KL divergence (Lin 1991). The task loss is:
f and 1", and 'D is a measure of statistical divergence, used to measure
Latio EM (KAAM))a (G1 (Zi,t 211,0) JSD(
where is the set (of size M) of time/instance indices for the randomly sampled pairs of repre- sentations. Using divergences allows us capture how two distributions measures differ. whereas a simple norm could only capture by how much two vectors differ. This nuance is important 3-dimensional vector that learned a hierarchical representation of
suppose the time-embedding time (equivalent to seconds, minutes, hours). A difference of 1.0 on all time scales (01: 01: 01) represents very different situation to difference of 3.0 hours and no difference in minutes and seconds (0.3 00 00), but could not be captured by simple vector norm.
4.2.2 TIME-EMBEDDINN-CCODDIII
FORECASTING
Our second pretext task aims to incorporate predictive information in the time-embedding vectors, well as context-awareneet in the representations, to encourage robustness missing data. The task takes in the representation of time series at a specific timestep, and tries to predict the repre- sentation vector of nearby point, conditioned the target's time-embedding.
5

---

Published as a conference paper at ICLR 2024
The imput used is the concatenation [zg,, T of the representation 21, R at time and the time-embedding of the target T++A RK Amaz is a hyperparameter to fix the range in which the prediction target can be sampled. The target is the encoded representation Z,,++A at a uniformly sampled timestep + A, A U[[Amazz Amar]. The input is forwarded through the task head 92 : R F+K ~ RR,, a 2-layer MLP with ReLU activations. The loss is a simple MSE given by:
1 MT
E Σ Jefty Lele (9o
((21d)
(c)) TutD ('
(cz) 2, d
)²
2
Speed=
(5)
where A Ul-Amaz: Amea]. ΩM and QT are the sets of randomly sampled instances and timesteps for each batch, whose respective cardinalities are controlled by hyperparameters M and T. The contexts e1 and 02 are chosen randomly from {e, d'.. so they may be identical or different, fur- ther encouraging contextual consistency. Conditioning this prediction task on the time-embedding of the target forces the model to extract as much information about the signal as possible from its position in time. This results in more information-denss time-embeddings, which can be leveraged when working with individual trajectories for forecasting and anomaly detection.
In practice, we choose short prediction range Amaz & 20, as the focus is not to build represen- tations tailored to forecasting but rather 'context-aware" representations. This context-awareness is enforced by making predictions backwards well forwards, encouraging representations to contain information about their surroundings, making them robust to missing timesteps. Longer pre- diction horizons would push representations contain more predictive features than spatial features, biasing the model away from use-cases around classification, clustering and other 'comparative' or instance-level downstream tasks.
A key objective of this pretext task is to build resilience to missing data. This is done by (1) learning information-dense time-embeddings, which are available even when data is missing, and (2) by learning context-aware representations, which can predict missing timesteps in their close vicinity.
5 EXPERIMENTS
This sections presents the experiments conducted to evaluate T-Rep's learned representations. Be- cause of the variety of downstream tasks, we perform no hyperparameter tuning, and use the same hyperparameters across tasks. Further, the same architectures and hyperparameters are used across all evaluated models where possible, to ensure fair comparison. Experimental details and guide- lines for reproduction are included in Appendix experiments has been made publicly available
A.2 and A.3 The code written to produce these
5.1 TIME SERIES ANOMALY DETECTION
We perform two experiments, point-based anomaly detection on the Yahoo dataset (Niko- lay Laptev/ and segment- based anomaly detection on the 2019 PhysioNet Challenge's s Sepsis dataset (Reyna et al., 2020a| Goldberger et al., 2000). We chose to include both types of tasks as segment-based anomaly detection tasks help avoid the bias associated with point-adjusted anomaly detection (Kim et al.) 2022). Yahoo contains 367 synthetic and real univariate time series, featuring outlier and change point anomalies. Sepsis a real-world dataset containing multivariate time series from 40, 336 patients in intensive care units,
Yahoo (F1)
Sepsis (F1)
Baseline TS2Vec
0.110
0.241
0.733
0.619
T-Rep (Ours) 0.757
0.666
Table 1: Time series anomaly detection F1 scores, on the Yahoo dataset for point-based anomalies and Sepsis datasets for segment-based anomalies. Anomalies include outliers as well as change- points. TS2Vec results are reproduced using of- ficial source code (Zhihan Yue 202T).
featuring noisy and missing data. The task consists in detecting sepsis, medical anomaly present in just 2.2% of patients. On both datasets, we compare T-Rep to the self-supervised model TS2Vec Yue et al. 2022), as well as baseline following the same anomaly detection protocol as TS2
https: thw.h com let CeeeettttCaareeteeee Rep

---

Published as a conference paper at ICLR 2024
and T-Rep each dataset, but using the raw data. We follow streaming anomaly detection procedure (Ren et al. 2019) on both datasets. Details on the procedures can found in Appendix A.2.3 which also details the prooresessing applied to Sepsis.
Table shows the evaluation results (more detailed results are presented in Appendix A.9) T- Rep achieves the strongest performance in both datasets, with an F1 score of 75.5% on Yahoo and 66.6% Sepsis. This respectively represents 2.4% and 4.8% increase on the previous SOTA TS2Vec (Yue et al., 2022). T-Rep's performance can be attributed to its detailed understanding of temporal features, which help it better detect iitto--distribttinn or anomalous behaviour. It achieves this performance with latent space dimension of 200 on Yahoo, which is smaller than the 320 dimensions used by TS2 Yue al. (2022), further showing that T-Rep learns more information- dense representations than its predecessors.
5.2 TIME SERIES CLASSIFICATION
The classification procedure is similar to that introduced by |Franceschi et all 2019): a repre- sentation z is produced by the self-supervised model, and an SVM classifier with RBF ker- nel is then trained to classify the represen- tations (see Appendix A.2 and A.3 for pro- cedure and reproduction details). We com- pare T-Rep to SOTA self-supervised models
30 UEA datasets
Method
Avg. Acc.
Avg. Difference (%)
T-Loss TS2Vec TNC TS-TCC DTW
0.657
33.6
0.699
2.1
0.671
12.3
0.667
10.0
for time series: TS2Vec (Yue al. 2022), Loss (Franceschi et al.) 2019). TS-TCC (EI- dele al.] (021). TNC (Tonekaboni et al. 2021). Minirocket Dempster et al. (2021) and a DTW-bbaed classifier (Müller 2007). These models evaluated on the UEA classification archive's 30 multivariate time series, coming from diverse domains such medicine, sensor systems, speech, and activity recognition Dau al. 2019))
0.650
43.6
Minirocket T-Rep (Ours)
0.707
0.706
Table 2: Multivariate time series classification re- sults on the UEA archive. DTW, TNC, and TS- TCC results are taken directly from Yue al. (002),, while TS2Vec and Minirocket results are reproduced using the official code. 'Avg. Differ- ence' measures the relative difference in accuracy brought by T-Rep compared given model.
Evaluation results are summarised in Table 2 and full results are shown Appendix A.7
along with more details on the chosen evaluation metrics. Table shows that T-Rep has an accuracy +2.1% higher than TS2Vec and +4.8% higher than Minirocket on average. In terms of average accuracy, T-Rep's accuracy outperforms all competitors except Minirocket, which has 0.001 lead. TS2Vee's performance is very close to T-Rep's, only 0.07 lower. T-Rep has positive "Avg. differ- ence' to Minirocket despite having a lower Avg. Acc.' because Minirocket often performs slightly better than T-Rep, but when T-Rep is more accurate, it is so by a larger margin. is important to note that Minirocket was developed specifically for time series classification (and is the SOTA of this date), while T-Rep is a general representation learning model, highlighting its strengths across applications. The latent space dimensionality is set 320 for all baselines, except Minirocket which uses 9996 dimensions (as per the official code). T-Rep uses only 128 dimensions, but leverages the temporal dimension of its representations (see Appendix A.2.4 for details).
5.3
TIME SERIES FORECASTING
We perform a multivariate forecasting task on the four public ETT datasets (ETTh1, ETTh2, ETTml, ETTm2), which contain electricity and power load data, as well as oil temperature (Zhou et al.)
2021 Forecasting is performed over multiple horizons, using procedure described Appendix We evaluate T-Rep against TS2Vee (Yue et al. 2022), SOTA self-supervised model for time series, but also Informer (Zhou et al. 2021). l). the SOTA in time series forecasting, as well as TCN (Bai et al. 2018), supervised model with the same backbone architecture T-Rep, and linear model trained on raw data. Aggregate results, averaged over all datasets and prediction horizons, presented in Table (full results in Appendix A.8)
A.2.2

---

Published as a conference paper at ICLR 2024
Table shows that T-Rep achieves the best average scores, in terms of both MSE and MAE, with a 24.2% decrease in MSE on the supervised SOTA Informer (Zhou et al. 2021). ). and a slight im- provement of 1.80% on the self-supervised SOTA Ve2 (Yue et al., 2022). It also achieves better average rank, ranking first more than any other model. Furthermore, the linear baseline the second model that most frequently ranks first, beating TS2Vec in this metric. However, this model does not perform well in all datasets and therefore ranks 3rd in terms of MAE and MSE. Inter- estingly, most existing self-supervised methods for time series use high-dimensional latent spaces (F = 320 dimensions per timestep) (Franceschi et al. 2019; Tonekaboni et al. (02I), which is thus used to produce baseline results in Table This can be an issue for downstream applications. which might face the curse of dimensionality (Verleysen & François 2005). T-Rep, however, outperforms all baselines with a latent space that almost 3 times smaller, using only
128 dimensions. T-Rep's s superior performance can be attributed to its comprehensive treatment of time, capturing trend, scasonality, or distribution shifts of more easily with its time-embedding module.
T-Rep (Ours)
TS2Vec
Informer
TCN
Linear Baseline
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Avg. Rank
1.90
1.85
2.40
2.45
3.3
3.55
4.35
4.1
3,05
3.05
Ranks 1sd
8
8
2
3
3
1
6
7
Avg.
0.986
0.702
1.004
0.712
1,300
0.820
2.012
1.205
1.017
0.727
Table 3: Multivariate time series forecasting results on the ETT datasets, over 5 different prediction horizons. The presented results are averaged over all datasets/prediction horizons. The 'Ranks 1st metric counts the number of times model ranks 1s amongst its competitors. Results for all models are based on our own reproductions, using the official code for each model (see Appendix A.22 A.2
5.4 ROBUSTNESS TO MISSING DATA
T-Rep was developed with medical and HAR applications in mind, requiring strong resilience to missing data, which we evaluate in two different experiments. In the first qualitative experiment, visualize representations of incomplete time series using the DodgerLoopGame dataset from the UCR archive Dau et al. (201)), which features a continuous segment of 25 missing timesteps (see top row of Figure 2 a). We then visualised heatmap of T-Rep and TS2Vec's time series rep- resentations (bottom row), showing the 15 dimensions with highest variance. T-Loss (Franceschi al. 2019) is not included it does not produce timestep-level representations. For TS2Vec, the representations of missing timesteps very much stand out (they are brighter, reflecting higher val- ues), illustrating that the model is struggling to interpolate and instead produces out-of -f-distribution representations. On the other hand, T-Rep produces much more plausible representations for these missing timesteps with smoother transitions in and out the area with missing points, as well as realistic interpolations, matching the data distribution of the surrounding area.
MOMAN
Malay
0.6
08-
04-
6.0
02-
T5OWG
Missing data cobastnesa visualiation
Performace robsotaeso minsing date
Figure 2: Illustration of T-Rep's robustness to missing data on UCR archive datasets. (a) shows heatmap representations of T-Rep and TS2 when faced with missing data, and (b) shows accu- racy against percentage of missing data in a classification task for T-Rep. TS2Vec (Yue et al. 2022) and T-Loss (Franceschi al. 2019). Error bars denote the standard deviation over 6 train-test runs.
8

---

Published as a conference paper at ICLR 2024
Secondly, we decided perform more quantitative experiment, examining classification accu- racy for different amounts of missing data, on the Articulary WordRecognition dataset of the UCR archive (Dau et al. 2019). We compare T-Rep's performance to TS2Vec (Yue et al. 2022 and T-Loss,, a self-supervised representation learning model for time series, specifically designed for downstream classification and clustering tasks (Franceschi al., 2019). The results unequivo- qual, T-Rep's s performance is extremely resilient to missing data, starting at 98% accuracy with the complete dataset, and dropping by only 1.3% when faced with 75% missing data, and finally reach- ing 86.5% with only 10% of the available data (green curve of Figure 2b. The performance of VS2 is also very strong (orange curve), following similar trend T-Rep with 2% less accuracy on average, and more pronounced dip in performance when 90% of the data is missing, dropping to 82.7%. On the other hand, T-Loss is much more sensitive to any missing data, Its performance decreases exponentially to reach 2.6% when presented with 90% missing data (red curve).
5.5 ABLATION STUDY
To empirically validate T-Rep's components and pretext tasks, we conduct ablation studies on forecasting and anomaly detection tasks using the ETT datasets for forecasting and the Phys- ioNet Challenge's Sepsis dataset for anomaly detection. Unless specified, Time2 Vec is the chosen time-embedding method. w/o TE-conditioned forecasting assigns a weight of 0.0 to the 'time- sebbedding-conditionns forecasting task, redistributing weights evenly. w/o TE divergence pre- diction behaves similarly, but for the 'timeeembedding divergence prediction' task. w/o New pre- text tasks retains only the timeeembedding module and the two Ve2 pretext tasks, isolating the impact on performance of different embeddddd architectures. We explore fully-connected two-layer MLP with ReLU activations (w/ MLP TE module) and vector of RBF features (w/ Radial Basis Features TE module). The original TS2Vec model (w/o TE module) is also included in the ablation study, lacking any of T T-Rep elements.
Forecasting
Anomaly Detection
T-Rep
0.986
0.665
Pretext tacks
w/o TE-connitioned forecasting TE divergence prediction
1.022. (+3.7%)
0.392 (-41%)
(+1.7%)
(-4.7%)
w/o New pretext tasks
0.999 (+1.3%
0,513 (-22.8%)
Architectore
w/ MLP TE module
1,008 (+2.2%%
0,443 (-33,3%)
Features TE module
(+2. 19%)
0.401 (-39.7%)
w/o TE module (=TS2Vec)
1.004 (+1.8%)
0,610 (.8.2%)
Table 4: Ablation study results on ETT forecasting datasets (measured in MSE) and the Sepsis anomaly detection dataset (measured in F1 score). Percentage changes are calculated as the relative difference between modified model's performance and T-Rep's.
Results in Table confirm that the proposed pretext tasks and the addition of time-embedding module to the encoder contribute to T-Rep's performance: removing any of these decreases the scores in both tasks. These results also illustrate the interdependency of both tasks, as in forecasting, only leaving one of the tasks obtains worse results than removing both pretext tasks. It also justifies our preferred choice of time-embedding. since Time2Vec (Kazemi et al.) 2019) outperforms the other 2 architectures in both tasks.
6 CONCLUSION
We present T-Rep, a self-supervised method for learning representations of time series at a timestep granularity. T-Rep learns vector embeddings of time alongside its encoder, to extract temporal fea- tures such as trend, periodicity. distribution shifts etc. This, alongside pretext tasks which leverage the time-embeddingss, allows our model to learn detailed temporal dependencies and capture any periodic or irregularly recurring patterns in the data. We evaluate T-Rep on classification, forecast- ing, and anomaly detection tasks, where outperforms existing methods. This highlights the ability of time-embeddings capture temporal dependencies within time series. Further, we demonstrate T-Rep's 's efficiency in missing data regimes, and provide visualisation experiments of the learned embedding space, highlight the interpretability of our method.

---

Published as a conference paper at ICLR 2024
REFERENCES
Abubakar Abid and James Zou. Autowarp: learning a warping distance from unlabeled time series using sequence autoencoders. arXiv preprint arXiv: 10107, 2018.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprini arXiv: /803.0120 2018
Hubert Banville, Omar Chehab, Aapo Hyvärinen, Denis-Alexander Engemann, and Alexandre Gramfort. Uncovering the structure of clinical eeg signals with self-supervised learning. Journal of Neural Engineering, 18(4):046020, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Joseph Y Cheng, Hanlin Goh, Kaan Dogrusoz, Oncel Tuzel, and Erdrin Azemi. Subject-aware contrastive learning for biosignals. arXiv preprint arXiv:2007.0487/ 2020.
Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi. Chotirat Ann Ratanamahatana, and Eamonn Keogh. The ucr time series archive. IEEE/CAA Journal of Automatica Sinica, 6(6): (6):1129311355 2019.
Shohreh Deldari, Daniel V Smith, Hao Xue, and Flora Salim. Time series change point detection with self-supervised contrastive predictive coding. In Proceedings of the Web Conference 2021, pp. 3124-3135, 2021.
Angus Dempster, Daniel F Schmidt, and Geoffrey I Webb. MiniRocket: A very fast (almost) de- terministic transform for time series classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 248-257, New York, 2021. ACM.
David A Dickey and Wayne A Fuller. Distribution of the estimators for autoregressive time series with a unit root. Journal of the American statistical association, 74(366a):427 431, 1979.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting, arXiv preprint arXiv:2106. 14112222 12, 2021.
Vincent Fortuin, Matthias Hüser, Francesco Locatello, Heiko Strathmann, and Gunnar Rätsch.
Som-vae: Interpretable discrete representation learning on time series. arXiv preprint 1rXiv:1806.02199. 2018.
Jean- Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. Advances in neural information processing systems, 32, 2019.
A L Goldberger, L A Amaral, L Glass, J M Hausdorff, P C Ivanov, R G Mark, J E Mietus, G B Moody, C Peng, and E Stanley. PhysioBank, PoosioTo and PhysioNet: components of a new research resource for complex physiologic signals. Circulation, 101(23):E215-20 June 2000.
Harish Haresamudram, Apoorva Beedu, Varun Agrawal, Patrick L Grady, Irfan Essa, Judy Hoffman, and Thomas Plötz. Masked reconstruction based sue-suupevi for human activity recognition. In Proceedings of the 2020 ACM International Symposium on Wearable Computers, pp. 45-49, 2020.
Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Mareus Brubaker. Time2vec: Learning vector representation of time. arXiv preprint (rXiv:1907.05321 2019.
Eamonn Keogh, Jessica Lin, Ada Waichee Fu, and Helga Van Herle. Finding unusual medical time- series subsequences: Algorithms and applications. IEEE Transactions on Information Technology in Biomedicine, 10(3):429 439, 2006.
10

---

Published as a conference paper at ICLR 2024
Siwon Kim, Kukjin Choi, Hyun-Soo Choi, Byunghan Lee, and Sungroh Yoon. Towards a rigorous evaluation of time-series anomaly detection. Proceedings of the AAAI Conference on Artificial Intelligence, (967119--221,, 06 2022. doi: http://aaiiv61..2000000
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: /412. 69800, 2014.
Diederik P Kingma and Max Welling, 1rXiv:1312.611 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Dani Kiyasseh, Tingting Zhu, and David Clifton, Cloes: across space, time, and patients. In International Conference on Machine Learning, pp. 5606- 5615. PMLR, 2021.
Contrastive learning of cardiac signals
Zhihan Li, Youjian Zhao, Jiaqi Han, Ya Su, Rui Jiao, Xidao Wen, and Dan Pei. Multivariate time ries anomaly detection and interpretation using hierarchical inter-metric and temporal embedding. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery data mining, pp.
3220-3230, 2021.
Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145-151, 1991.
Pankaj Malhotra, Vishnu TV, Lovekesh Vig, Puneet Agarwal, and Gautam Shroff. Timenet: Pre-trained deep recurrent neural network for time series classification. arXiv preprint arXiv:1706.08833 088388 2017.
L. McInnes, J. Healy, and J. Melville. UMAP: Dimension Reduction. ArXiv e-prints, February 2018
Uniform Manifold Approximation and Projection for
Meinard Müller. Dynamic time warping. Information retrieval for music and motion, pp. 69-84, 2007.
Saeed Amizadeh Nikolay Laptev, Y. B. A benchmark dataset for time series anomaly detec- tion, 2015. URL http/tttss/yyahoeeerrcc....... h0tpa//0o..o.ooooo1111555020100 --Perccciiak--ttttae---------e--
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint /007.:/80000300 2018
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetoto, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8033 8035. Curran Associates, Inc., 2019.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournaeeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-
2011.
Johannes Pöppelbaum, Gavneet Singh Chadha, and Andreas Schwung. Contrastive learning based self-supervised time-series analysis. Applied Soft Computing, 11 117:108397, 2022.
Hansheng Ren, Bixiong Xu, Yujing Wang, Chao Yi, Congrui Huang, Xiaoyu Kou, Tony Xing, Mao Yang, Jie Tong, and Qi Zhang. Time-series anomaly detection service at microsoft. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 3009-3017,
Matthew A. Reyna, Christopher S. Josef, Russell Jeter, Supreeth Shashikumar, M. Bran- don Westover, Shamim Nemati, Gari D. Clifford, and Ashish Sharma. Early pre- diction of sepsis from clinical data: The physionet/comppting in cardiology chal-
lenge 2019. Critical Care Medicine. 48(2), 2020a. ISSN 0090=3493. URL https:://journat 1ww. com X httpa//www.......ooooooooooooooa /tt.Le/xxx//////0000e111/ the aspx
prediction_od of sep__sepssss...... caaapsssssfoo__siiii_

---

Published as a conference paper at ICLR 2024
Matthew A Reyna, Christopher S Josef, Russell Jeter, Supreeth P Shashikumar, M Brandon West- over, Shamim Nemati, Gari D Clifford, and Ashish Sharma. Early prediction of sepsis from clinical data: the physionet/computing in cardiology challenge 2019. Critical care medicine, 48 (2):210-217, 2020b.
Alaa Sagheer and Mostafa Kotb. Unsupervised pre-training of a deep lstm-based stacked autoen- coder for multivariate time series forecasting problems. Scientific reports, 91):11933, 2019.
Alban Siffer, Pierre-Alain Fouque, Alexandre Termier, and Christine Largouet. Anomaly detection in streams with extreme value theory. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1067-1075, 2017.
Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2828-2837 2019.
Sindhu Tipimeni and Chandan K Reddy. Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series. ACM Transactions on Knowledge Discovery from Data (TKDD), 16(6):1-17, 2022.
Sana Tonckaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. arXiv preprint arXiv:2/06.00750 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Michel Verleysen and Damien François. The curse of dimensionality in data mining and time series prediction. In International conf-ccnfeee on artificial neural networks, pp. 758-770. Springer, 2005.
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey. arXiv preprint arXiv:2202.07/25, 2022.
Lingfei Wu, Ian En-Hsu Yen, Jinfeng Yi, Fangli Xu, Qi Lei, and Michael Withrock. Random warp- ing series: A random features method for time-series embedding. In International Conference Artificial Intelligence and Statistics, pp. 793-802. PMLR, 2018.
Qiang Yang and Xindong Wu. 10 challenging problems in data mining research. International
Journal of Information Technology & Decision Making, 5(04))597-604, 2006.
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8980-8987, 2022.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eick- hoff. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp- 2114-2124, 2021.
Kexin Zhang, Qingsong Wen, Chaoli Zhang, Rongyao Cai, Ming Jin, Yong Liu, James Zhang, Yuxuan Liang. Guansong Pang, Dongjin Song. al. Self-supervised learning for time series analysis: Taxonomy, progress, and prospects. arXiv preprini arXiv:2306. 10125, 2023.
Wenrui Zhang, Ling Yang, Shijia Geng, and Shenda Hong. Cross reconstruction transformer for self-supervised time series representation learning. arXiv preprint arXiv:2205.09922, 2022.
Lin Yiting Zhihan Yue.
yyuzhhhan/ts2vees Ts2vec github repository, 2021.
URL
com/
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li. Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11106-11115, 2021.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 LATENT SPACE STRUCTURE
This section aims assess the representations produced by T-Rep in different settings, and show their interpretability. We perform 2 experiments, the to examine the intra-instance latent space structure as well as the inter-instance structure.
For the first experiment, we choose datasets from the UCR archive (Dau al. 2019) and create one synthetic dataset. We then plot a time series sample for each dataset and show a heatmap of the corresponding T-Rep representation evolving through time (only the 15 dimensions with highest variance are shown for each representation vector).
20
M Mrallmmmm Mmm 20
0.5
0.0
-20
1-hxp
10
10
Figure 3: Time series representation trajectories in the latent space. The top row shows the input signal, and the bottom row shows heatmap of the representations through time. Only the 15 most varying dimensions are shown for each representation. Data for the two left-most figures comes from UCR archive datasets, and the righ-most figure's 's data is synthetic.
Figure 3 showcases a variety of input signal properties captured by T-Rep. The leftmost plot showcases a sample from the Phoneme dataset (Dau et al. 2019). and we see that periodicity is captured by T-Rep. in particular the change in frequency and decrease in amplitude. Plot (b) is an extract of the Pig Central Venous Pressure medical dataset (Dau et al.. 2019), where T-Rep accurately detects an anomaly in the input signal, represented by sharp change value (illustrated by color shift from green to blue) at the corresponding spike. Finally, the rightmost plot showcases a synthetic dataset with an abrupt distribution shift. T-Rep's representation captures this with observe how T-Rep produces coherent representations for missing data in Figure 2 a (see section 3.4for details).
lighter colors and change in frequeecy. in the original signal. Finally,
This experiment highlights the interpretability of the representations learned by T-Rep, accurately reflecting properties the original signal.
To study the global latent space structure (inter-instance). we look at UMAP plots (Mcinnes et al, 2018) of representations learned by T-Rep five UCR datasets Dau al. (2019). Each instance is represented by a point, colored by the class it belongs to, to show whether the latent space structure discriminates the datasets' different classes. We include representations learned by TS2Vec to see how the new components of T-Rep affect the learned latent space. Results are presented in Figure
Looking the UMAPs across the 128 UCR datasets, we noticed that the latent space structure of T-Rep and TS2Vec is often similar, and/or of similar quality. This is shown in the two rightmost plots of Figure 4 and can be explained by the fact that T-Rep focuses on learning temporal features (and uses TS2Vec' 's tasks for spatial features), resulting in stronger differences in the evolution of representations through time (see Figure 3)
To focus on learning detailed temporal features, T-Rep puts a lower weight on TS2Vec's spatial- features pretext tasks. While one might intuitively think this results in lower quality spatial features, this not what our experiments show. Indeed, for none of the tested datasets did we observe significant downgrade in latent space structure, which is further supported by the classification results presented in section where T-Rep outperforms TS2Vec by small margin. These results show that learning detailed temporal structure within instance constitutes useful feature to
13

---

Published as a conference paper at ICLR 2024
T-RRep
TS2Vec
(6) SamootSSusppae
ShapeletSim
(c) MixadSStggssSmllTTaii
(4) UWaveeiestture LibarrZZ
Figure 4:
UMAP visualisations of the learned representations T-Rep and TS2Vec on UCR datasets. Each color represents different class. TS2Vec results produced using the official code (Zhihan Yue 2021
compare various instances. This is illustrated by the two leftmost plots of Figure where we see much cohesive and separated clusters for T-Rep than Ve2.ec. This experiment truly highlights the versatile nature of T-Repp. improving the quality of representations at both the dataset scale (inter- instance discrimination) and individual time series (timestep granularity).
A.2 EXPERIMENTAL DETAILS
The implementation of the models done in Python, using Pytorch 1.13 Paszke et al., 2019) for deep learning and scikit-learn (Pedregosa et al. 2011) for SVMs, linear regressions, pre-processing 2014) throughout, and train all models on a single
etc. We use the Adam optimizer (Kingma NVIDIA GeForce RTX 3060 GPU with Cuda 11.7.
As mentioned in the Experiments section, no hyperparameter tuning was performed. We a batch size of 16, a learning rate of 0.001, set the maximum number epochs to 200 across datasets and tasks. We use 10 residual blocks for the encoder network, and set hidden channel widths to 128. The kernel size of all convolution layers is set to 3. The weighting of pretext tasks is detail in section I.2.I below. As for the choice of time-embedding, with ReLU activations for the classification tasks, and Time2Vec for the forecasting and anomaly detection tasks (see Appendix [..4 for additional information). For all experiments (unless stated otherwise), models are trained and tested 10 times dataset to mitigate stochasticity. No random seed is used in any experiments.
fully-connected two-layer MLP
A.2.1 PRETEXT TASK WEIGHTS
As shown in Figure I loss passed to hierarchical loss computation framework is a linear combi- nation of pretext task's s loss:
04
l
04
where a; designates task i's weight. Choosing the task weights be seen semantic well as an optimisation problem: the extracted features will of course depend on the weights attributed each task, but the choice of weights will also impact the optimisation landscape of the final
14

---

Published as a conference paper at ICLR 2024
loss. Balancing these two factors is complex and interesting research direction, which we leave as future work. For all pretext tasks (instance- wise contrasting, temporal contrasting, embeddind divergence prediction, time-embedding-oo -oneiiiinno forecasting), we use the same weights: a, 0.25 Vi E [1,4]. This is the case across all downstream tasks, no task-specific tuning is performed.
A.2.2 FORECASTING
The multivariate forecasting task consists in taking the previous points X (x.-b+1,..,, x: ] RR and using them to predict the next H points (x.+1,..., X(+H} } R#xC where is the prediction horizon. Such X and y are used to train the supervised fore- casting baselines. For self-supervised models, the forecasting model is simple ridge regression, taking as input z₁ (the representation of xt), and trying to predict y. The input time series are normalised using the z-score, each channel being treated independently. We use covariates such as minute, hour, day-of- the time-embeddinngs. To accurately evaluate the benefits of time-embeddings, we wanted to ensure the model would not rely on other temporal features (the covariates).
etc., to avoid interference with the temporal features learned by
The forecasting model is a ridge regression (linear regression with an Le regularisation term, weighted by parameter a) trained directly on the extracted representations. a is chosen using grid search on validation set, amongst the following values: a 0,, 5, 10, 20, 50, 100, 500, 1000).
The train//alidation/tess split for the ETT datasets is 12/4/4 months. The prediction horizon H ranges from 1 to 30 days for the hourly data (ETTh, and ETTh₂), and from 6 hours to 7 days for minute data (ETTm; and ETTm2). The evaluation metrics, MSE and MAE, are respectively given by:
MSE
22,
(6)
HB
1=
1
MAE
HB
where oi4t is the true value, #/+ is the prediction, is the number of time series instances indexed by j. The reported MSE and MAE are thus averaged over all prediction horizons and instances. To mitigate the effects of stochasticity during training, each model is trained and tested 10 times on each dataset and prediction horizon, and the reported results are the mean.
A.2.3 ANOMALY DETECTION
Yahoo
The Yahoo anomaly detection task follows streaming evaluation procedure (Ren et al., 2019). where given a time series segment (.,,..., Ft), the goal is to infer the last point Tú is anomalous. To avoid drifting, the input data is differenced d times, following the number of roots d found by an ADF test (Dickey & Fuller 1979). For fair comparison, we follow the anomaly detection procedure of Yue et al. (2022), which computes between representations where the observation of interest Z2 has been masked vs. unmasked. Since the model was trained with random masking, it should output coherent representation even with a masked input, resulting in a small a, However, the point H, presents an anomaly, the masked representation will be very different from the unmasked representation, it anticipated 'normal' behaviour, not an anomaly, resulting in a high value of 04 More precisely, da is given by:
anomaly
a for Tt. Or is given by the dissimilarity
= ||r - 1
(8)
where ri is the representation of the unmasked x, and rith is the representation of the masked observation. The anomaly score adjusted by the average of the preceding points:
odj
(9)
a
15

---

Published as a conference paper at ICLR 2024
where Ci 3 Ei=1-2 Oze- At inference time, anomalies are detected if the adjusted anomaly score Or adj above a certain threshold: !di > u+BBor, where and a are the mean and standard deviation of the historical scores, and / is hyperparameter. If anomaly detected within delay of timesteps from when it occured, it is considered correct, following the procedure introduced in Ren al. (2019). To mitigate the effects of stochasticity during training, each model is trained and tested 10 times, and the reported results are the mean.
Sepsis
The anomaly detection procedure followed for Sepsis is different from Yahoo's. We use super- vised method, since there is only one type of anomaly (sepsis) to detect, we can fit a model to it, obtaining higher scores than a purely unsupervised method (as is done for Yahoo). One could not do that with dataset like Yahoo, where there various anomalies which cannot be explicitly classi- fied. Despite being solved a supervised manner, we believe sepsis detection remains an anomaly detection task and is differs from classification because rather than entire time series, thus evaluating different characteristics of T-Rep. To perform well, T-Rep must thus produce representations with accurate intra-instance structure, where healthy timesteps are very different from abnormal (sepsis) timesteps (which is learned in self-supervised manner) This is further supported by the very low number of positive samples (2.2%), so training a classification model on raw data is not efficient. For these reasons, although we used supervised model on top of our representation, we consider this task to be an anomaly detection task.
classifying individual timesteps
The Sepsis dataset firstly pre-processed by forward-filiing all missing values and features which have lower granularity than the pysiological data, to ensure we have data at all timesteps for all features. The length of time-series is then standardised to 45 (patient ICU stay length varies in the raw data), corresponding to the last 45 hours of each patient in the ICU. We then subselect ten features based on medical relevance ("HR', 'Temp', 'Resp'. 'MAP', 'Creatinine', 'Bilirubin direct', 'Glucose', 'Lactate', 'Age', 'ICULOS"). We have found this be rather 'light' rrepprocessing compared to what top cop-performers in the 2019 PhysioNet Challenge did. This is done on purpose, to further highlight gaps between model performances.
We obtain dataset X e Nx45x10 We train self-supervised models TS2Vec and T-Rep on this dataset, and then encode individual time series in sliding windows of length 6 (this window length seemed like medically sensible lookback period, but performed no parameter tuning). We thus obtain latent representations
24-6:t
(o(x
(10)
where fe is the self-supervised model (T-Rep TS2Vec) and Z:-6at R6xF,, with latent dimen- sionality F = 32. A very small latent space used since we pre-selected 32 features. Given 2,-6:t+ we try to predict whether showcases sepsis, which is framed as binary classification problem. This classification task is then solved using an SVM classifier with RBF kernel is trained using the representations. The penalty hyperparameter C' is chosen through cross-validation, in the range {10* kk E [[-4, 4l]3- The linear baseline uses the same method, but using the raw data Xi-6it instead of learned representations Z:-6:t.
A.2.4 CLASSIFICATIO
The classification procedure for TS-TCC, TNC, and T-Loss is the same as in Franceschi et al. (011)). Instance-level representations are produced by the self-supervised model, and an SVM classifier with RBF kernel is trained using the representations. The penalty hyperparameter C is chosen through voss-valid,tion, in the range [10*|k e [4-4,4]]]:
For T-Rep and TS2Vec, the procedure is slightly different, leveraging the fact that the temporal granularity of representations controllable by the user (from timestep-level to instance-wide), thanks to the hierarchical loss framework Yue et al. (202)). We use this to produce representations which maintain a temporal dimension, whose length is controlled by hyperparameter W: the latent vectors z have dimension & This is achieved by applying a maxpool operation with [
and stride kernel size k- |
the timestep-level representation R The
16

---

Published as a conference paper at ICLR 2024
representation RI is then flattened to obtain ftat E R R. which is fed to SVM with RBF kernel for classification, following the exact same protocol as in Franceschi et al. (2019). We refer to this new procedure TimeDim in Table
Maintaining the temporal dimension in the representations is advantageous, it gives better snapshot of a time series' latent trajectory, compared to simply using one instance-wide vector. This especially important for T-Rep, whose innovation lies in the treatment of time and its incorporation in latent trajectories. Because keeping some temporality increases the dimensionality of the vector ffat passed to the SVM, we use a smaller latent dimension to compensate. In practice we use W = 10 and F = 128. but no hyperparameter tuning was performed, so these may not be optimal parameters.
in all other experiments, the reported results are the mean obtained after training and testing each model 10 times per dataset.
Interestingly, for Vec, the TimeDim procedure does not result in performance gain, so the scores obtained with the original procedure from Franceschi et al. (2019) (using instance-wide representations) are reported in Table 2 To summarise, for the results reported in Table 2 (in the paper's ' main body). T-Rep is the only model to use the TimeDim procedure. TS2Vec, TNC, T-Loss, TS-TCC all use the procedure introduced by Franceschi et al. (2019). TS2Vec results for both procedures are reported in Table 5 which contains scores on individual datasets, as well as aggregate metrics.
A.3 REPRODUCTION DETAILS FOR BASELINES
The results for TS2Vec Yue et al., 2022) have heen reproduced using the official code (found here). The hyperparameters used for TS2Vec are exactly those of the original paper (except for the batch size, set to 16 in our reproduction), and the same as T-Rep where possible. The only notable difference accommodate the additional use of eeneeddddding vector. The results published in the TS2Vec paper weren't used for the forecasting task because of slight changes we made the experimental setup, specifically removing the use of covariates in forecasting. For anomaly detection and classification, we used our own reproduction because the results slightly differed from the originally published results, which attribute to the following reasons. Firstly, the change in batch size from to 16 for us, which the original paper says to be important (Yue et al. 2022). Secondly, we introduced new classification procedure, train and test all models 10 times to mitigate stochasticity and obtain more stable results, which is the case for the results of the original TS2Vec paper (Yue al. 2022).. Also, we do not use any random seed, while the TS2Vec codebase Zhihan Yue (021) shows a single run with a random seed used produce their results.
TS2Vec:
hyperparameters
that T-Rep uses slightly wider hidden channels to
had to test TS2Vec in this new setup. Further,
Informer: For the ETTh ETTh; and ETTm; datasets, we use the results published in the Informer paper (Zho et al. 2021. However, the paper doesn't 't include results on ETTm,, so we obtain these results using the official code (found hhre) and the default hyperparameters (which are also used in the original paper).
TCN: T-Rep's encoder architecture is TCN network (Bai et al. 2018 So we reused our code to implement the forecasting TCN model, changing only the training part to use a supervised MSE loss. The hyperparameters (channel widths, learning rate, optimiser) are the same as for T-Rep.
For the classification tasks, the results of TS-TCC (Eldele al.) 2021). TNC TTonekaboni et al.) 2021], and DTW (Müller (007) are taken from the TS2Vec paper Yuue et al. 2022), based on their reproduction. The results for T-Loss (Franceschi et al. 2019) are taken from the T-Loss paper directly (Franceschi et al. 2019).
17

---

Published as a conference paper at ICLR 2024
For the anomaly detection tasks, the baseline results of SPOT, DSPOT (Siffer et al. 2017). and SR (Ren et al. 2019) are taken directly from the Ve2 paper (uue et al. 2022).
A.4 TIME-EMBEDDING CHOICE
The choice of architecture for the time-embedding varies depending the temporal features one wishes to extract. For users concerned with capturing periodicity and trend, a Teme2Vee- module ideal (Kazemi al., 2019). Foeusing these features has helped it become the top performer for anomaly detection and forecasting. If unsure about what features to extract, a fully-connect MLP may also be used, leading perhaps less interpretable but very flexible and expressive embeddings. We found this to be the best time-embedding for classification tasks, We have also implemented embedding model based on RBF features, which performs well in anomaly detection. is not well understood when and why different time-embeddings perform better than others, beyond the simple case of Time2 Vec (Kazemi al. 2019). We leave this as an area for future research.
A.5 CONTEXTUAL CONSISTENCY
Ve2 introduces a new pair-sampling method for contrastive learning: contextual consistency. The idea to sample two overlapping segments the time-series and encourage the representations in the overlapping timesteps to be similar. Random transformations are applied to each sampled segment, leading to different contexts for the overlapping window. Idcally, the encoder should extract representation of the window which is invariant the surrounding context.
Given an input time-series E RTxF, random cropping is applied to obtain two overlapping segments of x: Xayd, and Xaydy such that 0 & a1 & 02 < b1 < by < T. a1 02, bj, b2 are sampled uniformly in the allowed ranges each training iteration. Timestamp masking is then randomly applied within each segment by the encoder, which outputs the respective representations 7ayshy ( --. 2b,} and Zuzchg (zazz..., 2b, These representations in the overlapping interval [a2, b1] correspond to the same timesteps, but were obtained using different contexts. Timestep-wise and instance-wiss contrasting tasks are then applied them Yue et al. (2022).
The contextual consistency method very powerful as helps learn invariances different sub- sequence distributions (which might differ in the case of non non-stationary data), without making any underlying assumptions about the time-series distribution. This contrasts other SSL methods for temporal data, which often feature inductive biases in the form of invariance assumptions to noise, cropping, flipping etc. (Zhang et al. 2023).
A.6 HIERARCHICAL LOSS
The hierarchical loss framework applies and sums the model's loss function at different scales, starting from a per-timestep representation and applying maxpool operations to reduce the time- dimension between scales (Yue et al. 2022). This gives users control over the granularity of the representation used for downstream tasks, without sacrificing performance. It also makes the model more robust to missing data, as it makes use of long-range information in the surrounding represen- tations to reconstruct missing timesteps (Yue et al. 2022). Its precise algorithm given in Figure below, taken from Yue al. (2022):
18

---

Published as a conference paper at ICLR 2024
Algorithm Calculating the hierarchical contrustive loss
1:
procedure HIERLOSS(r, r")
2: Chier f- Leuua (r, r');
3:
1-1;;
4:
while time lenggli(r) do
5:
I! The maxpoolic operates along time axis.
6:
maxpoolld(r, kernel_size 2);
7:
maxpoolld(r" kernel size 2);
8:
Chier Lhier + Lduailr, r');
9:
d +-d+1:
10:
end while
11:
Clier +- LLiie/dd;
12:
return Cxier
13:
end procedure
Figure 5: Hierarchical loss algorithm Yue et al. 2022)
A.7 UEA CLASSIFICATION FULL RESULTS
Table presents the full results for the classification task on the 30 UEA multivariate time series datasets Dau al. (2019). The evaluated models T-Rep, following the TimeDim procedure
detailed in Appendix A.2.4 TS2Vec Yue et al. (2022). following both the TimeDim procedure and
the original procedure introduced in Franceschi al. 2019), T-Loss Franceschi al. (2019). TNC
Tonekaboni et al. (02T). TS-TCC Eldele et al. (2021 and DTW Müller (2007).
Dataset
T-Rep (Ours)
TS2Vec
TS2Vec (TimeDim)
T-Loss
TNC
TS-TCC
DTW
Minirocket
Articulary WordRecognition
0.968
0.974
0.969
0.943
0.973
0.953
0.987
0.980
AtrillFihillation
BasicMotions
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
Character CharacterTrajectories
0,989
0.988
0.988
0.993
0.967
0.985
0,989
0.988
Cricket
0.958
0.953
0,938
0.972
0.958
0.917
0.972
Dueeee
0.457
0.434
0.368
0.65
0.46
0.38
0.6
0,540
ERing
Eigen Worms
0.884
0.916
0.88
0.84
0.84
0.779
0.618
0.954
Fpilepsy
0.97
0.963
0.962
0.971
0.957
0.957
0.964
1.0
EihanolConceetrraion
0,333
0.303
0.298
0.205
0.297
0.285
0.322
0.357
FaceDetection
0
0.541
0.577
0,513
0.536
0.544
0.529
0.569
FingerMovements
0.47
0.46
HandMoovemeeDDieccioo
0.536
0.418
0.527
0.351
0.324
0.243
0.231
0.405
Handwriting
0.414
0.463
0.424
01451
0.249
0.498
0.286
0.241
Heartbeat
0.725
0.734
0.724
0.741
0.746
0,750
0.717
0.722
InsectWinghen
0.328
0.323
0.327
0.156
0.469
0.264
0.1
0.319
Jupunese Vowels
0,962
0.961
0.989
0.978
0.93
0.949
0.921
LSST
0.526
0.558
0.546
0.5009
0.595
0.474
0.551
0.668
Libras
0.829
0.859
0.833
0.883
0.817
0.822
0.87
0.944
MotorImagory
0.495
0.497
0.5
0.470
NATOPS
0.804
0.897
0.824
0.917
0.911
0.822
0.883
0.916
PEMS-SF
0.772
0.794
0.675
0.699
0.734
0.711
0.896
PenDigits
0.971
0.977
0.975
0.981
0.979
0.974
0.977
0.974
PhonemeSpectra
0.232
0.243
0.228
0.222
0.207
0.252
0.151
0.280
RacketSperts
0.883
0.893
0.865
0.855
0.775
0.802
0.875
SelfR SelfRegulationSCII
0.819
0.79
0.819
0.843
0.799
0.823
0.775
0.884
SelfRegulationSCP2
0,591
0.554
0.563
0.539
0.55
0.532
0.539
0.494
SpokenArabic Digits
0.994
0.992
0.993
0.905
0.934
0.97
0.963
0.989
StandWalJJump
0.441
0.407
0.3
0.332
0.4
0.332
0.2
0.333
UWaeeGesturLiibrayy
0.885
0.875
0.89
0.875
0.759
0.753
0.903
0.913
Avg. Accuracy
0.706
0.699
0.693
0.657
0.671
0.667
0.650
0.707
Ranks 1a
2
1
1
3
2
9
Avg. Difference (%)
12.3
Table 5; Classification accuracy of T-Rep and other self-supervised models for time series 30 UEA datasets.
The metrics used for evaluation are the following:
Avg. Accuracy: The measured test accuracy, averaged over all datasets.
19

---

Published as a conference paper at ICLR 2024
Ranks 1st This metric measures the number of datasets for which the given model is the best amongst all compared models. For example, T-Rep the best performing model on 10 out of 30 datasets.
Avg. Acc. Difference: This measures the relative difference between T-Rep and given model M, averaged over all datasets. calculated as:
30 Acc(T-Rep): Acc( Acc(MM; Acc(M),
Avg. difference 100
(11)
where Mc)(M); is the accuracy of model M on dataset i.
Avg. Rank: For each dataset, compute the rank of model compared other models based on accuracy. The average rank over all 30 datasets is then reported.
A.8 ETT FORECASTING FULL RESULTS
Table|6 the full results for the forecasting task on the ETT datasets Zhou et al. (2021). We evaluate T-Rep against TS2 Vec Yuee et al. 2022), SOTA self-supervised model for time series, but also Informer (Zhou et al. 2021). the SOTA in time series forecasting, as well as TCN (Bai et al. 2018), simpler model with the same backbone architecture T-Rep, but trained with supervised MSE loss.
T-Rep (Ours)
Te2
Informer
TCN
Linear Baseline
Dataset
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
24
0.511
0.496
0.575
0.529
0.577
0.549
0.767
0.612
0.873
0.664
48
0.546
0.524
0.608
0.553
0.685
0.625
0.713
0.617
0.912
0.689
ETTh
168
0.759
0.649
0.782
0.659
0.931
0.752
0.995
0.738
0.993
0.749
336
0.936
0.742
0.956
0.753
1.128
0.873
1.175
0.800
1.085
0.804
720
1.061
0.813
1.092
0.831
1.215
0.896
1.453
1.311
1.172
0.863
24
0.560
0.565
0.448
0.506
0.720
0.665
1.365
0.888
0.463
0.498
48
0.847
0.711
0.685
0.642
1.457
1.001
1.395
0.960
0.614
0.588
ETTh
168
2.327
1.206
2.227
1.164
3.489
1.515
3.166
1.407
1.738
1.016
336
2.665
1.324
2.803
1,360
2.723
1.340
3.256
1.481
2.198
1.173
720
2.690
1.365
2.849
1.436
3.467
1.473
3.690
1.588
2.454
1.290
24
0.417
0.420
0.438
0.435
0.323
0.369
0.324
0.374
0.590
0.505
48
0.526
0.484
0.582
0.553
0.494
0.505
0.477
0.450
0.813
0.637
ETTm;
96
0.573
0.516
0.602
0.537
0.678
0.614
0.636
0.602
0.866
0.654
288
0.648
0.577
0.709
0.610
1.056
0.786
1.270
1.351
0.929
0.697
672
0.758
0.649
0.826
0.687
1.192
0.926
1.381
1.467
1.008
0.746
24
0.172
0.293
0.189
0.310
0.147
0.277
1.452
1.938
0.275
0.364
48
0.263
0.377
0.256
0.369
0.267
0.389
2.181
0.839
0.363
0.434
ETTmg
96
0.397
0.470
0.402
0.471
0.317
0.411
3,921
1.714
0.441
0.484
288
0.897
0.733
0.879
0.724
1.147
0.834
3.649
3.245
0.754
0.664
672
2.185
1.144
2.193
1.159
3,989
1.598
6.973
1.719
1.796
1.027
Avg. Rank
1.90
1.85
2.40
2.45
3.3
3.55
4.35
4.1
3.05
3.05
Ranks 1st
8
8
2
1
3
3
1
1
6
7
Avg. MSE
0.986
0.702
1.004
0.712
1.300
0.820
2.012
1.205
1.017
0.727
Table 6: Multivariate time series forecasting results on the four ETT datasets. The 'Ranks 1 metric measures the number of situations where the given model is the best amongst all compared models.
20

---

Published as a conference paper at ICLR 2024
A.9 TIME SERIES ANOMALY DETECTION FULL RESULTS
F1
Prec.
Rec.
SPOT
0.338
0.269
0,454
DSPOT
0.316
0.241
0.458
SR
0.563
0,451
0.747
TS2Vec
0.733
0.706
0.762
T-Rep (Ours)
0.757
0.796
0.723
Table 7: Time series anomaly detection results, on the Yahoo Webscope dataset. Anomalies include outliers well as changepoints. TS2 results reproduced using the official source code (Zhi- han Yue 2021). while all other baseline results are taken directly from Yue et al. (2022).
Fi
Acc.
Raw data
0.241
0.676
TS2
0.619
0.771
T-Rep (Ours)
0.665
0.790
Table 8: Time series segment-based anomaly detection results the Sepsis dataset Reyna et al. 2020b). TS2Vec results are reproduced using the official source code (Zhihan Yue (021).
21


Reviews:
[{"review_id": "TV2DeFfwA2", "reviewer": "Reviewer_99iS", "summary": "The paper presents a self-supervised way of learning latent representations of variable length time series data that are useful for various downstream tasks like time series anomaly detection, forecasting and classification. The authors propose two surrogate loss functions in form of 'pretext tasks' that influence the learned representation/embedding to persist temporal consistency in latent space along with information to forecast, and thus be robust to missing data. The results are demonstrated on three downstream tasks where T-Rep outperforms the SOTA time series representation learning methods.", "questions": "My questions can be found in Weaknesses section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Below are the strengths of this proposed work: \n1. The problem is highly relevant to the time series research community and well motivated by the authors. They also do a good job at covering the related work and highlighting the necessity of temporal robustness in the representations that's usually overlooked when learning time series representation. \n2. The authors introduce surrogate loss functions to train the representation model in a self supervised manner which they refer to as 'pretext tasks'. The two loss functions (although not novel) are practically reasonable given the need of temporal consistency and predictability in the learned representations. \n3. Authors do a good job of testing T-Rep across three key downstream tasks which helps in validating the usefulness of learned representation as task-agnostic.", "weaknesses": "Following are weaknesses of this work: \n1. The effectiveness of the proposed method relies on the evaluation done mainly on UCR/UEA and Yahoo Datasets which are infamous for their incorrectness, and triviality in labels. Refer: https://arxiv.org/pdf/2009.13807.pdf. This questions the effectiveness of the proposed method and how well would it work in real-world scenarios?\n2. Similar to Pt.1, I feel the downstream benchmarks are bit too trivial to test the true effectiveness/usefulness of a representation learning method. For e.g., in the task of anomaly detection, the authors propose to use a simple point-based evaluation method which is not realistic, as real world anomalies are usually segment based and need more rigorous evalution to demonstrate usefulness. Refer: https://arxiv.org/pdf/2109.05257.pdf. \n3. In Sec 5.5, I get that one can manually look at the heatmaps to infer similarities between the patterns in raw time series signal and the learned representation but I didn't fully understand the usefulness of that? Can I just look at the heatmap and use that information? if so, how? Manually matching pattern is pleasing but I don't see any usability, and hence don't see any true interpretability coming out of it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nbe84tnJEQ", "reviewer": "Reviewer_rWLt", "summary": "The paper proposes a time series representation learning methodology based on self-supervision. The paper identifies two key issues with current time-series embedding using contrastive learning which -- a) aims to embed entire trajectories based on a binary notion of similarity vs. dissimilarity, ignoring temporal structure, and b) incompatibility with systems which switch between states, where notions of similarity vs. dissimilarity are based on states. They propose two pretext tasks to address these issues. \n\n\nOverall, I think the paper is very well written and motivated, except a few things that can really improve the quality. I hope authors can address these in the discussion.", "questions": "1. How may dataset are used for anomaly detection (sec 5.1)?\n2. How does the issues of similarity/dissimilarity play out in multivariate settings? What if the switching behaviour is only present in a few of the channels?\n3. Why does T-Rep perform better in some cases and not it others? For instance, in Table 5 in ETTh_2 T-Rep performs better at larger horizons, while being significantly worse at lower ones. This is counter-intuitive, since in section 4.2.2. the paper mentions that the pretext forecasting task specifically aims to predict shorter horizons.\n4. It seems that the methods requires supervision in pretest task 4.2.1 and not in 4.2.2 for the pretext task, can the authors comment on potential use of T-Rep in completely unsupervised pretraining?\n5. How much overlap do we need for contextual consistency? How is this parameter decided? \n6. In section 3, the terms such as contextual consistency and heirarchical loss are only defined intuitively. These need to be defined in terms of their mathematical formulation(s). Currently, these are difficult to understand. Furthermore, it is also unclear how this plays out mathematically in linear projection layer. \n7. What is the mathematical action of TCN layer, and what is its role?\n\n\nMinor: \n1. Fig. 1 uses yellow against a whote background is difficult to read.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is very well motivated, I congratulate the authors on explaining the issues with contrastive learning in the context of time-series. \n2. The paper is overall well-written and easy to follow. \n3. Experiments are set-up well except that monte-carlo simulations are missing.", "weaknesses": "1. The results are not repeated across random seeds, which significantly impacts the confidence in the method. \n2. Visualizations: since the paper centers around time series representation learning, it would have been extremely valuable to see t-SNE plots of the learned embeddings to see how T-Rep performs over SOTA (TS2VEC) and others. \n3. There are a number of moving parts, and some level of ablations are expected, but missing. For instance, impact of overlap in contexual consistency etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BbypRcymac", "reviewer": "Reviewer_znEu", "summary": "The paper proposes T-Rep, a self-supervised method for learning time series representations at the timestep level. The key innovation of T-Rep is the use of learnable time embeddings in pretext tasks to learn detailed temporal dependencies and robustness in time series representations. Experiments demonstrate improved performance.", "questions": "See the weekness.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper proposes T-Rep, a self-supervised method for learning representations of time series at the timestep level.\n\nT-Rep learns vector embeddings of time called \"time-embeddings\" alongside its feature extractor encoder. The time-embeddings help capture temporal features like trend, periodicity, distribution shifts.\n\nThe time-embeddings are incorporated into pretext tasks to learn fine-grained temporal dependencies and make the model robust to missing data. Two new pretext tasks are proposed: time-embedding divergence prediction and time-embedding-conditioned forecasting.\n\nT-Rep is evaluated on downstream tasks of classification, forecasting, and anomaly detection. It outperforms previous self-supervised methods like TS2Vec, showing the benefit of time-embeddings.\n\nT-Rep is more robust to missing data than methods like TS2Vec. Visualizations show T-Rep can produce smooth representations even with missing timesteps.", "weaknesses": "The choice of time-embedding architecture is not well motivated or analyzed. Different architectures are used for different tasks, but it is unclear why they perform best in each case. More ablation studies on the time-embedding design could strengthen this key component.\n\nThe pretext tasks using time-embeddings seem somewhat ad-hoc. While they demonstrate the utility of time-embeddings, developing more principled pretext tasks derived from intrinsic properties of time series could be beneficial.\n\nThe comparison to previous methods like TS2Vec is not entirely fair, as T-Rep uses a larger encoder architecture. Comparisons with a TS2Vec model of comparable complexity could better isolate the benefits of the time-embeddings. Besides, the composition on TimeNet and PatchTST on SOTA ETT dataset is necessary.\n\nThe treatment of missing data is a major claimed contribution, but the missing data experiments are limited. More systematic tests on real-world messy data with different missing data types could better showcase these abilities.\n\nThe interpretability of representations is claimed but only briefly demonstrated with some visualizations. More analysis connecting latent dimensions to meaningful time series properties could better support the interpretability claims.\n\nThe classification task does not standalone evaluate the quality of the learned representations. Adding unsupervised evaluations like clustering could help assess representation quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kfFZ8qbmDd", "reviewer": "Reviewer_5x3c", "summary": "The paper introduces a sef-supervised representation learning method with a temporal, timestep granularity. The model is a linear projection layer and time embedding module that is fed into a dilated convolutional encoder. This representation is then used by four pretext tasks, two of which (time-embedding divergence prediction and time-embedding conditioned forecasting) are introduced in this paper. A linear combination of the pretext task losses is then used to compute a hierarchical loss. The paper evaluates the approach on different downstream tasks, including anomaly detection, forecasting, and classification. Additionally, the model authors show that the model is robust to missing data and qualitatively visualize the time evolution of the representation.", "questions": "Appendix A.2 mentions that each models are run 10 times but also notes \"Also, we do not use any random seed, (...)\". I'm unsure what this means. Are all experiments started from the same seed or are ten (fixed) seeds used here? I would kindly ask the authors to clarify that.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "### Originality\nIntroducing the temporal structure into time series representations are an important problem. This paper attempts to capture the time evolution with two additional pretext tasks and a time embedding module. These components are well reasoned and appear to result into a representation that can be successfully leveraged in tasks that require timestep granularity (forecasting, anomaly detection) and tasks that might not necessarily need it (classification).  \n\n### Quality\nThe discussion of the related work covers related articles well and summarizes the gap in learning temporal resolution in representation learning for time series. The experimental section covers standard benchmarks for anomaly detection, forecasting, and classification. I think the experimental evaluation has some weaknesses, which I will elaborate in the Weaknesses section. \n\n### Clarity\nThe paper is well written. The introduction and related work clearly introduces the challenges and previous work in this space and points out the specific challenge that the paper aims to address (learning timestep granular representations that work for several downstream tasks). The experimental section is also well written and easy to follow. \n\n### Significance\nTime series representation learning is an important problem. Previous work focused on classification or other tasks that don't require timestep granularity of the embeddings, while some applications (like forecasting) would require this. This paper represents a significant step into this direction of universal time series embeddings that are useful for several different applications.", "weaknesses": "The paper has two weaknesses in the experimental evaluation: Missing ablation experiments and choice of baselines. \n\n### Ablation study \n\nThe method presented in this work shares several components with TS2Vec (linear projection layer, dilated convolution encoder, instance-wise/temporal contrasting, and the hierarchical loss). It would be beneficial for the readers to point out the common components of T-Rep and TS2Vec. It would also be interesting for the reader to understand, which of the additional components introduced in this work actually improved the performance, but an ablation study to investigate this is missing. My understanding is that the additional pretext tasks require the time embedding (TE) module, so factoring that out for ablation is probably difficult. However, testing the impact of the pretext tasks setting the weights of the linear combination loss for the TE-conditioned forecasting and/or TE-divergence prediction should be straightforward. I would kindly ask the authors to add these ablation results. I would consider raising my score if the ablation study is added. \n\n\n### Choice of baselines \n\nFor the forecasting experiments, the authors use the benchmark introduced by Zhou et al., AAAI 2021 is a common benchmark in forecasting. However, several iterations on transformer architectures have been published since this work, some of which are even outperformed by linear baselines (Zeng et al., AAAI 2022). Given that the forecasting task introduced here is a linear regression layer using the time series embeddings, it would be good to understand the gain in performance relative to the linear methods (N-Linear and D-Linear) that are introduced in Zeng et al., AAAI 2022. I would kindly ask the authors to consider these baselines. \n\nFor classification, Minirocket (Dempster at al., KDD 2021) is a simple and fast baseline for practical applications and it would be interesting to understand the gain in accuracy from T-Rep over Minirocket.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F1IjqZ1rlk", "reviewer": "Reviewer_9kuw", "summary": "This paper proposes a time embedding method in representation learning on unlabeled time series data. Following existing contrastive learning framework, it proposes two new regressive tasks to induce the learned time embeddings to play a good role in the final representation. Experiments on three different task categories  are performed to support the effectiveness of the proposed designs. An analysis on the robustness to missing data is also provided.", "questions": "1. While the authors claim that time embedding $\\tau$ are induced to learn such time-related features as trend, periodicity etc. , they are function of the time indices and are unaware of the time series values. How could the aforementioned features be captured from indices only?\n2. I understand that $\\tau$ is normalized so that their difference can be measured by JSD. But there are alternative ways to measure the divergence without confining the norm of time embeddings. \n3. The task in Sec 4.2.1 aims to regress difference in the representation to the time distance. As stated in point 1, again, various pattern difference might be contained in the representations from arbitrary pairs of time series, therefore it's very likely that they can not regress to the consistent time distance. Same concern applies to the other task\n4. The claim that the two proposed task complement each other needs to be elaborated.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The presentation of the paper is smooth and easy to follow. The limitation of the existing methods and the novel part in the proposed method is quite clear.\n2. Comprehensive experiment results are provided to show the versatility of the learned representations.", "weaknesses": "1. The biggest flaw of the paper is the lack of ablation study, in my opinion. While sufficient quantitative results are provided to showcase the advantage of the proposed T-Rep, there is no clear evidence that support the effectiveness of the time embedding module, the very core highlighted in the paper. While the evidence in the analysis of robustness to missing data claims that the use of time embedding is leveraged to improve the contextual awareness and to fill the gap of the missing interval, but it is not supported by any rigorous analyses and remains a conjecture. \nExtensive ablation studies should be provided to compare the T-Rep with the baseline of TS2Vec and versions with either of the regressive pretext task removed. \n2. Above the flaw in empirical study, the idea of using time embedding to improve contrastive learning is not well-motivated. It is not clear why time embedding would be the ideal way to improve the latent temporal structure. See details in questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)