Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency
Authors: Xiang Zhang
Ziyuan Zhao
Theodoros Tsiligkaridis
Marinka Zitnik
Venue: NeurIPS 2022 Accept (2022)
Keywords: time series
pre-training
contrastive learning
transfer learning
self-supervised learning

Abstract: Pre-training on time series poses a unique challenge due to the potential mismatch between pre-training and target domains, such as shifts in temporal dynamics, fast-evolving trends, and long-range and short-cyclic effects, which can lead to poor downstream performance. While domain adaptation methods can mitigate these shifts, most methods need examples directly from the target domain, making them suboptimal for pre-training. To address this challenge, methods need to accommodate target domains with different temporal dynamics and be capable of doing so without seeing any target examples during pre-training. Relative to other modalities, in time series, we expect that time-based and frequency-based representations of the same example are located close together in the time-frequency space. To this end, we posit that time-frequency consistency (TF-C) --- embedding a time-based neighborhood of an example close to its frequency-based neighborhood --- is desirable for pre-training. Motivated by TF-C, we define a decomposable pre-training model, where the self-supervised signal is provided by the distance between time and frequency components, each individually trained by contrastive estimation. We evaluate the new method on eight datasets, including electrodiagnostic testing, human activity recognition, mechanical fault detection, and physical status monitoring.  Experiments against eight state-of-the-art methods show that TF-C outperforms baselines by 15.4% (F1 score) on average in one-to-one settings (e.g., fine-tuning an EEG-pretrained model on EMG data) and by 8.4% (precision) in challenging one-to-many settings (e.g., fine-tuning an EEG-pretrained model for either hand-gesture recognition or mechanical fault prediction), reflecting the breadth of scenarios that arise in real-world applications. The source code and datasets are available at https://github.com/mims-harvard/TFC-pretraining.

Self-Supervised Contrastive Pre-Training for Time Series via Time-Frequency Consistency
Xiang Zhangi
Ziyuan Zhao"
Harvard University
Harvard University
@lang_ahang@hmms harvard.edu
ziyuanzhao@college harvard.
Theodoros Tsilikkariiss MIT Lincoln Laboratory ttsiliill mit. edu
Marinka Zitnik
Harvard University marinka harvard.edu:
Abstract
Pre-training on time series poses a unique challenge due to the potential mismatch between pre-training and target domains, such shifts in temporal dynamics, fast-evolving trends, and long-range and short-cyclic effects, which can lead to poor downstream performance. While domain adaptation methods can mitigate these shifts, most methods need examples directly from the target domain, making them suboptimal for pre-training. To address this challenge, methods need accommodate target domains with different temporal dynamics and be capable of doing so without seeing any target examples during pre- traininng. Relative to other modalities, in time series, we expect that time-based and frequency- based representations of the same example are located close together the time- frequency space. To this end, we posit that time-frequee frequency consistency (TF-C) embedding time-hased neighborhood of an example close to frequency-base neighborhood - is desirable for pre-training. Motivated by TF-C, we define a decomposable pre-training model, where the self-supervised signal is provided by the distance between time and frequency components, each individually trained by contrastive estimation. We evaluate the new method on eight datasets, including electrodiagnostic testing, human activity recognition, mechanical fault detection, and physical status monitoring. Experiments against eight state-of- the-art methods show that TF-C outperforms basclines by 15.4% (F1 score) average in one-to- one settings (e.g.. fine-tuning an GEG-pretrained model on EMG data) and by 8.4% (precision) in challenging one-to-many settings (e.g., fine-tuning EFG-pretrained model for either hand-gesture recognition or mechanical fault prediction), reflecting the breadth of scenarios that arise in real- woal-wrlld applications. The source code and datasets are available at http://iihuu..ooommmmmeee haatsss-aavarrTFFCC. preeranntai...g.
Introduction
Time series plays important roles in many areas, including clinical diagnosis, traffic analysis, and climate science [1, 2, 4, 5, 6]. While representation learning has considerably advanced analysis of time series [7, 8, 9] more broadly [10]. learning generalizable representations for temporal data remains fundamentally challenging problem [8, 11]. There numerous immediate benefits from generating such representations, of which pre-training capability is particularly desirable and of
'These authors contributed equally to this work. Present address: University of North Carolina at Charlotte, xiang zhang @ uncc.edu
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Self Self-supervised pre-training
One to-many independent fine Ene-luning
Scenario Gestura recagrition
Scenario Fault detection
Scenario 3 Seizura diagnasis
based
Frequency basedd contrestiva enpodar
contrastive enooder
Transfer
F(,
F(1
F(1 +)
F(1
Time- Frequercy Commiranyy (TF-C)
Well-rainee model considering TF-C
Jhy
Jm
Figure I: a. Illustration of Time- Time-Frequency Consistency (TF-C), Time-based embedding =! and frequency- based embedding z of time series sample wi, along with 2i and Σ learned from augmentations of æ/. should be close each other in the latent time- -reeffry space, b. Leveraging TF-C property in time series to optimize a pre-training model F with parameters θ that get fine-tuned to & on a small scenario speriici dataset.
great practical importance [12, 13]. Central to pre-training is a question of how to process time series in diverse dataset to greatly improve generalization new time series coming from different datasets [14, 15, 10]. By training a neural network model on a dataset and transferring it to a new target dataset for fine-tuning, i.e., without explicit retraining that target data, expect the resulting performance to be at least as good as that of state-of-the-ar models tailored to the target dataset.
However, unfortunately, the expected performance gains are often not realized for variety of reasons (e.g., distribution shifts, properties of the target dataset unknown during pre-training) [16, 17] that get compounded by the complexity of time series: large variations of temporal dynamics across datasets, varying semantic meaning, irregular sampling, system factors (e.g., different devices or subjects), etc. [18, 17]. This complexity of time series limits the utility of knowledge transfer for pre-training [19, 20J. For example, pre-training a model on a diverse time series dataset with mostly low-frequency components (smooth trends) may not lead to positive transfer downstream tasks with
components (transient events) [17]. Examining these challenges can provide clues to what kind of inductive biases could facilitate generalizable representations of time series this paper offers a strategy for that through a novel time- frequency consistency principle.
In addition, target datasets are not available during pre-training (different from domain adaption [21]; Appendix A), requiring that the pre-training model captures a latent property that holds true for previously unseen target datasets. At the center of this desideratum is the idea of property that would be shared between pre-training and target datasets and would enable knowledge transfer from pre- pre-training to fine-tuning. In computer vision (CV), pre-training is driven by findings that initial neural layers capture universal visual elements, such as edges and shapes, that are relevant regardless of image style and tasks [22]. In natural language processing (NLP), the foundation for pre-training is given by linguistic principles of semantics and grammar shared across different languages [23]. However, due the aforementioned temporal complexity, such a principle for pre-training on time series has not yet been established. Moreover, supervised pre-training requires access to large annotated datasets, which limits its use in domains where richly labeled datasets are scarce [24, 25). For example, in medical applications, labeling data at scale is often infeasible or can be expensive and noisy (experts can disagree on ground-true -tuth labeling [26, 27]. e.g., whether an ECG signal indicates a normal vs. abnormal rhythm) [28, 29]. To mitigate these issues, self-supervised learning emerged as a promising strategy to sidestep the lack of labeled datasets [30].
Present work. We introduce a strategy for self-supervised pre-training in time series by modeling Freq-nnyy Consistency (TF-C). TF-C specifies that time-based and frequency-based representa- tions, learned from the same time series sample, should be closer to each other in the time-frequency space than representations of different time series samples. Specifically, we adopt contrastive learn- ing in time-space to generate a time-based representation. In parallel, we propose a set of novel augmentations based on the characteristic of the frequency spectrum and produce a frequency-basse embedding through contrastive instance discrimination. This is the first work to develop frequency- based contrastive augmentation to leverage rich spectral information and explore time- freeufeeyynncy consistency in time series. The pre- pre-training objective is to minimize the distance between time-based and frequency- -based embeddings using a novel consistency loss (Figure 1 (a)). The self-supervised loss is used to optimize the pre-training model and enforce consistency between time and frequency domains in the latent space. The learned relationship encoded in model parameters are transferred to initialize the tuning model and improve performance in datasets of interest (Figure I (b)).

---

We evaluate the TF-C model on eight time series datasets under two evaluation settings (i.e., one- to-one and one-to-many). The eight datasets cover a large set of variations: different numbers of channels (from univariate to 9-channel multivariate), varying time series lengths (from 128 to 5, 20)), different sampling rates (from 16 Hz to 4,000 Hz), different scenarios (neurological healthcare, human activity recognition, mechanical fault detection, physical status monitoring, etc.) and diverse types of signals (EEG, EMG, ECG, acceleration, and vibration). We compare TF-C approach to eight state-of-the-art baselines. Results show that TF-C achieves positive transfer, outperforming all baselines by a large margin of 15.4% (FI score) on average. Further, the approach outperforms the strongest baselines with improvement of up to 7.2% in the Fl score. Finally, the TF-C approach improves prior work by 8.4% in precision (when pre-training the model on sleep EEG signals and fine-tuning it on hand-gesture recognition) in challenging one-to- many setups that apply the same pre-trained model to multiple independent fine-tuning datasets.
2 Related Work
Pre-training for time series. Although there are studies on self-supervised representation learning for time series [7, 8, 31, 32] and self-supervised pre-training for images [33, 34, 35, 24J, the intersection of these two areas, i.e., self-super pre- tre-irrinig for time series, remains underexplored. In time series, it's not obvious what reasonable assumptions can bridge pre-training and target datasets. Hence, pre-training models in CV [36, 37, 14] and NLP [10, 15, 38] are not directly applicable due to data modality mismatch, and the existing results leave room for improvement [31, 39, 40]. Shi et al. [12] developed the only model to date that is explicitly designed for self-supervised time series pre-training. The model captures the local and global temporal pattern, but it is not convincing why the designed pretext task can capture generalizable representations. Although several studies applied transfer learning in the context of time series [7, 8, 18, 41J, there is no foundation yet of which conceptual properties are most suitable for pre-training on time series and why. Addressing this gap, we show that TF-C, designed to be invariant to different time-series datasets, can produce generalizable pre- pre-training models.
Unlike domain adaptation [21. 42] that requires access to target datasets during training, pre- pre-training models do not have access to fine-tuning datasets. As a result, one needs to identify a generalizable time-series property to benefit from pre-training. Further, self-supervised domain adaptation does not need labels in the target dataset but still requires labels for model training [43, 44]. In contrast, TF-C does not need any labels during pre-training.
Contrastive learning with time series. Contrastive learning, a popular type of self-supervised learning, aims to learn an encoder that maps inputs into an embedding space such that positive sample pairs (original augmentation and another alternative augmentation/view of the same input sample) pulled closer and negative sample pairs (original augmentation and an alternative input sample augmentation) are pushed apart [30, 45]. Contrastive learning in time series is less investigated in comparison, partly due to the challenge of identifying augmentations that capture key invariance properties in time series data. For example, CLOCS defines adjacent time segments as positive pairs [41], and TNC assumes overlapping temporal neighborhoods have similar representations [46]. These methods leverage temporal invariance to define positive pairs which are used to calculate con- trastive loss, but other invariances, such as transformation invariance (e.g.. SimCLR [40], contextual invariance (e.g., TS2vec [47] and TS-TCC [48]) and augmentations are possible. In this work, we propose an augmentation bank that exploits multiple invariances to generate diverse augmentations (Sec. 1),, which adds richness to the pre-training model [48]. Importantly, we propose frequency- based augmentations by perturbing the frequency spectrum of time series (e.g., adding or removing the frequency components and manipulating their amplitude; more details in Sec. 4.2) to learn better representations by exposing the model to a local range of frequency variations. In previous work, CoST processes sequential signals through the frequency domain, but the augmentations are still implemented in time space [49]. Similarly, although BTSF [50] involves frequency domain, its data transformation is solely implemented in the time domain using instance- dropout. Additional commentary on differences between CoST and BTSF is in Appendix B. To the best of our knowledge, this is the first work that directly perturbs the frequency spectrum to leverage frequency-invarian for contrastive learning. Further, we develop a pre-training model that subjects to TF-C upon two individual contrastive encoders.
3

---

3 Problem Formulation
1, , N of unlabeled time series samples where sample se! has Krtet channels and [ timestamps. Let Diune = {{ar!ume :15) i 1..... ......... M) be a fine-tuning (i.e., target; target and fine-tuning are used interchangeably) dataset of labeled time series samples, each having Ktune channels and L L'uee timestamps. Furthermore, every sample xt! is associated with a label y/s E {I....,CI C'}, where C' is the number of classes. Without loss of generality, in the following descriptions, we focus on univariate (single-channel) time series, while noting that our approach can accommodate multivariate time series of varying lengths across datasets (shown in experiments in Sec. 5.2). We use superscript symbol to denote contrastive augmentations, We note that x. = z; denotes an input time series sample, and x denotes discrete frequency spectrum of z;-
We are given pre-training dataset 'Dpret = [ar poct
Problem Supervsd Contrastive Pre-Training For Time Series). Given an unlabeled pre-training dataset Do with N samples and a target dataset Diume with M samples (M & N). The goal is use Doret to pre-train model F so that by fine-tuning model parameters on Diune, the fine-tuned model produces generalizable representations z!!me = F(x: for every tume
We follow an established setup. e.g., [41]: for pre-training, only the unlabeled dataset Diret is available while, for fine-tuning, small labeled dataset Drune can be used. short, a model F is pre-trained on the unlabeled time series dataset Deee and its optimized model parameters Q are fine-tuned to go from F(, +) to F(; +) using the dataset Dtune The + denotes fine-tuned model parameters. Note that this problem (i.e., Dr is independent of the target dataset) is distinct from domain adaptation as fine-tuning dataset Dlune is not accessed during pre-training. As a result, the pre-trained model can be used with many different fine-tuning datasets without re-training.
Rationale for Fime-Frequency Consistency (TF-C), The central idea is to identify a general property that is preserved across time series datasets and use it to induce transfer learning for effective pre-training. The time domain shows how sensor readouts change with time, whereas the frequency domain shows how much of the signal lies within each frequency component over the entire spectrum [51]. Explicitly considering the frequency domain can provide an understanding of time series behavior that cannot be directly captured solely in the time domain [52]. However, existing contrastive methods (e.g.. [47, 48]) focus exclusively on modeling the time domain and ignore the frequency domain altogether. One can argue that approach is sufficient in the case of high-capacity methods as time and frequency domains are different views of the same data [53]. which can be transstrddl using transformation, such Fourier and inverse Fourier [54, 52]. The relationship between the two domains, grounded in signal processing theory, provides an invariance that is valid regardless of the time series distribution [55, 56] and thus serve as an inductive bias for pre- training. Appendix C provides a commentary with analogies for images. Approaching this invariance through the lens of representation learning, we next formulate Time-Frequency Consistency (TF-C). The TF-C property postulates there exists a latent time-frequency space such that for every sample 2, time-based representation z! and frequency-bas representation 2 of the same sample, together with their local augmentations (defined later), are close to each other in the latent space.
Representational Fime-EFequency Consistency (TF-C). Let x; be time series and be a model satisfying TF-C. Then, time-based representation z and frequency-based representation z as well as representations of x local augmentations are proximal in the latent time-freqnency space.
Our strategy is to use dataset Drret to induce TF-C in F's model parameters θ which, in turn, are used to initialize the target model on Dlune and produce generalizable representations for downstream prediction. The invariant nature of TF-C means that the approach can bridge Deret and Dlune even when large discrepancies exist between them (in terms of temporal dynamics, semantic meaning, etc.), providing a vehicle for general pre- pre-trainin on time series.
To realize TF-C, our model JF has four components (Figure 2): a time encoder Gr, frequency encoder Gr. and two cross-space projectors R2 and Re that map time-based and frequency- representations, respectively, to the same time-frequeny space. Together, the four components provide a way to embed æ; to the latent time-freen mene-freuency space such that the time-based embedding 2! = (xx()) and the frequency-basee based embedding x! = R2 (G(())) are close together.

---

Time
Crs
encoder
Ry
2
Gr
Xj
W
Time Series
Ca
X
Frequency encoder
GF
Ry
CF
Positive pair
Negative pair
Cr Time
Cp. Frequency loss
Cc Consistancy loss
Figure Overview TF-C approach. Our TF-C pre-training model F has four components: a time encoder Gr, a frequency encoder Gt, and two cross-space projectors RG and R.. For an input time series æı, the model produces time-bat based representations (i.e., x; and Σ of input and its augmented version, respectively) and frequency-basee bbase representations (i.e. 2 and & of input æ; and its augmented version, respectively). The TF-C property realized by promoting alignment time- and frequency-basee based representations in the latent time-frequency space, providing vehicle transferring a target dataset not seen hefore.
Our Approach
Next, we present the architecture of the developed self-supervised contrastive pre-training model F. Unless specified otherwise, the data mentioned in this section are from pre-training dataset and the superscript pret is omitted for simplification. Here we describe the model using univariate time series as example, but our model can be straightforwardly applied to multivariate time series (Sec 5),
4.1 Time-based Contrastive Encoder
For given input time series sample æ;, we generate an augmentation set X through time-based augmentation bank 8" x! XT. Each element कद. X augmented from æ; based on the temporal characteristics. Here, the time-based augmentation bank includes jittering, scaling, time-shifts, and neighborhood segments, all well-established in contrastive learning [40, 48, 41]. We develop an augmentation bank to produce diverse augmentations (rather than single type of augmentation) and expose the model complex temporal dynamics, which produces more robust time-based embeddings [48].
For the input æ; randomly select augmented sample zi E X and feed into contrastive
time encoder Gy that maps samples to embeddings. We have h = Gy (x:) and h: = (TT)) As Ø is generated based on af after passing through G7, assume the embedding of x! is close to the embedding of : but far away from the embedding of æ; and TJ that are derived from another
sample x. E Dpret [34, 47, 41]. In specific, we select the positive pair as (x. z!) and negative pairs as (x), ( and (x:, aJ) [34].
Contrastive time loss. To maximize the similarity within positive pair and minimize the similarity within a negative pair, we adopt the NT-Xent (the normalized temperature-scatl distance function d which is widely used in contrastive learning [34, 40]. In specific, define the loss function of the time-based contrastive encoder in terms sample x! as:
cross entropy loss)
exp(sim(hi) (h,, /T) Lr, d(h , N., D = -llo Ee, Dim expeexxpsiiiil (h), (,(z,))/T)
(1)
where immuu,0 v) n² u²/// ||a||| ||l||| denotes the cosine similarity, the lize is indicator function that equals to 0 when = and otherwise, and 7 is temporal parameter to adjust scale. The T Drrect refers to a different time series sample its augmented sample. This loss function
5

---

urges the time encoder Gr to generate closer time-basee embeddings for positive pairs and push the embeddings for negative pairs apart from each other.
4.2 Frequency- Contrastive Encoder
We generate the frequency spectrum æi from time series sample æ. through transform operator (e.g., Fourier Transformation [54]). The frequency information in time series is universal and plays key role in classic signal processing [57, 53, 55], but it is rarely investigated in self-supervised contrastive representation learning for time series [58]. this section, we develop augmentation method to perturb o based on characteristics of frequency spectra and show how to generate frequency-based representations.
As every frequency component in the frequency spectrum denotes basis function (e.g., sinusoidal function for Fourier transformation) with the corresponding frequency and amplitude, we perturb the frequency spectrum by adding or removing frequency components. A small perturbation in the frequency domain may cause large changes to the temporal patterns in the time domain [55]. To make sure the perturbed time series is still similar to the original sample (not only in frequency domain but also in time domain; Figure 6), we use a small budget E in the perturbations where E' denotes the number of frequency components we manipulate. While removing frequency components, we randomly select E frequency components and set their amplitudes to O. While adding frequency components, we randomly choose E frequency components from the ones have smaller amplitude than a Ams and increase their amplitude to a Am- The Am is the maximum amplitude in the frequency spectrum and is pre-defined coefficient adjust the scale of the perturbed frequency component (a = 0.5 in this work). We produce an augmentation set X for x through frequency- augmentation bank 1 æ! X². As described above, we have two augmentation methods (ie., removing or adding frequency components) in 13, | 2. Details on the exploration of frequency augmentation strategies are covered in Appendix J.
We utilize a frequency encoder Gp to map the frequency spectrum (e.g., x;)) to a frequency-based embedding (e.g., h Gr(z))). We assume the frequency encoder Gp can learn similar embedding for the original frequency spectrum of and a slightly perturbed frequency spectrum xi E X Thus, we set the positive pair as (x ã and the negative pairs as (x!, æτ)) and (æf, 2f).
Contrastive frequency loss. We calculate frequency- contrastive loss for sample æ; as:
exp(sim(hī, /T)TT) Le = d(h,, h., D = llg Ex,EDON EDo 1 exsjexp((iim(t) (h,, Gr(z))/TT)
In preliminary experiments, find that the value of has little effect on performance and the same r throughout all experiments. The Le yield frequency encoder Gp producing embeddings invariant to frequency spectrum perturbations.
4.3 Time-Frequent Consistency
We develop a consistency loss item Lc.; to urge the learned embeddings to satisfy TF-C: for a given sample, its time-based and frequency- to be close to each other (see Sec. for justification). To make sure the distance between embeddings measurable, we map h from time space and hξ from frequency space to a joint time-frequency space through projectors Rt and Re, respectively. In specific, for every input sample æ;, we have four embeddings, which are 2! = R2 (h.)). Σष Rr(h)) ) 2! = Rs (h()), and ž Rs (h.)) The first two embeddings are generated based on temporal characteristics and the latter two embeddings are produced based on the properties of frequency spectrum.
embeddings (and their local neighborhoods) are supposed
To enforce the embeddings the time-frequency space subject to TF-C, we design consistency loss Le that measures the distance between based embedding and frequency- based embedding. We use STF d(z], zz, Dare)) denote the distance between 2 and =!). Similarly, we define S STF. and غीं Note, this time-frequency space, we don't consider the distance between 2! and !! where the two embeddings are from the same domain (i.e., time domain). The same applies to pair the distance between 2! and zy. We have already considered information of above two pairs in the calculation of Lr, and Lr

---

Next, let's closely observe S:" and S. that involve three embeddings: z/ z;, and z Here, x and of are learned from the original sample (æ) and æ;) while x is learned from the augmented क.. Thus, intuitively, z should be closer to 2 in comparison to 2 Motivated by the relative relationship, we encourage the proposed model to learn a S that is smaller than S Inspired by the triplet loss [59], design (S!F ST 6) term of consistency loss Le,, where ō is given constant margin to keep negative samples far apart [60]. This term optimizes the model towards smaller S! and relatively larger S.a. Similarly, S. is supposed to be smaller than S and S In summary, calculate the consistency loss Le, for sample 2; by:
Le
Σ(s Speir
STD Sp + 6), Spair E (S S ST)
(3)
where S! denotes the distance between a time- based embedding (e.g., 2 or 2 and a frequency- based embedding (e.g., z or 2.). In each pair, there is least embedding that is derived from augmented sample instead of the original sample. The ò is a pre-defined constant. By combining all the triplet loss items, Le encourages the pre-training model to capture the consistency between time-based and frequency-basee embeddings in model optimization. Note, although the Eq. 3 does not explicitly measure the loss across different time series samples (e.g., 2; and æ.)). the cross-sample relationships are implicitly covered in the calculation of S and spair.
4.4
Implementation and Technical Details
The overall loss function in pre-training has three terms. First, the time-based contrastive loss D, urges the model learn embeddings invariant to temporal augmentations. Second, the frequeney- based contrastive loss Lp promotes learning of embeddings invariant to frequency spectrum-bii based augmentations. Third, the consistency loss Le guides the model to retain the consistency between time-based and frequency-basee embeddings. In summary. the pre-training loss is defined as:
LTF-C, = M(Cr,, + Lr,s) + Cc,
(4)
where À controls the relative importance of the contrastive and consistency losses. We calculate the total loss by summing LTr-.; across all pre-traine pre-training samples. In implementation, the contrastive losses are calculated within the batch. From our problem definition, the model J we want to learn is the combination of neural networks Gr. Rr. Gp. and Rs. When pre-training is completed, we store parameters of entire model, and denote it F(., +) where O represents all trainable parameters. When a sample tine is presented, fine-tuned model F generates an embedding x!ume via concatenation [z, T Ju where & are fine-tuned model's 's parameters.
as: Z!ue
F Fx: 2)
Experiments
We compare the developed TF-C model with 10 baselines on 8 diverse datasets. We investigate the time series classification tasks in the context of one-to-one and one-to-many transfer learning setups (the many-to-one setting is fundamentally different discussed in Appendix K). We also assess TF-C in extensive downstream tasks including clustering and anomaly detection.
Datasets. (1) SLEEPEEG [61] has 371,055 univariate brainwaves (EEG; 100 Hz) collected from 197 individuals. Each sample is associated with one of five sleeping stages. (2) EPILEPSY [62] monitors
the brain activities of 500 subjects with single-channel EEG sensor (174 Hz). A sample is labeled in
binary based on whether the subject has epilepsy or not. (3) FD-A [63] gathers the vibration signals from rolling bearing from a mechanical system aiming at fault detection. Every sample has 5,120 timestamps and an indicator for one out of three mechanical device states. (4) FD-B [63] has the same setting the FD-A but the rolling bearings
performed in different working conditions (e.g., varying rotational speed). (5) HAR [64] has 10,299 9-dimension samples from 6 daily activities. (6) GESTURE [65] includes 440 samples that are collected from 8 hand gestures recorded by an accelerometer. (7) ECG [26] contains 8.528 single-sensor ECG recordings with sorted into four classes based on human physiology. (8) EMG [66] consists of 163 EMG samples with 3-class labels implying muscular diseases. Dataset labels are not used in pre-training. Further dataset statistics are in Appendix D and Table 3.
Baselines. We consider baseline methods. This includes state-of-the methods: TS-SD [12], TS2vec [47], CLOCS [41], Mixing-up [18], TS-TCC [48], SimCLR [40], TNC [46], and CPC [30].
7

---

Table 1: One-to-one pre-training evaluation (Scenario 3). Pre-training is performed on HAR, followed by
fine-tuning on GESTURE. Results for other three scenarios are shown Tables 4-6,
Models
Accuracy
Precision
Recall
Fl score
AUROC
AUPRC
Non-DL (KNN)
0.6766=000
0.6500#0.000
0.6821
0.8190=0.000 0.6442+00000
0.523 =30000
Random Init.
0.421910.086
0,4751 10.0925
0.496310.108
0, 0.0096
0.7129
0.3358
TS-SD
0.6937200533
0.6806:00:446
0.6883=00555
0.8708 0.678
0.626 m00
TS2vec
0.6453
0.6287
0.6451
0.6261 +0.0294
0. 0.8890
0.6670=aot1
CLOCS
0.4731 =00029
0.4639+00432
0.4766=00026
0,4392++.019
0.8161
0.491 60
Mixing-up
0.7183
0.:00110:0166
0.7183
0.6991 10.0145
0.9127 10,0018
0.7654100071
TS-TCC
0.7593
0.7668 10:0257
0.7566110000
0.7457
0.8866:10.004
0.7217
SimCLR
0.43832200663
0.4255 1072
0.438332006s
0.3.13:200019
0.7721 1100055
0.41 16200011
TF-C (Ours)
0.782414.227
0.7982.10.066
0.8011
0.7991 1000296
0.9052.10.0136
1.7861
The TS2Vee, TS-TCC, SimCLR, TNC, and CPC are designed for representation learning on a single dataset rather than for transfer learning, we apply them to fit our settings and make the results comparable. As the training of TNC and CPC are very time-consuming and relatively less competitive (Table 4), we only compare them in the one-to-one setting (scenario 1) while not in other experiments. To examine the utility of pre-training, we consider two additional approaches that are applied directly to tinettning datasets without any pre-training: Non-DL DL (a non-deep learning KNN model) and Random Init. (randomly initializes the fine-tuning model). The evaluation metrics are accuracy, precision (macro-averaged), recall, F1 score, AUROC, and AUPRC.
Implementation. We use two 3-layer I-D ResNets [67] as backbones for encoders G- and GF- Our datasets contain long time series (samples in FD-A and FD-B have 5,120 observations), and preliminary experiments identified ResNet as a better option than a Transformer variant [68]. We use 2 fully connected layers for R, and R,, with no sharing of parameters. We set E = 1 and a 0.5 in frequency augmentations and r = 0.2, d = 1, = 0.5 in loss functions. Reported are mean and standard deviation values across independent runs (both pre-training and fine-tuning) data split. Results for KNN (K=2) do not change so the standard deviation is zero. Method details and parer-parameter selection are in Appendix E.
the same
5.1 Results: One-to-One Pre-Treee Tee-Trannig Evaluation
Setup. In one-to-one evaluation, we pre-train a model on one pre-training dataset and use it for fine-tuning one target dataset only. Scenario 1 (SLEEPEEG - EPILEPSY): Pre-training is done on SLEEPEEG and fine-tuning on EPILEPSY. While both datasets describe a single-channel EEG, the signals are from different channels/positions scalps, track different physiology (sleep vs. epilepsy), and are collected from different patients. Scenario 2 (FD-A + FD-B): Datasets describe mechanical devices that operate in different working conditions, including rotational speed, load torque, and radial force. Scenario 3 (HAR- + GESTURE): Datasets record different activities (6 types of human daily activities 8 hand gestures). While both datasets contain acceleration signals, HAR has 9 channels while GESTURE has 1 channel. Scenario 4 (ECG + EMG): While both are physiological datasets, the ECG records the electrical signal from the heart whereas EMG measures muscle response in response to a nerve's ' stimulation of the muscle. We note that the discrepancies between pre-training and fine-tuning datasets in the above four scenarios are substantial, and they cover diverse range of variation in time series datasets: varying semantic meaning, sampling frequency, time series length, number of classes, and system factors (e.g., number of devices or subjects). The setup further challenged by the relatively small number of samples available for fine-tuning (EPILEPSY: 60; FD-B: 60; GESTURE: 480; EMG: 122). Further details are in Appendix
Results. The results for the four scenarios are shown in Table and Tables 4-6. Overall, our TF-C model has won 16 out of 24 tests metries in 4 scenarios) and is the second-best performer in only 8 other tests. We report all metrics but discuss the Fl score in the following, On average, our TF-C model claims a large margin of 15.4% over all baselines. Although the strongest baseline is varying (such as TS-TCC in Scenario 2; Mixing-up up in Scenario 3), our model outperforms the strongest baselines by 1.5% across all scenarios. Specifically, as shown in Table (HAR - GESTURE; Scenario 3), TF-C achieves the highest performance of 79.91% % in F1 score, which yields margin of 7.2% over the best baseline TS-TCC (74.57%). One potential explanation is that Scenario involves a complex dataset (HAR has 6 classes while GESTURE has 8 classes) that can be difficult to model. The complexity of Scenario is further verified by poor performance of all models (+80%) relative to performance on other Scenarios (=90%): TF-C shows strong robustness by
8

---

Table 2: One-to-many pre-training evaluation. Pre-training is performed on SLEEPEEG, followed by an independent fine-tuning on EPILEPSY, FD-B, GESTURE, and EMG.
Scenarios
Models
Accuracy
Precision
Recall
score
AUROC
AUPRC
Non-DL. (KNN)
0.8525100000
0.643120000 0.86392000000
0.6791 100000
0.6434200000
0.6279
Random Init.
0.8963+0066
0.9213
0.7447
0.7959
0.8578:0.113
0.6489 1926
TS-SD
0.8952290822
1484
0.7767 as 1855
0.7677
0.7940gm 1825
SLEEPERG 1 EPILEPSY
TS2vee
0.9395:0004
0,9059;0.016
0,9039=00:18
0.9045
0,9587=00005
0.9430=00000
CLOCS
0,9507:am17
0,9301
20:007
0.9127
0.9206:0mm66
019803-20000
0.9669
Mixing-up
0.802
0.4011
20.0000
0.4451 0.5000=00000
+:00000
0.9743:00.008
0,9618
TS-TCC
0.9253+0009
0,9451
40.004p
0.
O. 0.8633
0,9842=0.004
0.9744
SimCLR
0.9071 1a00ess
0,9221 10.0106
0.78642. 1071
0.8178
0,9045 2000000
0.9128
TF-C (Ours)
0.9495
0.9456=:00us
0.8908
0.9149
0.9811=00037
0.9703
Non-DL. (KNN)
0.4470
0,2847 100000000000
0.3275
0,2284:100000
0.4946.000000
0.3308
Random Init.
0,4736:00025
0.4829#4.000
0.5235
0.491 +000500
0.7864
0.7528
SLEEPEEG 4
TS-SD
0.5566:00210
0.5710=0.883
02722
0,5703+000008
0.7 0.7191200113
n.5693
TS2vee
0.4790saa013
0.43339.0.000
0.4842.20.019
a..389
0.6463-a0.010
0,4442
FD-B
CLOCS
0,4927.a00m
0.4824400081
0.5873
0,4746100085
1.6992..0000
0.5501
Mixing-up
0.67892a1046
0,7146:00.844
0.7613::00198
0.7273
0.8209
2.77077::002
TS-TCC
0.5499gmmmm
0.5279 go000ss
0.63962.000
0.5418..00m8
0,7329g0000 20000m
0.58242.0088
SimCLR
0.4917:00437
0.5446:0..om
0.4760
0.4224
0.6619200021
0.5009
TF-C (Ours)
0.6938:00231
0.7559
#000349
0.7202=0x155
0.8965
0.7871=00227
Non-DL (KNN)
0.6833ga0000
0.6501
20.0000
0.6833
0.6443:a0000
0.8 0.81900000000
0.5232200000
Random Init.
0.4219+00229
0,4751
#00075
0.4963
0.4886+00459
0.7129=00166
0.3358
TS-SD
0.6922x0044
0.6698
0.6867200488
0.6656190445
0,8725 2000004
0.6185
SLEEPERG
TS2vee
0.6917+00033
+0,055
0.6854=.mm4
0.6570=0.092
0.8068
0.6989
CLOCS
0.44332005i8
0,4237 20,0794
0.4433:00:18
0.4014:0002
0.8073
0.4460100084
GESTURE
Mixing-up
0.6933+0x231
0.6719
0.6933=0m231
0.6497 +000006
0,8915
0.7279
TS-TCC
0.71882:0054
0.7135
2000052
0.7167
O. 2.6984
1.9099
2.7675
SimCLR
0.4804:000554
0.4955 0.5411=0.194
0.0031
0.5076=0.1588
TF-C (Ours)
0.764221umm6
0,7731
0.7429
0.7572.10001
0.9238
0.796
Non-DL (KNN)
0.4390
0.5143 0.3072
0.3979
0..025
0.4084200000
Random Init.
0.7780
0.590930002
0.666720013
0.6238.10007
209109.0.123
0.7771
TS-SD
0.4606+00000
0,1545
0.2111 0.
0.0005
0.3775=00010
SLEEPEEG 1.
TS2vee
0.7854200618
0,8040
0.6785
0.67661000901
0.9331:9900
0.8436::0072
CLOCS
0.69852a0ms
0.53062200700
0.53542mmm
0.513920049
0.792320002
0.64842m0m0
EMG
Mixing-up
0.30241005s
1.1099200126
0.2583201450
0.4506
0,36602
TS-TCC
0.78892am92
0.5851
20.0004
0.63102nm99
0.5904-20022
0.8851 200013
0.7939200mm
SimCLR
0.6146:.0882
0.5361 #0.1724
0.4990
0.4708 1486
0.7799
0.6392:a.156
TF-C (Ours)
0.8171
0.7265
0.8159
0.7683
0.9152a0021
0.8329a00:32
learning more generalizable representations. Additionally, we visualize the learned representations in fime-frequency space (Appendix I), and the analyses provide further support for the TF-C property.
5.2 Results: One-to-Many Pre-Training Evaluation
Setup. In one-to-many evaluation, pre-training done using one dataset followed by fine-tuning on multiple target datasets independently without starting pre- pre-training from scratch. Out of cight datasets, SLEEPEEG has most complex temporal dynamics [69] and is the largest 37,1055 samples). For that reason, we pre-train model on SLEEPEEG and separately fine-tune a well-pre-traines trained model on EPILEPSY, FD-B, GESTURE, and EMG.
Results. Results are shown in Table 2. As there are fewer commonalities between EEG signals vs. vibration, and acceleration vs. EMG, we expect that transfer learning will be less effective for them than one-to-one evaluations. The pre-training and fine-tuning datasets are largely different in the bottom three blocks (SLEEPERG + (FD-B, GESTURE, EMG)). The large gap reasonably leads to deterioration in baseline performances, however, our model has a noticeably higher tolerance to knowledge transfer across datasets with large gaps. Notably, We find that the proposed model with TF-C earned the best performance in 14 out of 18 settings in the three challenging settings: indicating our TF-C assumption is universal in time series. For example, our approach outperforms the strongest baseline by 8.4% (in precision) when fine-tuning on GESTURE. Our model has great potential to serve as a universal model when there is no large pre-training dataset that is similar to the small fine-tuning dataset. Furthermore, the TF-C consistently outperforms KNN and Random Init. (which are not pre- trained) by a large margin of 42.8% and 25.1% % (both in Fl score) on average.
Ablation study. We evaluate how relevant the model components are for effective pre-training. As shown in Table (SI.FFPERG > GESTURE; Appendix H), removing Lc, LT, and LF result in performance degradation (precision) of 6.1 6., 7.2%, and 6.7%, respectively. To validate that

---

the performance increment is not solely brought by a third loss term no matter what consistency it measures, we replaced consistency loss Le with loss term measuring the consistency within time space (named LTT-C) or within frequency space (named Lrr.c). Results show our consistency loss outperforms LTT.C and LFr C by 5.3% and 7.2% (accuracy), respectively.
5.3 Additional Downstream Tasks: Clustering and Anomaly Detection
Clustering Task. We evaluate the clustering performance of TF-C taking SLEEPEEG - EPILEPSY as an example. Specifically, we added a K-means (K=2), as Epilepsy has classes, on top of =quee in fine-tuning, We adopt commonly used evaluation metrics: Silhouette score, Adjusted Rand Index (ARI), and Normalized Mutual Information (NMI). Table shows our TF-C obtains the best clustering surpassing the strongest baseline (TS-TCC) by large margin (5.4% in Silhouette score). conveys that TF-C can capture more distinctive representations with the knowledge transferred from pre-training, which is consistent with the superiority of TF-C in the above classification tasks.
Anomaly Detection Task. We assess how TF-C performs on a sample-level anomaly detection task. Note we work on the sample-level rather than the observation- anomaly detection. Based on global patterns, the former aims to detect abnormal time series samples instead of outlier observations in sample (as in BTSF [50] and USAD [70]) which emphasizes local context. Specifically, In the scenario of FD-A + FD-B, we built a small subset of FD-B with 1,000 samples, of which 900 are from undamaged bearings, and the remaining 100 are from bearings with inner outer damage. Undamaged samples are considered "normal," and inner/outer damaged samples are "outliers." " In fine-tuning, used one-class SVM on top of learned representations 2! The experimental results (Table 8) show that our TF-C outperforms five competitive baselines with 4.5% in F-1 Score. Results show that the proposed TF-C is more sensitive anomalous samples and can effectively detect the abnormal status in mechanical devices.
Conclusion
We develop a pre-training approach that introduces time-frequency consistency (TF-C) as a mecha- nism to support knowledge transfer between time-series datasets. The approach uses self- superviede contrastive estimation and injects TF-C into pre-training, bringing time- based and frequency- based representations and their local neighborhoods close together in the latent space.
Limitations and future directions. TF-C property can serve universal property for pre-training on diverse time series datasets. Additional generalizable properties, such as temporal autoregressive processes, could also be helpful for pre-training on time series. Further, while our method expects as input regularly sampled time series. it can handle irregularly sampled time series by using an encoder (such as Raindrop [71] and SeFT [72]) that can embed irregular time series. For frequency encoder inputs of, alternatives include resampling or interpolation to obtain regularly sampled signals and using regular or non-uniform FFT operations. Furthermore, TF-C's current embedding strategy and loss functions are favorable for classification, leveraging global information over tasks that use local context (e.g., forecasting). Results show that the TF-C approach performs well across broad downstream tasks, including classification, clustering, and anomaly detection (Sec. 5.3).
Acknowledgments and Disclosure of Funding
We gratefully acknowledge support by US Air Force Contract No. FA8702-15-D-0001, Harvard Data Science Initiative, and awards from Amazon Research, Bayer Early Excellence in Science, AstraZeneca Research, and Roche Alliance with Distinguished Scientists. T.T. is supported by the Under Secretary of Defense for Research and Engineering under US Air Force Contract No. FA8702- 15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funders.
10

---

References
[1] Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data, 6(1):1-18, 2019.
[2] Shahbaz Rezaei and Xin Liu. Deep learning for encrypted traffic classification: An overview. IEEE communications magazine, 57(5)-6--81, 2019.
[3] Suman Ravuri, Karel Lene, Matthew Willson, Dmitry Kangin, Remi Lam, Piotr Mirowski, Megan Fitzsimons, Maria Athanassiadou, Sheleem Kashem, Sam Madge, et al. Skilful precipitation nowcasting using deep generative models of radar. Nature, 597(7878):62--67) 2021.
[4] Omer Berat Sezer, Mehmet Ugur Gudelek, and Ahmet Murat Ozbayoglu. Financial time series forecasting with deep learning: A systematic literature review: 2005-2019. Applied soft computing, 90: 10618,, 2020.
[5] Bing Su and Ji-Rong Wen. Temporal alignment prediction for supervised representation learning and few-shot sequence classification. In ICLR, 2022.
[6] Yixiang Deng, Lu Lu, Laura Aponte, Angeliki M Angelidi, Vera Novak, George Em Karniadakis, and Christos S Mantzoros. Deep transfer learning and data augmentation improve glucose levels prediction in type 2 diabetes patients. NPJ Digital Medicine, 4(1): 11)) 2021.
[7] Quentin Rebjock, Baris Kurt, Tim Januschowski, and Laurent Callot. Online false discovery rate control for anomaly detection in time series. NeurlPS, 34:26487-26498, 2021.
[8] Fan-Keng Sun, Chris Lang, and Duane Boning. Adjusting for autocorrelated errors in neural networks for time series. NeurlPS, 34:29806-2988
2021.
[9] Angus Dempster, François Petitjean, and Geoffrry I Webb. Rocket: exceptionally fast and accu- rate time series classification using random convolutional kemels. Data Mining and Knowledge Discovery. (4455):4444-11 1495, 2020.
[10] Wenyong Huang, Zhenhe Zhang, Yu Ting Yeung, Xin Jiang, and Qun Liu. Spiral: Self- supervised perturbation-invarian representation learning for speech pre-training. ICLR, 2022.
[11] Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre- Alain Muller. Deep learning for time series classification: review. Data mining and knowledge discovery, 33(4)-917-963, 2019.
[12] Pengxiang Shi, Wenwen Ye, and Zheng Qin. Self-supervised pre-training for time series classification. In IJCNN, pages 1-8, 2021.
[13] Weixia Dang, Biyu Zhou, Lingwei Wei, Weigang Zhang, Ziang Yang. and Songlin Hu. Ts- bert: Time series anomaly detection via pre-training model bert. In International Conference on Computational Science, pages 209-223, Springer, 2021.
[14] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, pages 3558- 3568, 2021.
[15] Kailai Sun, Zuchao Li, and Hai Zhao. Multilingual pre-training with universal dependency learning. NeurlPS, 34:8444 2021.
[16] Rui Ye and Qun Dai. Implementing transfer learning across different datasets for time series forecasting. Pattern Recognition, 109:10 2021.
[17] Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre- Alain Muller. Transfer learning for time series classification. In 2018 IEEE international conference big data (Big Data), pages 1367-1376. IEEE, 2018.
[18] Kristoffer Wickstrøm, Michael Kampffmeyer, Karl Øyvind Mikalsen, and Robert Jenssen. Mixing up contrastive learning: Self-supervised representation learning for time series. PRL. 155:54-61, 2022.
[19] Priyanka Gupta, Pankaj Malhotra, Jyoti Narwariya, Lovekesh Vig, and Gautam Shroff. Transfer learning for clinical time series analysis using deep neural networks. Journal of Healthcare Informatics Research, 4(2):112-137, 2020.
[20] Amiel Meiseles and Lior Rokach. Source model selection for deep learning the time series domain. JEEE Access, 266190-6200, 2020.

---

[21] Ankit Singh. Clda: Contrastive learning for semi- domain adaptation. NeurIPS. 34:5089-5101, 2021.
[22] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-train trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR. 2019.
[23] Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. OpenAI, 2018.
[24] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. In NeurlPS, volume 33, pages 22243-22255, 2020.
[25] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In NeurIPS, volume 33, pages 12449-12460, 2020.
[26] Gari Clifford, Chengyu Liu, Benjamin Moody, H Lehman Li-wei, Ikaro Silva, Qiao Li, AE Johnson, and Roger G Mark. Af classification from a short single lead ecg recording: The physionet/computing
cardiology challenge 2017. In 2017 Computing in Cardiology (CinC), pages 1-4, IEEE, 2017.
[27] Mitchell L Gordon, Kaitlyn Zhou, Kayur Patel, Tatsumori Hashimoto, and Michael S Bernstein. The disagreement deconvolution: Bringing machine learning performance metries in line with reality. In CHI, pages 1-14, 2021.
[28] Simon Rogers, Derek Sleeman, and John Kinsella. Investigating the disagreement between clinicians' ratings of patients in icus. IEEE Journal of Biomedical and Health Informatics, 17(4):843-852, 2013.
[29] Leonard M Horowitz, Rita de Sales French, Kirk D Wallis, David L Post, and Ellen Y Siegelman.
The prototype as a construct in abnormal psychology: li. clarifying disagreement in psychiatric judgments. Journal of Abnormal Psychology, 90(6):575, 1981.
[30] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. In 12XXv:1807.03744 2019.
[31] Pritam Sarkar and Ali Etemad. Self-supervised learning for ecg-based emotion recognition. In ICASSP, pages 3217-3221, 2020.
[32] Joseph Y Cheng, Hanlin Goh, Kaan Dogrusoz, Oncel Tuzel, and Erdrin Azemi. Subject-aware contrastive learning for biosignals. arXiv preprint arXiv:2007.0487 2020.
[33] Sriram Ravula, Georgios Smymis, Matt Jordan, and Alexandros Dimakis. Inverse problems
leveraging pre-trained contrastive representations. NeurlPS, 04:8753-8765 2021.
[34] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607, 2020.
[35] Zhigang Dai. Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers. In CVPR, pages 1601-1610, 2021.
[36] Hsin-Yine Lee, Jia-Bin Huang, Maneesh Singh, and Ming- Yang. Unsupervised repre- sentation learning by sorting sequences. Proceedings of the IEEE international conference on computer vision, pages 667-676, 2017.
[37] Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-training of image features non-curated data. In ICCV, pages 2959-2968. 2019.
[38] Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 181.000000 2018.
[39] Neo Wu, Bradley Green, Xue Ben, and Shawn O' O'Banion. Deep transformer models for time series forecasting: The influenza prevalence case. In 11X1v:2001.08317, 2020.
[40] Chi Ian Tang, Ignacio Perez-Pozuelo, Dimitris Spathis, and Cecilia Mascolo. Exploring con- trastive learning in human activity recognition for healthcare. arXiv preprint 21.. 2020.
[41] Dani Kiyasseh, Tingting Zhu, and David A Clifton. Cloes: Contrastive learning of cardiac signals across space, time, and patients. In ICML, pages 5606-5615 2021.
12

---

[42] David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin. Adamatch: A unified approach to cumi-supereiee learning and domain adaptation. ICLR. 2022.
[43] Guoqiang Wei, Cuiling Lan, Wenjun Zeng. Zhizheng Zhang, and Zhibo Chen. Toalign: Task- oriented alignment for unsupervised domain adaptation. NeurlPS, 34:13834-13844 2021.
[44] Tongkun Xu, Weihua Chen, Pichao Wang, Fan Wang, Hao Li, and Rong Jin. Cdtrans: Cross-
domain transformer for unsupervised domain adaptation. ICLR, 2022.
[45] Bernd Illing, Jean Ventura, Guillaume Bellec, and Wulfram Gerstner. Local plasticity rules can learn deep representations using supf-vuee contrastive predictions. NeurIPS, 34:30365-
30379, 2021.
[46] Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. In ICLR, 2021.
[47] Zhihan Yue, Yujing Wang. Juanyong Duan, Tianmeng Yang. Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In AAAL, volume 36, pages 1180-8987, 2022.
[48] Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In IJCAL, pages 2352-2359, 2021.
[49] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. In ICLR, 2022.
[50] Ling Yang and Shenda Hong. Unsupervised time-series representation learning with iterative bilinear temporal-spectral fusion. In ICML, pages 35038-250555 PMLR, 2022.
[51] Rob Hyndman and George Athanasopoulos. 2018.
Forecasting: principles and practice. OTexts,
[52] Ronald Newbold Bracewell and Ronald N Bracewell. The Fourier transform and its applications. volume 31999. McGraw-hill New York, 1986.
[53]
Leon Cohen. frequenyy analysis, volume 778. Prentice hall New Jersey, 1995.
[54] Henri J Nussbaumer. The fast fourier transform. In Fast Fourier Transform and Convolution Algorithms, pages 80-111. Springer, 1981.
[55] Patrick Flandrin. Fime-frequencyyline-scalll analysis. Academic press, 1998.
[56] Antonia
Applications in time-frequency signal processing. CRC
press, 2018.
[57] Ryan Soklaski, Michael Yee, and Theodoros Tsiligkaridis. Fouricr-based augmentations for improved robustness and uncertainty calibration. NeurlPS W. 2021.
[58) Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia
Makedon. A survey on contrastive self-supervised learning. Technologies, 9(1):2, 2020.
[59] Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In International workshop on similarity- pattern recognition, pages 84-92. Springer, 2015.
[60] Vassileios Balntas, Edgar Riba, Daniel Ponsa, and Krystian Mikolajezyk. Learning local feature descriptors with triplets and shallow convolutional neural networks. In Bmve, volume 1, page 3, 2016.
[61] Bob Kemp, Aeilko H Zwinderman, Bert Tuk, Hilbert AC Kamphuisen, and Josefien JL Oberye. Analysis of deeep-deeendent neuronal feedback loop: the slow- wave microcontinuity of the eeg. IEEE Transactions on Biomedical Engineering, 47(9): 11111185111111 1194, 2000.
[62] Ralph G Andrzejak, Klaus Lehnertz, Florian Mormann, Christoph Rieke, Peter David, and Christian E Elger. Indications of nonlinear deterministic and inite-dimensional structures in
time series of brain electrical activity: Dependence on recording region and brain state. Physical Review E, 64(6))06190) 1977, 2001.
[63] Christian Lessmeier, James Kuria Kimotho, Detmar Zimmer, and Walter Sextro. Condition monitoring of bearing damage in electromechanica. drive systems by using motor current signals of electric motors: A benchmark data set for data-driven classification. In PHM Society European Conference, volume 3, 2016.
13

---

[64] Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra Perez. and Jorge Luis Reyes Ortiz. A public domain dataset for human activity recognition using smartphones. In ESANN, pages 437-442, 2013.
[65] Jiayang Liu, Lin Zhong, Jehan Wickramasuriya, and Venu Vasudevan. uwave: Accelerometer- based personalized gesture recognition and its applications. Pervasive and Mobile Compuring. 5(6):657-675, 2009.
[66] Ary Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung Kang Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of a new research resource for complex
physiologic signals. circulation, 101(23):e215-e222 2000.
[67] Amrutha Ramanathan and James McDermott. Fall detection with accelerometer data using residual networks adapted to multi-variate time series classification. IJCNN, pages 1-8, 2021.
[68] George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. In KDD, pages 2114-2124, 2021.
[69] Xiang Zhang and Lina Yao. Deep Learning for REG-Based Brain CompuCC Interfaces: Representations, Algorithms and Applications. World Scientific, 2021.
[70] Julien Audibert, Pictro Michiardi, Frédéric Guyard, Sébastien Marti, and Maria A Zuluaga. Usad: Unsupervised anomaly detection on multivariate time series. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3395-3404, 2020.
[71] Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guided network for irregularly sampled multivariate time series. In ICLR, 2022.
[72] Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. Set functions for time series. In ICML, pages 4353-4363, 2020.
14

---

Broader Impacts
Our approach for self-super pre-training improves classification performance on target datasets in different application scenarios. The recognition of time- meneffruyenccy consistency universal property specific to time series data a weak assumption that enables effective, task- and domain- agnostic transfer learning. We believe our work will inspire the research community to uncover other universal properties for transfer learning. We also hope our work will also attract more researchers to the more general problem of time series representation learning which is still underappreciated relative to problems from CV and NLP fields.
On the society level, our work, along the line of transfer learning, can facilitate more efficient use of time series data in various settings. For example, medical settings, some diseases of clinical interest may have very small labelled dataset. In this case, unlabelled data from patients of different diseases but with similar underlying physiological conditions can be used to pre-train the model. However, practitioners need to be aware of the limitations of the model, including that it may make biased predictions. Specifically, bias may exist in the source dataset used for pre-training due to an imbalance of samples from subjects of different demographic attributes. Also, the standardized medical protocols for collecting these datasets might be unsuitable for subjects with certain physiological attributes, creating unforeseen bias that may be transferred to fine-tuning:
All datasets in this paper are publicly available and are not associated with any privacy or security concern. Furthermore, we have followed guidelines on responsible use specified by primary authors of the datasets used in the current work.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] In abstract and introduction, we claim that TF-C is generalizable property of time series that support pre-training, which is well- justified in Sec. 3 and experimentally demonstrated in Sec. 5 (our model consistently performs comparatively to or above baseline methods).
(b) Did you describe the limitations of your work? [Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Broader Impact on Page 10.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] Yes, we include an anonymous link (see Abstract) that provides the source codes with all implementa- tion details, implementation of baselines, and eight datasets. The link will be updated to non-anonymous link after acceptance.
(b) Did you specify all the training details (e.g.. data splits, hyper-paramette were chosen)? [Yes] See implementation details in Sec. 5. See Appendix E for baseline architectures and hyper-parameter settings. More details can be found the included
how they
URL.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] We run experiments for 5 times and report the average value with standard deviation. See Table Tables 4-6, and Table 2.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix E.
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
15

---

(a) If your work uses existing assets, did you cite the creators? [Yes] We used eight
existing datasets and 6 state-of-the-ar baselines contrastive learning and pre-training for time series. We cited the creators for every exist asset we used. See Sec. 5.
(b) Did you mention the license of the assets? [Yes] All dataset licenses are mentioned in the Appendix D.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] See the anonymous link in Abstract.
(d) Did you discuss whether
how consent was obtained from people whose data you're using/curating? [No] All data we use is freely available for download, without any requirement to re-contact the data curator.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] Our datasets are public, well-established, and do not contain PII or offensive content
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "llwLnbA5cHE", "reviewer": "Reviewer_hNJe", "summary": "This paper proposes a self-supervised contrastive pre-training target for time series via time-frequency consistency. Based on the assumption that time-based embedding and frequency-based embedding for a particular time series example should be close together in time-frequency space, consistency loss is introduced. Evaluation of eight real-world time series datasets under one-to-one and one-to-many transfer settings, the proposed pre-training target is proved to have better performance over existing methods.\n\n=====================\n\nUpdates in the rebuttal period: Thanks for the authors' responses about updated clarifications and extended experiments. I have read them carefully and raised my scores.", "questions": "1. What are the detailed augmentation and frequency-domain transformation settings in the experiments? How are different augmentations compared in the frequency domain? \n", "limitations": "1. The novelty should be discussed a step further. \n2. Theoretical analysis, further experiments, or case studies about frequency domain augmentations should be added. \n3. More baselines should be compared. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The paper is clearly written and well presented. \n2. Experiments on eight real-world datasets are comprehensive.\n\nWeakness:\n1. The novelty should be discussed a step further. This paper claims that \"this is the first work that develops augmentations in frequency domain\", however, this sentence is ambiguous. Does this paper first \"augment time series self-supervised learning in frequency domain\" or \"develops frequency domain augmentations\"? This should be awared because the reviewer has noticed that there is a relevant method BTSF to appear in ICML 2022 (first appeared in arxiv in 8 Feb 2022), using a close temporal-spectral fusion idea for time-series representation learning, which involves both time domain and frequency domain for contrastive learning as well (amazingly experimental SLEEPEEG, EPILEPSY, HAR datasets in this paper are also used in BTSF). Please consider rephrasing this claim based on the above fact. Besides, the differences and specific novel ideas of this paper to that of BTSF should be discussed.\n2. More baselines should be compared. Many baselines listed in the related work are supposed to be compared in the experiments as well. Especially some of them show better performance than the currently selected baselines. Besides, please consider update the references, for example, [45]: arxiv -> ICLR 2022, [46]: arxiv -> AAAI 2022, [47]: arxiv -> IJCAI 2021. \n3. The experiments only involves classification tasks without regression task, forecasting task, anomally detection task. Hence, it is suggested to point out that this paper aims at this specific task in the title and introduction. \n4. Further understanding of frequency augmentation is needed. As a fundamental concept and assumption of the paper, the time-frequency consistency's underlying mechanism is not discussed in this paper. This paper verified its effectiveness, but doesn't provide further analysis about how it works. Theoretical analyses are also missing. Besides, as mentioned in section 4.2, augentation in frequency domain may change the time series significantly in time domain. Does the augmented time series still make sence? Since no case study is given, the effectiveness and logic of this operation is questionable.\n5. Minors: - \"For\" -> \"for\" in the title. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "auzIGOTy6L", "reviewer": "Reviewer_UjqE", "summary": "This paper proposes a new method for self-supervised pre-training for time series datasets. These pre-trained models can then be fine tuned for classification on different downstream time series datasets.\n\nIn addition to using augmentations on time series data in a contrastive learning setting to obtain time series specific  representations , the authors also input frequency domain (FFT)  for input time series. These frequency domain inputs  are  also augmented through frequency domain perturbations. These frequency domain inputs and their corresponding perturbations are then used by a contrastive loss to learn frequency domain representations.\n\nThese time series and frequency domain representations are then projected to a  Time-freqeuncy space where the the time series representations and their corresponding frequency domain representations are encouraged to be closer than the time series representations and its corresponding augmented frequency domain representations (or frequency domain and its augmented time series representation) through a margin loss.\n\nThis methodology results in representations ,which when fine tuned on different types of time series datasets, leads to improved classification performance over other self-supervised representation learning baselines.", "questions": "Could the authors provide more information about the permutation budget needed for frequency domain perturbations ?\n The authors mention that the perturbation budget is to ensure that the perturbations do not change the input seignifcantly but more details on this could be helpful (as  some low frequency pertbuations might change the signal more than high frequency perturbations , though this might depend on the input sequence ). \nSome details on how these perturbations affect the learned representations could be helpful for the readers.\n\n", "limitations": "The authors have talked about some limitations in their work (e.g how the method can't directly work for irregulary sampled data but encoder based methods could help solve this)\n\n\nAlso I think mentioning how different time series provide different types of representations which are more suitable for certain downstream time series classification datasets (like human activity pre train and fine tune gesture) could be helpful when discussing the limitations.\n(The authors do mention in the results section, but mentioning in the limitations could be helpful ) \n\nAlso  would this method work for really high dimensional datasets ? (Like 100 dimensional?) \n", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n- The paper proposes a novel direction for self-superivison for time series datasets through both, perturbations in the frequency domain , as well as a join time-frequency representation space\n- There has been relatively little work on learning self-supervised representations for time series datasets (at least as compared to image/vision/nlp domains). This work addresses this and presents a method which could be useful for the community.\n- The method was extensively evaluated on different types of downstream time series classification tasks (that range from ECG, to human activity recognition and motor fault detection) and showed promising results\n- The method was mostly  clearly described and the paper was was to follow.\n\nCons\n(This isn't really  a con, \n- I had some issues with section 3, particularly the paragraph titled \"Rationale for Time-Frequency Consistency\" where some of the sentences were a bit confusing. E.g line 162. \"the relationship between them provides an invariance\" that can be exploited. I understand how the relationship between  time domain and frequency domain could be invariant , but I am having difficulty of seeing how this could be useful for self-supervised representations. I am thinking of this interns  invariant representations learned for images, where an image and its rotated form should have invariant representations, and is more intuitive to understand.\n  Maybe more details on how the time-frequency relationship invariant can be useful for self-supervision could be helpful (perhaps though a simple example) ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Xr0Mt5MmQ-K", "reviewer": "Reviewer_q9DY", "summary": "In this work, the authors present a self-supervised pre-training model for time series (TS) data by leveraging Time-Frequency Consistency (TF-C). The main idea is that time-based and frequency-based representations of the same time series sample are closer in the joint time-frequency space, and farther apart from the representations of other samples. The model has 4 components: a time encoder - to embed TS from time domain; a frequency encoder - to embed TS from frequency domain; and two cross-space projectors that map time-based and frequency-based representations to a unified time-frequency space. The model is trained using 3 types of losses: Contrastive time loss, Contrastive frequency loss, and Time-frequency consistency loss. To enable contrastive learning, the model uses augmentation banks for both time and frequency embeddings. The authors evaluate their method in two experimental scenarios, (a) pre-train on one dataset and test on one dataset (b) pre-train on one dataset and test on multiple datasets with 8 diverse datasets and compare it against 8 baselines methods to show the superiority of TF-C approach.", "questions": "1. The current embedding scheme and objective functions focus on global time series features which are suitable for the classification task. Can the proposed method be applied easily to time series forecasting task where modeling local features is relevant as well? \n\n2. Did you try pre-training the model on multiple datasets as well? \n\n", "limitations": "See weaknesses.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well written and easy to follow. I enjoyed reading it.\n2. While frequency domain has been modeled in recent methods like CoST, the idea of using Time-Frequency Consistency, in addition to individual domain losses, for self-supervision is novel.\n3. The method was compared to 8 baselines methods on 8 diverse datasets and showed strong empirical results. Through an ablation study of loss function components, the paper confirms that each loss term contribute towards the model performance.\n4. The paper performed multiple experimental settings (one-to-one and one-to-multiple) and dataset scenarios (transfer learning among datasets with substantial discrepancy) to show the method outperforms the other baselines.\n\nWeaknesses:\n1. The authors present the proposed method as universal pre-training method for time-series, however, only a single time series task (time-series classification) is explored in the paper. Therefore, it is not clear if the proposed method performs competitively on other time series tasks.\n\nTypos:\n\n(a) Line 173 - shouldn’t it be D^tune instead if D^pret?\n(b) Line 175 -  “D^pret and D^pret” ->  “D^pret and D^tune”\n(c) Line 221 - “sale” ->   scale\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DTdimuxM-v", "reviewer": "Reviewer_miad", "summary": "The authors propose a new way of training self-supervised models for time series, such that time-frequency consistency is enforced. To this end, the authors add two additional loss terms to an earlier-introduced contrastive loss. The two additional terms enforce time, respectively frequency consistency, inspired by the triplet loss. Performance of the proposed model is compared to several strong baselines, and the proposed model is shown to outperform most baselines on most metrics.", "questions": "Could the authors explain how it is possible that the random initialization method for one-to-many in sleepEEG --> Fd-B performs almost as good as the proposed model, and much better than most of the baselines on the AUPRC metric?\n\nHow sensitive is the performance to the setting of lambda?\n\nTwo additional minor things:\n- Fig 1: the symbols for the embeddings of the time-encoder have an F superscript instead of a T.\n- L277: \"have\" should be \"has\"\n", "limitations": "Yes, limitations were discussed.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": " Originality: The proposed approach is simple, but elegant.\n\nQuality: The paper is of good quality.\n\nClarity:  The paper is well-written and easy to follow, and the figures are also clear and informative, therewith helping the reader to gain a better understanding. \n\nSignificance: The field of self-supervised learning has so far mostly been focused on imaging tasks, while many real life data are time series. Development in self-supervised learning methods, dedicated to time series is, therefore, highly significant. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept