Multi-view Self-Supervised Contrastive Learning for Multivariate Time Series

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 08:37:21

Abstract: Learning semantic-rich representations from unlabeled time series data with intricate dynamics is a notable challenge. Traditional contrastive learning techniques predominantly focus on segment-level augmentations through time slicing, a practice that, while valuable, often results in sampling bias and suboptimal performance due to the loss of global context. Furthermore, they typically disregard the vital frequency information to enrich data representations. To this end, we propose a novel self-supervised general-purpose framework called Temporal-Frequency and Contextual Consistency (TFCC). Specifically, This framework first performs two instance-level augmentation families over the entire series to capture nuanced representations alongside critical long-term dependencies. Then, TFCC advances by initiating dual cross-view forecasting tasks between the original series and its augmented counterpart in both time and frequency dimensions to learn robust representations. Finally, three specially designed consistency modules —temporal, frequency, and temporal-frequency— aid in further developing discriminative representations on top of the learned robust representations. Extensive experiments on multiple benchmark datasets demonstrate TFCC's superiority over the state-of-the-art classification and forecasting methods and exhibit exceptional efficiency in semi-supervised and transfer learning scenarios. Code, data, and model checkpoints will be released after the review period.

Main paper: 
 related work: 
 Self-supervised Learning for Time Series. Self-supervised representation learning for time series has gained significant growth, but there remains considerable scope for improvement in this area.Inspired by the well-established self-supervised learning methods in computer vision and natural language processing, recent studies primarily utilize contrastive learning frameworks for time series representation learning. For instance, TST [24] adopted masked signal prediction as an auxiliary task for transformer model pretraining. TS-TCC [5] constructed a cross-view prediction on weak and strong augmentations in the temporal module to learn robust representations and proposes a context consistency module to learn discriminative representations. TNC [18] maintains temporal consistency by distinguishing between neighboring and nonneighboring signals. TS2Vec [23] optimizes hierarchical contrast learning at multiple scales. However, these time-based methods are often inadequate for long time series, failing to capture long-term dependencies effectively [22] . Besides, the effectiveness will significantly diminish when applied to downstream tasks involving periodic time series [25] .Temporal and Frequency Contrasting. Exploring the frequency domain can yield extra insights, and few studies have considered transformation invariance and investigated this area, but they may still have some aspects that could be improved. CoST [20] proposed contrastive learning in time and frequency domains to learn disentangled seasonal-trend representations for time series forecasting. BTSF [22] and CRT [25] also included the frequency domain, but they only perform augmentations in the time domain. Similarly, TF-C [26] and TS-TFC [11] incorporate frequency features. Nevertheless, TF-C failed to consider phase perturbation and contextual invariance, while TS-TFC overlooks the use of augmentations, which may affect their model's generalization. In this work, we first present two instance-level augmentation families to enrich the pre-trained models, and two cross-view forecasting tasks to learn robust representations, especially by introducing spectral perturbations and prediction strategy in the frequency domain. Besides, we propose contextual consistency and temporal-frequency consistency modules to capture discriminative information by crossdomain correlations. 
method: 
 Our proposed TFCC framework is shown in Figure 1 .Data augmentation is vital in contrastive learning for enhancing data diversity [3, 5, 12] . Contrastive learning aims to maximize the similarity between different views of the same sample while minimizing similarity with other different samples, making similar instances grouped and dissimilar instances separated. We create positive pairs by matching the original series with its augmented counterpart, treating other samples as negative pairs. Wen [19] thoroughly reviewed time series augmentation techniques. Data augmentation usually introduces synthetic disturbances that reflect the complexity of time series, which mitigates overfitting risks and improves model robustness and generalization. However, current methods often lose global semantic information and fail to capture long-term dependencies due to segment-level sampling bias. [22] .This work presents a novel data augmentation strategy designed to maintain data's raw properties and capture global context. We leverage various invariant transformations to build a time-based augmentation family (TAF) and a frequency-based augmentation family (FAF), enriching the representation learning during pretraining. For TAF, we utilize instance-level temporal augmentations, including scaling, jittering, and dropout mask to generate diverse data while keeping the entire series integrity, all well-recognized in [17, 22, 26] . Given a sample 𝑥, we input the sequence 𝑥 𝑇 into TAF to produce positive samples 𝑥 𝑇 𝑝𝑜𝑠 . Negative samples 𝑥 𝑇 𝑛𝑒𝑔 are generated by randomly selecting other variables. For the FAF, we introduce perturbations to the spectrum by adding or removing frequency components and phase and amplitude perturbations. 𝑥 𝐹 is derived through the Fast Fourier Transformation (FFT) [14] and then create 𝑥 𝐹 𝑝𝑜𝑠 by applying FAF. Negative samples 𝑥 𝐹 𝑛𝑒𝑔 are obtained from other variables' frequency. In experiments, the model will randomly Temporal domain Inspired by [5] , we developed our temporal contrasting module to capture the latent temporal features. For a given input sample 𝑥 𝑖 , 𝑖 is the timestamp of a time series sample, we generate an augmentation set 𝑋 𝑇 𝑖 using TAF:𝑥 𝑇 𝑖 → 𝑋 𝑇 𝑖 . Each mem- ber x𝑇 𝑖 ∈ 𝑋 𝑇 𝑖 is augmented from 𝑥 𝑇 𝑖 .We randomly select a member x𝑇 𝑖 to generate a positive pair with the original sample, and expose the model to complex, missing, and unstable temporal dynamics to derive more robust embeddings. The original and the augmented views are all passed through an encoder consisting of three convo- 𝑡 , f (h t+m , z t ) = exp (G m (z t )) T h t+m ,where G 𝑚 is a linear function that maps 𝑧 𝑡 back into the same dimension as ℎ 𝑡 . Two contrastive losses 𝐿 𝑇 and L𝑇 are designed to maximize the dot product between the original and the predicted of the same sample while minimizing the dot product with other samples N 𝑡,𝑚 in the batch, computed as: Figure 3 showcases the architecture of the Transformer used in our work, which is the same as that in [5] . It mainly consists of consecutive Multi-Head Attention (MHA) blocks and an MLP block. The MLP consists of two fully connected layers with a nonlinear ReLU function and filter, as well as two Dropout in the middle. Our Transformer employs pre-normalized residual connectivity, which produces more stable gradients. We add a token 𝑧 into the module and then output its state as a representation context. The Transformer operation begins by taking the features ℎ ≤𝑡 +𝑚 and passing them through a linear projection layer, which transforms these features into a hidden dimension, represented as 𝑃𝑟𝑜 𝑗𝑒𝑐𝑡𝑖𝑜𝑛 : R 𝑑→𝑘 . The result of this linear projection is then fed into the Transformer, resulting in 𝑃𝑟𝑜 𝑗𝑒𝑐𝑡𝑖𝑜𝑛, ℎ ′ ∈ R 𝑘 . Subsequently, we combine the context vector with the feature vector ℎ ′ , resulting in the input features becoming 𝜙 0 = [𝑧; ℎ ′ ], where the subscript 0 signifies that it serves as the input to the first layer. Following this, we propagate 𝜙 0 through the Transformer layers using the equations below:𝐿 𝑇 = -1 𝑀 𝑀 𝑚=1 log exp ( G 𝑚 (𝑧 𝑇 𝑡 )) 𝑇 ℎ 𝑇 𝑡 +𝑚 𝑛∈N 𝑡,𝑚 exp ( G 𝑚 (𝑧 𝑇 𝑡 )) 𝑇 ℎ 𝑇 𝑛 (1)EQUATIONEQUATIONFrequency domain Time and frequency domains are two different views of the same sequence, and the latter can offer another perspective for uncovering spectral characteristics in time series [25] . Recent studies like CoST, BTSF, TFDNet, and TS-TFC [11] have examined frequency but missed out on frequency augmentations. Our work is one of the few to explore frequency-based augmentations and the first to perturb phase and amplitude. When perturbing the phase of 𝑥, we first extract its original phase, then generate a random perturbation controlled by perturb_ratio, and sum them up to get the final perturbed spectrum. For sample 𝑥 𝑖 , the frequency spectrum 𝑥 𝐹 𝑖 is generated from 𝑥 𝑇 𝑖 through FFT. For 𝑥 𝐹 𝑖 , an augmentation set 𝑋 𝐹 𝑖 is built using FAF: 𝑥 𝐹 𝑖 → 𝑋 𝐹 𝑖 . We utilize the same encoder and Transformer to map 𝑥 𝐹 𝑖 to a frequency-based embeddings 𝑧 𝐹 𝑖 = 𝑇𝑟𝑎𝑛𝑠 𝑓 𝑜𝑟𝑚𝑒𝑟 (𝐸𝑛𝑐𝑜𝑑𝑒𝑟 (𝑥 𝐹 𝑖 )), z𝐹 𝑖 = 𝑇 𝑟𝑎𝑛𝑠 𝑓 𝑜𝑟𝑚𝑒𝑟 (𝐸𝑛𝑐𝑜𝑑𝑒𝑟 ( x𝐹 𝑖 )). We are the first to design crossview prediction to capture frequency patterns in the frequency domain, the process of which is similar to that in the time domain, 𝐿 𝐹 and L𝐹 are computed by:𝐿 𝐹 = -1 𝑀 𝑀 𝑚=1 log exp ( G 𝑚 (𝑧 𝐹 𝑡 )) 𝐹 ℎ 𝐹 𝑡 +𝑚 𝑛∈N 𝑡,𝑚 exp ( G 𝑚 (𝑧 𝐹 𝑡 )) 𝐹 ℎ 𝐹 𝑛 (5) L𝐹 = -1 𝑀 𝑀 𝑚=1 log exp ( G 𝑚 ( z𝐹 𝑡 )) 𝐹 ℎ 𝐹 𝑡 +𝑚 𝑛∈N 𝑡,𝑚 exp ( G 𝑚 ( z𝐹 𝑡 )) 𝐹 ℎ 𝐹 𝑛 (6)Prior studies have delved into contextual consistency [4, 5] 2 (a)), while the remaining 2(N-1) contexts within the batch are categorized as negative pairs, collectively forming 𝑐 𝑛𝑒𝑔 : 𝑐 -∈ 𝑐 𝑛𝑒𝑔 . As such, in this context, we employ D as the distance function to maximize the similarity between positive pairs while minimizing the similarity between negative pairs, enabling the representations to be discriminative. The temporal contextual consistency loss is defined as:EQUATIONwhere sim(𝒖, 𝒗) = 𝒖 𝑇 𝒗/∥𝒖 ∥∥𝒗 ∥ is the cosine similarity, 𝜏 is the degree of pull and push as the temperature parameter and shares the same value throughout all experiments.Frequency Consistency. We implement frequency consistency between the original and its augmented views to learn underlying patterns that the time domain might miss. We assume the model can learn similar features from the original spectrum 𝑥 𝐹 𝑖 and a perturbed spectrum x𝐹 𝑖 , thus the embeddings 𝑐 𝐹 𝑖 = 𝑃𝑟𝑜 𝑗𝑒𝑐𝑡𝑖𝑜𝑛(𝑧 𝐹 𝑖 ) and c𝐹 𝑖 = 𝑃𝑟𝑜 𝑗𝑒𝑐𝑡𝑖𝑜𝑛( z𝐹 𝑖 ), i.e., (𝑐 𝐹 𝑖 , c𝐹 𝑖 ) can be denoted as a positive pair (Figure 2 (b)), similar to temporal consistency, 𝑐 𝑛𝑒𝑔 : 𝑐 -∈ 𝑐 𝑛𝑒𝑔 , total 2(N-1) within the same batch. The frequency contextual consistency loss can be calculated as:EQUATIONIn the latent space, time and frequency embeddings from the same sample intuitively should be closer than other different samples.Inspired by this, we propose the temporal and frequency consistency, shown in Figure 2 (c). For each sample 𝑥 𝑖 , we generate four embeddings: time-based original 𝑐 𝑇 𝑖 , frequency-based original 𝑐 𝐹 𝑖 , and their augmented version c𝑇 𝑖 and c𝐹 𝑖 . These embeddings, derived from the data's temporal and frequency characteristics, emphasize both aspects' importance in time series analysis. Hence, we input both time and frequency domains into the model to strengthen the cross-domain interactions and enhance the learning of temporal and spectral details.To maintain consistency in time-frequency space, we introduce a consistency loss, 𝐿 𝑇 𝐹 𝐶 , to measure the distance between time and frequency embeddings. We utilize D 𝑇 𝐹 𝑖 = D (𝑐 [8, 16] is designed as the temporal-frequency consistency loss, enabling the model to learn smaller D 𝑇 𝐹 𝑖 . We calculate the consistency loss 𝐿 𝑇 𝐹 𝐶 by:𝐿 𝑇 𝐹 𝐶 = 𝑁 ∑︁ 𝑖=1 ( ∑︁ D 𝑝𝑎𝑖𝑟 (𝜀 + D 𝑇 𝐹 𝑖 -D 𝑝𝑎𝑖𝑟 𝑖 )), D 𝑝𝑎𝑖𝑟 ∈ (D T 𝐹 𝑖 , D 𝑇 F 𝑖 , D T F 𝑖 ) (9)where 𝜀 is a constant and set to 1. D 𝑝𝑎𝑖𝑟 denotes the distance between the time embeddings 𝑐 𝑇 𝑖 , 𝑐 T 𝑖 and the frequency embeddings 𝑐 𝐹 𝑖 , 𝑐 F 𝑖 . Each pair has at least one embedding from the augmented views. The 𝐿 𝑇 𝐹 𝐶 loss motivates the pre-training model to learn the consistency between the time and frequency embeddings during model optimization.Our self-supervised pre-training process has a total of seven losses. First, 𝐿 𝑇 , L𝑇 , 𝐿 𝐹 , L𝐹 are introduced for the cross-view prediction to acquire robust representations. Second, 𝐿 𝑇 𝐶 and 𝐿 𝐹 𝐶 are used to capture embedded information that is invariant to augmentations and extract discriminative representations. Last, the time-frequency consistency loss, 𝐿 𝑇 𝐹 𝐶 , ensures consistency between time and frequency latent embeddings. In summary, the overall loss is:EQUATION)where 𝜆1, 𝜆2, and 𝜆3 are fixed hyperparameters indicating the relative weight of each loss. 
experiments 4.1 experimental setup: 
 We conducted extensive experiments to assess TFCC's performance and performed ablation studies to highlight each component's contribution to the outcome. More detailed descriptions of augmentation methods and experiments are in Supplementary Materials.Datasets. Classification (1) Epilepsy was conducted preprocessing following previous works [5, 22] . The original dataset featured 5 classes, but 4 did not contain epileptic seizures. Therefore, we merged them into a single class and treated it as a binary classification problem. (2) HAR contains six human activities: walking, walking upstairs, walking downstairs, standing, sitting, and lying down collected by smartphones [1] . They use a waist-mounted Samsung Galaxy S2 device to record the sustained activity of each subject, with a 50Hz sampling rate. (3) Sleep-EDF records multisleep modes, utilized a single EEG channel, sampled at a rate of 100 Hz [7] , which is designed to categorize the input EEG signals into one of the following categories: wake (W), non-rapid eye movement (N1, N2, N3), and rapid eye movement.Forecasting (1) ETT1 comprises two distinct granularities: two hourly-level datasets (ETTh) and one 15-minute-level dataset (ETTm). The data was collected from separated countries in China for longsequence time series forecasting. Each dataset contains target values of 6 load features and "oil temperature" from July 2016 to July 2018.(2) Weather2 contains 11 local climatological features focusing on 1600 locations in the U.S. over 4 years from 2010 to 2013.Implementation Details. We run all experiments on one GeForce RTX 3090 Ti GPU, running 64-bit Linux 5.15.0-56-generic, and perform a 60/20/20 train/validation/test split. We set the epoch size to 50, the batch size to 128, Adam optimizer, a learning rate of 3e-4, weight decay of 1e-4, and dropout to 0.2, 𝛽 1 = 0.5, 𝛽 2 = 0.55. For temporal augmentations, the drop_mask ratio is 0.2, while the scale ratio and jitter ratio are different according to different datasets(𝑠𝑐𝑎𝑙𝑒 𝐸𝑝 =0.001, 𝑗𝑖𝑡𝑡𝑒𝑟 𝐸𝑝 =0.001, 𝑠𝑐𝑎𝑙𝑒 𝐸𝐷𝐹 =1.5, 𝑗𝑖𝑡𝑡𝑒𝑟 𝐸𝐷𝐹 =2, 𝑠𝑐𝑎𝑙𝑒 𝐻𝐴𝑅 =1.1, 𝑗𝑖𝑡𝑡𝑒𝑟 𝐻𝐴𝑅 =0.8). For frequency augmentations, the pertub_ratio is 0.1. The number of heads in the Transformer is 8.Baselines. Classification: Random Init.: A linear classifier on a randomly initialized encoder. Supervised: Supervised training of both encoder and classifier. CPC [15] . SimCLR [4] . MHCCL [13] . TS-TCC [5] . TSTCC-Floss [21] . TF-C [26] . BTSF [22] . CA-TCC [6] .Forecasting: 1) Representation learning: TNC [18] . TS-TCC. TS2Vec [23] . CoST [20] . BTSF [22] . 2) End-to-end models: TCN [2] . Informer [27] .Results. Classification: We evaluate the performance of these methods on three widely used datasets. For a fair comparison, we adhere to previous linear benchmark evaluation models [4, 6] and measure how well the learned representations are used to classify hidden states with two metrics: accuracy=ACC = TP+TN TP+TN+FP+FN , and the macro-averaged F1-score MF1= 2×PR P+R . Before training the linear classifier, we froze the self-supervised pre-trained encoder. Table 1 summarized the results, with the best results in bold and the second-best underlined. Our TFCC outperforms all baselines in all datasets, including the supervised method. This superior performance is ascribed to the instance-level augmentations and multiview contrasting strategy, helping to capture the long-term dependencies and the learned transformation-invariant representations. Additionally, our TFCC surpasses three TS-TCC-based methods (TS-TCC, TSTCC-Floss, CA-TCC), especially in the HAR dataset, with improvements of 2.36%, 1.87%, and 1.05%, and outperforms the TF-C and BTSF by 4.58% and 7.29%, which illustrates TFCC's effectiveness and superiority. Forecasting: We conduct a comprehensive linear evaluation to access the performance on four benchmarks across a range of forecasting lengths 𝑇 ∈ {24, 48, 168, 336, 720} using two metrics: the mean square error (MSE) and the mean absolute error (MAE). Specifically, we designed a linear regression model with an L2 paradigm penalty, trained on top of a fine-tuned self-supervised pre-trained encoder. The training phase includes two stages: (1) learning representations through the TFCC framework, and ( 2) training a linear regressor for each 𝑇 on top of the learned representations. Table 2 shows that TFCC excels in multivariate forecasting, outperforming all baselines across varying datasets and lengths. Notably, TFCC's performance improves with increasing sequence length compared to the baselines, indicating our model can better use global context information and is more capable of capturing long-term dependencies in time series than other baseline models. More forecasting results compared to other baselines are in Supplementary Materials.To assess the effectiveness of each component in TFCC, we conducted ablation studies by comparing TFCC and its different variants. Table 3 shows the ablation study on four datasets. We can see the temporal cross-view forecasting task boosts feature robustness, leading to ∼2% improvement in Epilepsy and ∼6% in HAR, ∼10% in ETTh1 and 3% in Weather (in Accuracy and MSE). Additionally, introducing contextual consistency (temporal, frequency) further enhances performance by strengthening feature discriminability with ∼1% boost in Epilepsy and 16.6% in ETTh1. Performance improvements are observed when adding frequency cross-view forecasting, with further 0.59% and 1.11% improvement in Epilepsy and HAR, 30.9% and 20.6% in ETTh1 and Weather, meaning frequency can help to learn periodic features of time series. Furthermore, incorporating temporal-frequency consistency in our TFCC framework demonstrates the best performance, further enhancing the model's ability to capture discriminative representations. Last, our TFCC with both T-Aug and F-Aug is superior to only using T-Aug or F-Aug. These findings suggest that each self-supervised contrasting view contributes significantly to learning more useful and valuable representations.We further validate the transferability of our proposed method by designing transfer learning experiments to verify whether it learns general representations that can be applied to different domains. We evaluate the transferability using the Fault Diagnosis (FD) datasets from four different domains, denoted as domains A, B, C, and D. we select one domain (source domain) for training and test it on the other three domains (target domains). Our experiment performs two steps on the source domain: (1) self-supervised training to get a pre-trained encoder, and (2) fine-tuning on the pre-trained encoder. Finally, we utilize the fine-tuned model to obtain accuracy on the test set in each target domain. Table 4 shows the performance of supervised, TS-TCC, and TFCC in 12 cross-domain scenarios. The representations of our fine-tuned method on the source domain can be well adapted to the target domains. TFCC achieves best in 9 out of 12 cross-domain scenarios. Notably, TFCC has 100% accuracy in B → D and D → B. Overall, TFCC improves the transferability of the learned representations over the supervised, TS-TCC, and CA-TCC training, by ∼8%, ∼4%, and ∼2% in terms of accuracy in 12 scenarios, respectively.To evaluate the effectiveness of TFCC in a semi-supervised setting, we fine-tuned the pre-trained encoder with 1%, 5%, 10%, 50%, 75%, and 100% of randomly selected instances of training data. Figure 4 presents the results of our fine-tuned TFCC (i.e., blue curves) compared to supervised training on the three classification datasets under the above settings. The results indicate our model performs significantly better than supervised training with limited labeled data. For example, with only 1% labeled data, TFCC fine-tuning still achieves 76.04% in HAR and 90.74% in Epilepsy. Especially for the unbalanced Sleep-EDF, our method can also reach 65.25%, while supervised training is only 32.24%. Moreover, TFCC's performance steadily improves as the percentage of labeled data increases, proving its applicability to label-scarce environments.TFCC incorporates several hyperparameters requiring careful tuning. To achieve optimal performance, we conducted a sensitivity analysis on two datasets, HAR and ETTh1, focusing on five key parameters: dropout mask ratio, temperature 𝜏, and the weights of contrastive losses 𝜆1, 𝜆2, and 𝜆3. As delineated in Table 5 , optimal performance is attained with a dropout mask ratio of 0.2. This is because too high value may lose the original properties of time series and too low may lead to representation collapse. Table 6 shows that TFCC gets the best performance when 𝜏= 0.15. A reasonable 𝜏 will facilitate the optimization of the training process and make the representation more discriminative as it is adjusted. FD-A FD-B FD-C FD-D AVG A → B A → C A → D B → A B → C B → D C → A C → B C → D D → A D → B DAugmentations selection is less explored, which is still an open and pending problem. Table 7 presents the augmentation effects. it can be seen that MSE and MAE reached 0.346 and 0.482 when slicing was used alone. Furthermore, the MSE increased from 0.297 to 0.301, MAE rose from 0.453 to 0.464 when slicing is used in multi-augmentations, with worse performance, which may be due to slicing destroying the long-term dependence, leading to sampling bias. In contrast, when adding "drop_mask" and "phase perturbation", the model's performance has a significant improvement, with the MSE decreasing from 0.304 to 0.294, and further to 0.281. Specifically, when adding phase perturbation, the MSE decreased from 0.294 to 0.271, and the MAE decreased from 0.451 to 0.428, which are improved by 7.8% and 5.1%, respectively, indicating that the introduction of phase perturbation can significantly improve the performance of the model. These findings reveal that TFCC can produce representations invariant to temporal and spectrum perturbations, and appropriate time and frequency augmentations can improve model generalization.  
conclusion: 
 This study proposes TFCC, a novel framework specifically engineered for unsupervised representation learning from time series data. In TFCC, we use the entire series as input and construct two sets of instance-level augmentation families based on the temporal and frequency characteristics to generate different views for training. Then, the framework injects multi-view contrasting modules into pre-training to learn properties invariant to diverse perturbations within various complex dynamics. Specifically, two challenging cross-view forecasting tasks between the original and its augmented series in time and frequency domains are first designed for robust representation learning. Contextual consistency brings the temporal-based and frequency-based representations along with their local neighbors close together in their latent space. Temporalfrequency consistency fosters mutual representation learning automatically and unveils temporal-spectral correlations through cross-domain dependencies, thus promoting discriminative representation learning. Comprehensive experiments validate TFCC's exceptional prowess in both classification and forecasting tasks, alongside its notable efficiency in label-scarce and transfer learning scenarios.  
: 
 tasks [22] . The endeavor to derive generalized representations from such constrained time series data presents an intricate challenge; thus, unsupervised representation learning in time series analysis paves the way for label-scarce scenarios. Lately, numerous studies [5, 9, 10, 26] have utilized contrastive loss to uncover the intrinsic patterns of time series, but they still face some notable limitations.First, segment-level representations introduce sampling bias. Previous efforts, such as TNC [18] , CPC [15] , TS-TCC [5] , and CA-TCC [6] , have yielded charming contributions, driving significant strides in representation learning. These methods mainly apply segment-level sampling policy and create contrastive pairs along the temporal axis, which may fall short of capturing the global context and undermine the preservation of temporal dependencies within time series [25] . Segment-level methods are highly dependent on the way to construct contrastive pairs, which usually assume neighboring segments as positive pairs and distant ones as negative, may not perform well in long-term scenarios and may not be able to capture the complete semantic information, thus affecting the stability and effectiveness of downstream tasks [22] .Next, frequency domain information is not fully explored. The aforementioned approaches have exhibited their strong ability for self-supervised representation learning, but they all overlook the frequency properties and the temporal-spectral affinities in time series, limiting the discriminative and expressive representation learning. CRT [25] , CoST [20] , and BTSF [22] provided valuable insights for mining frequency information, but only implemented augmentations in the time domain. The time domain focuses on sequential order and continuity, whereas the frequency domain emphasizes periodic patterns. Solely focusing on time-domain information often neglects important features, which can significantly hinder model performance, particularly in complex and noisy series. TF-C [26] considered the time and frequency augmentations, yet it lacks considerations for phase augmentation and sequence context information, indicating it was not fully mining frequency patterns and ill-suited for prediction tasks. Exploring frequency information can unearth vital insights and improve the effectiveness and generalizability of learned representations. However, how can we fully capture frequency domain information? Extracting robust information through prediction has been proven effective in the time domain [6] , but what about the frequency domain?To address these challenges, we present TFCC, a novel multiview self-supervised contrastive learning framework that enhances time series representation learning through three aspects. First, we applied instance-level augmentations to create contrastive pairs in both time and frequency domains, especially perturbating the frequency spectrum, which would not alter the raw properties of the entire series and effectively reduce false negatives and false positives. Additionally, we designed two cross-view forecasting modules in the time and frequency domains to encourage the model to learn robust representation using the past augmented samples to predict the future of the original samples without data loss and vice versa, which may differ from TS-TCC. To our knowledge, this is the first work that explores phase perturbation augmentation and performs prediction to capture spectrum patterns in the frequency domain. Last, we leverage three consistency modules in latent space: two contextual consistencies and one temporalfrequency consistency. Contextual consistency aims to maximize intra-sequence similarity and minimize inter-sequence similarity in the same domain, fostering discriminative representation learning. Temporal-frequency consistency is designed to pull together the time-based and frequency-based representations of the same series and push away the representations of different series in the shared latent space, achieving mutual learning from cross-domain interactions. As a result, TFCC utilizes multiple views to empower the exploration and extraction of generalizable time series representations and capture the embeddings invariant in various temporal dynamics and semantic meaning, leading to superior performance in downstream tasks, and serving as a vehicle for semi-supervised classification domain adaption and label-scarce scenarios.In summary, the contributions of our work are as follows.• A novel multi-view contrastive learning framework is proposed for unsupervised general-purpose representation learning in time series data.  
 
 Appendix: https://github.com/zhouhaoyi/ETDatasethttps://www.ncei.noaa.gov/data/local-climatological-data/ 


Decision: This paper introduces TFCC, a multi-view self-supervised contrastive learning framework for time series representation learning, incorporating both temporal and frequency domain information. While reviewers appreciated the comprehensive experiments, clear writing, and innovative combination of temporal and frequency domain augmentations, they raised several concerns. Key limitations include the modest novelty of the approach, lack of detailed hyperparameter tuning, potential overfitting issues, and limited discussion on computational complexity and scalability. Additionally, the baselines used for comparison are outdated, and the paper lacks visualization and deep analysis of key claims. Despite these issues, the paper demonstrates slight improvements over state-of-the-art methods and has potential applications in real-world scenarios. The ratings are mixed.

After carefully reading the comments and the paper, AC leans towards borderline acceptance. Some reviewers challenged the work to be not  well aligned with MM theme, which AC finds it aligned.

Reviews: review: paper_summary: paper_summary: This paper introduces TFCC, a multi-view self-supervised contrastive learning framework for time series representation learning. TFCC addresses limitations of existing methods by incorporating both temporal and frequency domain information. It leverages instance-level augmentations, cross-view forecasting in both domains, and consistency modules to learn robust and discriminative representations. Experiments demonstrate TFCC's slightly improved performance compared to state-of-the-art methods in classification, forecasting, semi-supervised learning, and transfer learning settings.
strength_weakness: strength_weakness: 1) The paper leverages both temporal and frequency domain information for time series representation learning. This approach tackles limitations of existing methods that focus solely on the temporal domain or fail to fully exploit frequency information.

TFCC combines various elements:
- Instance-level augmentations for preserving global context and long-term dependencies.
- Cross-view forecasting in both time and frequency domains to learn robust representations.
- Contextual consistency modules for discriminative learning.
- Temporal-frequency consistency for aligning time-based and frequency-based embeddings.

2) The paper presents extensive experiments on diverse datasets covering both classification and forecasting tasks. It also demonstrates effectiveness in semi-supervised and transfer learning settings.
3) The paper provides a thorough explanation of the proposed framework, including the motivations behind each component and its contribution to the overall performance.
4) The authors perform thorough ablation studies to evaluate the impact of each component of the framework, providing valuable insights into the effectiveness of individual modules.
limitations: limitations: 1) While the paper mentions sensitivity analysis, it lacks a detailed discussion on the process of hyperparameter tuning and how it impacts performance. It appears that if the hyperparameters are not optimized, the performance of the model can fall below the state of the art.
2) The paper could benefit from a discussion on potential overfitting issues related to the complex architecture and multiple loss functions.
3) The paper does not address the computational complexity and scalability of the TFCC framework, particularly for large-scale datasets or high-dimensional time series. The proposed framework adds considerable complexity compared to vanilla contrastive learning methods, for only modest improvements in accuracy. It is unclear if the accuracy improvements justify the added complexity.
4) The paper acknowledges augmentation selection as an open problem, but provides limited analysis on the impact of different augmentation strategies on model performance.
5) The paper primarily focuses on benchmark datasets. Discussing potential applications of TFCC in real-world scenarios and providing case studies would enhance the paper's impact.
paper_summary: paper_summary: This paper addresses issues of sampling bias and suboptimal performance caused by the loss of global context and neglect of crucial frequency information by introducing a self-supervised framework named Temporal-Frequency and Contextual Consistency (TFCC). The framework incorporates dual instance-level augmentations across the entire series, dual cross-view forecasting tasks in both time and frequency dimensions, and three specially designed consistency modules—temporal, frequency, and temporal-frequency—to robustly enhance data representation.
strength_weakness: strength_weakness: (1) This paper introduces a method named TFCC that employs instance-level augmentations in both time and frequency domains, specifically through frequency spectrum perturbation. This approach retains the raw properties of the series and effectively reduces false negatives and false positives, thereby enhancing the accuracy of the contrastive learning model.

(2) The paper designs two cross-view forecasting tasks aimed at developing robust representations, and strengthening the framework's analytical capabilities.

(3) The paper provides a comprehensive set of experiments to support its findings.
limitations: limitations: (1) The paper lacks innovation, as the fusion method combining time and frequency domains has been previously addressed, notably in the study "Unsupervised Time-Series Representation Learning with Iterative Bilinear Temporal-Spectral Fusion."

(2) The enhancement techniques employed in this paper, specifically the temporal contrast learning method, have already been explored in other research, such as TS2Vec: Towards Universal Representation of Time Series.

(3) The focus of the paper on single modality does not align well with the main theme of the conference.
paper_summary: paper_summary: This paper propose a contrastive learning-based method to learn the representation of time series data by leveraging both time and frequency domain information.
strength_weakness: strength_weakness: 1. The paper is well written and easy to follow.

2. Compared with the frequency domain-based contrastive learning, the paper considers the phase information.

3. Extensive experiments demonstrate that the method achieves superior performance in classification and forecasting.
limitations: limitations: 1.  The important claims seem to be lack of visualizations and deep analysis. For example , why the phase information is important for the representation learning? And what a role it played in the representation learning?

2. The baselines chosen for time series forecasting are too old. Informer is published in 2021 and TCN is in 2018.
paper_summary: paper_summary: This paper proposes Multi-view Self-Supervised Contrastive Learning for Multivariate Time Series
strength_weakness: strength_weakness: 1. well written
2. Ample experiment
limitations: limitations: 1. batch size is a very important hyperparameter for Contrastive Learning. I suggest the author evaluate it

2. Lack of comparison with state-of-the-art baselines [1] [2] [3].

3. I believe that the transferability of representation learning is crucial. Can TFCC achieve good performance with other backbone models as well?

4. The idea of integrating both time-domain and frequency-domain concepts has been previously explored in related works [4,5], and the authors are not the first to propose it.

[1] SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling

[2] A TIME SERIES IS WORTH 64 WORDS: LONG-TERM FORECASTING WITH TRANSFORMER

[3] Parametric Augmentation for Time Series Contrastive Learning

[4] Self-supervised contrastive pre-training for time series via time-frequency consistency

[5] TimesURL: Self-Supervised Contrastive Learning for Universal Time Series Representation Learning