ShuffleMTM: Learning Cross-channel Dependence in Multivariate Time Series from Shuffled Patches

Authors: Hyunwoo Seo, Byeongyeon So, Chiehyeon Lim
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Self-supervised learning, masked modeling, multivariate time series, cross-channel dependence

Abstract:
Masked time-series modeling has widely gained attention as a self-supervised pre-training method for multivariate time series (MTS). Recent studies adopt a channel-independent (CI) strategy to enhance the temporal modeling capacity. Despite the effectiveness and performance of this strategy, the CI methods inherently overlook cross-channel dependence, which is inherent and crucial in MTS data in various domains. To fill this gap, we propose ShuffleMTM, a simple yet effective masked time-series modeling framework to learn cross-channel dependence from shuffled patches. Technically, ShuffleMTM proposes to shuffle the unmasked patches from masked series across different channels, positioned at the same index. Then, Siamese encoders learn two views of masked patch representations from original and shuffled masked series, simultaneously capturing the temporal dependence within a channel as well as spatial dependence across different channels. ShuffleMTM pre-trains the Siamese encoders to reconstruct the original series by incorporating cross-channel information with intra-channel cross-time information. Our proposed method consistently achieves superior performance in various experiments, compared to advanced CI pre-training methods and channel-dependent methods in both time series forecasting and classification tasks.

Decision: reject
Reviewer scores: 5, 8, 6, 3

Reviews and discussion:
<thread>
creation date:
1729693076817

thread_type:
official_review

<thread_object>
creator: reviewer_riey

<content>
summary:
This paper proposes a simple method to improve the multivariate time series self-supervised learning paradigm. In order to capture cross-channel dependencies, it proposes method named ShuffleMTM, which first shuffle unmasked patch along the channel dimension for data processing, and secondly performs information extraction based on cross attention based on the original mask sequence and the shuffled mask sequence.

strengths:
1. The problem is clearly defined and the method is clear and simple；
2. Experimental results show that although this method is simple, it brings certain gains on some data sets.

weaknesses:
1. This paper lacks novelty. Except for the shuffle operation on the unmasked time series , the use of Siamese network does not give me a bright feeling. At the same time, as the author expressed, part of the work differs from patchstst in that the linear layer is replaced by the transformer decoder structure, which seems to be an operation driven by experimental results and has nothing novel.
2. This paper claims to enhance the prediction ability of multivariate time series by capturing cross-channel dependencies. However, in Table 1 we can find that the experimental results do not seem to prove this point. It is also a data set with strong channel correlation, and the weather dataset shows Good results, however, the electricity dataset and traffic dataset performed poorly. Firstly, experimental results prove that this method cannot capture cross-channel dependencies well. Secondly, it cannot prove that the improvements to other data sets are based on cross-channel information. Gains from extraction.
3. The method proposed in this paper has problems in the reasoning stage. In the training and verification stage, some data operations can indeed be performed through shuffle to improve model performance. However, in the reasoning stage, if the random seeds are different each time, then for the same input, due to the uncertainty of other channel patches introduced by the shuffle operation, the output results are inconsistent each time, which is unacceptable in real scenarios.
4. In the robustness exploration experiments based on different proportions of missing values, all selected baseline methods, including this paper, showed that as the proportion of missing values increases, the performance first decreases, and then the performance increases. The author does not give a reasonable explanation for this phenomenon.

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730466751716

thread_type:
official_review

<thread_object>
creator: reviewer_5atj

<content>
summary:
This paper proposes a simple time series masking modeling method based on a Siamese network and a specially designed decoder. It establishes the relationship between masked patches within a channel and shuffled patches across channels at the patch level, enabling learning of both intra-channel and inter-channel temporal dependencies. This approach demonstrated superior performance on the main temporal analysis task.​

strengths:
- The proposed method effectively models temporal correlations and dependencies within and across channels through a simple yet effective masking approach.
- Comprehensive experiments and analyses support the method.

weaknesses:
- Limited generalizability: The method relies on a channel-independent time series model as the backbone, limiting its applicability to more advanced temporal models, especially those that inherently model inter-channel relationships.

- The motivation of the paper could be more precisely stated. The CI method does not completely ignore inter-channel modeling; Rather, they construct these relations implicitly rather than explicitly, as multiple channels are simultaneously optimized in a single iteration.​

questions:
- **Main Experiments**: 

Effectiveness should be validated on larger datasets. Numerous masking and generative pre-training methods, such as Moirai [1] and Timer [2], have achieved notable results on large-scale time series data. Comparisons of these approaches could shed light on the effectiveness of this mask modeling approach as data size scales, and whether it holds advantages over other unsupervised pre-training paradigms in larger data scenarios.​

- **Ablation Study**: 

The performance of ShuffleMTM is most likely affected by the masking ratio and the number of channels. The masking ratio controls the potential shuffled candidate locations, while the number of channels determines the diversity of patch replacements. A more detailed examination of their relationship is recommended. For example, on high-channel datasets (such as Traffic and ECL), it would be beneficial to explore how model performance changes with increasing masking ratios.

​Another ablation experiment could evaluate the effect of enforcing that the current position in one channel is always replaced by another channel during shuffling.​

[1] Unified Training of Universal Time Series Forecasting Transformers

[2] Timer: Generative Pre-trained Transformers Are Large Time Series Models
</content>
</thread_object>

</thread>
<thread>
creation date:
1730652872071

thread_type:
official_review

<thread_object>
creator: reviewer_wko4

<content>
summary:
This paper proposes a simple strategy called ShuffleMTM for handling multi-channel time series. During pre-training, time series patches are randomly masked and shuffled across channels. Two views of each channel's univariate time series (the first is the one after random masking, and the second is after both random masking and shuffling) goes into a Siamese encoder followed by a decoder that aims to reconstruct the raw input time series. The paper demonstrates that ShuffleMTM works very well in practice.

strengths:
- The proposed strategy is well-explained and very simple.
- The experimental results of Section 4 show that ShuffleMTM works quite well in practice.
- I think the experiments in Sections 5 and 6 are very helpful in checking on the robustness of ShuffleMTM to various conditions (such as limited data, missing data, hyperpameters) and in capturing patch-level and channel-level dependence.

weaknesses:
- While ShuffleMTM is fairly simple to describe, I think the key question in my mind is, from a theoretical viewpoint, what sort of cross-channel structure can be captured by the specific shuffling performed, and what sort of cross-channel structure can *not* be captured. I think discussing this in more detail would really strengthen the paper. Just to give a simple example, suppose that the patch size is 1 so that each patch is just a single time step, and that it turns out that the true underlying structure is that each time step depends on *previous but not the current* time step's information in other channels. If I understand ShuffleMTM correctly, it seems like the shuffling done by ShuffleMTM in this case would not be helpful in capturing the cross-channel structure? Or maybe I'm misunderstanding something about ShuffleMTM? In any case, what I'm trying to do here is to just try to think of a setting where ShuffleMTM wouldn't work well (I'd guess that the authors could come up with various examples).
- Figure 2 I think is very important and helpful, but it took me some time to figure out what I am looking at. I'd suggest making the caption more descriptive and maybe adding some text (or some sort of drawing over the diagram) to better explain what is going on. If I understand the figure correctly, each colored cell is a patch. Blue/pink/green correspond to three channels. Gray cells are masked. Only some columns are randomly shuffled (I'm inclined to suggest that you circle or box the columns that were chosen for random shuffling across channels). Meanwhile, the two views that are inputs to the Siamese encoder correspond in this case to channel 1's data (actually stating this to be the case would be helpful to the reader).
- The experimental results would benefit from having error bars reported (such as standard deviation across some fixed number of experimental repeats with different random seeds) in Tables 1-4 and Figures 4-6. If there is a concern about tables becoming too large, one solution is to just defer, for instance, MAE results (with newly added error bars) to the appendix and keep MSE results (with newly added error bars) in the main paper.

Minor:
- There are a number of English grammar issues and typos. For example, in the first sentence of the introduction (line 33), the word "traffics" is used incorrectly. Perhaps this part of this first sentence can be reworded as "including energy, transportation, and medicine" (note that "medicines" here should be singular). At line ~66-67, "Unlike previous methods that recovers" should say "Unlike previous methods that recover". I'd suggest proofreading the paper carefully and using, for instance, a grammar checker.

questions:
Please address the weakness points raised.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688621289

thread_type:
official_review

<thread_object>
creator: reviewer_g464

<content>
summary:
This paper addresses the challenge of capturing cross-channel dependencies in multivariate time series pre-training. Unlike previous channel-independent methods, the proposed model, ShuffleMTM, masks a multivariate instance to create an original masked view and then shuffles the channels within each time step to generate a shuffled masked view. The original view serves as the query, while the shuffled view serves as the key and value in a cross-attention decoder, with the objective of reconstructing the original instance. Experimental results demonstrate that ShuffleMTM outperforms several channel-independent baselines in both forecasting and classification tasks. Additional experiments assess its performance in limited data scenarios, robustness, and other settings.

strengths:
1. The paper is well-structured and accessible, making it easy to follow.

2. The focus on capturing cross-channel dependencies in multivariate time series pre-training is both important and relevant.

3. A capacity-robustness analysis is provided, offering insights into the model’s forecasting accuracy as well as its robustness, training-testing gap, and other performance metrics.

weaknesses:
1. Some crucial parts about the proposed model is missing: how to adapt the pre-trained model to downstream tasks. Is the shuffled view utilized during fine-tuning? If not, it is unclear how the channel-independent encoder effectively captures cross-channel dependencies.

2. The proposed shuffling method is  not straightforward. Why not simply apply 2D attention, as done in Crossformer, to the masked patches, or flatten the time series into a 1D sequence and use self-attention? What specific advantages does shuffling offer over these approaches, aside from efficiency?

3. The ablation study could be improved. The following experiment should be included: removing the shuffled view by using the original view as the query, key, and value in the decoder, and comparing this configuration with the proposed method, which uses the original view as the query and the shuffled view as the key and value.

4. In the forecasting task, the lookback window is fixed to 96, which is shorter than the future window to forecast. It would be valuable to assess the model’s performance with an extended lookback window.

5. Please explain Section 6.1 in detail, including:
 - How to comupte the "self-attention map" and "patch-correlation" in patch-level dependence paragraph as there is no cross-channel attention in the proposed method?
 - How to compute the "channel correlations in the raw time series" in channel-level dependence?

questions:
Please see the Weaknesses section for detailed questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733193400519

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Meta Response to all reviewers

comment:
Dear Reviewers,

We sincerely appreciate your insightful and constructive feedback. The discussions with the reviewers have been invaluable in providing meaningful clarifications and enhancing the quality of our work. Based on your comments, we have made the following improvements: (1) clarified the motivation and academic contributions of our work within the context of time-series research; (2) strengthened the technical contributions of ShuffleMTM; (3) conducted additional ablation experiments; (4) improved the overall presentation of our work.

We would like to use this space to summarize the key points raised by the reviewers and provide our meta responses. These meta responses, addressing the four aspects mentioned above, aim to offer a clearer understanding of the overall revisions made to our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper aims to address the limitations of existing channel-independent (CI) masked time series modeling methods by learning cross-channel dependence from shuffled patches. It trains Siamese encoders to learn two views of masked patch representations to capture the temporal dependence within a channel and spatial dependence across multiple channels.

Major strengths:
- Learning cross-channel dependence effectively is an important topic in multivariate time series modeling.

Major weaknesses:
- The effectiveness of the proposed pipeline which involves a multi-channel pre-training stage followed by a single-channel fine-tuning stage for modeling channel dependency needs to be justified more thoroughly.
- Since no theoretical justification for the proposed method is presented in the paper, it would need strong experimental validation to demonstrate the effectiveness of the method. However, for datasets with strong correlation between channels, no significant improvement can be observed.
- The effectiveness of the shuffle operation is dataset-dependent, but it may not be easy to know in advance whether it will help or not before trying it out.

Despite studying an important topic in multivariate time series modeling, the effectiveness of the somewhat counter-intuitive method needs to be justified more convincingly, either through theoretical justification or based on comprehensive experimentation. Among other things, some additional experiments can use synthetic datasets in which some channels are really correlated.

The authors are encouraged to improve their paper for future submission by considering the comments and suggestions of the reviewers.