TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysis

Authors: Shiyu Wang, Jiawei LI, Xiaoming Shi, Zhou Ye, Baichuan Mo, Wenze Lin, Ju Shengtong, Zhixuan Chu, Ming Jin
Primary area: learning on time series and dynamical systems
Keywords: time series, pattern machine, predictive analysis

Abstract:
Time series analysis plays a critical role in numerous applications, supporting tasks such as forecasting, classification, anomaly detection, and imputation. In this work, we present the time series pattern machine (TSPM), a model designed to excel in a broad range of time series tasks through powerful representation and pattern extraction capabilities. Traditional time series models often struggle to capture universal patterns, limiting their effectiveness across diverse tasks. To address this, we define multiple scales in the time domain and various resolutions in the frequency domain, employing various mixing strategies to extract intricate, task-adaptive time series patterns. Specifically, we introduce TimeMixer++, a general-purpose TSPM that processes multi-scale time series using (1) multi-resolution time imaging (MRTI), (2) time image decomposition (TID), (3) multi-scale mixing (MCM), and (4) multi-resolution mixing (MRM) to extract comprehensive temporal patterns. MRTI transforms multi-scale time series into multi-resolution time images, capturing patterns across both temporal and frequency domains. TID leverages dual-axis attention to extract seasonal and trend patterns, while MCM hierarchically aggregates these patterns across scales. MRM adaptively integrates all representations across resolutions. TimeMixer++ achieves state-of-the-art performance across 8 time series analytical tasks, consistently surpassing both general-purpose and task-specific models. Our work marks a promising step toward the next generation of TSPMs, paving the way for further advancements in time series analysis.

Decision: accept (oral)
Reviewer scores: 10, 8, 6

Reviews and discussion:
<thread>
creation date:
1729529064966

thread_type:
official_review

<thread_object>
creator: reviewer_9h69

<content>
summary:
The paper introduces a time series pattern machine method called TimeMixer++ for processing multiscale time series. The method transforms time series into multi-resolution time images to enable pattern extraction with respect to temporal and frequency domains followed by three dominant modules - (1) input projection; (2) a stack of Mixerblocks; (3) output projection.  Extensive experiments show the proposed method obtains improvement over the well-established competing methods.

strengths:
The methods used in the paper (e.g., time imaging and image decomposition) are very interesting. The evaluation is comprehensive: the authors discuss long and short-term forecasting, zero-short forecasting, classification, and anomaly detection.

weaknesses:
In terms of the forecasting results shown in Tables 3 and 4, the performance gain is negligible, and such minor improved performance certainly can be attributed to the parameter tuning, e.g., a well-tuned parameter settings for TimeMixer++ while a weak parameter settings for other competing methods.  

The paper barely offers insights both theoretically and experimentally. The theoretical understanding of the improvement as well as its time imaging and multi-resolution mixing is lacking, mostly based on intuition and simply blending the models. 

There are some papers that already discussed the use of frequency analysis and the frequency components extraction for model deployment (e.g., [1][2][3]) to capture the periodic patterns, and they all claim it can capture the global interaction and patterns among time series, so what is the benefits of introducing multi-resolution time imaging, and it is worthwhile to compare them in ablation study? In addition, it is encouraged to cite the papers [1][2][3] if not yet in the references. 



References:
[1] Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictors https://arxiv.org/pdf/2305.18803
[2] TFDNet: Time-Frequency Enhanced Decomposed Network for Long-term Time Series Forecasting https://arxiv.org/abs/2308.13386
[3] FEDNET: FREQUENCY ENHANCED DECOMPOSED NETWORK FOR OUT-OF-DISTRIBUTION TIME SERIES CLASSIFICATION https://openreview.net/forum?id=OVu9DsOjgH

questions:
[1] Questions follow from the points listed in the weakness section.
[2] What is the naming of the method, i.e., TimeMixer++ in terms of? or just because both the methods target processing multi-scale time series? 
[3] What is the connection with the method TimeMixer?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730190526493

thread_type:
official_review

<thread_object>
creator: reviewer_lkya

<content>
summary:
The paper presents TimeMixer++, an advanced framework designed to enhance general time series analysis. TimeMixer++ integrates multi-resolution time imaging, multi-scale mixing, and dual-axis attention mechanisms to effectively capture and adapt to diverse patterns within time series data. This innovative approach allows for robust and flexible analysis across various temporal scales and resolutions.

strengths:
1. the authors introduce a robust framework TimeMixer++ that leverages multi-resolution time imaging, multi-scale mixing, and dual-axis attention to enhance general time series analysis. They present SOTA results on four different tasks.
2. the integration of both multi-scale and multi-resolution mixing strategies for adaptive pattern extraction demonstrates innovation.
3. the manuscript and appendix are well-prepared, but the authors have not yet released the promised code.

weaknesses:
1. the fonts in the figures should be enlarged for better readability. For example, in Figure 1 (right), the label "Benchmarking model performance across representation analysis in four tasks" appears blurred. Additionally, consider using a single set of legends for all four tasks to enhance clarity.
2. the source code repository has not released for reproducing, i will consider raising the score if the released repository and the consistency of the results.
3. more detail on how it compares to recent models like TimesNet and iTransformer on specific time series tasks would strengthen the paper’s claims.
4. including a discussion on computational efficiency (e.g., FLOPs, memory usage) for different tasks could enhance the paper’s utility.

questions:
See weaknesses (W2, W3, W4).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705154475

thread_type:
official_review

<thread_object>
creator: reviewer_e5jj

<content>
summary:
The paper presents TIMEMIXER++, a general-purpose model for various time series tasks, including forecasting, classification, anomaly detection, and imputation. Utilizing a multi-scale, multi-resolution framework, the proposed method transforms time series data into multi-resolution images to capture complex temporal and frequency-domain patterns, enabling flexibility across analytical applications. The model’s approach includes dual-axis attention for decomposing seasonal and trend components and hierarchical multi-scale and multi-resolution mixing to integrate patterns across scales. The proposal achieves strong performance across eight benchmark tasks, outperforming both general-purpose and task-specific models. This work contributes to advancing time series analysis with new state-of-the-art benchmarks across settings.

strengths:
S1. The proposed model captures both short- and long-term dependencies by transforming time series data into multi-resolution images, enabling the analysis of complex temporal and frequency-domain patterns that challenge traditional models. The authors validate this with experimental results showing the new architecture outperforms SOTA models on most standard benchmarks. The ablation study helps validate the importance of the individual parts of the architecture – the channel mixing, image decomposition, and multi-scale and multi-resolution mixing. This approach continues to validate the benefits of integrating image analysis techniques with time series tasks.

S2. The architecture is flexible for supporting different kinds of time-series tasks. The hierarchical multi-scale and multi-resolution mixing modules enable the model to flexibly adapt across various time series tasks, from forecasting to anomaly detection, promoting robust and accurate performance across applications.

S3. Empirical Validation: the testing in this paper on eight benchmark time series tasks, including the hyperparameter ablation results, shows TIMEMIXER++ consistently surpasses both general-purpose and task-specific models, affirming its potential as a high-performance, general-purpose solution for time series analysis. The experiments were very thorough.

weaknesses:
W1. There is little exploration of scaling of model size, which would be an interesting avenue for validating the model architecture in a zero shot setting. The current zero-shot experiments are primarily in-domain and not cross-task.

questions:
Q1. The proposed architecture adds significant computational cost to the internal representation of the model compared to vanilla transformers and some of the previously proposed models. It seems this does not have a significant effect on the training time compute and memory complexity of the model. Have the authors conducted any studies to compare the inference-time cost of TM++ compared to other methods? 

Q2. As mentioned by authors, some time series tasks (imputation, anomaly detection) benefit more from diverse representations while others like forecasting and classification benefit from consistent representation. Given this, is there any way to leverage a routing model dependent on the proposed task type, which could lower the inference-time cost of this model?

Q3. MTS-Mixer (Li et. al., 2023) presents another approach to channel decomposition which similarly outperformed competing models, but they found the approach worked best with MLPs rather than attention-based models. Have the authors explored this technique for separating from attention mechanisms which could lead to further efficiency and model performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732417281703

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revisions

comment:
We sincerely thank all the reviewers for their detailed reviews, 
which are instructive for us to improve our paper further.
We have revised the paper according to the comments, and the 
edits have been highlighted in **RED**. 

This paper introduces TimeMixer++, a novel and general time series pattern machine for universal predictive analysis. 
TimeMixer++ disentangles seasonality and trend patterns 
in latent space through Time Image Decomposition (TID) and adaptively integrates these patterns using 
Multi-Scale Mixing (MCM) and Multi-Resolution Mixing (MRM). **It achieves state-of-the-art performance 
across eight diverse time series tasks, outperforming 27 baselines on 30 benchmarks**.

The reviewers generally held positive opinions of our paper, noting that the proposed method **"demonstrates innovation"**
and
is **"very interesting"**; **"the manuscript and appendix are well-prepared"**;
**"the experiments were very thorough"** and **"comprehensive"**;  and that
we **"present SOTA results"**.


The reviewers also raised insightful and constructive concerns. Here is the summary of the major revisions:

- **Provide cross-domain zero-shot forecasting results (Reviewer e5Jj)**: 
Following the reviewer's suggestion, we conducted a zero-shot forecasting experiment 
on two additional cross-domain datasets, M3 and M4. TimeMixer++ continues to perform best
in this setting. For complete results, please refer to Appendix D.

- **Add inference time comparison (Reviewer e5Jj)**:
To address the reviewer's request, we have reported the inference time of TimeMixer++ 
along with seven baselines on long-term forecasting tasks to provide a more comprehensive 
understanding of the efficiency analysis. TimeMixer++ achieves an inference speed of 90 ms/iter, 
which matches that of TiDE and surpasses other models. For the complete results, please refer to Appendix D.

- **Add ablations on the channel mixing module (Reviewer e5Jj)**:
We have included ablation results evaluating the choice of channel mixing based on a multi-layer 
perceptron (MLP). The results demonstrate that our proposed attention-mixing approach consistently 
outperforms MLP-mixing. The updated results can be found in Appendix D.

- **Update the figures (Reviewer LKyA):**
Following the reviewer's suggestion, we have updated Figure 1 in the main text and Figure 12 in Appendix E.


- **Clarify differences among Koopa, TFDNet, and FEDNet (Reviewer 9h69)**:
We clarify that TimeMixer++ is distinct from Koopa, TFDNet, and FEDNet in its technical design, pattern 
learning capabilities, and respective objectives within the literature.
Additionally, we have updated the Introduction and Related Work sections, adding relevant citations to 
highlight our contributions.

The valuable suggestions from reviewers are very helpful for us to revise the paper to a better shape. We'd be 
happy to answer any further questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732533845260

thread_type:
official_comment

<thread_object>
creator: area_chair_eu5n

<content>
title:
Acknowledge the author responses

comment:
Dear Reviewers,

Thank you very much for your effort. As the discussion period is coming to an end, please acknowledge the author responses and adjust the rating if necessary.

Sincerely,
AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1739365945993

thread_type:
public_comment

<thread_object>
creator: ~kashif_rasul1

<content>
title:
MixerBlock questions

comment:
Why are you referring to the inputs of the mixer block as time series? presumably, at that point, the inputs are representations of the original multivariate data of feature-length $d_\mathrm{model}$  coming from the channel attention?

why are seq of representations $\mathbf{x}^l_m$ a 1-D time series in line 243? If it's the output from the channel attention it has shape $d_\mathrm{model} \times T/2^m$ no?

What is the motivation for decomposing the representations from the channel attention layer? My intuition was that at that point the output from the channel attention would not resemble time series, like trend or seasonality, even more so up the layers?

Did you compare your model to prototypical mixer architectures like MLP-mixer, Conv-Mixer, and ViT style models that can easily be up-cylced for the multivariate forecasting task?

thank you!
</content>
</thread_object>

</thread>
<thread>
creation date:
1740838915167

thread_type:
public_comment

<thread_object>
creator: ~danny_dongyeop_han1

<content>
title:
Questions regarding the paper

comment:
First, thank you for your amazing work! I am interested in applying your model in our domain (medical timeseries), and had a few questions if you don’t mind. 

**Questions regarding pretraining** : In Appendix L “Limitations and Future Work”, you say TimeMixer++ can be an effective backbone model, and will explore scaling the model. We also want to do this for our domain, but realized that some aspects of TimeMixer++ are not well suited for pretraining and wanted to ask your opinion if you don’t mind : 
1. **Do you have any suggestions on what type of SSL pretext task could be used?**  
    - As far as I know, current timeseries foundation models (e.g., MOIRAI, TimeMOE, Timer-XL) use GPT/BERT-like pretext tasks for pretraining. However, because your model don’t use patches (tokens), using GPT/BERT-like pretext tasks seems impossible. Do you have suggestions on how to resolve this (either by modifying the model or choosing a different pretext task?)
    - Also, it seems that current time-series foundation models only use single patch sizes as inputs. As an outsider this perplexes me. If possible, could you please tell me why? Your paper and others seem to argue that using single sized tokens to represent timeseries is not ideal, yet many foundation models use tokens.
2. **How would you modify your model to adapt to varying number of input variables?**
    - Due to how input projection is designed, it seems that the model cannot be trained on datasets with different number of channels. Do you have any suggestions on how to resolve this?
    - Also, was there a reason why you mixed the channels early (during input projection from $C$ to $d_{model}$), and keep the channel dimension and model them in the MixerBlock?
3. **Do you think TimeMixer++ could adapt to different input lengths?**
    - As far as I understand, it seems that at least computationally the model can be applied to different input lengths. Is my understanding correct? Do you think that fine-tuning the model to a task with much shorter input length compared to the pretraining dataset will be ok? 
 


**Questions regarding the model itself :**

1. **Would your model still work well if the data always has FFT amplitudes with a 1/f trend?**  : Due to the inherent 1/f trend in our data, selecting the top-k frequencies during MRTI would always return the k lowest frequency values, making the model ignore higher frequency features (which are crucial for certain classification tasks in our field). Do you think it will be OK for us to use a different top-k frequency selection method (e.g., selecting the top-k after detrending the 1/f component)? Or should we stick to the original method? 
2. **Is $p_k$ actually $p_{k,m}$? :** It seems that $p_k$, obtained from running FFT and picking the Top-K periods is the value with respect to the scale $M$. Wouldn't it be that for different scales the same period length would be different number of timepoints (for example, $p_k$ obtained in scale $M$ should equal $2p_k$ in scale $M-1$?)
3. **Details on the 2D convolutions you used in Time Image Decomposition and Multiscale Mixing? :** Could you please tell the kernel sizes and such that were used for the 2D convolutions you used in your model? It would greatly help in understanding your paper!

Thank you!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents TimeMixer++, a general-purpose model for various time series tasks, including forecasting, classification, anomaly detection, and imputation.  The proposed model achieves state-of-the-art performance across 8 time series analytical tasks, consistently surpassing both general-purpose and task-specific models.  These results are obviously impressive, and this paper is worthwhile to receive further attention.  Thus, I would like to recommend an accept as a spotlight paper.