VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasters

Authors: Mouxiang Chen, Lefei Shen, Zhuo Li, Xiaoyun Joy Wang, Jianling Sun, Chenghao Liu
Primary area: learning on time series and dynamical systems
Keywords: time series forecasting, foundation models, computer vision

Abstract:
Foundation models have emerged as a promising approach in time series forecasting (TSF). Existing approaches either repurpose large language models (LLMs) or build large-scale time series datasets to develop TSF foundation models for universal forecasting. However, these methods face challenges due to the severe cross-domain gap or in-domain heterogeneity. This paper explores a new road to building a TSF foundation model from rich and high-quality natural images. Our key insight is that a visual masked autoencoder, pre-trained on the ImageNet dataset, can naturally be a numeric series forecaster. By reformulating TSF as an image reconstruction task, we bridge the gap between image pre-training and TSF downstream tasks. Surprisingly, without further adaptation in the time-series domain, the proposed VisionTS could achieve superior zero-shot forecasting performance compared to existing TSF foundation models. With fine-tuning for one epoch, VisionTS could further improve the forecasting and achieve state-of-the-art performance in most cases.  Extensive experiments reveal intrinsic similarities between images and real-world time series, suggesting visual models may offer a "free lunch'' for TSF and highlight the potential for future cross-modality research. Our code is available in the Supplementary Material.

Decision: reject
Reviewer scores: 5, 8, 3

Reviews and discussion:
<thread>
creation date:
1730579781097

thread_type:
official_review

<thread_object>
creator: reviewer_tfqf

<content>
summary:
This paper proposes using MAE, a masked autoencoder pretrained on 2D images with mask modeling pretraining, as a zero-shot forecasting model for time series data. The proposed method reformulates the time series forecasting task into a reconstruction task. Experimental results show that this method can perform zero-shot forecasting and achieve state-of-the-art performance in most cases after fine-tuning.

strengths:
- This paper provides an interesting view of using MAE pretrained on vision tasks as a zero-shot time series forecasting model. 
- Results show that this model is better than text-based time series forecasting in zero-shot settings. 
- An interesting approach is proposed to achieve this transformation from image to text.

weaknesses:
# After reading the rebuttal:
I appreciate the authors' feedback on my concerns. However, as the responses still fail to resolve my concerns, I have decided to maintain my initial score.

- The statement, “VisionTS can work well for multivariate forecasting by using the channel independence strategy,” does not demonstrate that VisionTS effectively handles the relationships among multivariate data. While employing a channel independence strategy for multivariate handling is straightforward, it is not necessarily the optimal approach. Achieving good performance with this strategy might be more indicative of limitations in the benchmark rather than evidence that multivariate relationships are unnecessary.

- I appreciate that the authors provided some possible explanations for why MAE can be used for zero-shot learning on time series. However, none of these explanations serve as solid evidence. For instance, the motivation and case study could equally apply to other models, such as LLMs and time series. Additionally, the Modality Analysis and Figure 6 highlight the dissimilarity between images and most time series data, which contradicts the claim of “a relatively small gap between images and some time series.”

- The authors use LaMa as an example of another vision model. Given that LaMa employs a relatively weak vision backbone, I wonder whether using a vision model with stronger representation capabilities would result in better performance. Alternatively, carefully tuning the introduced hyperparameters might allow other models, such as LLMs or randomly initialized models, to achieve comparable performance.

- I appreciate the authors’ explanation regarding the tuning of hyperparameters. However, these results remain empirical and lack theoretical analysis.

# Initial review:
- Using MAE for forecasting only works for univariate forecasting, while the more challenging and important multivariate forecasting is not supported.
- It doesn't really make sense that MAE, which is pretrained solely on images, can perform reasonable zero-shot forecasting, as no time series information is encoded in the MAE pretraining. This work only transforms the format of time series data to make it compatible with MAE.
- The transformation between a univariate input X and the image-like 2D matrix is unclear. The explanation of how P is selected should be provided. How is P obtained from sampling frequency and Fast Fourier Transform? Why is this the right way to select P, or is P not important for the final performance?
- There are tricks and hyperparameters involved in aligning MAE with time series forecasting, but the explanations are unclear.
  - The standard deviation of the input is controlled by a hyperparameter r. While an initial explanation is given, more detailed experiments and justification should be provided for the choice of this value.
  - Why does setting c = 0.4 perform well, aside from the experimental results?
- Given the many tricks and unclear explanations, I doubt whether MAE can truly perform zero-shot time series forecasting or if it merely overfits to these time series datasets to achieve a favorable evaluation number.
- The inference cost comparison is only provided for these LLM-time series models, while a comparison with native time series models should also be included.

questions:
- Why does MAE work for zero-shot learning on time series data?
- How do these tricks work, and why are these hyperparameters chosen aside from experimental results?- 
- For more questions, please refer to the weakness sections.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730647077397

thread_type:
official_review

<thread_object>
creator: reviewer_bbkw

<content>
summary:
This paper provides a new perspective that treating time series forecasting problems as image reconstruction problems. And directly take the pretrained MAE model to conduct such tasks. Specifically, the paper just performs segmentation, normalization and alignment to transform 1D time series data into 2D data, and predict masked tokens for TSF tasks. The results shows that this algorithm has decent performance under both zero-shot and full-shot settings.

strengths:
1. This paper provides a novel view for TSF problems, and explain with reasonable motivations in Fig.2.
2. The method part provides sufficient details about the proposed method, making it easy to read and follow.
3. This paper provides experiments on a wide range of tasks to validate the proposed method.

weaknesses:
1. In the alignment step, the scaling process is influenced by both the context length $L$ and the prediction length $H$. It seems that the forecasted result at $L+1$ may change due to whether we will predict $L+2$ or not, which is counter-intuitive. 

Since the method map any-length series to a fixed resolution, I suppose it requires the model to understand patterns of different frequency. This might be hard for a pretrained vision model. It is trained only with natural images, which cannot cover drastic changes such as noises.

2. If the similarity between image and TSF holds true, maybe the authors should try other image inpainting models for the TSF tasks. We may assume that thoses models can have a better performance for image reconstruction.

3. In full-shot experiments, the authors decide to tune all the LN parameters. It is uncommon for the validation in vision pretrain model. Usually we either finetune the full model or only the last linear layer (e.g. linear probe). I wonder the reason that the authors prefer the LN tuning.

questions:
My main concerns are listed in weaknesses. Regarding these comments, I would like the authors to check if MAE is suitable for TSF.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687552902

thread_type:
official_review

<thread_object>
creator: reviewer_3mcw

<content>
summary:
The paper titled "VISIONTS: VISUAL MASKED AUTOENCODERS ARE FREE-LUNCH ZERO-SHOT TIME SERIES FORECASTERS" introduces a approach to time series forecasting (TSF) by leveraging pre-trained visual masked autoencoders (MAEs) on natural images. The key idea is that the pixel variations in images can be interpreted as temporal sequences, which share intrinsic similarities with time series data. The authors reformulate TSF as an image reconstruction task, allowing the pre-trained MAE to perform zero-shot forecasting without further adaptation. Extensive experiments demonstrate that VISIONTS achieves superior zero-shot forecasting performance compared to existing TSF foundation models and can further improve with minimal fine-tuning. The findings suggest that visual models may offer a "free lunch" for TSF and highlight the potential for future cross-modality research.

strengths:
1. The authors provide extensive experimental results across 43 TSF benchmarks, demonstrating VISIONTS's effectiveness in zero-shot and few-shot settings. 

2. The paper offers a theoretical analysis of the similarities between images and time series, providing a solid foundation for the proposed method. This contributes to a deeper understanding of cross-modality transferability.

3. The paper is well-written, with clear explanations of the methodology, experiments, and results. The figures and tables are informative and support the narrative effectively.

weaknesses:
1. The current approach is limited to univariate forecasting, which may restrict its applicability in real-world scenarios where multivariate time series are common. This limitation could be a barrier to broader adoption.

2. While the use of pre-trained models is a strength, it also introduces a dependency on external datasets for performance, which may not always be ideal or ethically sound. Did you use EVA-CLIP?

3. The paper does not fully address how VISIONTS scales with larger and more complex time series datasets, which is a critical consideration for practical applications.

questions:
Please refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731674739859

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Comments

comment:
## Global comment

We thank the reviewers for their insightful comments and constructive feedback. We're pleased that they agree our paper offers a `novel and interesting perspective` (R BBkW, R Tfqf). The paper is recognized for its `strong motivation` (R BBkW), `solid theoretical foundation` (R 3mCw), and `extensive experiments` (R 3mCw, R BBkW). The reviewers also appreciate the paper’s `clarity and detail` (R 3mCw, R BBkW). **We believe that our work opens a new road to building TSF foundation models and can inspire future cross-modality research.** We value their suggestions and will incorporate all feedback into the final version.

Our proposed VisionTS is initially based on a visual MAE. To explore its potential with other vision models, we also tested LaMa [1], a visual inpainting model. The MSE results (averaged across four prediction lengths) are as follows.

|                               | **ETTh1** | **ETTh2** | **ETTm1** | **ETTm2** | **Average** |
| ----------------------------- | --------- | --------- | --------- | --------- | ----------- |
| ***Traditional baselines***   |           |           |           |           |             |
| ARIMA                         | 0.912     | 0.516     | 0.707     | 0.405     | 0.635       |
| Seasonal Naive                | 0.600     | 0.483     | 0.489     | 0.358     | 0.482       |
| ***Few-shot methods***        |           |           |           |           |             |
| TimeLLM                       | 0.556     | 0.370     | 0.404     | 0.277     | 0.402       |
| GPT4TS                        | 0.590     | 0.397     | 0.464     | 0.293     | 0.436       |
| ***Zero-shot methods***       |           |           |           |           |             |
| Moirai (Small)                | 0.400     | 0.341     | 0.448     | 0.300     | 0.372       |
| Moirai (Large)                | 0.510     | 0.354     | 0.390     | **0.276** | 0.382       |
| **VisionTS (backbone: MAE)**  | **0.390** | **0.333** | **0.374** | 0.282     | **0.345**   |
| **VisionTS (backbone: LaMa)** | 0.425     | 0.376     | 0.400     | 0.294     | 0.374       |

Importantly, **all baselines, except VisionTS, were trained on time series datasets**. The results show that VisionTS with LaMa performs similarly to Moirai (Small/Large) in the zero-shot setting, and outperforms other few-shot models and traditional baselines. This suggests that the performance is driven by the inherent similarity between images and time series, not solely by the MAE model.

We also found that LaMa's performance lags behind MAE's. A possible reason is that LaMa may have overfitted image-specific nuanced features during training, as discussed in Section 4.2 (Scaling Analysis).

Next, we will address each reviewer's concerns individually.

*References*

[1] LaMa: Resolution-robust Large Mask Inpainting with Fourier Convolutions. WACV 2022.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes VisionTS, an approach that reveals the intrinsic similarities between time series and images, reformulates time series forecasting into a masked image modeling task and employs ImageNet-pretrained masked autoencoder (MAE) for zero-shot and full-shot forecasting to deliver superior performance.

This paper provide a very interesting and novel perspective on time-series forecasting. The technical insight is well above the bar of ICLR.

However, both Reviewer Tfqf and Reviewer 3mCw raised concerns that VisionTS is limited to univariate forecasting, especially in its highlighted zero-shot setting. While the ignorance of multivariate relationships is the "limitations in the benchmark", the strong experimental results only on such limited benchmarks indeed weaken the claim of the intrinsic connection between time series and images.