Dual-Forecaster: A Multimodal Time Series Model Integrating Descriptive and Predictive Texts

Authors: Wenfa Wu, Guanyu Zhang, tan zheng, Yi Wang, Hongsheng Qi
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: time series forecasting, multimodal time series model, multimodal comprehension

Abstract:
Time series forecasting plays a vital role for decision-making across a wide range of real-world domains, which has been extensively studied. Most existing single-modal models rely solely on numerical series, which suffer from the limitations imposed by insufficient information. Recent studies have revealed that multimodal models can address the core issue by integrating textual information. However, these models focus on either historical or future textual information, overlooking the unique contributions each plays in time series forecasting. Besides, these models fail to grasp the intricate relationships between textual and time series data, constrained by their moderate capacity for multimodal comprehension. To tackle these challenges, we propose Dual-Forecaster, a pioneering multimodal time series model that combines both descriptively historical textual information and predictive textual insights, leveraging advanced multimodal comprehension capability. We begin by developing the historical text-time series contrastive loss to align the descriptively historical textual data and corresponding time series data, followed by encoding multimodal text-time series representations between them through the history-oriented modality interaction module, and then combining predictive textual data through the future-oriented modality interaction module to ensure textual insights-following forecasting. Our comprehensive evaluations on synthetic dataset and captioned-public datasets demonstrate that Dual-Forecaster is a distinctly effective multimodal time series model that outperforms or is comparable to other state-of-the-art models, highlighting the superiority of integrating textual information for time series forecasting. This work opens new avenues in the integration of textual information with numerical time series data for multimodal time series analysis.

Decision: reject
Reviewer scores: 3, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1729666744668

thread_type:
official_review

<thread_object>
creator: reviewer_cjgf

<content>
summary:
Based on transformer, this paper introduces Dual-Forecaster, a multimodal time series forecasting model designed to enhance prediction accuracy by integrating both numerical time series data and corresponding textual information, where the data is used and aligned to help time-series prediction task. To effectively merge these modalities, Dual-Forecaster incorporates three cross-modality alignment techniques: Historical Text-Time Series Contrastive Loss, History-oriented Modality Interaction Module, Future-oriented Modality Interaction Module.

strengths:
1. Point out the information insufficiency problem, which I agree.
2. The model is clean and simple, and it works well in synthetic data. The use of pre-trained model and its application is acceptable.

weaknesses:
1. I wonder how to use the model as the textual data is well-designed, and it seems that it performs bad when I don't input. For most of the users, I just wanna use the model as quick as possible.
2. What about I use wrong information in texts, this interests me. 
3. The example is too simple in Figure2
4. Compare with some multi-modal approach, (TIME-LLM eg.). the comparison is weak.
5. give more ablation study including pre-trained model.

questions:
1. explain why textual data, in particular, is the most effective supplementary modality
2. why use the contrastive loss ?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730296279672

thread_type:
official_review

<thread_object>
creator: reviewer_e6mb

<content>
summary:
The paper introduces Dual-Forecaster, a novel multimodal time series forecasting model that integrates both historical and future textual data with numerical time series data. The model is designed to improve forecasting performance by leveraging rich semantic information from textual data, which traditional time series models often lack.

strengths:
1. The paper presents a novel approach to time series forecasting by integrating both historical and future textual data, which is a creative extension of existing multimodal time series models. This work has the potential to significantly impact the field of time series forecasting by demonstrating the value of incorporating textual data. 
2. The extensive experiments and ablation studies provide robust evidence of the model's effectiveness and the importance of each component.

weaknesses:
1. Although utilizing predictable future textual information is a commendable effort, I am concerned that in synthetic data and captioned datasets, the inclusion of predictable future textual information may lead to information leakage (where they represent future ground truth), potentially resulting in unfair comparisons with other models and an overestimation of this model's performance.
2. While the paper focuses on time series forecasting, it does not explore the potential of Dual-Forecaster in other multimodal time series analysis tasks (Imputation, Anomaly Detection, etc.). Expanding the scope could provide a more comprehensive evaluation of the model's capabilities.

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730479321571

thread_type:
official_review

<thread_object>
creator: reviewer_j1sz

<content>
summary:
This paper introduces a time series forecasting model that incorporates both historical and future textual data to improve forecasting accuracy. The proposed model, Dual-Forecaster, uses two main branches: a textual branch for embedding historical and predictive texts and a temporal branch for numerical time series data. It employs three cross-modality alignment techniques—contrastive loss, history-oriented, and future-oriented modality interaction modules—to align and integrate information across these modalities. Experimental results on synthetic and public datasets demonstrate that Dual-Forecaster outperforms baseline models by leveraging multimodal comprehension, thus underscoring the value of textual insights in time series forecasting. The study opens new pathways for integrating diverse data types to enhance predictive performance across various real-world applications.

strengths:
- The paper is well-structured and clearly written, with a logical organization that makes it easy to follow the development of ideas. The authors include illustrative figures, such as the model architecture and case studies, which aid in understanding the Dual-Forecaster model’s interactions and multimodal integration strategies.
- The methodology is clearly presented, and the cross-modality alignment techniques—contrastive loss, history-oriented, and future-oriented interaction modules—are well-integrated within the model. These mechanisms aim to capture complex relationships between text and time series data, contributing to the overall approach.
- The experimental results are strong, including evaluations on synthetic and real-world datasets as well as ablation studies. The results indicate that the proposed components have a positive impact on performance, providing evidence of Dual-Forecaster’s potential effectiveness in multimodal time series forecasting.

weaknesses:
- **Missing Baseline Comparison**: The authors reference the paper "Beyond Trend and Periodicity: Guiding Time Series Forecasting with Textual Cues" by Xu et al. (2024), yet they do not conduct a comparison experiment with this approach. Given the conceptual similarity between the two models, it is crucial to include this as a baseline to demonstrate any distinct advantages of the proposed method.
- **Over-reliance on Synthetic Data and Limited Real-world Testing**: While the paper includes evaluations on both synthetic and public datasets, the reliance on synthetic data limits the applicability of the results. Synthetic setups with artificial textual descriptions do not reflect the complexities and nuances of real-world time series data. The public datasets used (e.g., ETTm1, ETTm2) are also limited in variety and complexity. To strengthen the paper, the authors should include more challenging datasets, such as exchange rates or stock indices, which better represent real-world scenarios. Additionally, the paper lacks evaluation on how the model handles noisy or misleading textual data—an issue highly relevant in real-world, high-stakes contexts.
- **Limited Justification for Dual Textual Modality** The inclusion of both historical (descriptive) and future (predictive) textual information is presented as a core feature, yet the paper provides limited justification for this design choice. It is unclear whether this dual modality substantially enhances forecasting or if similar results could be achieved using only one type of text. A direct ablation study comparing the use of single versus dual text modalities would help clarify this.
- **Error Bounds Missing** The paper does not report error bounds or standard deviations for the results across multiple runs. This omission makes it difficult to assess the consistency and robustness of the reported performance improvements. Including error bounds would provide a more rigorous evaluation of the model's stability.

questions:
- Have you considered simply combining the historical and future texts into a single input rather than using separate modules to handle them? This approach could reduce model complexity. How do you anticipate that combining the texts would affect forecasting performance, and is there evidence that using separate modules significantly enhances results?
- How would your captioning method perform on more complex and volatile datasets, such as exchange rates or stock indices, where trends and patterns are less predictable? Given that these types of data often contain abrupt changes and noise, would your approach to generating textual descriptions still produce reliable or meaningful captions?
- Real-world applications, especially in high-stakes domains like finance, often include noisy or misleading textual data. How does your model handle such cases where the text may not accurately reflect trends in the time series? Have you tested the robustness of Dual-Forecaster with noisy or contradictory textual inputs, and if not, could you discuss any mechanisms that could be added to mitigate potential misdirection from unreliable text?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695773170

thread_type:
official_review

<thread_object>
creator: reviewer_uk6h

<content>
summary:
The paper introduces a novel text-guided time series forecasting model, named Dual-Forecaster, which leverages both historical and predictive textual data to enhance forecasting accuracy. This model has been evaluated using synthetic and real-world datasets, demonstrating superior performance.

strengths:
- The paper is well-organized and easy to follow. 

- The model architecture is well explained and seems to be solid. 

- The ablation study, particularly the alignment heatmap between time series patterns and textual descriptions, provides in-depth understanding on how the model actually works.

weaknesses:
- The task definition may need further discussion. The statement right now is clear. However, the practicality of the task definition could be better justified, specifically regarding the availability of caption data in real-world applications.

- There may be potential information leakage in this setting/dataset used. According to the TGTSF [1] cited in related work, to avoid directly using the information in the future time series, it is advisable to use the text information from external sources that is related to the system we are analyzing. 

- The captioning process of public dataset needs further discussion. Directly asking the GPT to describe the time series may not be a promising method. 
    - The GPT's ability of processing numerical values still remains under discussion. Asking it to generate description of time series which is also a sequence of float values, may suffer from severe hallucination problem. 
    - As mentioned above, since the textual descriptions are derived directly from the time series data, there is a risk of information leakage, particularly if these descriptions are used for future predictions.
    - It seems that the captioned textual information also contains numerical values. It is still questionable that if the information in these values still remains after the text embedding model, i.e. RoBERTa in this paper. 

- According to the ablation study, this work seems to fall into the category of aligning time series pattern and textual description. Maybe it is a better idea to apply this to classification and anomaly detection task, which is not very sensitive to the information leakage problem. 

[1] Zhijian Xu, Yuxuan Bian, Jianyuan Zhong, Xiangyu Wen, and Qiang Xu. Beyond trend and periodicity: Guiding time series forecasting with textual cues.

Minor:

- Some figures are hard to read, e.g. Figure 3 & 5.

questions:
- How are the future-oriented predictive text generated? It seems that authors use the known time series for caption to generate the description for trend and periodicity. This is reasonable for the look-back window part of each training sample, but using the caption for forecasting horizon may lead to severe information leakage. 

- See Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732176517075

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response #1: Misleading Textual Data

comment:
**We have to emphasize that our work aims to illustrate that integrating textual data does indeed improve forecast performance.** Therefore, we focus on designing a multimodal information fusion framework with advanced multimodal comprehension capability. To verify the effectiveness of the framework, it is necessary to eliminate the influence of data caused by inaccurate textual information. Therefore, the data we use contains accurate textual information. Experimental results demonstrate the superiority of our model in multimodal information fusion, with advanced multimodal comprehension capability. Moreover, the model enables textual insights-following forecasting, thereby providing more reliable forecasts. 

As for the common response of Misleading Textual Data raised by reviews, I will answer from the following two aspects: 
1. After training Dual-Forecaster using data containing accurate textual information, during the inference phase, if the textual data contains incorrect information, the Dual-Forecaster will tend to provide forecasts that match the textual information due to its ability of textual insights-following forecasting, which may deviate from the true value. 
2. If we **train our model using textual data that contains both accurate and incorrect information**, the model will adaptively learn the forecasting mode under both accurate and incorrect textual data inputs during the training process. During the inference stage, if the input textual data contains incorrect information, the model can still provide accurate forecasts. Additionally, using textual data containing inaccurate information to **fine-tune Dual-Forecaster** can also enable the model to provide accurate forecasts even when inputting textual data containing inaccurate information. 

In summary, our current work falls under the first point mentioned above. When inputting textual data containing inaccurate or misleading information, the model's forecasts may deviate from the true values. **The second point mentioned is our future work.** We believe that Dual-Forecaster can be easily extended to time series forecasting scenarios involving textual data input with inaccurate information and provide accurate forecasts.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732176643845

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response #2: Potential Information Leakage

comment:
Regarding the common concern of potential information leakage raised by reviewers, we would like to make the following clarifications: 
1. We acknowledge that using information within the prediction interval to assist time series data for forecasting would result in information leakage in traditional time series forecasting framework.
2. However, as we mentioned in the INTRODUCTION section, the current traditional time series forecasting models that rely solely on time series data are **constrained by the limitation of insufficient information**, which has reached a bottleneck in improving the forecasting performance. Therefore, it is crucial to introduce supplementary data to assist time series data for accurate forecasting.
3. The reason why these supplementary data can further improve the model’s forecasting capability is attributed to its ability to **provide more information that is not included in time series data**, such as relatively accurate predictions of future trends. This information is beneficial for accurately grasping future trends and providing more reasonable forecasts, which are also **more in line with the needs of practical time series forecasting scenarios**. However, traditional time series forecasting models are unable to provide accurate forecasts due to the lack of this information when facing distribution changes caused by external events. 
4. To simulate actual time series forecasting scenarios, we construct future textual data based on the time series in prediction interval, which contains accurate future trend information. It should be emphasized that the future textual data we constructed does not contain any information reflecting the amplitude of the predicted values (see Figures 6 and Figure 7). Moreover, the future textual data we use is a shape-base textual description of time series trend, which is a relatively universal textual description. Any other form of textual data (such as news, weather forecasting reports, etc.) can be summarized into similar textual descriptions through specific methods, which can always be obtained before forecasting. **Therefore, we believe that there is no information leakage from the perspective of the requirements of real-world time series forecasting scenarios.**
</content>
</thread_object>

</thread>


Metareview: