CoMRes: Semi-Supervised Time Series Forecasting Utilizing Consensus Promotion of Multi-Resolution

Authors: Yunju Cho, Jay-Yoon Lee
Primary area: learning on time series and dynamical systems
Keywords: Time series forecasting, Multi-scale, Semi-supervised learning

Abstract:
Long-term time series forecasting poses significant challenges due to the complex dynamics and temporal variations, particularly when dealing with unseen patterns and data scarcity. Traditional supervised learning approaches, which rely on cleaned and labeled data, struggle to capture these unseen characteristics, limiting their effectiveness in real-world applications. In this study, we propose a semi-supervised approach that leverages multi-view setting on augmented data without requiring explicit future values as labels to address these limitations. By introducing a consensus promotion framework, our method enhances agreement among multiple single-view models on unseen augmented data. This approach not only improves forecasting accuracy but also mitigates error accumulation in long-horizon predictions. Furthermore, we explore the impact of autoregressive and non-autoregressive decoding schemes on error propagation, demonstrating the robustness of our model in extending prediction horizons. Experimental results show that our proposed method not only surpasses traditional supervised models in accuracy but also exhibits greater robustness when extending the prediction horizon.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1729488580300

thread_type:
official_review

<thread_object>
creator: reviewer_gung

<content>
summary:
This paper discusses multi-resolution long-term time-series forecasting (LTSF). This work extends the Pathformer architecture with two components: data augmentation, and consensus-based training loss for multiple MLP outputs. Comparison between the ablated components and Pathformer is held on widely used time-series prediction benchmarks. Proposed CoMRes improves over the baselines.  Further analysis is done on resource-efficient case.

strengths:
1) The multi-resolution time-series forecasting task is an important direction.

2) Authors have done an in-depth study on the ablation study.

weaknesses:
1) I am uncertain about the novelty of the proposed paper. This paper borrows most of the architecture from the Pathformer, except the MLP for the individual component output and the aggregation layer. Furthermore, the consensus-based loss that minimizes the relative difference can be naturally formulated while utilizing the ensemble. I feel like this paper is a sheer extension of the Pathformer and the previous methods that utilize data augmentation. 

2) The experiment results do not convince the efficacy of the method. MRes with consensus training often underperform over the MRes baseline. Furthermore, performance differences between the data augmentations are small. This questions the efficacy of each component as a general method. Finally, ComRes drastically underperforms Pathformer in one dataset, which questions the efficacy of the whole algorithm in general.

3) The extra MLP layer may induce an extra computation burden on the training and inference stage, especially since (M+1) calculations are all required for the final computation. Maybe further analysis of the computation efficiency will further help to understand the limitations.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730498260377

thread_type:
official_review

<thread_object>
creator: reviewer_iocg

<content>
summary:
This paper proposes a "consensus promotion" learning objective to enhance the consistency of multi-scale time series data predictions. Additionally, given this learning objective is well-suited for semi-supervised learning, the authors introduce data augmentation strategies aligned with the proposed framework to further improve model generalizability. Extensive experiments are conducted on various datasets and ablation settings.

strengths:
(1) This paper proposes and investigates a novel perspective on improving time-series prediction tasks: the consistency of predictions across different patch sizes of time-series data, which represents an interesting and promising research direction.

weaknesses:
The main weakness of this paper lies in its evaluation. 

(1) The variance of model performance and metrics is not reported for any experimental results, making it difficult for readers to assess whether the minor improvements are consistent and significant or simply due to stochastic gradient descent. The authors are encouraged to report variances for all results and conduct comprehensive statistical tests to demonstrate whether the improvements are statistically significant.

(2) There is no experimental evidence to demonstrate whether the proposed consensus promotion is actually beneficial. For example, what is the exact prediction MSE for each "individual-view model"? Do they actually become more consistent after applying the unsupervised consistency loss? Additionally, in Table 1, if I understand correctly, the only difference between "MRes (SL)" and "MRes w. consensus" is that "MRes w. consensus" includes additional consensus promotion learning objectives. However, there are no consistent patterns indicating which model's MSE is generally smaller. The authors are encouraged to design clear experiments that demonstrate whether the proposed consensus promotion works as expected and whether it is beneficial for the prediction task.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730539621599

thread_type:
official_review

<thread_object>
creator: reviewer_gebu

<content>
summary:
The authors propose CoMRes, a long-term forecasting architecture based on Pathformer, which leverages a consensus promotion mechanism for multi-resolution views. The core idea is to align the supervised predictions of multiple resolutions with the aggregated prediction. Furthermore, they use augmentation strategies (Time Warping, Interpolation and Noise Injection) to construct the different views.  The work also presents an evaluation using autoregressive and non-autoregressive decoding schemes, aiming to show the robustness of the forecasts in longer prediction horizons. The authors claim that CoMRes outperforms traditional supervised models, exhibiting greater robustness and accuracy on multiple popular datasets.

strengths:
- The paper introduces an interesting concept of promoting consistency among multi-resolution views, which has potential in capturing complex temporal relationships.

- The amount of datasets used in the experimental section is exhaustive, which adds impact to the of the experimental results.

- The addition of limited-resource scenarios is valuable, as this is often the case in time series applications.

- In general i think the writing of the paper is good and understandable.

- The discussed related work seems to be exhaustive.

weaknesses:
- The title and abstract do not clearly explain why the proposed approach is semi-supervised. It is challenging to understand how forecasting can be performed in a "semi-supervised" manner, given that "labels" can be constructed by shifting the time window. It would be helpful to provide more concrete explanations on this aspect, particularly regarding the proposed use of augmented data and the challenges with temporal dynamics.

- In lines 031-033, the statement about the superiority of transformers in forecasting is misleading. Multiple recent works (Das et al., 2023; Wang et al., 2024; Zeng et al., 2023) have shown that transformers are not always state-of-the-art. This statement should be reconsidered or at least clarified with respect to specific contexts or benchmarks.

- The comparison against relevant baselines is lacking. The paper does not include benchmarks like TimeMixer, TiDE, PatchTST, D/NLinear, and other advanced transformer-based models (e.g., iTransformer). Including these baselines would significantly strengthen the evaluation and provide a better context for the proposed model.

- The tables, particularly Table 1, are difficult to read. I'm not sure what red or blue means at least its not stated in the caption.

- The baseline method achieves the best result multiple times without the proposed augmentations, which weakens the argument for using the augmentations. In general I'm pretty confused about what the idea of multiple resolutions vs augmentations is in this work.

- While the limited-resource scenario is a useful addition, it does not effectively demonstrate how the method performs on datasets with many variables (e.g., Traffic or Electricity). This should at least be stated in the limitations/conclusion section.

## Refs
Das et al., Long-term forecasting with TiDE: Time-series dense encoder. (TMLR 2023)

Wang et al., TimeMixer: Decomposable multiscale mixing for time series forecasting. (ICLR 2024)

Zeng et al., Are transformers effective for time series forecasting? (AAAI 2023)

questions:
1. In line 032, the paper cites Wang et al., suggesting multi-resolution methods "overlook consistency." Could you provide specific details about what was missed in the multi-resolution reconciliation methods used by Wang et al.?

2. In Section 3, you mention that augmentations are less effective in forecasting compared to classification and anomaly detection. Could you elaborate on why this is the case? Specifically, how does your chosen augmentations differ?

3. Why do you assume a Euclidean distance to measure consensus between the aggregated prediction and each view? Would other measures, such as soft-DTW, work better?

4. Does your framework also work with more "complex" augmentation strategies beyond the three mentioned (time warping, noise injection, interpolation)? Maybe discussing the limitations is a valuable contribution.

5. Could you clarify whether MRes is your method (with ablated components)? I would not count that as a baseline.

6. I'm a bit confused, at what points do you use augmentations, and at what different resolutions (scales)? Could you describe this process in more detail?

7. The paper claims superior robustness compared to traditional supervised learning approaches. Could you clarify how exactly robustness was demonstrated?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659381321

thread_type:
official_review

<thread_object>
creator: reviewer_2k6g

<content>
summary:
This paper addresses challenges in long-term time series forecasting, particularly in managing unseen patterns and data scarcity. The authors propose CoMRes, a semi-supervised, multi-view approach to enhance forecasting by promoting consensus among multiple models on augmented, unseen data. This method aims to improve prediction accuracy and reduce error accumulation over extended horizons. Additionally, the study examines both autoregressive and non-autoregressive decoding schemes, emphasizing the robustness of non-autoregressive decoding in minimizing long-term error propagation. Experimental results highlight CoMRes's superior performance and stability compared to traditional supervised models.

strengths:
The strengths are as follows:

- The paper is clear, well-written and presented.
- The authors cover most of the commonly used datasets in the field, and some relevant baselines: the empirical constributions appear sound as a result.
- Results indicate the method does indeed provide some life over the baselines.

weaknesses:
The weaknesses are as follows:

- Absence of submitted code making reproducibility more difficult.
- The paper could benefit from more detailed comparisons with other time-series forecarsting methods.
- It would be interesting to include error bars on the results table, especially given the proximity of different methods.

questions:
- Could the authors add error bars?
- What are the author's plans wrt. the release of code?

Edit: the authors have addressed the questions and points I raised to a level that I find satisfactory. I am raising my score as a result.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731926107376

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Rebuttal Revisions

comment:
We sincerely thank all the reviewers for their insightful reviews and valuable comments, which are instructive for us to improve our paper further.

This paper proposes **CoMRes**, a semi-supervised framework that leverages consensus promotion on augmented data without requiring explicit future values as labels. Our approach addresses a different aspect of multi-scale (multi-view) modeling and long-term time series forecasting (LSTF) compared to existing studies, **focusing on improving consistency and generalization when encountering unseen data**. Individual-view models often produce divergent predictions due to variability in single-scale representations. To address this, CoMRes enhances agreement among multiple single-view models by leveraging augmented unseen data.

Addressing the effective utilization of unlabeled data in time series forecasting is a relatively unexplored yet crucial area, and we believe our approach paves the way for further research.

We have made every effort to address all concerns by providing detailed clarifications and requested results. Below is a summary of the major revisions:

**1. Main text** - All revisions have been incorporated in the revised paper, with changes highlighted in blue.
- **introduction and related works** : Revised to better articulate our contributions, with changes highlighted in blue.
- **4.1 Time Series Forecasting**: To highlight the primary objective of our experiments
  - Rephrased the "Baseline" section for clarity.
  - Add the "Ablations" section for clarity. 
  - Elaborated on the performance comparisons between CoMRes and the baseline methods in "Results" section and Table 1.
- **4.3.1 Limited-Resource Scenarios** : Add Traffic result to demonstrate how the method performs on datasets with many variables

**2. Appendix** - Following the reviewers' suggestions, we have included the following:
- A.2 Variance of Individual-view Model (**Reviewer iocg**)
- A.4 Error bars (**Reviewer 2k6g, iocg**)
- A.5 Additional baselines (**Reviewer 2k6g, GeBU**)
- A.7 Discussion - More Related Works (**Reviewer GunG**)
- A.8 Discussion - Result on ILI dataset (**Reviewer GunG**)
- A.9 Discussion - Comparison on a Per-Augmentation Basis (**Reviewer GeBU**)
- A.10 Discussion - Computation Cost (**Reviewer GeBU, GunG**)
- A.12 Limitation and Future Work  (**Reviewer GeBU**)

**3. Code for Reproducibility** (Reviewer 2k6g):
The code has been made available at the following anonymous repository: [https://anonymous.4open.science/r/anonymous-9FB7/](https://anonymous.4open.science/r/anonymous-9FB7/)

We sincerely hope that the above revisions adequately address the reviewers' concerns and provide clarity, thereby strengthening the soundness and highlighting the contributions of our paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733237048497

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response and Revision Summary

comment:
We sincerely thank all the reviewers for their thoughtful feedback and constructive comments. Your insights have been invaluable in helping us refine and improve the quality of our submission.

We greatly appreciate the reviewers’ positive recognition of our work. CoMRes has been acknowledged for introducing an interesting and **novel perspective of promoting consistency** among multi-resolution views, **representing a promising direction in time-series forecasting (GeBU, iocg)**. The paper addresses **an important direction of multi-resolution time-series forecasting(GunG)**, a critical area for advancing temporal modeling techniques. The sound empirical contributions, **in-depth ablation studies, extensive datasets, and inclusion of limited-resource scenarios(2k6g, GeBU, GunG)** further provide valuable insights into the method’s effectiveness, enhancing credibility and practical value for real-world applications. Additionally, we are pleased that **the clarity and presentation of our work were well-received. (2k6g, GeBU)**.

In response to the insightful and constructive concerns raised, we have made every effort to address them comprehensively and incorporated all suggested improvements. Key revisions include:
- **Enhancing Evaluation and Analysis:**
  - Added comparisons with additional baselines to contextualize current progress (Appendix A.5) (2k6g, GeBU).
  - Included error bars and statistical significance tests to rigorously evaluate performance variability (Appendix A.4) (2k6g, iocg).
  - Conducted variance analysis of multi-view models to examine the effectiveness of the unsupervised loss function (Appendix A.2) (iocg).
  - Detailed per-augmentation analysis to address varying augmentation strategies (Appendix A.8) (GeBU).
  - Expanded discussion on computation costs, including runtime comparisons, to clarify computational efficiency (Appendix A.10) (GeBU, GunG).
  - Added results on the Traffic dataset to demonstrate performance on datasets with many variables (Section 4.3.1, Limited-Resource Scenarios) (GeBU).
- **Clarifications:**
  - Revised the introduction and related works section to articulate contributions more effectively (GeBU, iocg, GunG).
  - Expanded the discussion of related works in Appendix A.7 (GunG).
  - Rephrased the “Baseline” section in Section 4.1, added an “Ablations” section, and elaborated on performance comparisons between CoMRes and ablations in the "Results" section and Table 1 (GeBU).
  - Included discussions on the ILI dataset (Appendix A.8) and limitations/future work (Appendix A.12) to address additional feedback (GeBU, GunG).

All revisions have been highlighted in blue in the revised manuscript. **We are sincerely grateful for the reviewers’ constructive engagement, which has greatly enhanced the quality of this work.** We appreciate the reviewers' recognition of our efforts and increased scores reflecting the improvements.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces CoMRes, a semi-supervised time series forecasting framework that promotes consensus across multi-resolution augmented data views without requiring explicit labels. The authors claim this method improves forecasting accuracy and robustness, particularly in long-term predictions, while mitigating error accumulation. Strengths include a novel application of consensus promotion to time series forecasting, robust empirical evaluation across diverse datasets, and effective handling of limited-resource scenarios. However, weaknesses include limited methodological novelty as the framework adapts existing approaches like Pathformer, and some reviewers raised concerns about computational overhead and insufficient clarity on the generalization of augmentations. Missing elements include direct comparisons with advanced baselines and deeper analysis of computational trade-offs. While the method demonstrates practical utility and robust results, the concerns about novelty and computational efficiency contributed to mixed reviews, but overall improvements during the rebuttal and strong empirical performance led to a borderline acceptance recommendation.