Title: TimePerceiver: An Encoder-Decoder Framework for Generalized Time-Series Forecasting
Venue: NeurIPS 2025
Accepted: yes
Keywords: Time Series Forecasting

Transformer

Attention

Self-Supervised Learning
Abstract: In machine learning, effective modeling requires a holistic consideration of how to encode inputs, make predictions (i.e., decoding), and train the model. However, in time-series forecasting, prior work has predominantly focused on encoder design, often treating prediction and training as separate or secondary concerns. In this paper, we propose TimePerceiver, a unified encoder-decoder forecasting framework that is tightly aligned with an effective training strategy. To be specific, we first generalize the forecasting task to include diverse temporal prediction objectives such as extrapolation, interpolation, and imputation. Since this generalization requires handling input and target segments that are arbitrarily positioned along the temporal axis, we design a novel encoder-decoder architecture that can flexibly perceive and adapt to these varying positions. For encoding, we introduce a set of latent bottleneck representations that can interact with all input segments to jointly capture temporal and cross-channel dependencies. For decoding, we leverage learnable queries corresponding to target timestamps to effectively retrieve relevant information. Extensive experiments demonstrate that our framework consistently and significantly outperforms prior state-of-the-art baselines across a wide range of benchmark datasets.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new attention based encoder-decoder architecture along with a new training regime to unify the development of encoder and decoder as a single task (Generalized Formulation). This helps the encoder to attend to multiple arbitrary sampled segments of the input. Their experiments show that they outperform various (transformer focused) baselines on the standard multivariate long-term forecasting benchmark datasets. Furthermore, the authors propose a novel latent bottleneck  that compresses input patches into a small set of learnable tokens to improve from O(n^2) to O(NM), with M << N.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths:\n    \n    - The paper introduces a novel unified encoder-decoder framework and an interesting idea of sampling multiple temporal segments.\n        \n    - The figures are clear and aid understanding of the architecture\n    - The manuscript is easy to follow.\n    - The experiments are extensive, although focused on long-term forecasting.\n        \n    - The authors outline research questions, which improves the structure.\n        \n- Weaknesses:\n    \n    - The authors omit recent architectures (e.g., CNN- and MLP-based models such as CycleNet, TimeMixer, TimeMixer++, and ModernTCN), as well as entire model families (RNNs) and newly introduced (x)LSTMs (xLSTM-Mixer, Tirex, SegRNN), all of which achieve SOTA results in time series forecasting.\n        \n    - Masking and reconstruction tasks are common pretraining strategies but not the only ones; prior time series pretraining works include TS-TCC, TFC, TS2Vec, and others that utilize multiple self-supervised losses.\n        \n    - The significance and novelty of the architecture are unclear, as the work appears to build directly on previous pretraining methods while ignoring modern foundation models that already use an encoder-decoder architecture (for example, Lag-Llama and Chronos on top of the T5 architecture). In general, the related work section is not exhaustive. The authors should include foundation models such as TimesFM (decoder-only) and Chronos or Chronos-Bolt (patch-based), as well as some of the earlier time series pretraining works.\n        \n    - While the unified view is appealing, it is confusing how this relates to general time series model analysis; if a unified view is employed, methods from imputation should also be included in the evaluation or the related work section.\n        \n    - The experiments over different input lengths are appreciated, but results without multiple seeds are less meaningful.\n        \n    - The notation for the generalized framework is confusing; equations 5–7 require multiple readings to understand.\n        \n    - The number of latent tokens is not sufficiently explored; reporting that 8, 16, and 32 tokens work best is not enough. What happens when decreasing or increasing this number? From a practical perspective, it is unclear how to choose this hyperparameter.\n\n\n### Refs\n\nKraus, Maurice, et al. \"xLSTM-Mixer: Multivariate Time Series Forecasting by Mixing via Scalar Memories.\" _arXiv preprint arXiv:2410.16928_ (2024).\n\nAuer, Andreas, et al. \"TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning.\" _arXiv preprint arXiv:2505.23719_ (2025).\n\nLin, Shengsheng, et al. \"Segrnn: Segment recurrent neural network for long-term time series forecasting.\" _arXiv preprint arXiv:2308.11200_ (2023).\n\nEldele, Emadeldeen, et al. \"Time-series representation learning via temporal and contextual contrasting.\" IJCAI (2021)\n\nZhang et al. \"Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency\" NeurIPS (2022)\n\nYue, Zhihan, et al. \"Ts2vec: Towards universal representation of time series.\" _Proceedings of the AAAI conference on artificial intelligence_. Vol. 36. No. 8. 2022.\n\nAnsari, Abdul Fatir, et al. \"Chronos: Learning the language of time series.\" _arXiv preprint arXiv:2403.07815_ (2024).\n\nDas, Abhimanyu et al. \"A decoder-only foundation model for time-series forecasting\" ICML (2024)\n\nRasul, Kashif, et al. \"Lag-llama: Towards foundation models for time series forecasting.\" _R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models_. 2023.\n\nLin, Shengsheng, et al. \"Cyclenet: Enhancing time series forecasting through modeling periodic patterns.\" _Advances in Neural Information Processing Systems_ 37 (2024): 106315-106345.\n\nWang, Shiyu, et al. \"Timemixer: Decomposable multiscale mixing for time series forecasting.\" ICLR (2024).\n\nWang, Shiyu, et al. \"Timemixer++: A general time series pattern machine for universal predictive analysis.\" ICLR (2025).\n\nLuo, Donghao, and Xue Wang. \"Moderntcn: A modern pure convolution structure for general time series analysis.\" ICLR (2024).\n\nQUESTIONS:\n- Q1. How does your architecture differ from T5? Please discuss how your generalized formulation can be projected to other architectures (or is this impossible?).\n    \n- Q2. In Table 2, could the authors provide all combinations? It is unclear what impact positional embedding sharing has on the standard formulation.\n    \n- Q3. While the focus is on transformer-based methods, a global view should include other architectures. Please provide results for modern RNN methods (xLSTM-Mixer), MLPs (CycleNet, TimeMixer, TimeMixer++), and convolutional models (ModernTCN).\n    \n- Q4. In your ablation, you report that ETTh1 works best for contiguous sampling. Could the authors provide a rationale? Simply stating that this highlights the importance of sampling ratio is insufficient, especially since one of the main claims is that previous pretraining methods do not model this behavior correctly.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author propose TimePerceiver, a generalized Transformer structure for time series forecasting. The authors propose three key innovations: latent bottleneck representations-based encoder, learnable query-based decoder and a hybrid architecture supporting flexible sequence length forecasting. \n\nExperiments on benchmarks (e.g., ETT, Traffic, Weather) demonstrate state-of-the-art (SOTA) performance.The model exhibits strong generalization and deterministic forecasting capabilities.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.Well-written and clearly presents its contributions. 2. Interesting approach, and experimental results are soundness. 3. The analysis and discussion are convincing. \n\nWeaknesses:\n1. The novelty is weak, since the latent bottleneck representations of encoder is introduced in time series forecasting, “Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting” (ICLR2023). Also, there is no theoretical proofs for the bottleneck representation, please add. Therefore, the novelty is the learnable query-based decoder.\n2. As a generalized forecasting model, some generalized baselines should compare, \"One Fits All: Power General Time Series Analysis by Pretrained LM\", \"TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis\" and \"UniTS: A Unified Multi-Task Time Series Model\"\n3. In the learnable query of decoder, did the author consider adding covariates corresponding to the prediction target to enhance the expressiveness of the query? \n4. Since the author states that TimePerceiver can encompasses extrapolation, interpolation, and imputation tasks along the temporal axis in lines 54-56, could the author do some comparisons within these tasks, \"Optimal Transport for Time Series Imputation\", \"Csdi: Conditional score-based diffusion models for probabilistic time series imputation\".\n\nQUESTIONS:\nPlease refer to weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a fresh approach to tackling a variety of time-series forecasting challenges—not just predicting the future, but also filling in gaps and making sense of missing data. Unlike older methods that mostly focus on just the encoder part of the model, TimePerceiver brings the encoder, decoder, and training process together into one unified system. Its design uses clever attention mechanisms to understand complex patterns across both time and different data channels, and the decoder is flexible enough to handle any target points you care about. Instead of relying on a two-step training process, TimePerceiver is trained end-to-end, which streamlines things and helps it perform better. The authors back up their claims with thorough experiments, showing that their model consistently outperforms other top methods. Overall, the paper makes a strong case for rethinking how we design and train models for time-series forecasting by taking a broader, more integrated approach.\n\nSTRENGTHS AND WEAKNESSES:\nThis authors present how it brings together different parts of time-series forecasting—encoding, decoding, and training—into one well-designed, unified approach. The authors use attention mechanisms and bottleneck representations to help the model capture complex patterns, and they back up their claims with thorough experiments and diagrams that make even the vague concepts easier to understand. The authors expand the scope of forecasting to include not just predicting the future, but also filling in gaps and handling missing data, making their method useful for a wide range of real-world problems. The way the authors combine existing ideas in a new way shows real innovation and could have an impact across fields like health, finance, and climate science.\n\nOn the other hand, the model’s complexity could be hard to use in situations where resources are limited or real-time results are needed, and the paper doesn’t discuss these practical challenges. While the framework’s generality is a good idea, it would be more convincing if the authors gave more examples of where all these forecasting tasks are needed in the real world. Lastly, many of the individual building blocks come from other areas, so the true originality is in the combination rather than the components themselves.\n\nQUESTIONS:\n1. What are the computational costs (training time, memory footprint) of TIMEPERCEIVER compared to the baselines, especially given the dual attention mechanisms and bottleneck structure?\n\nSuggestions: Please include a runtime and memory usage comparison across different sequence lengths. This is important to assess the trade-off between performance and scalability.\n\n2. Under what conditions does TIMEPERCEIVER struggle? Are there failure cases or performance drop-offs in certain data regimes (e.g., small datasets, irregular sampling)?\n\nSuggestions: Discuss known limitations and include a sensitivity analysis or case examples showing where the model might be less effective.\n\n3. How does each architectural component (e.g., latent bottlenecks, learnable queries, self-attention in bottlenecks) individually contribute to performance?\n\nSuggestions:  Please clearly  discuss each innovation's role. For example, what is the performance delta when removing learnable queries alone?\n\n4. Can the authors elaborate on specific real-world scenarios where the simultaneous handling of extrapolation, interpolation, and imputation is essential?\n\nSuggestions:  Add at least one case study or detailed scenario (e.g., in health, finance, or sensor networks) where this generalized forecasting capability provides a clear advantage over traditional models.\n\nLIMITATIONS:\nThe authors do not address the limitations of the proposed framework. While the technical contributions are clearly presented, the following points  should be addressed:\n\na) The paper should explicitly discuss scenarios where TIMEPERCEIVER might fail or underperform, such as:  sparse time-series data, extremely long sequences where attention-based architectures struggle with memory and speed.\n\nb) The authors should comment on the training and inference costs. High computational demands can have environmental and accessibility implications.\n\nc) If deployed in domains like health or finance, biases in training data could propagate through the model. The authors should acknowledge these risks and suggest monitoring or mitigation strategies.\n\nd) If used in sensitive settings (e.g., medical records, financial logs), how does the model handle privacy risks, particularly if used for imputation of missing data?", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents **TimePerceiver**, a unified encoder-decoder framework for multivariate time series forecasting. The core idea is to generalise the training task beyond standard extrapolation tasks to include **interpolation** and **imputation** for arbitrary missing input.  \n\nTo that end, the authors propose a Transformer-based model using patch-based tokenisation and cross-variant attention. First, the context patches are transformed into a token representation augmented with temporal (patch position) and channel (index of dimension) information. The encoder compresses context patches into latent tokens (with size and dimensions to be chosen) and reprojects the contextualised input tokens into the original token space. The decoder uses tokens only based on temporal and channel information and uses cross-attention to transform these tokens into output tokens that are projected back into signal space.\n\nThe model is evaluated across 8 datasets with multiple forecasting horizons and consistently demonstrates strong performance, often achieving state-of-the-art or near-best results, with favourable computational and parameter efficiency. Furthermore, the authors show through ablation studies the benefits of their choices and show that the seasonality is recovered through the cross attention patterns.\n\nSTRENGTHS AND WEAKNESSES:\n- **Originality**:  While many of the ideas exist in other works: related ideas for the pretraining (e.g., masking-based pretraining) ([1,2]) and of having per channel and per patch attention by explicitely tokenizing them individually [3],  to the best of my knowledge, this is the first model not doing it in a two steps setting while limiting the computation and number of parameters needed. Thus, the learnable latent bottleneck is an interesting addition. \n\n- **Quality**:  \n  - (+)The model is built on clear motivations, and the experimental results are convincing. \n  - (-) Since the results were averaged over 5 runs, it could have been interesting to show the standard deviation of the results to show the statistical significance of the results (though it is not standard in the field).\n  - (+) The ablation study is compelling to experimentally show the benefits of their methods.\n  - (+) The visual analysis of the cross-attention map recovering the seasonality of the TS is an interesting addition. \n  - (-) Multiple latent sizes are tested, I suppose the choice is based on performance, but the procedure is not detailed. Studying the sensitivity of the model to this parameter should be included in the study. Furthermore, the reprojection in the original size is not motivated either.\n  - (-) Since the method was trained on an extended task able to handle imputation and interpolation, it could have been interesting to evaluate the performance on such tasks. \n\n\n- **Clarity**:  \n    The paper is well-structured with intuitive visualisations (e.g., Figures 1 and 2) that effectively explain the generalised forecasting setup and encoder-decoder flow. \n    \n- **Significance**:  Though many ideas seem to have been around already, this architecture with a learnable latent bottleneck allowing faster computation seems of interest for the community. Furthermore, the simplicity of the architecture, the very compelling results on different datasets and the superiority of the method against many SOTA methods make this contribution significant.\n\n\n    [1]|Dong, J., Wu, H., Zhang, H., Zhang, L., Wang, J., & Long, M. (2023). Simmtm: A simple pre-training framework for masked time-series modeling. _Advances in Neural Information Processing Systems_, _36_, 29996-30025.\n\n    [2] Kollovieh, M., Ansari, A. F., Bohlke-Schneider, M., Zschiegner, J., Wang, H., & Wang, Y. B. (2023). Predict, refine, synthesize: Self-guiding diffusion models for probabilistic time series forecasting. _Advances in Neural Information Processing Systems_, _36_, 28341-28364.\n\n    [3] Zhang, Y., & Yan, J. (2023, May). Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In _The eleventh international conference on learning representations_.\n\nQUESTIONS:\n- The decoder operates by projecting contextualised query tokens back into full patch outputs. It remains unclear whether bypassing re-expansion to full resolution (i.e., using latent tokens directly) might yield a more efficient architecture. Have the authors tried directly using the encoded latent token as input for the decoder? If so, how much was the performance degrading?\n- I am not sure I understand the difference between interpolation and imputation in the description of the project.\n- I am not fully convinced that this framework would be able to directly handle arbitrary time segments as is, as this would blur the captured seasonality. I think additional information on the size of the patch should be integrated into the representation, perhaps through the same system as temporal and covariate index information. Have the authors tried anything in that regard?\n\nLIMITATIONS:\nThe limitations of the models are correctly addressed but the statistical significance could be improved by including the standard deviation of the 5 runs of the experiments.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}