Generalizable autoregressive modeling of time series through functional narratives

Authors: Ran Liu, Wenrui Ma, Ellen L. Zippi, Hadi Pouransari, Jingyun Xiao, Christopher Michael Sandino, Behrooz Mahasseni, Juri Minxha, Erdrin Azemi, Eva L Dyer, Ali Moin
Primary area: generative models
Keywords: autoregressive modeling, transformers, time series

Abstract:
Time series data are inherently functions of time, yet current transformers often learn time series by modeling them as mere concatenations of time periods, overlooking their functional properties. In this work, we propose a novel objective for transformers that learn time series by re-interpreting them as temporal functions. We build an alternative sequence of time series by constructing degradation operators of different intensity in the functional space, creating augmented variants of the original sample that are abstracted or simplified to different degrees. Based on the new set of generated sequence, we train an autoregressive transformer that progressively recovers the original sample from the most simplified variant. Analogous to the next word prediction task in languages that learns narratives by connecting different words, our autoregressive transformer aims to learn the Narratives of Time Series (NoTS) by connecting different functions in time. Theoretically, we justify the construction of the alternative sequence through its advantages in approximating functions. When learning time series data with transformers, constructing sequences of temporal functions allows for a broader class of approximable functions (e.g., differentiation) compared to sequences of time periods, leading to a 26$\%$ performance improvement in synthetic feature regression experiments. Experimentally, we validate NoTS in 3 different tasks across 22 real-world datasets, where we show that NoTS significantly outperforms other pre-training methods by up to 6\%. Additionally, combining NoTS on top of existing transformer architectures can consistently boost the performance. Our results demonstrate the potential of NoTS as a general-purpose dynamic learner, offering a viable alternative for developing foundation models for time series analysis.

Decision: reject
Reviewer scores: 8, 3, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1730428668839

thread_type:
official_review

<thread_object>
creator: reviewer_3cgn

<content>
summary:
This paper proposes NoTS, a novel autoregressive pre-training method for time series. NoTS reinterprets time series as sequences of temporal functions generated by applying degradation operators of varying intensity levels to the original time series. The authors propose a "next-function prediction" task analogous to the next-word prediction task in language modelling and demonstrate the superior performance of NoTS on synthetic and real-world datasets.

strengths:
* The proposed approach of next function prediction for time series pretraining is novel and very interesting. The authors motivate their method very well. The authors bring a very different perspective for modelling time series.
* The implementation of their approach with degradation operators is also well-theoretically justified. The authors theoretically show that degradation operators avoid the problem of discontinuities when time series are broken up into functions.

weaknesses:
* Clarity: I am unable to understand many parts of the paper due to unclear presentation; especially relating to the empirical evaluation of the method.  

* Evaluation: The protocol used for evaluation in Sec 5.2 is very much unclear. There are several references to Wu et al (2022) but the authors should not expect a reader to look into another paper, especially for the protocol of experimental setup, and the paper should be self-contained. I don't understand the protocol used even after looking into the paper. Do you pretrain on another dataset? If so, which dataset?

* Insufficient details on the training time, cost etc: It would be important to add the cost of using the architecture, both during pretraining and finetuning. In terms of training time / inference time / number of parameters etc.

questions:
* Line 104: "giving a consistent performance boost when performing dataset-specific pre-training" - I don't understand what is dataset-specific pretraining. I have only heard of pretraining on a large dataset, and then dataset specific adaptation. Please clarify.

* Line 123: "trained while maintaining 82% average performance" - average performance on what? is it the pretraining dataset? it should be made clear.

* Line 294: "In time series domain, it is common to encounter datasets with new channel graphs or new tasks at test time. To apply the pre-trained weights in such situations, we build two types of adaptors" - you should give evidence (cite a paper) or provide an example for the first sentence. I don't understand what is meant by "encountering new channels at test time". 

* Line 303: "We initialize and append prompt tokens to the transformer architecture along with data tokens following the deep visual prompt tuning plan as detailed in Jia et al. (2022)". Again, the authors should not expect a reader to look into another paper for such a crucial aspect of the model. I did not understand this part; the authors must explain the mechanism better.

* How sample-efficient is your model (does it do well few-shot)? Experiments like these can potentially substantiate your contribution.

* Why not test on the forecasting task too, which is, equally or maybe even more important than classification/imputation tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730510671980

thread_type:
official_review

<thread_object>
creator: reviewer_fbrg

<content>
summary:
The paper presents a novel approach to time series analysis by treating data as functions of time. It introduces a pre-training framework called NoTS, which uses degradation operators to create augmented data, allowing an autoregressive transformer to learn interrelationships among functions. Experimental results show that NoTS outperforms existing methods across various tasks on synthetic and real-world datasets. Additionally, a lightweight model, NoTS-lw, is introduced, achieving high performance with minimal training parameters. Overall, the work emphasizes the benefits of a functional perspective in enhancing time series modeling.

strengths:
The method shifts the focus from traditional time series modeling to viewing data as functions of time, which enhances the understanding of temporal dynamics. The NoTS framework effectively utilizes degradation operators to create augmented data, improving the learning process for autoregressive transformers.

weaknesses:
While the paper demonstrates the effectiveness of the NoTS framework across various tasks and datasets, the experimental validation is primarily focused on a limited number of synthetic and real-world datasets. This raises questions about the generalizability of the findings. The performance of the proposed method in more diverse and complex scenarios, such as high-dimensional time series or those with significant noise, remains untested. Expanding the range of datasets and tasks could provide a more comprehensive evaluation of the model's robustness and adaptability; The introduction of degradation operators and the functional perspective may complicate the implementation of the NoTS framework. Practitioners may find it challenging to adapt the proposed methods to their specific use cases without a deep understanding of the underlying theory. This complexity could limit the accessibility of the framework for researchers and practitioners who are not well-versed in advanced time series modeling techniques, potentially hindering its adoption in real-world applications; Although the paper discusses the scalability of the NoTS-lw model, it does not provide extensive insights into how the framework performs with significantly larger datasets or more complex models. The scalability of the approach in terms of computational efficiency and memory usage is crucial for practical applications, especially in industries dealing with large-scale time series data. Without thorough analysis and benchmarks on scalability, it is difficult to ascertain whether the proposed methods can be effectively utilized in resource-constrained environments or for large-scale deployments.

questions:
1, How does the NoTS framework compare to traditional time series modeling techniques in terms of interpretability and ease of use for practitioners who may not have a strong background in advanced machine learning methods?

2, What specific strategies could be implemented to enhance the scalability of the NoTS framework for large-scale time series datasets, and how might these strategies impact the performance and computational efficiency of the model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730511031777

thread_type:
official_review

<thread_object>
creator: reviewer_wkqh

<content>
summary:
This paper introduces NoTS, a novel pre-training framework that reinterprets time series data through a functional lens. The framework constructs alternative sequences through two types of degradation operators: local smoothing (using averaging kernels with lengths $p_k$) and global smoothing (using sinc functions with frequency cutoff values $0.5p_k$), which create progressively simplified variants of the original signal while preserving functional properties.

The pre-training works by training an autoregressive transformer to recover increasingly detailed versions of the signal. A channel-independent encoder first converts each degraded signal $S_k$ into token groups $R_k$, then the transformer predicts the next resolution level using previous levels, with a decoder reconstructing the predicted signals. The training combines an AR reconstruction loss comparing $S'_{k+1}$ with $S_k$ and a latent consistency term for the original signal.

For real-world deployment, NoTS uses channel adaptors (linear layer, retrainable channel tokens) and task adaptors (prompt tuning, task-specific linear layers) requiring <1% new parameters while maintaining 82% performance. Experiments across 22 datasets spanning classification, anomaly detection, and imputation tasks are evaluated.

strengths:
1. Reformulates time series modeling from a functional perspective and introduces degradation operators to create multiple views.
2. Proposes autoregressive modeling between resolution levels instead of traditional next-time-point prediction.
3. Provides function approximation analysis for transformer limitations on time series.

weaknesses:
1. Limited Scope of Evaluation
- While the paper's focus on generalizable time series modeling, the crucial forecasting task is missing. The next-time-point prediction paradigm has clear connections to dynamic evolution, but the benefits of resolution-based prediction for forecasting tasks remain unexplored. This gap makes it difficult to fully assess the method's practical advantages.
- Performance improvements are moderate except for classification tasks.
- Limited datasets for imputation task. The ETT datasets are homogenous, and the model’s performance should be evaluated across other commonly used datasets as well, like Electricity and Weather.
- Limited ablation studies on key components. It would be better to show the impact of different degradation operators, degradation levels, and frequency cutoff points $p_k$ and diverse real-world datasets as well.
2. Minor parts:
- Technical clarification is needed on loss alignment choices, embedding usage for downstream tasks, and prompt tuning plan.

questions:
1. Which embeddings from the pre-trained model are used for downstream tasks? 
2. While following Jia et al. (2022), the adaptation of visual prompt tuning to time series data requires a more detailed explanation. Time series-specific prompt token structure and organization should be elaborated.
3. Please justify the alignment of $S_{k+1}'$ with $S_k$ in the loss function rather than $S_{k+1}$. The alternative of matching with $S_{k+1}$ should be discussed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730600460922

thread_type:
official_review

<thread_object>
creator: reviewer_dxpz

<content>
summary:
This paper proposes a pre-training method for time series data called Narratives of Time Series (NoTS). NoTS frames time series modeling by considering time series samples as a sequence of functions rather than a sequence of time periods. NoTS utilizes degradation functions to generate augmented variations of a time series signal. These augmented signals have varying degrees of simplification and together form a sequence that is used to train an autoregressive transformer. The authors provide theoretical justification for framing time series as functional sequences, showing that this approach can approximate a broader class of functions compared to traditional segmentation into periods. The authors validate NoTS on a variety of tasks, including classification, anomaly detection, and imputation, using both synthetic and real-world datasets.

strengths:
- Originality: the notion of "Narrative of Time Series" by predicting the "next function" in a functional decomposition of the time series is interesting and modestly novel. However, it is strongly patterned after the N-BEATS decomposition (see below), and links and connections to this earlier literature should be made.
- Quality: The experimental results are interesting, although the validity of results is called into question (below).
- Clarity: The paper is at times hard to follow, in particular with respect to notational sloppiness and shortcuts taken in detailing the model.

weaknesses:
- Eq. 1 is sloppily written for the case $i=1$
- Section 2.1: there should be more rigor in the notation. The paper mixes discrete and continuous time, and does not specify how discrete time indices map onto functional arguments in continuous time. For instance, the notion of a "calendar" containing this mapping could be introduced. 
- Since the functional representation of time series is so important to the paper, Section 2 lacks a survey of the rich literature on the topic, e.g. from the Gaussian processes literature (e.g. https://proceedings.neurips.cc/paper/2007/hash/81e74d678581a3bb7a720b019f4f1a93-Abstract.html ) and classical statistics (e.g. “Functional Data Analysis” by J.O. Ramsay and B.W. Silverman (1997)).
- The proposed decomposition is strongly reminiscent of the N-BEATS approach, which is not acknowledged. The paper should contrast and compare their proposed approach to N-BEATS: 
```
@inproceedings{
Oreshkin2020N-BEATS:,
title={N-BEATS: Neural basis expansion analysis for interpretable time series forecasting},
author={Boris N. Oreshkin and Dmitri Carpov and Nicolas Chapados and Yoshua Bengio},
booktitle={International Conference on Learning Representations},
year={2020},
url={https://openreview.net/forum?id=r1ecqn4YwB}
}
```

questions:
- Line 142: "Typically, each dataset has its unique channel-wise relationships" ==> not clear what this sentence means.
- If the discrete-time signal is obtained by sampling at irregular intervals or contains missing data, discuss how convolution operations would work for the degradation function.
- Line 256: "encoder produces groups of tokens from each signal" ==> not clear how the signal is chunked into tokens in the first place. Is it through a PatchTST-like mechanism, or some other approach? 
- Lines 264-265: The specifics of this model should be given (perhaps in appendix), or a citation provided. The overall data flow, ranging from tokenization, encoder, decoder is very hand-wavy and hard to follow.
- Lines 282-283: The argument made to explain the addition of the latent consistency term is not clear and should be further explained.
- Section 3.3, test-time adaptors: it is not clear whether these adaptors enable zero-shot handling of new time series, or if adaptor fine-tuning is required.
- Eq. (3) specifies that the absolute error is used as the training loss. This makes the model sensitive to the scale of the time series — if the series are not scale-normalized, training will overweight series with the largest absolute magnitude. Further, section 5.2 does not discuss this issue, which calls into question the validity of experimental results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730615865753

thread_type:
official_review

<thread_object>
creator: reviewer_ttya

<content>
summary:
This paper presents Narratives of Time Series (NoTS), a transformer-based framework that reinterprets time series data as temporal functions, capturing both local and global properties of time series through a novel pre-training objective. The authors introduce three main innovations:

Progressive Functional Degradation: NoTS generates simplified versions of time series by applying degradation operators that progressively smooth the signal. Each degradation level creates a coarser function, enabling the model to learn time series as a sequence of simplified functions rather than individual time points, preserving the functional structure over time.

Autoregressive Transformer with Next-Function Prediction: Inspired by next-word prediction in NLP, NoTS trains the transformer to reconstruct the original time series from its degraded variants. By predicting the next function in a sequence, each step adds more information, allowing the model to capture nuanced temporal dependencies across multiple scales.

Adaptation via Channel and Task Adaptors: NoTS includes adaptable modules to handle different time series tasks and varying channel structures, which improves model generalization and reusability across datasets. These adaptors support context-aware adaptation and help the model perform well with minimal parameter updates, making it versatile across tasks.

Experiments and Analysis:
The authors begin by presenting a theoretical analysis that shows standard transformers struggle to generalize when applied to time series tasks that involve complex operations, such as differentiation, due to their reliance on discretized data. Specifically, the authors prove that transformers encounter high approximation errors when attempting to model certain mathematical operations (e.g., differentiation) on time series. Authors also prove that their proposed method viewing time series data as continuous functions in "function space" rather than as discrete can enable the transformer to better capture the underlying properties of the time series.

NoTS is evaluated on both synthetic and real-world time series datasets, covering tasks such as classification, imputation, and anomaly detection. NoTS demonstrates strong performance across diverse datasets, outperforming established baselines such as masked autoencoders and next-period prediction models. Ablation studies further validate the importance of each component, and scalability experiments indicate that NoTS’s performance improves predictably with model size, in line with power-law trends seen in other fields like NLP.

strengths:
- The borrowed functional narrative approach is innovative for time series modality, addressing key limitations in time series modeling due to discretization. By leveraging function-based degradations, NoTS effectively captures temporal relationships, making it a strong framework for autoregressive analysis.

- Clear theoretical grounding in approximation theory supports the functional degradation and autoregressive prediction framework. Visualizations (e.g., degradation sequences) help convey complex concepts effectively.

- Experimental results show NoTS’s robustness across diverse tasks and datasets, with context-aware adaptors allowing minimal retraining. These adaptors facilitate handling different channel mappings and task-specific nuances, enhancing practical application.

weaknesses:
- The paper lacks an analysis of how sensitive NoTS is to the choice of the degradataion operators or guidance on selecting appropriate ones. This dependence could impact performance if the degradations are not well-suited to certain datasets or applications. The ablation study could be improved by including more ablation tests on this aspect.

- While NoTS is evaluated on classification, anomaly detection, and imputation, it lacks experiments on long-horizon forecasting tasks, which are crucial for assessing a model’s ability to capture long-range dependencies and temporal features (as highlihted in the intro). This addition would also reinforce the paper’s motivation, presented in the introduction, which critiques patch-based transformer architectures focused on forecasting tasks (e.g., Moirai and Lagllama).

questions:
- Could the authors provide an analysis on how sensitive NoTS is to the choice of degradation operators? Specifically, how does the performance vary when different types of degradation operators are used, or when the balance between local and global smoothing is adjusted?
- Is there any guidance the authors could offer on selecting suitable degradation operators for different datasets or applications? A more detailed ablation on this aspect would clarify the robustness of NoTS across varying conditions.
- Is there any limitations sourced from the pretraining objective that stops authors testing on the forecasting task? If not do the authors plan to evaluate NoTS on long-horizon forecasting tasks? Given that capturing long-range dependencies is highlighted as a core strength in the introduction, an assessment on forecasting (especially long-range) would provide valuable insights.
- Could the authors provide an analysis of how NoTS’s performance contribution varies with increasing time series lengths? Understanding how NoTS scales with longer time series would be valuable for assessing its capability to capture long-range dependencies and its potential applications to extended temporal data.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732774653285

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response #1 - Scope of the work

comment:
We sincerely thank the reviewers for their valuable feedback.

We are especially encouraged by reviewers’ recognition of the innovation and theoretical results provided by the work. We are particularly grateful to reviewer tTyA for highlighting our contribution as **innovative, addressing key limitations**, reviewer 3Cgn for describing our work **novel, very interesting, bring a very different perspective**. Even reviewers with critical feedback summarized our work as novel (WkQH and fbrG) or interesting (DXpz). Additionally, we really appreciate the positive remarks from reviewer tTyA and 3Cgn on the theoretical rigor of our approach, describing our work as **theoretically grounded** and **well-theoretically justified**.

## Scope of the work

Before presenting additional experimental results, we would like to address some major misunderstandings about the scope and central ideas of our work as follows:

- **The use and analysis of transformers are integral to our work** and represent a non-neglectable contribution, which distinguish our work from previous works. Building on the idea that transformers capture relationships in sequences, our paper proposes and validates the idea that the sequential input to the transformer could be constructed as an alternative functional form, rather than simply segmenting time series into patches. 

- In this work, we propose the above idea, theoretically justify it by analyzing transformers’ capacity to approximate functions when constructing different sequences, directly demonstrate its advantages through synthetic experiments for feature approximation, and further validate it on real-world datasets across diverse tasks.

- We have stated that the experimental results on real-world datasets are **an initial attempt** to demonstrate the idea’s feasibility in real life, as noted both in the experimental section (line 518-520) and the limitation section (line 536-538). **In the revised manuscript, we have further tone down the contribution claims in the abstract and introduction**. Our focus is on validating the idea in a theoretically grounded way, rather than conducting large-scale or scaled-up experiments. We believe the current results, combined with theoretical analysis, sufficiently justify the method. While we believe the proposed method provides a viable alternative for building time series foundation models, the goal of this work is not to develop a ChatGPT-like product or similar large-scale application.

- Our work focuses on time series analysis, with the autoregressive objective operating in the functional domain as the training objective. **Forecasting tasks are not the primary focus of this study**, nor do we place significant emphasis on them. While forecasting tasks are currently popular in the community, we believe that identifying suitable time series datasets for forecasting requires a separate, comprehensive investigation, which is beyond the scope of this paper.

We have revised the introduction and abstract sections in the manuscript to help clarify the above misunderstandings for future readers.

In the following responses, we address the key criticisms by providing:
1. Three sets of additional ablation results.
2. Initial forecasting results comparing our next-function prediction with traditional next-patch prediction pre-training methods.
3. Clarifications and detailed descriptions of our experimental setup.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes NoTS, a transformer-based method for pretraining time series data by reinterpreting them as sequences of functions at different levels of detail. This is achieved through "degradation operators" that progressively smooth the time series. NoTS uses an autoregressive transformer to predict the next, more detailed "function" in the sequence. The authors claim this approach leads to better generalization and performance on various time series tasks, supported by theoretical analysis and empirical results on classification, imputation, and anomaly detection.

Strengths of the Paper:
Novel Approach: NoTS offers a new perspective on time series representation using functional degradation and autoregressive prediction.
Theoretical Foundation: The paper provides a theoretical analysis to support its claims.
Clear Presentation: The paper generally presents complex concepts effectively.

Weaknesses of the Paper:
Limited Evaluation Scope: The lack of experiments on long-horizon forecasting is a major weakness, hindering a complete assessment of NoTS's ability to capture long-range dependencies, a key claim in the introduction.
Sensitivity to Degradation Operators: The paper lacks a thorough analysis of how the choice of degradation operators affects performance and provides limited guidance on their selection. This dependence on the choice of operators raises concerns about the method's robustness and generalizability.
Unclear Comparisons: The comparison to N-BEATS, a relevant prior work, is insufficient. A more detailed discussion of the similarities and differences is needed to highlight NoTS's contributions.
Missing Details: Reviewers pointed out a lack of clarity and detail in the model architecture, data flow, and specific implementation choices, making it difficult to fully understand and reproduce the work.

While NoTS presents a novel approach with some theoretical and empirical support, the weaknesses outweigh the strengths. The limited evaluation scope, particularly the absence of forecasting experiments, raises serious concerns about the validity of the authors' claims. The lack of analysis on the impact of degradation operators and unclear comparisons to existing work further weaken the paper.  Therefore, rejection is recommended.