Generalizable autoregressive modeling of time series through functional narratives

## Abstract

Abstract Time series data are inherently functions of time, yet current transformers often learn time series by modeling them as mere concatenations of time periods, overlooking their functional properties. In this work, we propose a novel objective for transformers that learn time series by re-interpreting them as temporal functions. We build an alternative sequence of time series by constructing degradation operators of different intensity in the functional space, creating augmented variants of the original sample that are abstracted or simplified to different degrees. Based on the new set of generated sequence, we train an autoregressive transformer that progressively recovers the original sample from the most simplified variant. Analogous to the next word prediction task in languages that learns narratives by connecting different words, our autoregressive transformer aims to learn the Narratives of Time Series (NoTS) by connecting different functions in time. Theoretically, we justify the construction of the alternative sequence through its advantages in approximating functions. When learning time series data with transformers, constructing sequences of temporal functions allows for a broader class of approximable functions (e.g., differentiation) compared to sequences of time periods, leading to a 26 % percent \% % performance improvement in synthetic feature regression experiments. Experimentally, we validate NoTS in 3 different tasks across 22 real-world datasets, where we show that NoTS significantly outperforms other pre-training methods by up to 6%. Additionally, combining NoTS on top of existing transformer architectures can consistently boost the performance. Our results demonstrate the potential of NoTS as a general-purpose dynamic learner, offering a viable alternative for developing foundation models for time series analysis.

## 1 Introduction

Recent advances in large language models (LLMs)
demonstrate the advantage of large-scale pre-training, providing a generalizable way for modeling complex systems (Radford et al., ).
At the core of state-of-the-art LLMs is the next-token prediction task (Achiam et al., ), where each data sample (sentence) is segmented into tokens (words), and the next word is predicted based on the previous words using the transformer architecture
(Vaswani, ).
By completing samples of sentences based on partial information in an autoregressive (AR) way, LLMs build generalizable data representations that can be rapidly adapted to new datasets and tasks (Brown, ).

Many works in time series analysis mimic the transformer-based modeling approaches in language by building sequences from samples through segmenting time series into periods of time points (Figure 1(A) (Nie et al., ; Liu et al., ; Zhang & Yan, ). An AR transformer on top of it would predict the next time period based on the existing ones (Figure 1(B) (Garza & Mergenthaler-Canseco, ).
However, this modeling approach
has two issues: (1) slicing time series into periods breaks nonlocal functional properties like trend or periodicity, and often requires special remedies to compensate for the issues (Zhou et al., ); (2) the predicted time periods lack generalizability, as the prediction is sensitive to the length of chunks, the position of where the slicing happens, and the characteristics of datasets.
To compensate for the issues of patching and build generalizability into the transformer,
recent works rely on the usage of operators like Fourier neural operators or Koopman operators, but they either require specially engineered coding blocks (Liu et al., ), or a specific set of predetermined bases that
may vary across datasets (Liu et al., ).

Inspired by Tian et al. () that replaces next-patch prediction with a next-resolution prediction task in computer vision,
in this work,
we re-think alternative approaches to build a coarse-to-fine sequence of time series by considering them as functions of time.
Instead of slicing time series into periods, we consider time series samples ${\mathbf{S}}$ as a sampled version of an underlying function $g(t)$ that can be structurally simplified in its functional form (Figure 1(A)).
Instead of mapping the sample onto fixed sets of basis like Taylor or Fourier series, we isolate functional components in a data-dependent way by building degradation operators $d_{k}(\cdot)$ of different intensity levels $k$ and progressively applying them on the signals.
By doing so, we generate an alternative sequence of samples consisting of augmented variants of the signal with increasing amount of information, offering an interconnected yet simplified representation of the original signal.
We train an autoregressive transformer to learn the connection of the different set of functionals, building a knowledge map of different functional components (^1^11For example, in brain decoding tasks, signals often contain cross-frequency coupling where low-frequency components drive the high-frequency components (Klimesch, ; Donoghue et al., ).).
Analogous to the next word prediction task that learns narrative in languages by completing sentences, we denote our method as the Narratives of Time Series (NoTS) because it learns the functional narrative of temporal signals (Figure 1(C)).

We first justify the construction of the alternative sequence using an intuitive function approximation analysis in Section 4.
When learning time series with transformers, under the universal approximation framework (Yun et al., ),
we show that learning time series as sequences of periods of time points would cause approximation issues, as performing sampling operation on commonly encountered time series signal processing operators (e.g., differentiation) creates discontinuous sequence-to-sequence functions.
Instead, such limitations can be bypassed by forming and learning from the alternative function sequences, as long as either (1) the constructed sequence is expressive, or (2) an expressive tokenizer (Ismailov, ) is used before learning with transformers.
The analytical result is validated experimentally through a feature regression task on synthetic datasets. We show that NoTS significantly outperforms other pre-training methods when approximating features with real-world characteristics, showing its superior expressiveness both theoretically and experimentally.

We further validate NoTS in real-world time series datasets in a multitask setting, where we consider performance across 22 real-world datasets consisting of classification, imputation, and anomaly detection tasks.
Across the board, NoTS improves the average performance of other pre-training methods by up to $6\%$,
significantly outperforming the state-of-the-arts including next-period prediction (Garza & Mergenthaler-Canseco, ), masked autoencoder (MAE) (Dong et al., ), and MAE with Fourier neural operators (Liu et al., ).
Moreover, we show that NoTS can improve the performance of existing transformer architectures (Nie et al., ; Liu et al., ), giving a consistent performance boost when performing dataset-specific pre-training. Interestingly, we present a synthetically pre-trained lightweight model NoTS-lw,
which can be efficiently adapted to real-world tasks and achieve $82\%$ average performance with only <$1\%$ parameters trained,
showing the potential
of NoTS on learning dynamics that can be transferred across datasets and tasks.

The main contributions of this paper are summarized as follows:

- •
An alternative approach to form sequences from time series data by considering them as functions of time and isolating functional components with constructed degradation operators.
- •
Analytical results showing that learning time series from the functional perspective allows the approximation of a broader class of functions when compared to learning across periods of time.
- •
A novel transformer-based pre-training framework
NoTS that progressively reconstruct time series from their degraded variants, and thus learn the interrelationships across functions.
- •
Experimental results on 2 synthetic and 22 real-world datasets, including 4 different classes of tasks (classification, regression, anomaly detection, and imputation), showing that NoTS significantly outperforms other pre-training methods from next-period predictors to Fourier-informed masked autoencoders, giving a stable performance boost on top of existing architectures.
- •
A synthetically pre-trained lightweight model NoTS-lw that can be efficiently adapted on new datasets and tasks with <$1\%$ parameters trained while maintaining $82\%$ average performance.

## 2 Preliminaries and related works

### 2.1 Preliminaries

##### Autoregressive (AR) transformers

AR transformers have revolutionized natural language processing by building next-token prediction-based language models (Ray, ). The transformer architectures learn the interactions across different elements (tokens) in a sequence ${\mathbf{X}}=[{\mathbf{x}}_{1},{\mathbf{x}}_{2},...,{\mathbf{x}}_{N}]$,
and the AR objective is defined as follows: The probability of obtaining the next token ${\mathbf{x}}_{i}$ can be deduced from the observed subsequence $[{\mathbf{x}}_{1},{\mathbf{x}}_{2},...,{\mathbf{x}}_{i-1}]$. Thus, the probability of obtaining the whole sample is the product of a sequence of unidirectional conditional probabilities:

$$ $p\left({\mathbf{x}}_{1},{\mathbf{x}}_{2},\ldots,{\mathbf{x}}_{N}\right)=\prod_ {i=1}^{N}p\left({\mathbf{x}}_{i}\mid{\mathbf{x}}_{1},{\mathbf{x}}_{2},\ldots,{ \mathbf{x}}_{i-1}\right)$ (1) $$

where the AR relationship is learned by a transformer model $p_{\theta}$ parameterized by $\theta$. In the language domain, ${\mathbf{x}}_{i}\in\mathcal{V}$ is typically a discrete token of a word from a given vocabulary, which forms next-word predictors with impressive in-context generalization capabilities (Brown, ).

##### Notations for time series

Time series samples are sequences of data points from multiple channels.
A multivariate time series sample of $C$ channels and a length of $T$ is represented as
${\mathbf{S}}=[{\mathbf{v}}_{1},{\mathbf{v}}_{2},...,{\mathbf{v}}_{T}]\in
\mathbb{R}^{C\times T}$. Typically, each dataset has its unique channel-wise relationships.

To apply transformers on time series, one needs to form sequences from the given signal ${\mathbf{S}}$. A naive approach is to directly treat ${\mathbf{v}}_{i}$ as tokens and then apply transformers (Zerveas et al., ). The drawback is that the token representation space is dependent only on ${\mathbf{v}}_{i}$, which varies across datasets, making it less generalizable. Recently, many time series framework produce tokens through cutting time series into different periods of time with a length of $L$, which creates tokens ${\mathbf{x}}_{i}=\operatorname{Tokenizer}([{\mathbf{v}}_{iL},...,{\mathbf{v}}_
{(i+1)L}])$ that contain more dynamics (Ren et al., ).
To further eliminate the negative impact of channel-wise relationships on generalizability,
Nie et al. (); Liu et al. () considers the channel-independent design, which processes each channel (row) of ${\mathbf{S}}$ independently, producing tokens based on individual channels for transformers. While the approach demonstrated more generalizability, it is computationally expensive in high-density settings, which was later discussed by other works (Zhang & Yan, ).

### 2.2 Pretraining methods for time series

Pre-training on large-scale datasets has proven effective in helping models learn generalizable patterns, which is particularly advantageous in the time series domain where downstream datasets are often small-scale
(Liu et al., ; Zhang et al., ; Woo et al., ). There are two prominent approaches for reconstruction-based pre-training in transformers: masked modeling and next-period prediction.
Masked modeling trains transformers by randomly masking elements, and predicting the masked values with the remaining sequence. Representative works include SimMTM (Dong et al., ), which implements masked modeling through aggregating neighboring points, and bioFAME (Liu et al., ), which employs Fourier-based kernels to achieve the same objective.
However, these methods suffer from the loss of nonlocal information. Several approaches are proposed to mitigate this issue, such as leveraging multi-resolution patches (Das et al., ; Woo et al., ).
More recently, the next-period prediction approach has gained attention for pre-training, particularly in the development of foundation models.
For instance, Time-GPT1 (Garza & Mergenthaler-Canseco, ) implements next-period prediction in a straightforward manner, while Chronos (Ansari et al., ) applies scaling and quantization techniques to tokenize time series data and model the categorical distributions.
Despite their success, these models also suffer from the challenge of losing nonlocal information, which is partially addressed through the use of lagged features and temporal covariates in Lag-Llama (Rasul et al., ).
Based upon previous works, our work aims to fundamentally address the issue through building and learning sequences from the function perspective.

An alternative line of research seeks to adapt language models directly for time series applications.
Some approaches transfer pre-trained weights from language models and retrain the tokenization layers to handle time-series-specific tasks (Zhou et al., ; Cao et al., ; Liu et al., ).
Another line of work focuses on reprogramming time series data into text, and applying LLMs to process the textual inputs with time series prompts (Jin et al., ; Xue & Salim, ).
While these works primarily aim to bridge the modality gap between pre-trained LLMs and time series applications, our approach is specifically designed for time series data to capture the subtle variations and dynamics inherent in temporal signals.

## 3 Methods

Alternative to modeling time series as sequences of fragmented time periods, our framework is built on the idea to model time series as sequences of constructed temporal functions with transformers.
We begin by introducing the high-level objective in Section 3.1, and then introduce the pre-training method NoTS in Section 3.2 as well as how to adapt it in real-world tasks in Section 3.3.

### 3.1 The next-function prediction task

We assume that each signal ${\mathbf{S}}=[{\mathbf{v}}_{1},{\mathbf{v}}_{2},...,{\mathbf{v}}_{T}]\in
\mathbb{R}^{C\times T}$ is intrinsically controlled by a temporal function $g(t):\mathbb{R}\to\mathbb{R}^{C}$, where ${\mathbf{v}}_{i}=g(i)$ is the product of a sampling process in time.
To train a transformer with awareness of the functional perspective, we build sequences of functions, where each element is a simplified version of the original sample.
Practically, we ask two questions:

- •
How to build meaningful functional elements, as they tend to change across different datasets?
- •
How to form a meaningful sequence for the transformer, so that we can enforce the transformer to learn generalizable representation from the constructed sequences of functional components?

In this work, we propose to construct *degradation functions* $d_{k}(\cdot)$ of intensity $k$, that generate augmentations of the original function $g(t)$ with varying levels of partial information. Applying degradation functions $d_{k}(\cdot)$ on signals generates data-dependent functions as tokens for transformer, removing the need for a fixed set of bases. By controlling the intensity of degradation, we create $g_{k}(t)=(d_{k}\circ g)(t)$, where $g_{k+1}(t)$ contains strictly more or an equal amount of information than $g_{k}(t)$ about the original sample $g(t)$, establishing a progressive relationship for the transformer to learn. Based on the constructed sequence, the new modeling approach becomes the following:

$$ $\displaystyle p\left(g_{1}(t),g_{2}(t),\ldots,g_{K}(t)\right)=\prod_{k=1}^{K}p \left(g_{k}(t)\mid g_{1}(t),g_{2}(t),\ldots,g_{k-1}(t)\right),$ (2) $$

where $K\rightarrow\infty$, ${\mathbf{v}}_{i}=g_{\infty}(i)$ forms the actual time series ${\mathbf{S}}$ under the sampling operation.

Figure: Figure 2: Narrative of Time Series (NoTS). (A) To perform autoregressive pre-training of NoTS, we first generate a sequence of time series from the raw signal that progressively simplifies the sample. The generated signals are passed into an encoder, added with position and resolution embeddings before fed into the AR transformer, which is trained with a decoder to reconstruct the signal of the next resolution. The raw signal was passed into a latent consistency loss directly. (B) To apply a pre-trained model on real-world dataset, we construct channel adaptor and task adaptor that handles unseen channel graphs and new tasks, respectively. The channel adaptor consists of a multi layer perceptron that pre-process channel maps and new additive channel embeddings. The task adaptor is newly initialized tokens that are prompted into the transformer following Jia et al. (). The produced task tokens and reconstructed samples are later used in multitask applications through this context-aware adaptation pipeline.
Refer to caption: extracted/5917591/imgs/NoTS_fig1.png

### 3.2 NoTS: A novel pre-training objective for transformers

Based on the framework, the pre-training objective NoTS consists of the following components.

##### Local and global degradation functions

Practically, we construct degraded signals with convolution operators: ${\mathbf{S}}_{k}=d_{k}({\mathbf{S}})=({\mathbf{S}}*w_{k})[n]$,
where $*$ represents discrete convolution between rows of signal ${\mathbf{S}}$ and a kernel $w_{k}$. We use two different kernels as defined below:

- •
Local smoothing: A simple averaging kernel of different lengths $p_{k}$ is used for local smoothing. Specifically, $w_{k}[n]=1/p_{k}$ for $-0.5p_{k}\leq n\leq 0.5p_{k}$ and $w_{k}[n]=0$ elsewhere. The set of even numbers $\{p_{k}\}$ is selected as hyperparameters with descending order as $k$ increases.
- •
Global smoothing: A low-pass filter with different frequency cutoff values is used for global smoothing. Specifically, we build $w_{k}[n]=\operatorname{sinc}(p_{k}n)$, where $\{p_{k}\}$ is a set of values that control the frequency cutoff of $0.5p_{k}$ as hyperparameters with descending order as $k$ increases.

By constructing both local and global smoothing degradation functions, the proposed method can simultaneously model autoregressive relationships across different smoothness and frequencies, covering a prevalent range of tasks in both time and frequency.

##### Autoregressive modeling of groups of tokens in the latent space

We build tokenizers to convert the constructed signals ${\mathbf{S}}_{k}$ into recognizable embeddings for transformers. Ideally, we want to use one token for each function, yet this is computationally infeasible as the length of signals increases. Instead, we rely on existing encoder/decoder architectures as tokenizers to convert signals into group of tokens in the latent space to perform AR modeling.
Specifically, the encoder produces groups of tokens from each signal ${\mathbf{R}}_{k}=\mathcal{E}({\mathbf{S}}_{k})$ and the decoder reconstructs signals from groups of tokens ${\mathbf{S}}_{k}^{\prime}=\mathcal{D}({\mathbf{R}}_{k}^{\prime})$,
where each ${\mathbf{R}}_{k}$ and ${\mathbf{R}}_{k}^{\prime}$ consists of multiple tokens for the transformer.
The AR modeling is enforced by applying group-wise masking to the transformer attention map, where:

$$ $[{\mathbf{R}}_{2}^{\prime},...,{\mathbf{R}}_{K}^{\prime}]=\operatorname{ Transformer}([{\mathbf{R}}_{1},...,{\mathbf{R}}_{K-1}]),\text{ with } \operatorname{mask}[\Omega_{k}]=\begin{cases}0,&\bigcup_{m=1}^{k}\Omega_{m}\\ -\infty,&\text{elsewhere}\end{cases}$ $$

and $\Omega_{k}$ represents the set of sequence positions corresponding to ${\mathbf{R}}_{k}$.
While $\mathcal{E}$ and $\mathcal{D}$ can be any encoder/decoder architectures, we implement a lightweight model NoTS-lw with a simple channel-independent 1D-ResNet encoder/decoder block to maintain fidelity in the token space. We also report results with different encoder/decoder architectures in Table 2.

##### Positional embeddings

Since transformers are inherently invariant to the order of sequences, it is critical to embed sufficient information about the raw position into the transformer to ensure ${\mathbf{R}}_{k}$ includes sufficient information about samples. Thus, we add the following embeddings:

- •
Group embeddings. To help transformers learn sufficient information about each function embedded by ${\mathbf{R}}_{k}$, we apply rotary positional embedding (Su et al., ) on each group of tokens to encode the relative sequence position within each group or cohort of tokens.
- •
Degradation embeddings. We add learnable absolute positional embeddings that encode the relative degree of degradation of each augmented variant of signals. In other words, the degradation embeddings encode information $k$ of the degradation function $d_{k}(\cdot)$.
- •
Channel embeddings (optional). When applying channel-independent architectures as the encoder, we add an additional set of learnable absolute positional embeddings along with the group embeddings to help encode channel-wise relationship within each group of tokens.

##### Training objective

We perform a self-supervised autoregressive reconstruction task to learn meaningful representations of time series using the proposed framework.
To achieve this, we minimize the differences between ${\mathbf{S}}_{k}$ and the reconstructed ${\mathbf{S}}_{k+1}^{\prime}$ for every $k<K$.
If only the AR loss is considered, the latent of the raw data ${\mathbf{R}}_{K}$ would remain unused by encoder/decoder throughout the training process. To avoid the resulting distributional shift, we add a latent consistency term, that regularizes the consistency between the latent of the raw data to be able to be reconstructed back.
Thus,
for each input signal matrix ${\mathbf{S}}\in\mathbb{R}^{C\times T}$, the training optimizes the following loss:

$$ $\mathcal{L}=\sum_{k=1}^{K-1}\underbrace{\mathcal{L}_{\text{recon}}({\mathbf{S} }_{k+1}^{\prime},{\mathbf{S}}_{k})}_{\text{AR reconstruction}}+\underbrace{ \mathcal{L}_{\text{recon}}(\mathcal{D}(\mathcal{E}({\mathbf{S}}_{K})),{\mathbf {S}}_{K})}_{\text{latent consistency term}}$ (3) $$

where $\mathcal{L}_{\text{recon}}$ is the reconstruction loss (mean absolute error is used throughout the paper).

### 3.3 Context-aware adaptation through channel and task adaptors

In time series domain, it is common to encounter datasets with new channel graphs or new tasks at test time. To apply the pre-trained weights in such situations, we build two types of adaptors:

##### Channel adaptors

To learn new channel-wise relationship at test time, we build channel adaptors as follows: (1) To encourage information exchange across channels, we add a data embedding layer before applying the encoder $\mathcal{E}$. The data embedding layer is a simple linear layer that encodes on the channel dimension $\mathbb{R}^{C}\rightarrow\mathbb{R}^{C^{\prime}}$ to mix channel-wise information at an early stage. (2) We also re-initialize and re-train additive channel tokens for each dataset when applicable.

##### Task adaptors

To apply the pre-trained models on a diverse set of tasks, we build task adaptors as follows:
(1) We initialize and append prompt tokens to the transformer architecture along with data tokens following the deep visual prompt tuning plan as detailed in Jia et al. (); (2) We also add task-specific linear layers at the end of the transformer for inference purpose.

Given the two adaptors, we can perform context-aware adaptation of NoTS on new channel maps and tasks, allowing the transfer of general-purpose dynamics that are learned at the pre-training stage. Interestingly, the adaptation pipeline is also parameter-efficient: The adaptors add <$1\%$ new parameters in comparison to the original model consists of encoder, transformer, and decoder.

## 4 An intuitive example: Approximating functions

To justify the construction of functional sequences, we provide an intuitive example to investigate the expressive power of transformers under the context of time series domain. Following Yun et al. (); Luo et al. (), we consider the standard transformer architectures:

$$ $\mathcal{T}^{h,m,r}:=\left\{f:\mathbb{R}^{d\times n}\rightarrow\mathbb{R}^{d \times n}\mid f\text{ consists of Transformer blocks }t^{h,m,r}\right\}.$ $$

where $t^{h,m,r}$ consists of one self-attention layer of $h$ heads of size $m$ and one feed-forward layer with $r$ hidden dimensions (see definitions in Appendix Eq. 6). To remove the restriction of permutation equivariance, we consider adding absolute positional embedding(^2^22Refer to Luo et al. () for a case study of relative positional embedding under the UA framework.)
to the transformer that creates $\mathcal{T}_{\mathrm{P}}^{h,m,r}:=\{f_{\mathrm{P}}(\bm{X})=f(\bm{X}+\bm{E})\}$ where $f\in\mathcal{T}^{h,m,r}$ and $\bm{X},\bm{E}\in\mathbb{R}^{d\times n}$. Detailed in Appendix A, our analysis is an extension of previous results in Yun et al. (); Ismailov ().

### 4.1 Time series in the function space

This work assumes time series data as functions in time $g(t)$, which forms function space $\mathcal{F}_{g}(\mathbb{R})$. An operator on the function space maps the original function $g(t)$ to a target function $h(t)$, creating a mapping across function spaces $A:\mathcal{F}_{g}(\mathbb{R})\rightarrow\mathcal{F}_{h}(\mathbb{R})$.
Assume that the signal is produced with a sampling plan $\{t_{i}\}_{i=1}^{T}$, the sampling operation on top of the functions discretizes the mapping into a set of output $\{A[g(t_{i})]\}_{i=1}^{T}$, which forms a sequence-to-sequence function $f_{(A)}:\mathbb{R}^{d\times T}\rightarrow\mathbb{R}^{d\times T}$.
When breaking time series into concatenations of time periods, ${\mathbf{S}}$ is directly treated as inputs to the transformer ${\mathbf{X}}$, where one aims to find a transformer network $f_{P}\in\mathcal{T}_{P}^{h,m,r}$ to approximate $f_{(A)}$.

##### Example: The differential operator

It is intuitive that one can easily construct a linear but discontinuous mapping $A$, which is not necessarily approximable by transformers. See below:

Theorem 1. Given $T>2$, and $\mathcal{D}\subseteq\mathbb{R}^{d\times T}$. Consider the differential operator $A$ that forms a sequence-to-sequence function $f_{(A)}$ under sampling plans $\{t_{i}\}_{i=1}^{T}$ with its initial starting point $t_{1}\in\mathbb{R}$ and fixed intervals. There exists a ${\mathbf{X}}\in\mathbb{R}^{d\times T}$, such that:

$$ $\sup_{\bm{X}\in\mathcal{D}}\left\|f_{P}(\bm{X})-f_{(A)}(\bm{X})\right\|_{2}^{2 }\geq T$ (4) $$

holds for any transformer network $f_{P}\in\mathcal{T}_{P}^{h,m,r}$.

Proof. We construct a negative example with $d=1$. Consider a set of input functions $g_{M}(t)=\sin(Mt)/M$, the target functions under the differential operator are $h_{M}(t)=\cos(Mt)$. As $M$ increases, the input function converges uniformly to a constant zero function, which gives a sampled input matrix $\bm{X}\rightarrow\mathbf{0}\in\mathcal{D}$.
At limit, the studied transformer network $f_{P}(\bm{X})$ converges to a fixed matrix ${\mathbf{B}}$ (see Appendix A.2). Thus, given a sampling plan of interval $t_{i+1}-t_{i}=\pi/M$ and two initial starting points $t_{1}^{(1)}=0$ and $t_{1}^{(2)}=\pi$, we form $\bm{X}_{1}$ and $\bm{X}_{2}$ that give:

$$ $\lim_{M\rightarrow\infty}\left\|f_{P}(\bm{X}_{1})-f_{(A,M)}(\bm{X}_{1})\right \|+\left\|f_{P}(\bm{X}_{2})-f_{(A,M)}(\bm{X}_{2})\right\|\geq\sum_{i=1}^{T}2=2T$ (5) $$

where $f_{(A,M)}$ denotes the function formed from $\{A[g_{M}(t_{i})]\}_{i=1}^{T}$, which leads to Eq. 4.

### 4.2 Two sufficient conditions to approximate the differential operator

When considering signals as functions in time, sampling from simple signal processing operators may create discontinuous sequence-to-sequence functions, causing approximation issues of transformer if one directly considers ${\mathbf{S}}$ as inputs ${\mathbf{X}}$ to the transformer. Instead, by constructing signals sequences of length $T$ using ${\mathbf{S}}_{k}=d_{k}({\mathbf{S}})$, and performing dimensionality reduction with an encoder $\mathcal{E}$, we create two sufficient conditions to address the approximation issue as follows:

Proposition 1. Given a signal ${\mathbf{S}}\in\mathbb{R}^{d\times T}$ and an encoder $\mathcal{E}:\mathbb{R}^{d\times T}\rightarrow\mathbb{R}^{d}$, there exists two sufficient conditions to approximate $\{A[g(t_{i})]\}_{i=1}^{T}$ with the construction of ${\mathbf{X}}=[\mathcal{E}({\mathbf{S}}_{1}),\mathcal{E}({\mathbf{S}}_{2}),...,
\mathcal{E}({\mathbf{S}}_{T})]$:

- •
The constructed ${\mathbf{S}}_{i}$ is expressive such that there exists a continuous mapping between a fixed element of ${\mathbf{S}}_{i}$ and the $i$-th element of the target output $A[g(t_{i})]$;
- •
Given any distinguishable ${\mathbf{S}}_{i}$, there exists an expressive tokenizer $\mathcal{E}$ that preprocess ${\mathbf{S}}_{i}$ to create a continuous mapping between $\mathcal{E}({\mathbf{S}}_{i})$ to the target.

Proof. See Appendix A.3.

##### Example solutions

We provide example solutions to approximate the differential operator. Under the first condition, a trivial solution can be constructed by performing phase transition with degradation operator, creating $(d_{i}\circ g)(t)=\sin(Mt+\frac{i\pi}{2T})/M$. While the first condition requires data-specific degradation operators,
the second condition provides more flexibility. In the case of differential operator, we rely on the result from Ismailov () and use a Kolmogorov’s mapping three-layer neural network to approximate an arbitrary (continuous or discontinuous) function $f:\mathbb{I}^{T}\to\mathbb{R}$, where $\mathbb{I}$ is a compact interval $[0,1]$.
Thus, one can construct a simple degradation operator of value shifts as:
$(d_{i}\circ g)(t)=\sin(Mt)/M+(1+i\delta/T)/M$,
where $\delta\in(0,M-2]$ is an arbitrary number that distinguishes ${\mathbf{S}}_{i}$.
In this case, there exists an encoder $\mathcal{E}$ that can create a continuous sequence-to-sequence function to the desired target (e.g., $\mathcal{E}({\mathbf{S}}_{i})=Mt_{i}$),
where the existence of a solution is guaranteed by previous results in Yun et al. ().

## 5 Experimental Results

While the approximation analysis posts strong assumptions on the solution including the minimal length $T$ of the constructed sequence and the use of specific encoder $\mathcal{E}$. In this section, we show that, experimentally, NoTS works in both synthetic and real-world applications with relaxed assumptions.

**Table 1: Feature approximation results on synthetic datasets. We compare the function approximation ability of different pre-training methods given the same architecture and pre-training pipeline. All presented numbers are averaged across three runs and scaled by 100 for better readability. Lower numbers are better.**
|  | Fractional Brownian motion (fBm) | Autocorrelated sinusoids |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Regression ($\downarrow$) | $\mathcal{H}$-index (1D) | SSC (32D) | WAMP (32D) | SSC (32D) | WAMP (32D) | b. power (96D) |
| VQVAE | 3.78 $\pm$ 0.45 | 38.93 $\pm$ 0.70 | 65.77 $\pm$ 3.72 | 26.24 $\pm$ 0.61 | 29.13 $\pm$ 0.90 | 14.37 $\pm$ 0.03 |
| MAE | 2.01 $\pm$ 0.61 | 25.78 $\pm$ 0.11 | 26.34 $\pm$ 0.03 | 25.29 $\pm$ 0.31 | 28.81 $\pm$ 2.86 | 14.90 $\pm$ 0.02 |
| FAMAE | 1.99 $\pm$ 0.24 | 33.85 $\pm$ 0.53 | 45.76 $\pm$ 0.24 | 28.26 $\pm$ 0.57 | 24.82 $\pm$ 0.84 | 13.92 $\pm$ 0.02 |
| Next-period pred. | 1.75 $\pm$ 0.11 | 27.38 $\pm$ 0.12 | 26.66 $\pm$ 0.19 | 24.44 $\pm$ 0.11 | 28.97 $\pm$ 1.37 | 13.96 $\pm$ 0.04 |
| NoTS (Ours) | 1.27 $\pm$ 0.16 | 23.78 $\pm$ 0.34 | 20.04 $\pm$ 0.12 | 23.13 $\pm$ 0.19 | 24.58 $\pm$ 0.48 | 13.62 $\pm$ 0.05 |
| Improvement | $\uparrow$ 37.80$\%$ | $\uparrow$ 8.41$\%$ | $\uparrow$ 31.44$\%$ | $\uparrow$ 5.66$\%$ | $\uparrow$ 0.98$\%$ | $\uparrow$ 2.20$\%$ |

### 5.1 Synthetic experiments: A feature regression task

##### Datasets

We build two synthetic datasets with AR components in time and frequency spaces.

(1) Fractional Brownian motion (fBm).
fBm is a generalized Gaussian process with special covariance structure that was found to be similar to many types of time series datasets such as traffic, stock prices, and biosignals (Rivero et al., ). Unlike the classic Brownian motion, fBm has interdependent increments across time that are controlled by the Hurst index $\mathcal{H}\in(0,1)$, which creates autoregressive components in time that exhibit long-range ($\mathcal{H}>0.5$) or short-range ($\mathcal{H}<0.5$) dependencies.
We simulate the fBm process 20,000 times to create signals of length 1024 using the Cholesky’s Method (Dieker & Mandjes, ) with $O(l^{3})$ complexity, and remove the generated signals with abnormal values due to simulation instability (around 0.5% of all data).

(2) Superposition of autocorrelated sinusoids. We extend previous synthetic datasets based on sinusoids (Yoon et al., ; Das et al., ), and build a new synthetic dataset of sinusoids with AR components in the frequency space.
Specifically, we sample the set of $\{f_{i}\}_{i=1}^{B}$ based on five random AR(B/2) processes, where we set $B=\{20,16,10,8,4\}$.
We generate amplitude following the $a_{i}=1/f_{i}$ frequency distribution, uniformly sample the phase $p_{i}\in(0,2\pi]$, and add Gaussian noise $0.05*\mathcal{N}(0,1)$ to the signal.
We randomly initialize each process 10,000 times, creating a dataset of 50,000 samples where each sample is of length 1024.

##### Results on feature regression

We estimate different pre-training methods’ capability of approximating functions with the feature regression task.
The ground truth of features are built based on common signal processing analysis methods Slope Sign Change (SSC, 32D) and Willison Amplitude (WAMP, 32D), and we also include the Hurst index ($\mathcal{H}$-index, 1D) for the fBM dataset and the band power (b. power, 96D) for the sinusoids (Appendix B.1).
Note that SSC and WAMP are both implemented with global thresholding, making them discontinuous sequence-to-sequence functions.
Following Section 3.3, we train a VQVAE (Van Den Oord et al., ), masked autoencoder (MAE) (Dong et al., ), frequency-aware MAE (FAMAE) (Liu et al., ), next-period prediction transformer (Garza & Mergenthaler-Canseco, ), and NoTS-lw on the synthetic datasets by appending them with a regression task adaptor to validate the performance of our proposed method.

As shown in Table 1, across the board, NoTS-lw significantly outperforms all other pre-training methods given the same architecture and training pipeline. The relative improvement is especially pronounced in the fBm dataset, where data has complicated covariance architecture that was found relevant in many real-world applications, where we have $26\%$ improvements across the features.

##### Visualizing the next-function prediction process

We present the data and latent visualizations in Figure 3. In Figure 3(A), we show the original data sequence $\{{\mathbf{S}}_{i}\}_{i=1}^{K}$ and the reconstructed data sequence $\{{\mathbf{S}}_{i}^{\prime}\}_{i=2}^{K}$. Note that the original signal ${\mathbf{S}}_{K}$ was not passed into the transformer. We can see that the predicted sequence has information that is not presented in previous signals, showing the function prediction capacity of the transformer.
In Figure 3(B), we plot the token space before or after the AR transformer using the PCA reduction on $\{{\mathbf{R}}_{i}\}_{i=1}^{K}$ and $\{{\mathbf{R}}_{i}^{\prime}\}_{i=2}^{K}$, respectively. When coloring the tokens differently based on their degradation parameter $i$, we observe that: (1) In original token space $\{{\mathbf{R}}_{i}\}_{i=1}^{K}$, severely degraded signals generates more clustered tokens, and the tokens would gradually disperse as signals become more realistic; (2) The predicted tokens $\{{\mathbf{R}}_{i}^{\prime}\}_{i=2}^{K}$ would generate a token space with similar behaviour without seeing the original set of tokens ${\mathbf{R}}_{K}$. This behaviour demonstrates the autoregressive capacity of the transformer.

**Table 2: Comparisons between NoTS and other pre-training methods on real-world datasets. We categorize the results based on (a) if adaptors are used, and (b) if the weights of the pre-trained models are frozen. We compute an average error rate ($\downarrow$) to compare the final performance of different methods in each condition.**
|  |  | Classification ($\uparrow$) | Anom. Det. ($\uparrow$) | Imputation ($\downarrow$) | Avg. ($\downarrow$) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | (a) | (b) | UCR-9 | UEA-5 | SMD | MSL | SWaT | PSM | ETTm1 | ETTm2 | ETTh1 | ETTh2 | error rate |
| SimMTM | ✓ | ✓ | 68.70 | 55.36 | 84.06 | 83.90 | 91.20 | 96.07 | 0.164 | 0.126 | 0.264 | 0.183 | 19.43 |
| bioFAME | ✓ | ✓ | 62.63 | 60.32 | 83.09 | 84.28 | 91.21 | 95.94 | 0.203 | 0.122 | 0.258 | 0.178 | 19.87 |
| Next-pred | ✓ | ✓ | 65.95 | 58.30 | 82.96 | 83.75 | 90.47 | 96.54 | 0.306 | 0.178 | 0.465 | 0.270 | 24.39 |
| NoTS-lw (Ours) | ✓ | ✓ | 71.88 | 62.78 | 83.63 | 84.28 | 93.26 | 96.27 | 0.164 | 0.126 | 0.286 | 0.196 | 18.51 |
| SimMTM | ✓ | ✗ | 81.65 | 61.23 | 83.48 | 84.11 | 91.35 | 96.36 | 0.123 | 0.107 | 0.201 | 0.166 | 16.14 |
| bioFAME | ✓ | ✗ | 81.53 | 63.57 | 83.59 | 83.98 | 91.46 | 96.88 | 0.129 | 0.107 | 0.202 | 0.178 | 16.05 |
| Next-pred | ✓ | ✗ | 80.62 | 62.76 | 83.00 | 84.09 | 91.00 | 96.87 | 0.130 | 0.119 | 0.228 | 0.188 | 16.82 |
| NoTS-lw (Ours) | ✓ | ✗ | 88.08 | 66.38 | 84.19 | 84.15 | 91.26 | 96.88 | 0.122 | 0.116 | 0.218 | 0.163 | 15.10 |
| PatchTST | ✗ | ✗ | 83.57 | 63.31 | 78.96 | 78.81 | 83.75 | 78.07 | 0.181 | 0.126 | 0.347 | 0.187 | 21.78 |
| +NoTS (Ours) | ✗ | ✗ | $\uparrow$1.71 | $\uparrow$1.65 | $\uparrow$2.20 | $\uparrow$3.96 | $\uparrow$5.97 | $\uparrow$11.25 | $\downarrow$.003 | $\downarrow$.003 | $\downarrow$.064 | $\downarrow$.006 | 18.33 |
| iTransformer | ✗ | ✗ | 82.67 | 67.62 | 85.18 | 83.04 | 91.88 | 97.07 | 0.162 | 0.111 | 0.240 | 0.168 | 16.07 |
| +NoTS (Ours) | ✗ | ✗ | $\uparrow$1.26 | $\uparrow$0.65 | $\uparrow$0.17 | $\uparrow$0.11 | $\uparrow$0.05 | $\uparrow$0.01 | $\uparrow$.005 | $\downarrow$.002 | $\downarrow$.013 | $\downarrow$.004 | 15.70 |

### 5.2 Real-world experiments: Context-aware generalization

##### Experimental setups

To examine the performance of NoTS in real-world applications, we perform multi-task validation following the setups in Wu et al. (). Specifically, we perform the classification task on the UCR subset (Dau et al., ) and UEA subset (Bagnall et al., ); the imputation task on the ETDataset (Zhou et al., ), and the anomaly detection task on MSL (Hundman et al., ), PSM (Abdulaal et al., ), SWaT (Mathur & Tippenhauer, ), and SMD (Su et al., ) datasets.
We follow Wu et al. () for standard data pre-processing and task deployment pipeline, except for the imputation task where we tested a more challenging variant of channel-wise imputation (see Appendix B.2.2 and B.2.3 for details and original imputation results).

To validate that NoTS is a superior pre-training method, we perform two sets of experiments:
First, we compare the performance of NoTS-lw against the next-period AR transformer, a MAE (Dong et al., ), and a frequency-aware MAE (Liu et al., ) by pre-training them on synthetic datasets and deploying the prompt tuning pipeline for all pre-trained base models.
Second, we append the pre-training pipeline NoTS on top of existing architectures PatchTST (Nie et al., ) and iTransformer (Liu et al., ), and compute the performance benefits from adding NoTS.

##### Experimental results

As shown in Table 2, with or without parameters frozen, NoTS-lw significantly outperforms all other pre-training methods. Specifically, given the same pre-training pipeline and architecture, NoTS-lw outperforms other method across all tasks by up to $6\%$ average. Interestingly, we note that NoTS-lw show comparable performance on imputation tasks, where MAE-like architectures are trained to perform the task. Additionally, when attaching NoTS on existing architectures PatchTST (Nie et al., ) and iTransformer (Liu et al., ), NoTS improves their performance without specific backbone or adaptors, showing the versatility of the pre-training method.

Interestingly, we should like to emphasize on the context-aware generalization ability of NoTS. With the architecture frozen (first 4 rows of Table 2), we only train <$1\%$ of the parameters, yet it performs $82\%$ performance, potentially demonstrating the context-aware generalization.

Figure: Figure 3: Visualizations of AR performance and loss. (A) We visualize the autoregressive inference process of NoTS on the synthetic dataset. From bottom to top, the signal variance is gradually recovered through the prediction of the AR transformer. (B) The token space is visualized through principal component analysis, where tokens of the simplified signals gradually disperse to a larger region when colored in different degradation degrees. When colored with relative group positions, the distribution does not shift as much on the direction of another principal component. (C) A pilot study shows that training larger NoTS models leads to lower reconstruction loss on the test set, potentially following the power law behaviour of AR models.
Refer to caption: extracted/5917591/imgs/NoTS_fig3_1data.png

### 5.3 Ablation experiments and model analysis

##### Ablation of effective components in NoTS

In Table 3, we perform ablations of NoTS by isolating the effective components of NoTS-lw in the feature regression task (the $\mathcal{H}$-index). Specifically, we train three variants of NoTS-lw by (1) removing the latent consistency term in training loss, (2) removing the autoregressive masking within transformer, creating a transformer that merely bridges tokens of the augmented samples, (3) removing the connections among constructed augmentations, and using degradation operator only as augmentations. As expected, removing the latent consistency term would cause distributional shift as the model never sees raw data, and would result in severe performance degradation. Interestingly, training a transformer that connects augmented samples can also provide improved performance, as observed in other works (Hou et al., ; Liu et al., ).

**Table 3: Ablation experiments. Columns denote: If the model sees original signal (orig.), if the model uses autoregressive modeling (AR), if the signal variants are connected (conn.), and if a Gaussian-based degradation operator is used ($d_{k}^{(\mathcal{N})}$).**
|  | orig. | AR | conn. | $d_{k}^{(\mathcal{N})}$ | error ($\downarrow$) |
| --- | --- | --- | --- | --- | --- |
| (1) | ✗ | ✓ | ✓ | ✗ | 1.75 |
| (2) | ✓ | ✗ | ✓ | ✗ | 1.48 |
| (3) | ✓ | ✗ | ✗ | ✗ | 1.82 |
| (4) | ✓ | ✓ | ✓ | ✓ | 1.69 |
| NoTS | ✓ | ✓ | ✓ | ✗ | 1.27 |

##### Connection to diffusion models

One might relate our work with diffusion models (Ho et al., ) by using a stochastic additive Gaussian noise of varying degrees as a degradation operator. We attempted this as model variant (4) in Table 3, yet the performance is inferior to the convolution-based degradation operators. One hypothesis is that time series data is inherently noisy, and adding Gaussian noise instead of performing smoothing or filtering can be less effective, as observed in audio signals (Dieleman, ). Building connections between NoTS and cold diffusion models with deterministic degradation operators (Bansal et al., ), or more recent diffusion models (Chen et al., ) can be an exciting future research direction.

##### Scalability analysis

While this work aims only to provide an initial experimental exploration of the proposed pre-training methodology NoTS, we attempted a pilot study to increase the size of NoTS-lw to demonstrate its potential given more parameters. We trained four models with 127k, 243k (used in all previous experiments), 641k, 2.1M parameters to observe their performance. As shown in Figure 3(C), when fixing the amount of training data, training the models to convergence with increased parameters leads to increased performance, potentially following a power law curve of AR frameworks in language and computer vision (Kaplan et al., ; El-Nouby et al., ).

## 6 Conclusion

In this paper, we propose a novel autoregressive pre-training method NoTS for time series. Our work aims to provide an alternative view of time series by considering them as functions of time instead of concatenations of time periods. This novel perspective allows us to construct degradation operators, which build an alternative sequence as inputs to the transformer.
The transformer is pre-trained with an autoregressive loss to encourage the learning of cross-function relationship, building a model that can recover signal variability from their simplified variants.
We validated the performance of NoTS with experimental results on 2 synthetic and 22 real-world datasets, demonstrating its superiority among existing pre-training methods across multiple tasks, showing a viable alternative for developing foundation models for time series analysis in the future.

##### Limitations

Future works may extend the existing results through: (1) Expanding our initial experimental efforts to larger models, larger-scale datasets, and more challenging tasks. (2) Building in-depth theoretical connection to diffusion-based models, connecting NoTS with recent works (Dieleman, ) from the audio and computer vision domain. (3) Understanding how NoTS performs
in stochastic events as detailed in (Kidger et al., ) based on rough path theory.

#### Acknowledgments

We gratefully acknowledge Apple for supporting this project: This work was supported by a grant from Apple, Inc. Any views, opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and should not be interpreted as reflecting the views, policies or position, either expressed or implied, of Apple Inc.
We also acknowledge the NSF CAREER Award (IIS-2146072) and the CIFAR Azrieli Global Scholars Program, which supported Dr. Dyer’s contributions to this work.
We are grateful to all data contributors, including all the people who have contributed to the UCR and the UEA time series archives; the iTrust Centre for Research in Cyber Security at the Singapore University of Technology and Design; and all others. We also thank Keyu Chen and Tianrong Chen for their inspiring discussions.

## Appendix

## Appendix A Analytical results

### A.1 Preliminaries

##### Transformers

We consider the standard transformer architecture as defined in (Luo et al., ).
The transformer network is the stack of transformer blocks, each of them consists of a self-attention layer $\operatorname{Attn}(\cdot)$ and a feed forward layer $\operatorname{FF}(\cdot)$. Given an input ${\mathbf{X}}\in\mathbb{R}^{d\times T}$,
they are written as:

$$ $\displaystyle\operatorname{Attn}(\bm{X})$ $\displaystyle=\bm{X}+\sum_{i=1}^{h}\bm{W}_{O}^{i}\bm{W}_{V}^{i}\bm{X}\cdot \sigma\left[\left(\bm{W}_{K}^{i}\bm{X}\right)^{T}\bm{W}_{Q}^{i}\bm{X}\right]$ (6) $\displaystyle\mathrm{FF}(\bm{X})$ $\displaystyle=\operatorname{Attn}(\bm{X})+\bm{W}_{2}\cdot\operatorname{ReLU} \left(\bm{W}_{1}\cdot\operatorname{Attn}(\bm{X})\right)$ $$

where $\bm{W}_{O}^{i}\in\mathbb{R}^{d\times m}$, $\bm{W}_{V}^{i},\bm{W}_{K}^{i},\bm{W}_{Q}^{i}\in\mathbb{R}^{m\times d}$, $\bm{W}_{2}\in\mathbb{R}^{d\times r}$, and $\bm{W}_{1}\in\mathbb{R}^{r\times d}$.

We denote $t^{h,m,r}:\mathbb{R}^{d\times T}\rightarrow\mathbb{R}^{d\times T}$ as a transformer block with an attention layer with $h$ heads of size $m$, and a feed-forward layer with $r$ hidden nodes. Thus, the transformer can be written as:

$$ $\mathcal{T}^{h,m,r}:=\left\{f:\mathbb{R}^{d\times T}\rightarrow\mathbb{R}^{d \times T}\mid f\text{ is a composition of transformer blocks }t^{h,m,r}\right\}.$ (7) $$

Similarly, transformer with absolute positional embedding is:

$$ $\mathcal{T}_{\mathrm{P}}^{h,m,r}:=\left\{f_{\mathrm{P}}(\bm{X})=f(\bm{X}+\bm{E })\mid f\in\mathcal{T}^{h,m,r}\text{ and }\bm{E}\in\mathbb{R}^{d\times T}\right\}$ (8) $$

##### Universal Approximator (UA)

The universal approximation framework considers the feasibility or existence of a neural network that can approximate different types of functions with arbitrarily small error.
Consider a transformer network $f_{1}$ and an arbitrary function $f_{2}$, where $f_{1},f_{2}:\mathbb{R}^{n\times T}\rightarrow\mathbb{R}^{n\times T}$ are both sequence-to-sequence functions. We define a distance between $f_{1}$ and $f_{2}$ as:

$$ $\mathrm{d}_{p}\left(f_{1},f_{2}\right):=\left(\int\left\|f_{1}(\bm{X})-f_{2}( \bm{X})\right\|_{p}^{p}d\bm{X}\right)^{1/p}$ (9) $$

being a UA means that for any given $f_{2}\in\mathcal{F}$, let $1\leq p<\infty$ and $\epsilon>0$, there exists a network $f_{1}$, such that $\mathrm{d}_{p}(f_{1},f_{2})\leq\epsilon$.
Several prior works have explored the concept of universal approximators (UAs) and whether transformers qualify as UAs. Below, we outline the key results from the literature that will be referenced in this paper:

Theorem 2 (informal, see Yun et al. ()).
Given $1\leq p<\infty$ and $\epsilon>0$, for any function $f\in\mathcal{F}_{\mathrm{PE}}$, where $\mathcal{F}_{\mathrm{PE}}$ consists of all continuous permutation equivariant functions with compact support, there exists a Transformer network $f\in\mathcal{T}^{2,1,4}$ where $\mathrm{d}_{p}(f,g)\leq\epsilon$.

Theorem 3 (informal, see Yun et al. ()).
Given $1\leq p<\infty$ and $\epsilon>0$, for any function $f\in\mathcal{F}_{\mathrm{CD}}$, where $\mathcal{F}_{\mathrm{CD}}$ consists of all
continuous functions with compact support, there exists a Transformer network $f\in\mathcal{T}_{\mathrm{P}}^{2,1,4}$ where $\mathrm{d}_{p}(f,g)\leq\epsilon$.

Theorem 2 discussed that transformers without positional embeddings are UAs for all continuous permutation equivariant functions; and Theorem 3 discussed that transformers with absolute positional embeddings (APE) are UAs for all continuous functions with compact support. Note that the latter results may be overruled by modifying the transformer architectures as follows:

Theorem 4 (informal, see Luo et al. ()). Consider relative positional encoding (RPE) that modifies the attention as $\operatorname{Attn}(\bm{X})=\bm{X}+\sum_{i=1}^{h}\bm{W}_{O}^{i}\bm{W}_{V}^{i}
\bm{X}\cdot\operatorname{softmax}\left[\left(\bm{W}_{K}^{i}\bm{X}\right)^{T}
\bm{W}_{Q}^{i}\bm{X}+{\mathbf{E}}\right]$, where ${\mathbf{E}}\in\mathbb{R}^{T\times T}$ encodes the relative position within attention maps. Given $T>2$, there always exists a continuous function $f_{M}:\mathcal{D}\to\mathbb{R}^{d\times T}$, such that $\sup_{\bm{X}\in\mathcal{D}}\left\|f_{\operatorname{RPE}}(\bm{X})-f_{M}(\bm{X})
\right\|_{2}^{2}\geq M$ holds for any modified RPE-based transformer.

While UA framework is typically used for understanding the approximation problem towards continuous functions, more recently, it is used to understand the approximation problem towards discontinuous functions. Specifically, in this work, we reference the results in Ismailov () that shows any discontinuous function may be implemented by a three-layer Kolmogorov type neural network:

Theorem 5 (informal, see Ismailov ()). Given $d\geq 2$ and any function $f:\mathbb{I}^{d}\to\mathbb{R}$, where $\mathbb{I}$ is a closed unit internal $[0,1]$, then function $f$ can be implemented exactly by a three-layer Kolmogorov neural network with $d$, $2d+1$, and $1$ processing units in the first, second, and final layer, respectively.

As stated in Ismayilova & Ismailov (); Ismailov (), the expressiveness of simple neural networks can be extended by constructing more diverse activation functions, which helps us understand the approximation ability towards discontinuous functions which are more prevalent in real-world complex systems. Note that other works also discuss the approximation problems towards functions that may be discontinuous (Kidger & Lyons, ; Pinkus, ).

### A.2 Additional proof of theorem 1

We only study the convergence property of self attention layers as the convergence property of feed forward networks has been extensively studied in previous works.
To prove convergence, we build an input sequence ${\mathbf{X}}+\Delta_{n}$, where $\Delta_{n}$ is defined as a bounded perturbation matrix $\Delta\in\mathcal{D}$ that is uniformly scaled by a positive value $n$. Given a self attention layer $\operatorname{Attn}({\mathbf{X}})$, we show the following:

Lemma 1. Given $n\geq N$, ${\mathbf{X}}\in\mathcal{D}$, $\Delta_{n}=\Delta/n$, there exists an $\epsilon$ such that:

$$ $\sup_{\Delta\in\mathcal{D},\|\Delta\|_{1}\leq 1}\|\operatorname{Attn}({\mathbf {X}})-\operatorname{Attn}({\mathbf{X}}+\Delta_{n})\|_{2}<\epsilon.$ (10) $$

holds for any self attention layer parameterized by $\bm{W}_{O}^{i}\in\mathbb{R}^{d\times m}$, $\bm{W}_{V}^{i},\bm{W}_{K}^{i},\bm{W}_{Q}^{i}\in\mathbb{R}^{m\times d}$.

Proof. First, we re-write the activation component $\sigma\left[\left(\bm{W}_{K}^{i}\bm{X}\right)^{\top}\bm{W}_{Q}^{i}\bm{X}\right]$ in Eq. 6 as column-wise softmax operation $\operatorname{softmax}({\mathbf{X}}^{\top}{\mathbf{W}}{\mathbf{x}}_{j})$, where ${\mathbf{x}}_{j}$ is a random column of ${\mathbf{X}}$. We have:

$$ $\displaystyle\|$ $\displaystyle\operatorname{softmax}({\mathbf{X}}^{\top}{\mathbf{W}}{\mathbf{x} }_{j})-\operatorname{softmax}(({\mathbf{X}}+\Delta_{n})^{\top}{\mathbf{W}}({ \mathbf{x}}_{j}+\delta_{j,n}))\|_{2}$ (11) $\displaystyle\leq\|\operatorname{softmax}({\mathbf{X}}^{\top}{\mathbf{W}}{ \mathbf{x}}_{j})-\operatorname{softmax}(({\mathbf{X}}+\Delta_{n})^{\top}{ \mathbf{W}}({\mathbf{x}}_{j}+\delta_{j,n}))\|_{1}$ $\displaystyle\leq 2\|{\mathbf{X}}^{\top}{\mathbf{W}}{\mathbf{x}}_{j}-({\mathbf {X}}+\Delta_{n})^{\top}{\mathbf{W}}({\mathbf{x}}_{j}+\delta_{j,n})\|_{\infty} \text{ (Corollary A.7 in \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{ edelman2022inductive}{\@@citephrase{(}}{\@@citephrase{)}}})}$ $\displaystyle=2\max_{i}(\frac{1}{n}({\mathbf{x}}_{i}^{\top}{\mathbf{W}}\delta_ {j}+\delta_{i}^{\top}{\mathbf{W}}{\mathbf{x}}_{j})+\frac{1}{n^{2}}\delta_{i}^{ \top}{\mathbf{W}}\delta_{j})$ $\displaystyle\leq 2\max_{i}(\frac{1}{n}({\mathbf{x}}_{i}^{\top}{\mathbf{W}} \mathds{1}+\mathds{1}^{\top}{\mathbf{W}}{\mathbf{x}}_{j})+\frac{1}{n^{2}} \mathds{1}^{\top}{\mathbf{W}}\mathds{1})=\epsilon_{h}$ $$

which shows that the attention map converges given deterministic ${\mathbf{X}}$ and ${\mathbf{W}}=({\mathbf{W}}_{K}^{i})^{\top}{\mathbf{W}}_{Q}^{i}$. Thus, Eq. 10 holds by considering the self attention operator $\operatorname{Attn}({\mathbf{X}})$ as a convex combination of attention heads given deterministic ${\mathbf{X}}$, $\bm{W}_{O}^{i}\in\mathbb{R}^{d\times m}$, $\bm{W}_{V}^{i}\in\mathbb{R}^{m\times d}$.

Thus, given $\bm{X}\rightarrow\mathbf{0}$, the transformer network $f_{P}({\mathbf{X}})$ converges to a deterministic matrix ${\mathbf{B}}$ that is dependent on the network parameters and the positional embedding ${\mathbf{E}}$.
Note that, under the context of network optimization, a more generalized version of the convergence property of transformers has been proved in other previous works (Wu et al., ; Gao et al., ).

### A.3 Additional proof of proposition 1

Based on the results in Theorem 3, it is known that as long as the constructed sequence ${\mathbf{X}}=[\mathcal{E}({\mathbf{S}}_{1}),\mathcal{E}({\mathbf{S}}_{2}),...,
\mathcal{E}({\mathbf{S}}_{T})]$ forms a continuous sequence-to-sequence function between input ${\mathbf{X}}$ and the target $\{A[g(t_{i})]\}_{i=1}^{T}$, it is guaranteed that there exists a transformer network $f_{P}\in\mathcal{T}_{P}^{2,1,4}$ that can approximate the constructed sequence-to-sequence function.
Thus, we show how the presented two conditions are sufficient to meet the above requirement:

- •
When there exists a continuous mapping between a fixed element $p$ of ${\mathbf{S}}_{i}$ and the $i$-th element of the target output $A[g(t_{i})]$, one can construct a simple linear encoder $\mathcal{E}({\mathbf{S}})={\mathbf{S}}{\mathbf{v}}$, where ${\mathbf{v}}[i]=0$ when $i\neq p$ and ${\mathbf{v}}[p]=1$, that creates the continuous sequence-to-sequence function.
- •
Based on the results in Theorem 5, if there exists an expressive tokenizer $\mathcal{E}$ (that may be a discontinuous function) that preprocess ${\mathbf{S}}_{i}$ to create a continuous mapping between $\mathcal{E}({\mathbf{S}}_{i})$ to the target, the existence of the transformer is guaranteed for a continuous sequence-to-sequence function.

## Appendix B Experimental details

### B.1 Synthetic experiments

#### B.1.1 Synthetic datasets details

##### Fractional Brownian motion (fBm)

Given a Hurst index $\mathcal{H}$ and two time steps $i$ and $j$, a fBm process is a continuous-time Gaussian process $B_{\mathcal{H}}(t)$ with the following covariance structure:

$$ $E\left[B_{\mathcal{H}}(i)B_{\mathcal{H}}(j)\right]=\frac{1}{2}\left(|i|^{2 \mathcal{H}}+|j|^{2\mathcal{H}}-|i-j|^{2\mathcal{H}}\right)$ (12) $$

Define function $\gamma(i,\mathcal{H})=0.5(|i-1|^{2\mathcal{H}}+|i+1|^{2\mathcal{H}}-2|i|^{2
\mathcal{H}})$, a fBm process can be simulated through the Cholesky decomposition method detailed as follows:

Figure: Algorithm 1 Simulation of fBm processes using the Cholesky’s method

##### Autocorrelated sinusoids

The autocorrelated sinusoids dataset is generated with AR processes in the frequency space. Given an integer $k$, a randomly initialized set of weights $\{\phi_{i}\}_{i=1}^{k}$, an AR($k$) process defines the sequence of frequency values as follows:

$$ $f_{t}=\sum_{i=1}^{k}\phi_{i}f_{t-i}$ (13) $$

The AR process ensures that the frequency components in the synthetic dataset are correlated, creating an autoregressive frequency structure that NoTS can effectively learn from.

#### B.1.2 Feature regression task details

We detail the feature extraction methods as follows. We use them as the ground truth for the feature regression task.
Define the indicator function $\mathbf{1}_{A}(x)$, where $\mathbf{1}_{A}(x)=1$ if $x\in A$ and $\mathbf{1}_{A}(x)=0$ otherwise. Given a single-channel signal ${\mathbf{s}}\in\mathbb{R}^{T}$ with $v_{i}$ as the value on $i$-th timestamp, all features are extracted on each channel of the signal as follows:

##### Slope Sign Change (SSC)

SSC measures directional slope changes in a signal, indicating the intensity of fluctuations. Given a threshold value $\delta$ as hyperparameter, a period of time series sequence ${\mathbf{s}}$, we extract SSC value with the following equation:

$$ $\operatorname{SSC}({\mathbf{s}})=\sum_{i=2}^{T-1}\mathbf{1}_{(v_{i}-v_{i-1})(v _{i}-v_{i+1})<0}\left(v_{i}\right)\cdot\mathbf{1}_{\max\left(|v_{i}-v_{i+1}|,| v_{i}-v_{i-1}|\right)\geq\delta}\left(v_{i}\right)$ $$

In practice, we extract the SSC values on top of segmented signals with a length of $32$.

##### Willison Amplitude (WAMP)

WAMP is a similar feature that focuses on counting significant amplitude changes between consecutive steps. Given a threshold value $\delta$ as hyperparameter, a period of time series sequence ${\mathbf{s}}$, WAMP is computed through $\operatorname{WAMP}({\mathbf{s}})=\sum_{i=1}^{T-1}\mathbf{1}_{|v_{i+1}-v_{i}|
\geq\delta}\left(v_{i}\right)$. In practice, we also extract the WAMP values on top of segmented signals with a length of $32$, creating a 32-dimensional feature for each studied synthetic data sample.

##### Band power (b. power)

The band power quantifies the energy within a specific selected range of frequencies. It is computed by first performing the Fourier transform of ${\mathbf{s}}$, creating a frequency representation ${\mathbf{s}}(f)$. The band power within frequency range $[f_{1},f_{2}]$ is later extracted as $\operatorname{BP_{(f_{1},f_{2})}}({\mathbf{s}})=\int_{f_{1}}^{f_{2}}|{\mathbf{
s}}(f)|^{2}\,df$.
In this work, we consider 3 unique frequency range $\{[5,10],[15,30],[30,80]\}$ as hyperparameters to extract a 96-dimensional feature for each studied synthetic data sample.

### B.2 Real-world experiments

#### B.2.1 Dataset information

##### Classification

We selected 9 univariate datasets from the UCR archive (Dau et al., ), filtering out all datasets with less than $140$ series length or less than $350$ training samples. The dataset selection is performed to ensure each dataset has both sufficient samples and dynamics.
The detailed information about the selected datasets is provided in Table 4.

**Table 4: Detailed information about the selected datasets from the UCR archive.**
| Dataset | Train | Test | Series Length | Classes |
| --- | --- | --- | --- | --- |
| FordA | 3601 | 1320 | 500 | 2 |
| FordB | 3636 | 810 | 500 | 2 |
| ScreenType | 375 | 375 | 720 | 3 |
| ECG5000 | 500 | 4500 | 140 | 5 |
| Wafer | 1000 | 6164 | 152 | 2 |
| StarLightCurves | 1000 | 8236 | 1024 | 3 |
| UWaveGestureLibraryAll | 896 | 3582 | 945 | 8 |
| HandOutlines | 1000 | 370 | 2709 | 2 |
| EthanolLevel | 504 | 500 | 1751 | 4 |

We also selected 5 multivariate datasets from the UEA archive (Bagnall et al., ), excluding those with a series length below $100$ and the training sample size below $200$. The detailed information about the selected datasets is provided in Table 5.

**Table 5: Detailed information about the selected datasets from the UEA archive.**
| Dataset | Channel | Train | Test | Series Length | Classes |
| --- | --- | --- | --- | --- | --- |
| EthanolConcentration | 3 | 261 | 263 | 1751 | 4 |
| Heartbeat | 61 | 204 | 205 | 405 | 2 |
| PEMS-SF | 963 | 267 | 173 | 144 | 7 |
| SelfRegulationSCP1 | 6 | 268 | 293 | 896 | 2 |
| SelfRegulationSCP2 | 7 | 200 | 180 | 1152 | 2 |

##### Imputation

For the imputation tasks, we use the ETDataset (Zhou et al., ), where ETTm1 and ETTm2 are sampled at minute intervals, and ETTh1 and ETTh2 are sampled at hourly intervals. The detailed information about the selected datasets is provided in Table 6.

**Table 6: ETDataset for imputation tasks.**
| Dataset | Channel | Series Length | Train | Validation | Test |
| --- | --- | --- | --- | --- | --- |
| ETTm1, ETTm2 | 7 | 96 | 34465 | 11521 | 11521 |
| ETTh1, ETTh2 | 7 | 96 | 8545 | 2881 | 2881 |

##### Anomaly detection

The detailed information about the selected datasets is provided in Table 7.

**Table 7: Detailed information about the selected datasets for the anomaly detection tasks.**
| Dataset | Channel | Series Length | Train | Validation | Test |
| --- | --- | --- | --- | --- | --- |
| SMD | 38 | 100 | 566724 | 141681 | 708420 |
| MSL | 55 | 100 | 44653 | 11664 | 73729 |
| SWaT | 51 | 100 | 396000 | 99000 | 449919 |
| PSM | 25 | 100 | 105984 | 26497 | 87841 |

#### B.2.2 Model training and architecture details

##### Training details

For pre-training on synthetic datasets, we use a learning rate of $0.05$, a MultiStepLR scheduler with a multiplicative factor $\gamma=0.3$, and two milestones on epoch $30$ and $150$. We perform all pre-training for a total of $300$ epochs on both of the synthetic datasets, where we set batch size as $1024$ for the reconstruction task.

For pre-training on real-world datasets, we use a learning rate of $0.005$. We perform all pre-training for either a total of $300$ epochs, or a total of $6000$ steps, whichever finishes the first. We set batch size as $32$ for imputation and anomaly detection tasks, and batch size as $64$ for classification tasks.

**Table 8: Complete classification results on the UCR datasets.**
|  | Parameter efficient tuning | Full-scale fine-tuning |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | NoTS-lw | Next-pred | bioFAME | SimMTM | NoTS-lw | Next-pred | bioFAME | SimMTM |
| HandOutlines | 71.62 | 64.32 | 64.05 | 88.92 | 93.51 | 72.16 | 91.62 | 89.73 |
| EthanolLevel | 28.60 | 26.60 | 25.20 | 29.00 | 91.40 | 48.60 | 41.20 | 38.00 |
| StarLightCurves | 91.66 | 87.34 | 85.15 | 88.59 | 97.21 | 97.49 | 97.56 | 97.39 |
| UWave-GL-All | 67.39 | 56.28 | 37.38 | 76.35 | 96.57 | 96.90 | 87.83 | 94.72 |
| FordA | 81.52 | 77.27 | 71.74 | 51.59 | 94.02 | 94.09 | 93.49 | 94.55 |
| FordB | 68.27 | 63.83 | 57.41 | 64.57 | 83.70 | 86.17 | 85.19 | 83.58 |
| Wafer | 98.78 | 89.21 | 89.21 | 89.23 | 99.81 | 99.87 | 99.64 | 99.85 |
| ECG5000 | 91.31 | 89.00 | 92.96 | 87.38 | 94.13 | 88.47 | 94.29 | 93.82 |
| ScreenType | 47.73 | 39.73 | 40.53 | 42.67 | 42.40 | 41.87 | 42.93 | 43.20 |
| Average | 71.88 | 65.95 | 62.63 | 68.70 | 88.08 | 80.62 | 81.53 | 81.65 |

We apply the same set of hyperparameters for both parameter efficient fine-tuning and full-scale fine-tuning, where we perform hyperparameter selection on learning rate $\{0.005,0.001,0.05\}$ and batch size $\{32,64,128\}$. We perform the fine-tuning for 300 epochs on imputation, anomaly detection, and feature regression tasks, and perform the fine-tuning for $4000$ steps on classification tasks.

The settings are applied consistently across all models to ensure a fair comparison. All models are optimized with an Adam optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.99$, and a weight decay of $1\times 10^{-5}$.

##### Model architectures

For all pre-training methods including NoTS-lw, we use a same channel-independent 1D-ResNet encoder for fair comparison. The encoder has $3$ ResNet layers of channel size $\{16,32,64\}$, each has $2$ ResNet blocks. The first convolutional layer has a kernel size of $7$, and the rest layers have a kernel size of $3$.
We append an additional convolutional layer after the ResNet blocks to alter the dimensionality $d$ of the token embeddings, where model variant $d=32$ is used for all experiments, and $d=16,64,128$ is trained for the scalability pilot study.
We use a $3$-layer $4$-head transformer with a token dimension of $d$, and $4\times$ size in the feed forward layer.
The decoder is built to be symmetric to the encoder architecture.

**Table 9: Complete classification results on the UEA datasets.**
|  | Parameter efficient tuning | Full-scale fine-tuning |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | NoTS-lw | Next-pred | bioFAME | SimMTM | NoTS-lw | Next-pred | bioFAME | SimMTM |
| EthanolConcentration | 28.14 | 25.48 | 28.14 | 25.48 | 30.04 | 29.28 | 27.76 | 28.90 |
| Heartbeat | 73.66 | 73.17 | 72.68 | 72.20 | 74.63 | 73.66 | 73.17 | 73.17 |
| PEMS-SF | 75.72 | 58.38 | 77.46 | 64.16 | 80.35 | 67.63 | 75.15 | 72.25 |
| SelfRegulationSCP1 | 79.18 | 77.82 | 69.97 | 59.39 | 89.08 | 86.01 | 85.67 | 74.06 |
| SelfRegulationSCP2 | 57.22 | 56.67 | 53.33 | 55.56 | 57.78 | 57.22 | 56.11 | 57.78 |
| Average | 62.78 | 65.95 | 62.63 | 68.70 | 88.08 | 80.62 | 81.53 | 81.65 |

**Table 10: Complete imputation results with masking ratio $12.5\%$ and $25\%$.**
|  | Parameter efficient tuning | Full-scale fine-tuning |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | NoTS-lw | Next-pred | bioFAME | SimMTM | NoTS-lw | Next-pred | bioFAME | SimMTM |
| 12.5% masking ratio |  |  |  |  |  |  |  |  |
| ETTm1 | 0.1556 | 0.2832 | 0.1957 | 0.1573 | 0.1194 | 0.1219 | 0.1251 | 0.1207 |
| ETTm2 | 0.1232 | 0.1774 | 0.1183 | 0.1243 | 0.1110 | 0.1164 | 0.1038 | 0.1041 |
| ETTh1 | 0.2764 | 0.4569 | 0.2471 | 0.2545 | 0.2091 | 0.2126 | 0.1966 | 0.1947 |
| ETTh2 | 0.1917 | 0.2692 | 0.1746 | 0.1796 | 0.1615 | 0.1886 | 0.1751 | 0.1632 |
| 25% masking ratio |  |  |  |  |  |  |  |  |
| ETTm1 | 0.1730 | 0.3280 | 0.2103 | 0.1697 | 0.1246 | 0.1377 | 0.1325 | 0.1244 |
| ETTm2 | 0.1294 | 0.1789 | 0.1257 | 0.1269 | 0.1205 | 0.1218 | 0.1095 | 0.1093 |
| ETTh1 | 0.2957 | 0.4738 | 0.2695 | 0.2734 | 0.2266 | 0.2440 | 0.2068 | 0.2078 |
| ETTh2 | 0.1994 | 0.2707 | 0.1824 | 0.1861 | 0.1653 | 0.1872 | 0.1806 | 0.1681 |

In our experiments, iTransformer (Liu et al., ) is implemented to transform inputs to an embedding dimension of $16$, where we build the decoder to be symmetric and linear. We used a $3$-layer $4$-head transformer network with a token dimensionality of $128$ and $4\times$ size in the feed forward layer.
PatchTST (Nie et al., ) is implemented with a patch length of $16$, stride of $8$, and token dimension of $32$. In cases where PatchTST becomes too computationally heavy (e.g., anomaly detection tasks), we adjust batch size to be $1$ and increase patch length to be $32$.

#### B.2.3 Complete experimental results

We show the complete classification results in Table 8 and Table 9 and the complete imputation results in Table 10. The averaged results are presented in Table 2.

##### Imputation task details

We perform a channel-wise imputation task instead of the traditional random imputation task. Specifically, when performing the masking, instead of uniformly sample random elements from $C\times T$ entries of ${\mathbf{S}}\in\mathbb{R}^{C\times T}$, we sample uniformly from $T$ columns and cover inputs from all channels. This is to eliminate the overfitting issues from the data embeddings.

#### B.2.4 Additional visualizations

We present additional data and token space visualizations in Figure 4 and Figure 5, respectively.

Figure: Figure 4: Additional data space visualizations.
Refer to caption: extracted/5917591/imgs/appendix_datavisual.png

Figure: Figure 5: Additional token space visualizations.
Refer to caption: extracted/5917591/imgs/appendix_latentvisual.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - Could the authors provide an analysis on how sensitive NoTS is to the choice of degradation operators? Specifically, how does the performance vary when different types of degradation operators are used, or when the balance between local and global smoothing is adjusted?
- Is there any guidance the authors could offer on selecting suitable degradation operators for different datasets or applications? A more detailed ablation on this aspect would clarify the robustness of NoTS across varying conditions.
- Is there any limitations sourced from the pretraining objective that stops authors testing on the forecasting task? If not do the authors plan to evaluate NoTS on long-horizon forecasting tasks? Given that capturing long-range dependencies is highlighted as a core strength in the introduction, an assessment on forecasting (especially long-range) would provide valuable insights.
- Could the authors provide an analysis of how NoTS’s performance contribution varies with increasing time series lengths? Understanding how NoTS scales with longer time series would be valuable for assessing its capability to capture long-range dependencies and its potential applications to extended temporal data.
soundness: 3
strengths: - The borrowed functional narrative approach is innovative for time series modality, addressing key limitations in time series modeling due to discretization. By leveraging function-based degradations, NoTS effectively captures temporal relationships, making it a strong framework for autoregressive analysis.

- Clear theoretical grounding in approximation theory supports the functional degradation and autoregressive prediction framework. Visualizations (e.g., degradation sequences) help convey complex concepts effectively.

- Experimental results show NoTS’s robustness across diverse tasks and datasets, with context-aware adaptors allowing minimal retraining. These adaptors facilitate handling different channel mappings and task-specific nuances, enhancing practical application.
summary: This paper presents Narratives of Time Series (NoTS), a transformer-based framework that reinterprets time series data as temporal functions, capturing both local and global properties of time series through a novel pre-training objective. The authors introduce three main innovations:

Progressive Functional Degradation: NoTS generates simplified versions of time series by applying degradation operators that progressively smooth the signal. Each degradation level creates a coarser function, enabling the model to learn time series as a sequence of simplified functions rather than individual time points, preserving the functional structure over time.

Autoregressive Transformer with Next-Function Prediction: Inspired by next-word prediction in NLP, NoTS trains the transformer to reconstruct the original time series from its degraded variants. By predicting the next function in a sequence, each step adds more information, allowing the model to capture nuanced temporal dependencies across multiple scales.

Adaptation via Channel and Task Adaptors: NoTS includes adaptable modules to handle different time series tasks and varying channel structures, which improves model generalization and reusability across datasets. These adaptors support context-aware adaptation and help the model perform well with minimal parameter updates, making it versatile across tasks.

Experiments and Analysis:
The authors begin by presenting a theoretical analysis that shows standard transformers struggle to generalize when applied to time series tasks that involve complex operations, such as differentiation, due to their reliance on discretized data. Specifically, the authors prove that transformers encounter high approximation errors when attempting to model certain mathematical operations (e.g., differentiation) on time series. Authors also prove that their proposed method viewing time series data as continuous functions in "function space" rather than as discrete can enable the transformer to better capture the underlying properties of the time series.

NoTS is evaluated on both synthetic and real-world time series datasets, covering tasks such as classification, imputation, and anomaly detection. NoTS demonstrates strong performance across diverse datasets, outperforming established baselines such as masked autoencoders and next-period prediction models. Ablation studies further validate the importance of each component, and scalability experiments indicate that NoTS’s performance improves predictably with model size, in line with power-law trends seen in other fields like NLP.
weaknesses: - The paper lacks an analysis of how sensitive NoTS is to the choice of the degradataion operators or guidance on selecting appropriate ones. This dependence could impact performance if the degradations are not well-suited to certain datasets or applications. The ablation study could be improved by including more ablation tests on this aspect.

- While NoTS is evaluated on classification, anomaly detection, and imputation, it lacks experiments on long-horizon forecasting tasks, which are crucial for assessing a model’s ability to capture long-range dependencies and temporal features (as highlihted in the intro). This addition would also reinforce the paper’s motivation, presented in the introduction, which critiques patch-based transformer architectures focused on forecasting tasks (e.g., Moirai and Lagllama).

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: - Line 142: "Typically, each dataset has its unique channel-wise relationships" ==> not clear what this sentence means.
- If the discrete-time signal is obtained by sampling at irregular intervals or contains missing data, discuss how convolution operations would work for the degradation function.
- Line 256: "encoder produces groups of tokens from each signal" ==> not clear how the signal is chunked into tokens in the first place. Is it through a PatchTST-like mechanism, or some other approach? 
- Lines 264-265: The specifics of this model should be given (perhaps in appendix), or a citation provided. The overall data flow, ranging from tokenization, encoder, decoder is very hand-wavy and hard to follow.
- Lines 282-283: The argument made to explain the addition of the latent consistency term is not clear and should be further explained.
- Section 3.3, test-time adaptors: it is not clear whether these adaptors enable zero-shot handling of new time series, or if adaptor fine-tuning is required.
- Eq. (3) specifies that the absolute error is used as the training loss. This makes the model sensitive to the scale of the time series — if the series are not scale-normalized, training will overweight series with the largest absolute magnitude. Further, section 5.2 does not discuss this issue, which calls into question the validity of experimental results.
soundness: 2
strengths: - Originality: the notion of "Narrative of Time Series" by predicting the "next function" in a functional decomposition of the time series is interesting and modestly novel. However, it is strongly patterned after the N-BEATS decomposition (see below), and links and connections to this earlier literature should be made.
- Quality: The experimental results are interesting, although the validity of results is called into question (below).
- Clarity: The paper is at times hard to follow, in particular with respect to notational sloppiness and shortcuts taken in detailing the model.
summary: This paper proposes a pre-training method for time series data called Narratives of Time Series (NoTS). NoTS frames time series modeling by considering time series samples as a sequence of functions rather than a sequence of time periods. NoTS utilizes degradation functions to generate augmented variations of a time series signal. These augmented signals have varying degrees of simplification and together form a sequence that is used to train an autoregressive transformer. The authors provide theoretical justification for framing time series as functional sequences, showing that this approach can approximate a broader class of functions compared to traditional segmentation into periods. The authors validate NoTS on a variety of tasks, including classification, anomaly detection, and imputation, using both synthetic and real-world datasets.
weaknesses: - Eq. 1 is sloppily written for the case $i=1$
- Section 2.1: there should be more rigor in the notation. The paper mixes discrete and continuous time, and does not specify how discrete time indices map onto functional arguments in continuous time. For instance, the notion of a "calendar" containing this mapping could be introduced. 
- Since the functional representation of time series is so important to the paper, Section 2 lacks a survey of the rich literature on the topic, e.g. from the Gaussian processes literature (e.g. https://proceedings.neurips.cc/paper/2007/hash/81e74d678581a3bb7a720b019f4f1a93-Abstract.html ) and classical statistics (e.g. “Functional Data Analysis” by J.O. Ramsay and B.W. Silverman (1997)).
- The proposed decomposition is strongly reminiscent of the N-BEATS approach, which is not acknowledged. The paper should contrast and compare their proposed approach to N-BEATS: 
```
@inproceedings{
Oreshkin2020N-BEATS:,
title={N-BEATS: Neural basis expansion analysis for interpretable time series forecasting},
author={Boris N. Oreshkin and Dmitri Carpov and Nicolas Chapados and Yoshua Bengio},
booktitle={International Conference on Learning Representations},
year={2020},
url={https://openreview.net/forum?id=r1ecqn4YwB}
}
```

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Which embeddings from the pre-trained model are used for downstream tasks? 
2. While following Jia et al. (2022), the adaptation of visual prompt tuning to time series data requires a more detailed explanation. Time series-specific prompt token structure and organization should be elaborated.
3. Please justify the alignment of $S_{k+1}'$ with $S_k$ in the loss function rather than $S_{k+1}$. The alternative of matching with $S_{k+1}$ should be discussed.
soundness: 2
strengths: 1. Reformulates time series modeling from a functional perspective and introduces degradation operators to create multiple views.
2. Proposes autoregressive modeling between resolution levels instead of traditional next-time-point prediction.
3. Provides function approximation analysis for transformer limitations on time series.
summary: This paper introduces NoTS, a novel pre-training framework that reinterprets time series data through a functional lens. The framework constructs alternative sequences through two types of degradation operators: local smoothing (using averaging kernels with lengths $p_k$) and global smoothing (using sinc functions with frequency cutoff values $0.5p_k$), which create progressively simplified variants of the original signal while preserving functional properties.

The pre-training works by training an autoregressive transformer to recover increasingly detailed versions of the signal. A channel-independent encoder first converts each degraded signal $S_k$ into token groups $R_k$, then the transformer predicts the next resolution level using previous levels, with a decoder reconstructing the predicted signals. The training combines an AR reconstruction loss comparing $S'_{k+1}$ with $S_k$ and a latent consistency term for the original signal.

For real-world deployment, NoTS uses channel adaptors (linear layer, retrainable channel tokens) and task adaptors (prompt tuning, task-specific linear layers) requiring <1% new parameters while maintaining 82% performance. Experiments across 22 datasets spanning classification, anomaly detection, and imputation tasks are evaluated.
weaknesses: 1. Limited Scope of Evaluation
- While the paper's focus on generalizable time series modeling, the crucial forecasting task is missing. The next-time-point prediction paradigm has clear connections to dynamic evolution, but the benefits of resolution-based prediction for forecasting tasks remain unexplored. This gap makes it difficult to fully assess the method's practical advantages.
- Performance improvements are moderate except for classification tasks.
- Limited datasets for imputation task. The ETT datasets are homogenous, and the model’s performance should be evaluated across other commonly used datasets as well, like Electricity and Weather.
- Limited ablation studies on key components. It would be better to show the impact of different degradation operators, degradation levels, and frequency cutoff points $p_k$ and diverse real-world datasets as well.
2. Minor parts:
- Technical clarification is needed on loss alignment choices, embedding usage for downstream tasks, and prompt tuning plan.

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 4
questions: 1, How does the NoTS framework compare to traditional time series modeling techniques in terms of interpretability and ease of use for practitioners who may not have a strong background in advanced machine learning methods?

2, What specific strategies could be implemented to enhance the scalability of the NoTS framework for large-scale time series datasets, and how might these strategies impact the performance and computational efficiency of the model?
soundness: 3
strengths: The method shifts the focus from traditional time series modeling to viewing data as functions of time, which enhances the understanding of temporal dynamics. The NoTS framework effectively utilizes degradation operators to create augmented data, improving the learning process for autoregressive transformers.
summary: The paper presents a novel approach to time series analysis by treating data as functions of time. It introduces a pre-training framework called NoTS, which uses degradation operators to create augmented data, allowing an autoregressive transformer to learn interrelationships among functions. Experimental results show that NoTS outperforms existing methods across various tasks on synthetic and real-world datasets. Additionally, a lightweight model, NoTS-lw, is introduced, achieving high performance with minimal training parameters. Overall, the work emphasizes the benefits of a functional perspective in enhancing time series modeling.
weaknesses: While the paper demonstrates the effectiveness of the NoTS framework across various tasks and datasets, the experimental validation is primarily focused on a limited number of synthetic and real-world datasets. This raises questions about the generalizability of the findings. The performance of the proposed method in more diverse and complex scenarios, such as high-dimensional time series or those with significant noise, remains untested. Expanding the range of datasets and tasks could provide a more comprehensive evaluation of the model's robustness and adaptability; The introduction of degradation operators and the functional perspective may complicate the implementation of the NoTS framework. Practitioners may find it challenging to adapt the proposed methods to their specific use cases without a deep understanding of the underlying theory. This complexity could limit the accessibility of the framework for researchers and practitioners who are not well-versed in advanced time series modeling techniques, potentially hindering its adoption in real-world applications; Although the paper discusses the scalability of the NoTS-lw model, it does not provide extensive insights into how the framework performs with significantly larger datasets or more complex models. The scalability of the approach in terms of computational efficiency and memory usage is crucial for practical applications, especially in industries dealing with large-scale time series data. Without thorough analysis and benchmarks on scalability, it is difficult to ascertain whether the proposed methods can be effectively utilized in resource-constrained environments or for large-scale deployments.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: * Line 104: "giving a consistent performance boost when performing dataset-specific pre-training" - I don't understand what is dataset-specific pretraining. I have only heard of pretraining on a large dataset, and then dataset specific adaptation. Please clarify.

* Line 123: "trained while maintaining 82% average performance" - average performance on what? is it the pretraining dataset? it should be made clear.

* Line 294: "In time series domain, it is common to encounter datasets with new channel graphs or new tasks at test time. To apply the pre-trained weights in such situations, we build two types of adaptors" - you should give evidence (cite a paper) or provide an example for the first sentence. I don't understand what is meant by "encountering new channels at test time". 

* Line 303: "We initialize and append prompt tokens to the transformer architecture along with data tokens following the deep visual prompt tuning plan as detailed in Jia et al. (2022)". Again, the authors should not expect a reader to look into another paper for such a crucial aspect of the model. I did not understand this part; the authors must explain the mechanism better.

* How sample-efficient is your model (does it do well few-shot)? Experiments like these can potentially substantiate your contribution.

* Why not test on the forecasting task too, which is, equally or maybe even more important than classification/imputation tasks?
soundness: 2
strengths: * The proposed approach of next function prediction for time series pretraining is novel and very interesting. The authors motivate their method very well. The authors bring a very different perspective for modelling time series.
* The implementation of their approach with degradation operators is also well-theoretically justified. The authors theoretically show that degradation operators avoid the problem of discontinuities when time series are broken up into functions.
summary: This paper proposes NoTS, a novel autoregressive pre-training method for time series. NoTS reinterprets time series as sequences of temporal functions generated by applying degradation operators of varying intensity levels to the original time series. The authors propose a "next-function prediction" task analogous to the next-word prediction task in language modelling and demonstrate the superior performance of NoTS on synthetic and real-world datasets.
weaknesses: * Clarity: I am unable to understand many parts of the paper due to unclear presentation; especially relating to the empirical evaluation of the method.  

* Evaluation: The protocol used for evaluation in Sec 5.2 is very much unclear. There are several references to Wu et al (2022) but the authors should not expect a reader to look into another paper, especially for the protocol of experimental setup, and the paper should be self-contained. I don't understand the protocol used even after looking into the paper. Do you pretrain on another dataset? If so, which dataset?

* Insufficient details on the training time, cost etc: It would be important to add the cost of using the architecture, both during pretraining and finetuning. In terms of training time / inference time / number of parameters etc.

Meta-review: The paper proposes NoTS, a transformer-based method for pretraining time series data by reinterpreting them as sequences of functions at different levels of detail. This is achieved through "degradation operators" that progressively smooth the time series. NoTS uses an autoregressive transformer to predict the next, more detailed "function" in the sequence. The authors claim this approach leads to better generalization and performance on various time series tasks, supported by theoretical analysis and empirical results on classification, imputation, and anomaly detection.

Strengths of the Paper:
Novel Approach: NoTS offers a new perspective on time series representation using functional degradation and autoregressive prediction.
Theoretical Foundation: The paper provides a theoretical analysis to support its claims.
Clear Presentation: The paper generally presents complex concepts effectively.

Weaknesses of the Paper:
Limited Evaluation Scope: The lack of experiments on long-horizon forecasting is a major weakness, hindering a complete assessment of NoTS's ability to capture long-range dependencies, a key claim in the introduction.
Sensitivity to Degradation Operators: The paper lacks a thorough analysis of how the choice of degradation operators affects performance and provides limited guidance on their selection. This dependence on the choice of operators raises concerns about the method's robustness and generalizability.
Unclear Comparisons: The comparison to N-BEATS, a relevant prior work, is insufficient. A more detailed discussion of the similarities and differences is needed to highlight NoTS's contributions.
Missing Details: Reviewers pointed out a lack of clarity and detail in the model architecture, data flow, and specific implementation choices, making it difficult to fully understand and reproduce the work.

While NoTS presents a novel approach with some theoretical and empirical support, the weaknesses outweigh the strengths. The limited evaluation scope, particularly the absence of forecasting experiments, raises serious concerns about the validity of the authors' claims. The lack of analysis on the impact of degradation operators and unclear comparisons to existing work further weaken the paper.  Therefore, rejection is recommended.

Decision: Reject