Large Pre-trained time series models for cross-domain Time series analysis tasks
Authors: Harshavardhan Kamarthi
B. Aditya Prakash
Venue: NeurIPS 2024 poster (2024)
Keywords: Time-series
Self-supervised Learning
Primary area: other
TL;DR: Novel time-series segmentation and self-supervised training for building general pre-trained time-series models capable of time-series analysis across multiple domains.

Abstract: Large pre-trained models have been vital in recent advancements in domains like language and vision, making model training for individual downstream tasks more efficient and provide superior performance. However, tackling time-series analysis tasks usually involves designing and training a separate model from scratch leveraging training data and domain expertise specific to the task. We tackle a significant challenge for pre-training a foundational time-series model from multi-domain time-series datasets: extracting semantically useful tokenized inputs to the model across heterogeneous time-series from different domains. We propose Large Pre-trained Time-series Models (LPTM) that introduces a novel method of adaptive segmentation that automatically identifies optimal dataset-specific segmentation strategy during pre-training. This enables LPTM to perform similar to or better than domain-specific state-of-art model when fine-tuned to different downstream time-series analysis tasks and under zero-shot settings. LPTM achieves superior forecasting and time-series classification results taking up to 40% less data and 50% less training time compared to state-of-art baselines.

Large Pre-trained time series models for cross-domain Time series analysis tasks
Harshavardhan Kamarthi
B. Aditya Prakash
College of Computing Georgia Institute of Technology harsha. beagha..kaaacc....mmmmom edu
College of Computing Georgia Institute of Technology badityappec. gatech. edu
Abstract
Large pre-trained models have been vital in recent advancements in domains like language and vision, making model training for individual downstream tasks more efficient and provide superior performance. However, tackling time-series analysis tasks usually involves designing and training a separate model from scratch leveraging training data and domain expertise specific to the task. We tackle a significant challenge for pre-training foundational time-series model from multi- domain time-series datasets: extracting semantically useful tokenized inputs to the model across heterogenous time-series from different domains. We propose Large Pre-trained Time-series Models (LPTM) that introduces novel method of adaptive segmentation that automatically identifies optimal dataset-specific segmentation strategy during pre-training. This enables LPTM to perform similar to or better than domain-specific state-of-art model when fine-tuned to different downstream time-series analysis tasks and under zero-shot settings. LPTM achieves superior forecasting and time-series classification results taking up to 40% less data and 50% less training time compared to state-of-at of-art baselines.
Introduction
Time-series analysis tasks are important well-studied problems such forecasting (Hyndman and Athanasopoulos 2018) and classification (Chowdhury et al. 2022) with applications wide-ranging domains such as retail, meteorology, economics, and health. Recent works (Chen et al.) 2021 Wang
ct al. 2022 Zeng et al. 2023) have shown the efficacy of purely data-driven deep learning models. However, most state-of-art neural models known to be data-hungry and require substantial training data. Motivated by language and vision foundational models Bommasani et al. (2021), recent body of works build pre-trained time-series models Das et al. (2023b); Ansari al. (2024); Jin al. (2024): Rasul et al. (2023). These models are trained on diverse datasets from different domains during pre-training. They require less training resources and data and produce superior performance. These models can also be deployed without any training, in a zero-shot or few-shot setting.
These foundational models, however, require large amounts of data for pre- training, which is still challenge for time-series datasets. Moreover, they do not provide consistent performance improvement across all the domains. We identify important challenge to building unified pre-trained model for time-series that is pre-trained on and deployed to multiple domains: representation of diverse time-series input into these models.
Most neural sequential models input time-series values for each time-step separately. However, unlike text data, each individual time stamp may not provide enough semantic meaning about local temporal patterns the time series. To tackle this, [his,[Nie et al. (2022) proposed to segment the time series and input each segment as individual tokens to their transtormer-based model. This simple segmentation method tokenizing time-series has been used by recent mre-trained models Woo et al.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(2024); Das et al. (2023b) to provide superior performance across multiple applications. However, segmenting imput time-series uniformly with fixed-length segments, while simple, can be a very inflexible tokenizing method, especially when dealing with datasets from multiple domains with different set of underlying generative dynamics, sampling rate, noise, etc. For example, among two datasets, a dataset with a lower sampling rate (such as GDP time-series) may require longer segments than those with higher sampling rates to capture similar patterns in the model (such as heart sensors collecting data in milliseconds). However, note that the dynamics of the same time-series may vary with time Liu et al. (024a). For, time intervals that are smoother with less complex dynamics, using longer segment sizes may suffice but intervals where time-series temporal patterns may require finer-graine segmentation. For example, for seasonal epidemics, time-series is smoother during the off-season and has more complex dynamics during outbreaks and times of higher incidence.
complex and have multiple
We tackle this important problem of representing diverse time-series datasets when training a pre- trained foundational model for time-series and propose Large Pre-trained Time-series Models (LPTM), novel method for generating pre-trained models for time-series data across multiple domains. LPTM uses a simple transformer-based architecture and leverages masking-based self- supervised pre-training to train on multiple datasets from different domains. Our main contribution focuses on how we input time-series segments as tokens to the transformer. To overcome the challenges associated with segmentation on diverse datasets discussed above, we propose novel adaptive segmentation module that segments the time-series of each domain based on how well performs supervupeee pre- pre-training. The segmentation module uses novel scoring mechanism during pre-training to identify an effective segmentation strategy for a domain. L.PTM can be fine- tuned or deployed in zero-shot setting to various forecasting and classification tasks in domains such as epidemiology, energy, economics, behavioural datasets, etc. LPTMM provides performance on par with state-of-art models with lesser pre-training data, training data and fewer training steps. Our main contributions can be summarized as follows:
Multi-domain Pre-trained time-series model We propose framework for generating foundational pre-trained models for time-series that are trained on multiple datasets across varied domains. LPTM solves the tokenization problem for cross-domain time-series data and proposes novel adaptive segmentation module which is important to build pre-trained models for time-series similar to foundational models for text and vision.
Adaptive segmentation for cross-domain pre-training optimally extract semantically useful information from time-series of different domains with varied dynamics and sampling rates for pre-training, we propose a novel adaptive segmentation module that learns segmentation strategy for each domain based on losses from self-supervised learning tasks.
State-of-art and efficient performance in diverse downstream time-series tasks We evaluate LPTM on downstream forecasting and classification tasks from multiple domains and observe that LPTM consistently provides performance similar or better than previous state-of-art models usually under zero-shot evaluation as well as when fine-tuned with lesser training data and compute time. Overall, we also observe that LPTM typically requires less than 80% of training data used by state-of-art baselines to provide similar or better performance.
2 Problem Setup
Time-series analysis tasks Our pre-trained model can be used for many time-series tasks including forecasting and classification from multiple benchmarks and domains. For given downstream task let D" be the time-series dataset consisting of time series yi.... A time-series analysis task's goal is to predict important properties of the time-series. For example, the forecasting task involves predicting p the future values y I...T+ whereas classification involves predicting the class label of the input time-series based on labelled training data.
Self-supervised pre-training on knowledge and patterns from time-series datasets from time-series from different domains. Formally, we have access to time-series datasets from K domains where the datasets of domain le is denoted as D/ (D), (D/ h= where N(k) is the number of datasets in domain k. Examples of these domains include epidemiology, energy forecasting, macrocconomics, traffic prediction, etc. The entire set of
datasets The goal of our work is to learn useful

---

heterogenous multi-domaij pre-train dataset is denoted as pre - (Di. effectively pre-train LPTM on D pre we formulate the problem as set of supervieed learning tasks Tp { on the set pre-training datasets 2 pre During pre-training, we sample (Di, k), dataset and its domain label from pre and train the model on each of the self-supervised learning tasks in Tee- The tasks in Tie are self-supervised and do not require additional labels or other ground truth. These tasks mask patches of the input data and train the model to recover the original input.
In order to
Therefore, our problem can be formally stated as: Given a heterogeneous set of multi-domaii datasets pre and their domain labels, we train model leveraging SSI. lasks Tpe that learns important patterns and knowledge that can be leveraged on fine-tuning the model to any time-series analysis task on any novel dataset from any of the domains d (1,2...., K}.
3 Methodology
Segment Ecoring and pouning segments
Outpul embeddings
-00
GRU
A(2)
Inaralorerr
2
Chosen subsot
subsequencees 5ly0.
Figure 1: Overview of LPTM. The input time-series y (I...T) is first segmented based on a scoring function optimized using SSL loss. The segments are fed as individual tokens to the transformer encoder to get output embeddings of time-series that are used for downstream tasks.
Similar model pipelines used in NLP and vision, first train pre-trained model M (0 on multiple pre-training datasets Dpe- Most parameters of the pre-trained model Opee are trained over all the datasets and tasks. However, domain to capture varied sizes of segments that differ across datasets. These segments are used as tokens for a transformer model that shares the parameters across all the tasks. For each pre-trained and downstream task, we append final linear layer on the output embeddings of the transformer to generate the final prediction. Note that during fine-tuning on downstream tasks we update the parameters of all the modules of L.TMM.
use separate segmentation module for each dataset
3.1 Adaptive Segmentation module
Previous works input each time-step time-series as individual tokens or fixed-sized segments. Unlike text, individual time-steps do not typically provide any semantic meaning about the temporal patterns of the time-serics. Therefore, Nie et al. (022) proposed to segment the input time-series into uniform length segments and use each of the segments tokens to the transformer model. Segments fixed size are also too inflexible to capture semantics of sequences that show varied behaviour across time and across datasets. Different pre-trained datasets may have varied temporal scales, periodicity and other temporal dynamics that cannot be encompassed by a single uniform segmentation strategy. For example, epidemic time-series have characteristic properties like seasonality, peaks and sudden outbreaks that should be captured by segmentation. Economic time-series, in contrast, more monotone with sudden anomalies and changes in data distribution. Moreover, using a uniform segmentation may not be ideal for time series with multi-scale trends with some time-stamps having denser temporal information requiring finer-graned segmentation than others. Therefore, our goal is identify an independent segmentation strategy for each domain of time-series dataset.
usually observed weekly and may
typically captured every quarter and
Overview LPTM automatically identifies effective segmentation strategy for each dataset
domain during pre-training. The overarching goal of the segmentation module involves identifying segments that lead to lower SSL loss. The segmentation module first scores all possible segments of

---

the input time-series and the segments with the highest scores are then chosen as the output segments used tokenise the time-series. The scoring function trained such that the score the segments will reflect how likely the chosen segments will lower the SSL loss.
Details For a given input time-series y2....), we pass it through a GRU to get hidden embeddings {z())= Ji=1 that models the temporal patterns of the input:
{z J (G
(I)
We then introduce a segment score finction that provides a scalar score for any subsequence the input time-series:
s(i,)) = = tanh ( z, W2z, + b).
Intuitively, the score si.,j) j) for subsequence from time-stamp to denotes how good the given segment is for the dataset when optimizing for the SSL loss.
In next step, we sample subset (t. of subsequences over the time-series that a) covers the entire input time-series, b) has high score function value. While retrieving the optimal ((y(1...)) is an interesting combinatorial optimization problem, we generate (Iy(...)) using simple process as follows: for each E {1,2..... 1 1}. we denote h(i) = arg ( s(t,j) as the best segment starting from time-step Then generate the set of segments ((y {(i, H( To reduce the number of segments, we iteratively remove the lowest-scoring segments until we cannot remove any more segments without having time-steps not covered by any segments in the set. The final set of segments after pruning is denoted as (()()..))) The segmentation procedure is summarized in Alg.
Algorithm 1: Adaptive Segmentation Module
Input :ime-series y(I...t) = (1 Procedure GetScores ().....))
2 {z + GRU (y (1) M==); for i E {1,2.....
// Encode time-series with GRU
....,
} do
for j E (i (i+1) ........,. 1) do
((,,)) v tanh (Wiz w2z(f) + b): :// Scores for all segments
return {s(i,))) icj
i.j. (1..
(4t Procedure ChooseSegments (1s(i, Kgj (E( for (1.2...... 1) do h(i) + arg (it s(i,j); Š+ ((i)())))
// Best segment starting at index i
11
i t arg min HAAA)E) h();;
// Select lowest scoring segment
12
while Segments in S - {i,, h(i)) doesn't cover all time-steps 21, .t do
13
5 ((i, (())):
14
arg min H(A Mj);
return S S:
15
To generate the token embeddings é(i.j) for each segment (i.j), we pass the embeddings (z()) (++1) ....,, 200} through self-attention layer used in transformers and aggregate the output embeddings.
e(i.j) = Σ Self-Atten (2 (02 (+1)) 200)
Additionally, concatenate the following features the token embedding of each segment token provide information regarding the position and length of the segment: (1) Positional encoding of the starting time-step of the segment pis(i) defined as:
pos(i) ( (0ssi//00)/)))) cos(i (oos(i/105(-1)/p)
if i is even
if is odd.

---

where D is the dimensions of output embedding of self-attention over {ê,, ē,+1..... ,r) (2) Po- sitional encoding of the length of the segment pos(j i). The final output the segmentation module is sequence (e, R := where R number of segments. The segments are arranged based on the ascending order of the first time-stamp of each segment. The token embeddings are fed into stack of transformer layers similar to the encoder of the transformer Vaswani et al. (2017). The output of the transformer layers a sequence of output embeddings denoted as {o,
3.2 Self-supervised learning Tasks
Pre-tre tre-training on a wide range of heterogeneous datasets from multiple domains helps LPTM learn from useful patterns and latent knowledge across these domains that be generalized to range downstream tasks on different domains. We propose two general self-supervised learning tasks motivated by pre-trained language models enable L.TM to learn from all pre-trained datasets. We leverage a transformer model and use the segment token embeddings of the segmentation module. The two pre-training SSL tasks are Random Masking (RANDMASK, and Last token masking (LASTMASK). RANDMASK allows the model to extrapolate and interpolate masked segments of the input time-series. RANDMASK has also been explored for representation learning in previous works (Zerveas et al. 2021; Nie et al., 2022) but are applied on the same dataset as that used for training unlike our data and task aagnosiic pre-training setup. Formally, we mask each input segment token with a probability of y and decode the values of time-series of the masked segments from the output embeddings of the transformer. We use a simple GRU with a single hidden layer on the output embedding of the masked token to decode the values of the segment. We use mean-squared similar RANDMASK ASK except we mask last y fraction of the
error as the loss LSsz. LASTMASK segments. This allows the model to forecast the future values of the seriss, an important task in many time-series domains.
3.3 Training details
The values of the time-series of each dataset can vary widely based on the time-series domain. Therefore, as part of pre-processing, first normalize the time-series of each dataset of pre-train datasets independently. Moreover, the data distribution and the magnitude of the time-sent series et al.] (2021). REVIN performs instance normalization on the input time-series and reverses the normalization of the output values. The normalization step is part of the neural model and gradients are calculated over the normalization and reverse normalization layers.
Instance normalization
vary across time. We use reversible instance normalization (REVIN) layer Kim
Training the score function We use the loss from the SSL tasks to train the score function of the segmentation module and GRU in Eqn. Since there no direct gradient flow between the score function and the final predictions, due to the discrete nature of choosing the segments, we match the aggregated scores of all the chosen segments in (()(...)) to the negative logarithm of the total MSE loss of both SSL tasks:
C, (
Σ (tg..)
s(i.j) +lg((sst))
(5)
where LSSL is the total loss of both SSL tasks. We also backpropagate over Lg once every 10 batches to stabilize training since changing the segmentation strategy for every batch leads to unstable and inefficient training.
Linear-probing and fine-tuning Kumar al. (022) showed that fine-tuning all the parameters of the pre-trained model for specific downstream task can perform worse than just fine-tuning only the last layer (linear probing), especially for downstream tasks that are -ut-tf-distribution to pre-traine trained data. To alleviate this, they suggest performing two-stage fine-tuning process: we first perform linear probing followed by fine-tuning all the parameters.
4 Related Works
Neural models for time-series analysis DeepAR Salinas et al. (022)) a popular forecasting model that trains auto-regressive recurrent network to predict the parameters of the forecast
5

---

distributions. Deep Markov models Krishnan et al. (2017): Rangapuram et al. (2018) Li et al. (2021);
Gu et al. (202T) model the transition and emission components with neural networks. Recent works have also shown the efficacy of transformer- based models on general time-series forecasting Oreshkin jet al. (2019; Zhou et al. (202T); ;; Chen et al. (202T); /hou et al. (2022); Liu et al. (202T). However, these methods do not perform pre-training and are trained independently for each application domain.
therefore, they do not leverage cross cross-domain datasets to generate generalized models that can be used for wide range of benchmarks and tasks.
Self-supervised learning for time-series Recent works have shown the efficacy of self-supervised representation learning for time-series for various classification and forecasting tasks in a wide range of applications such as modeling behavioral datasets Merrill and Althoff (2022); Chowdhury et al.
(2022), power generation Zhang et al.] (2019), health care Zhang et al. (2022) J. Franceschi et al.
(2019) used triplet loss to discriminate segments of the same time-scries from others. These works use methods such as contrastive losses Eldele et al. (202): Yue et al. (2022) or other similarity betric-bbsed techniques Toneeabooi et al. (2021p. However, all these methods apply SSL on the same dataset that is used for training and may not adapt well to multi nolti-domain datasets. There has been some recent works leveraging foundational models like LLMs for time-series forecasting across multiple applications. One set of works directly uses LLMs without fine-tuning to perform time-series forecasting via careful prompting Grunnerrrrrrrrrr al. (024); :Ti al. (2024); Liu et al. (2024d). Other works fine-tune LLMMs specifically for time-series forecasting /hou et al. (2023)) Rasul et al. (2023) y Ansari al. (224)). The time-series representation used by these models includes using individual time-steps as input Ansari et al. (2024), converting each digit of the time-series to character embeddings to be directly used by LLMs Gruver et al. (2024); ): JJin al. (2024) and uniform segmentation Das al.)
(2023b); Woo et al. (2024); Zhou et (2023). LPTM performs superior to these methods while being 10x to 100x smaller than the large LLMs used as backbones.
5 Experiment Setup
Datasets We derive pre-train time-series datasets from multiple domains: + Epidemics: We use a large number of epidemic time-series aggregated by Project Tycho (van Panhuis et al.) 2018). from 1888 to 2021 for different diseases collected at state and city levels in the US. We remove time series with missing data and use time series for 11 diseases of very diverse epidemic dynamics such as seasonality, biology, geography, etc.: Hepatitis A, measles, mumps, pertussis, polio, rubella, smallpox, diphtheria, influenza, typhoid and Cryptosporidiosis (Crypto.). Electricity: We use ETT electricity datasets (ETTI and ETT2) collected from (Zhou et al., 202T) at hour intervals over 2 years. We use the default 12/4/4 train/val/test split and use the train split for pre-training as well. * Traffic Datasets: We use 2 datasets related to traffic speed prediction. PEMS-Bays (PEM-B) and METR-LA (Li et al. 2017) are datasets of traffic speed at various spots collected by the Los Angeles Metropolitan Transportation Authority and California Transportation Agencies over 4-5 months. Demand Datasets: We use bike and taxi demand datasets (NY-B, NY-T) from New York City collected from April to June 2016 sampled every 30 minutes. We all but the last 5 days of data for training and pre-training. Stock forecasting: We also collect the time-series of daily stock prices of Nasdaq and S&P 500 index using Yahoo finance data (Xu and Berkely 2014) from July 2014 to June 2019. We train and pre-train using the first 800 trading days and use the Tast 400 for testing. M4 competition time-series: We also used the 3003 time-series of M4 forecasting competition (Makridakis and Hibon 2000) which contains time-series from multiple domains including demographics, finance, and macroeconomics. Motion and behavioral sensor datasets: We use the of sensor datasets extracted from UEA archive (Bagnall al., 20188) and UCI Machine learning repository (Asuncion and Newman 2007) similar to (Chowdhury et al. 2022). Note that our publicly accessible pre-training dataset is significantly smaller than other pre-trained datasets used by past work (nsari et al. (2024): Das et al. (2023b) some of which use confidential
data inaccessible to us. We also do not use any synthetic datasets like Das et al. (2023b); Ansari et al. (2024).
Downstream tasks We test the pre-trained LPTM trained on datasets discussed above multiple forecasting and time-series classification tasks. We perform forecasting on the influenza incidence time series in US and Japan. Specifically, we use the aggregated and normalized counts of outpatients

---

exhibiting influenza-like symptoms released weekly by CDC For influenza in Japan, we use influenza-affected patient counts collected by N We forecast up 4 weeks ahead over the period of 2004 to 2019 flu seasons using a similar setup as Flusight competitions Reich et al. (2019).
We also perform electricity forecasting mentioned previously. The last 10% of PEM-Bays dataset is used for traffic forecasting to 1 hour ahead and the last days of New York demand datasets for demand forecasting up 120 minutes in the future. We also perform forecasting on the Nasdaq dataset for up to 5 days ahead and M3 time-series for I month ahead. We use 6 of the sensor datasets from Asuncion and Newman (007) for time-series classification tasks. We use an 80-20 train-test split similar to Chowdhury et al. 2022).
the ETTI and ETT2 datasets using the train/test split
Baselines We compared LPTM's performance in various time-series tasks against twenty two state-of-the-art general forecasting and domain-speciff baselines. First, we compare against recent pre-trained foundational time-series models: (1) LLM-Time Gruver et al. (2024). (2) TimesFM Das
et al. (2023b). (3) Lag-LLAMA Rasul al. (202)). (4) Chronos Ansari et al. (2024) and (5) MOIRAI Woo et al. (2024). We skip models like Lime-LLM/ et al. (2024). MOMENT Goswami Goswanioswwami et al. (2024) and Autotimes Liu et al. (2024b) which cannot perform zero-shot forecasting across domains and are outperformed by the aforementioned recent models when fine-tuned. We compared
with (6) Informer /Zhou et al. (2021), (7) Autoformer [Chen al. (2021), (8) i'Transformer' Liu et al.)
(2023) and (9) PStch' Nie et al. (022), four state-of-the-an transformer-basee forecasting models.
We also compare against other recent model (10) MICN (Wang et al. 2022), (11) TiDE Das al.
(2023a) (12) TFT Lim et al. (2021) and (13) TimesNeT Wu et al. (2023 We also compare with (14) N-HITS Challu al. (2023) which uses multi-scale interpolation and (15) AutoARIMA Hyndman and Khandakar (2008) a ARIMA based model that does automatic hyperparameter search. We also compare it against three other state-of art self-super methods for tume-series: (16) TS2Vec Yue
ct al. 2022). (17) TS-TCC (Eldele et al. 2021 and (18) SimMTM Dong et al. (2024) uses masking as trained task for time-series classincation.
Finally, we compared against the best models for individual tasks for each domain. For influenza forecasting, compared against previous state-of-art models (19) Epi Kamarthi et al.) (2021) and (20) ColaGNN Deng et al. (2020) respectively. We also compare against (21) STEP Shao et al. (2022) SOTA model for demand forecasting, traffic prediction, and stock prediction benchmarks among the baselines by automatically modelling sparse relations between multiple features of the time-series. For classification tasks, we compare against (22) TARNet Chowdhury al. (2022)
Results
Table 1: Average zero-shot forecast performance (measured as RMSE over 10 runs) LPTM and pre-trained baselines. The best model is in bold.
Model
Flu-US
Flu-japan
ETTI
ETT2
PEM-B
NY-B
NY-T
Nasdag
M4
LLMe-Tmme
1,38
1411
0.57
0.54
4.3
4.5
13.53
0.29
1.189
TimesIiM
1.35
1259
0.61
0.59
3.9
3.9
13.11
0.29
1.211
Lag-LLAMA
1.52
1488
0.83
1.06
5.3
3.8
12.84
0.24
1.311
Chronos
1.29
1274
0.62
0.56
4.2
3.6
13.74
0.29
1.125
MOIRAI
1.39
1411
0.69
0.52
4.2
4.4
13.82
0.27
1.192
TS2Vec
1.94
1233.1
1.33
1.82
3.7
4.1
14.39
0.87
1.616
TS-TCC
2.17
1356.15
1.14
1.57
4.1
3.8
15.72
0.92
1.492
SimMTM
2.17
1356.15
1.14
1.57
4.1
3.8
15.72
0.92
1.492
LPTM
1.14
996
0.53
0.49
3.4
3.2
13.12
0.22
0,972
The code for implementation of LPTM and datasets are provided anonymized link and hyperpa- rameters are discussed in the Appendix. LPTM consists of 10 layers for the transformer and overall has about 100M parameters which 2x to over 10x smaller than other pre-trained time-series models.
Zero-shot forecasting An important benefit of foundational models in language and vision is their ability to adapt to novel tasks without any fine-tuning in a zero-shot setting Brown et al. 2020). We
html
http::////aaaaaappppaaaaa/////aa htmll

---

Table Average forecast performance (measured RMSE over runs) LPTM and baselines over different domains. The best model is in bold and the second best is underlined.
Model
Flu-US
Flu-Japan
ETTI
ETT2
PEM-B
NY-B
NY-T
Nasdaq
M4
Rank
AutoARIMA
2.14
1344
0.73
0.64
4.1
4.13
16.43
0.62
1.89
25,06
Informer
1.62
1139
0.57
0.71
3.1
2.89
12.33
0.83
1.0S5
15.89
Autoformer
1.41
1227
0.72
0.82
2.7
2.73
12.71
0.19
0.887
13.67
PatchTST
0.96
1113
0,52
11.95
0.877
7.5
N-HITS
1211
0.53
0.57
0.968
13.0
TiDE
1.21
1186
0.49
0.49
3.5
3.86
11.95
0.57
1.078
13.44
MICN
0.95
1145
0.49
0.57
3.6
2.61
11.56
0.13
0.931
7.77
TimesNet
1.04
1194
0.56
0.62
3,9
2.83
11.82
0.19
1.055
12.11
TFT
1.21
1876
0.52
0.51
4.6
2.95
12.55
0.24
1.18
16.11
Transformer
1256
LLM-Time
1.21
1319
0.52
0.49
3.9
12.11
0.21
1.064
14.05
TimesFM
1.32
1214
0.58
0.49
3.7
2.8
12.19
0.22
1.07
13.44
Lag-LLAMA
1.46
1416
0.61
0.57
3,9
2.9
13.43
0.28
1.33
20.16
Chronos
1.21
1228
0.59
0.52
3.7
3.1
12.82
0.27
1.04
15.55
MOIRAL
1.31
1336
0.62
0.55
3.9
13.71
1.21
19.22
STEP
1.17
983
0.54
1.331
10.33
EpiENP
0.52
872
0.81
1.25
4.1
2.98
12.11
0.28
1.281
16.77
ColaGNN
1.65
694
0.72
1.19
3,9
3.19
14.97
0.25
1.185
19.22
TS2Vcc
1.55
905,9
0.99
1.74
3.5
3.11
13.48
0.94
1.344
21.94
SimMTM
1.31
1289
0.61
0.55
3.4
3.1
12.79
0.28
1.284
17.94
TS-TCC
1134.6
0.75
1.274
21
LPTM
0.79
704
0.49
0,872
2.55
LPTM-NoSegment
0.93
766
0.57
0.55
3.2
3.17
14.96
0.27
1.146
13.72
Trann
0.96
827
0.46
0.57
3.7
2.66
12.43
0.25
1.271
11.66
LPTMMNNoLinProl
0.92
885
0.43
0.53
3.1
2.49
12.17
0.19
1.032
6.55
LPTM-Only Malskakka
0.87
895
0.51
0.52
2.8
2.42
12.36
0.21
1.076
8.0
Lass
0.79
773
0.44
2.31
12.04
0.19
0.916
3.77
evaluate the zero-shot performance of LPTM and other pre-trained baselines. Similar to Gruver et al. (2024) we use the last 20% of the datasets for zero-shot evaluation. We not fine-tune the models but only input the normalized time-series for new tasks directly during inference. The results are summarized Table LPTM outperforms all the baselines significantly. Moreover, the baselines such TS2Vec, TS-TCC and SimMTM which not designed to handle datasets from multiple domains perform much worse than other pre-trained methods. This shows the importance of adaptive eemmenation-basee tokenization of LPTM to better generalize across multiple domains.
Fine-tuned forecasting We also train the models with training data to evaluate fine- senettned perfor- mance. The forecasting performance are shown in Table 2 We evaluated LPTM against seventeen other forecasting baselines. L.PTM is either the first or a close second best-performing model in all the benchmarks. LPTM generally outperforms general time-series forecasting methods well as 2recent pre-trained models in all benchmarks despite its much lower parameter count (2-10x lower). Further, competitive or superior in performance to domain-specific methods designed specifically for the given domains (such as EpiENP and STEP). LPTM beats the previous state-of-art domain-specific baselines in five of the benchmarks and comes second in four more, Finally, LPTM improves upon the state-of-an art on electricity forecasting, traffic forecasting, and M3 datasets.
Time-series classification Unlike other autoregressive foundational models designed for forecast- ing, LPTM can be used for classification due to its encoder-style architecture. We add single classification layer over pooled output embeddings (o 1
]==1 to predict the class logits. We evaluate LPTM and baselines the classification of 35 sensor and behavioral datasets from UCI classification
repositiry (Asuncion and Newman, 2007). We report the accuracy scores in Table We observe that LPTM has highest mean rank and largest number of times outperforms all basclines.
Data efficiency A significant advantage of leveraging pre-trained models is that we do not require large datasets for fine-tuning specific task. We evaluate the efficacy of LPTM to train with fraction of training data. For each time-series analysis task, we fine-tune the model using only k% of training data for different values of We use the first 4% the timestamps' values. We do not choose a random sample prevent data mixing from the rejected portion of training data. We also performed the similar experiment on the best baseline for each task and compare data efficiency of baseline with LPTM. The comparison plots shown in Figure With lesser data, the performance

---

Table 3: Average classification performance (measured accuracy score 10 runs) of LPTM and baselines over different domains. The best model in bold and the second best underlined. The best model is statistically significant over the baselines (p S 0.05) when it beats the previous state-of-art.
Informer
Autoformer
TimesNet
TARNet
TS2Vec
TS-TOC
TST
SimMTM
CRT
LPTM
BasicMotions
0.95
0.93
0.92
1,00
0.99
1,00
0.92
0.86
1.00
FaceDetectton
0.51
0.49
0.59
0.63
0.51
0.54
0.62
0.73
0.78
0.79
FingerMovement
0,47
0.59
0.78
PEMS-SF
0.67
0,71
0.84
0.94
0.75
0.73
0.93
0.86
0.89
0.93
RacketSports
0.85
0.79
0,93
EigenWorns
0.49
0,62
0.73
0.89
0.84
0.77
0.72
0.82
0.79
0.94
Articulary WordReecognitionn
AtrialFibrillation
0.57
0.55
0.68
1,00
0.44
0.37
0.72
0.85
0.89
0.93
Erajetoe/Essscctories
0.97
Cricket
0.94
0.87
0.88
1,00
0.98
0.97
0.84
0.96
0.94
0.99
DuckGeese
0.54
0.44
0.56
0.75
0.39
0.57
0.74
0.58
0.55
0.79
Epilepsy
0.58
0.57
0.61
1.00
1.00
0.98
0.94
0.78
0.55
0.97
ERing
0.23
0.29
0.46
0.92
0.89
0,78
0.86
0.64
0.75
0,97
concentratonn
0.14
0,27
0.34
0.32
0.45
0.37
0.46
0.34
0:21
0.53
HandMovemenn NowemeenDiieeiioonn
0.14
0.27
0.34
0.32
0.45
0.37
0.46
0.34
0.21
0,53
Handwriting
0.16
0,18
0.29
0.24
0.52
0.55
0.37
0.64
0.52
0.51
Hearthear
0.53
0.66
0.61
0.78
0.71
0.09
0.74
0.74
0.62
0.74
InnceeWingbcatt
0.13
0.16
0,14
0.14
0.18
0.22
0.69
0.62
0.18
0.72
Nowels
0.87
0.96
0.94
0.99
0.97
0,98
0.99
0.94
0.99
0.98
Libras
0.75
0,92
0.95
LSST
0.36
0.44
0.32
0.97
0.54
0.62
0.65
0.53
0.48
0.98
MotorLmagery
0.62
NATOPS
0.75
0.69
0.84
0.92
0.91
0.92
0.94
0.85
0.88
0.94
PonDigits
Phoneme
0.11
0.13
0.15
0.17
0.28
0.226
0.29
0.26
0.18
0.32
SelfRegulation
0.65
0.57
0.81
0.77
0.84
0.89
0.96
0.92
SpokenArabicDiiie
0.92
0.96
0.92
0.98
0.94
0.97
0.99
0.96
0.94
1.00
StandWalkJump
0.21
0.34
0.53
0.28
0.31
0.61
0.52
0.58
U'WaveGesturre
0,79
0,86
0.82
0.87
0.91
0,92
0.86
0.74
0.82
0.94
PAMAP2
0.73
0.87
0.84
0,97
0,93
0,94
0.96
0.93
0.88
0.97
Opportunity Gestures
0.73
0.68
0.83
0.92
0,74
0.74
0.62
0.72
0.92
Opportunity occometion
0.84
0.78
0.85
0.91
0.75
0.82
0.84
0.87
0.89
0.89
SelfRegulationSCP2
0.562
0.598
0.612
0.622
0,593
0.575
0.604
0.614
0.625
Occupancy
0.774
0.739
0.814
0,833
0.876
0.865
0.881
0.826
0.814
0,836
MaquiioSound
0.662
0.691
0.715
USu:US
ET12
NY-Bike
M3
2.5
UIN
Best Base Ihe
UPM Best Baseline
Best Boseine
2.5
Best Base the
2.0
ROSE
1.5
2.5
10
0.5
0.4
0.5
0.5
0.8
1.0
0.4
0.5
0.0
0.8
Thining
Training
Figure 2: Performance of LPTM and best baseline with varying fractions of training data. In most cases LPTM significantly outperforms baselines with lower amount of data.
of the baseline is much worse whereas LPTM typically requires much less data provide similar performance to when we have access to the full dataset. This shows the importance of pre-training to quickly ramp up the performance of the model with much less data, a problem we encounter many real-world settings such as when we need to deploy forecasting model on novel applications such as new pandemic with sparse data availability.
Training efficiency An important advantage of pre-trained models is that they require much less training time and resources to fine-tune to a downstream task compared to time required for pre- training or even training from scratch. We compare the fine-tuning time of LPTM with baselines on benchmarks from different domains. We also measure the average time required by LPTM reach the performance of the best baseline in cases where we eventually outperform them. The training times are summarized in Appendix Table We observe that the time taken by LPTM to reach performance of best best-performing baseline (LPTM-TB) is significantly smaller than the time taken by any other baselines. Even when L.TTM doesn't 't outperform the best baseline, typically converges much faster.

---

Ablation and Sensitivity Studies We study the impact of our adaptive segmentation as well as pre-training and linear probing via the ablation models J.PTM-NNSeemmnt, LPTM-NoPre and LPTM-Nol LinProh. We also investigate the individual impact of both SSL task via the ablation models L.PTM-OnlyRandMask and L.PTM-Only' Anta..aaa..... The performance of the ablation variants are also shown in T We observe that the ablation variants' performances are significantly worse than the variants, underperforming some of the baselines. The worst performing variant is usually LPTM- NoSegment, showing the importance of deriving good time-series segments to improve representation learning of time-series for each dataset. We also examined the sensitivity hyperparameter ry for SSL tasks and found the optimal value 0.4 for LASTMASK and 0.2 for RANDMASK. The sensitivity analysis plots are in Appendix Fig. 5
Segments generated by LPTM We also visualized the segments in Fig. 3 segment sizes smaller in regions of high variance important parts of the time-series such as peak of the epidemic whereas simpler trends have longer segments which matches our intuition.
We observe that the
Valla
0.0
100
(a) Flu-US
ETTI
BasicMotions
Figure 3: Segmentation learned by LPTM
Conclusion
We make significant contribution towards general pre-trained models for time-series analysis tasks replicating the success large pre-trained models language and vision domains. We introduce LPTM, general tre-trained model that provides state-of-art performance on wide range of forecasting and classification tasks from varied domains and applications. LPTM provides similar to or better performance to state-of-art domain-speciffe models in applications such as epidemiology. energy, traffic, and economics and significantly beats recent time-series foundational models. We also observe that LPTM required significantly lesser pre-training and training data to reach optimal performance compared to other baselines most benchmarks.
Our work mainly focuses on the important challenge of providing semantically meaningful inputs to model that caters to learning time-series segmentation strategies specific to each domain. This crucial when pre-training on diverse datasets, a key challenge for time-series data. The underlying model architecture is a straightforward transformer encoder that uses well-known masking techniques for self-supervised pre-training Therefore. method can be extended to leverage novel time-series model architectures and SSL methods. Extending our methods to provide calibrated forecasts that provide reliable uncertainty measures is also another important direction of research. We can also extend it to leverage multimodal datasets like text that provide important contextual information about the dataset Liu al. (2024c).
Since our model can applied any generic time-series analysis tasks including those in critical domains such as public health, medicine, economies, etc., important steps need to be taken to address potential misuse the our methods such as testing for fairness, data quality issues, ethical implications of predictions, etc.
10

---

Acknowledgements
This paper was supported in part bythe NSF (Expeditions CCF-1918770, CAREER IIS 2028586, Medium IIS-1955883, Medium IIS-2403240, Medium IIS-210691I, PIPP CCF-2200269), CDC MInD program, Meta faculty gifts, and funds/computing resources from Georgia Tech.
References
Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. 2024. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07815 (2024).
Arthur Asuncion and David Newman. 2007. UCI machine learning repository.
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. 2018. The UEA multivariate time series classification archive, 2018. arXiv preprint T0f.:::81110007.. (2018).
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108. 07258 (2021).
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877-1901.
Cristian Challu, Kin Olivares, Boris Oreshkin, Federico Garza Ramirez, Max Mergenthaler Canseco, and Artur Dubrawski. 2023. Nhits: Neural hierarchical interpolation for time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 6989-6997.
Minghao Chen, Houwen Peng. Jianlong Fu, and Haibin Ling. 2021. Autoformer: Searching transform- ers for visual recognition. In Proceedings of the IEEE/CVF international conference on computer vision. 12270-12280.
Ranak Roy Chowdhury, Xiyuan Zhang, Jingbo Shang, Rajesh K Gupta, and Dezhi Hong. 2022. TARNet: Task-Aware Reconstruction for Time-Series Transformer. In Proceedings the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 212-220.
Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan Mathur, Rajat Sen, and Rose Yu. 2023a Long-term forecasting with tide: Time-series dense encoder. arXiv preprint arXiv:2304.0844
(2023).
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou, 2023b. A decoder-only foundation model for time-series forecasting. arXiv preprint arXiv:2310. 10688 (2023).
Songgaojun Deng, Shusen Wang, Huzefa Rangwala, Lijing Wang, and Yue Ning 2020. Cola- GNN: Cross-location Attention based Graph Neural Networks for Long-term ILI Prediction. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 245-254.
Jiaxiang Dong, Haixu Wu, Haoran Zhang, Li Zhang, Jianmin Wang, and Mingsheng Long. 2024. Simmtm: simple pre-training framework for masked time-series modeling. Advances Neural Information Processing Systems 36 (2024).
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. 2021. Time-series representation learning via temporal and contextual contrasting. arXiv preprini orXiv:2106 /4/ (2021).
Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. 2019. Unsupervised scalable represen- tation learning for multivariate time series. Advances neural information processing systems 32 (2019).
11

---

Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. 2024. Moment: A family of open time-series foundation models. arXiv preprint arXiv:2402.03880 03885 (2024).
Nate Gruver, Mare Finzi, Shikai Qiu, and Andrew G Wilson. 2024. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems 36 (2024).
Albert Gu, Karan Goel, and Christopher Ré. 2021. Efficiently modeling long sequences with structured state spaces, arXiv preprint arXiv:2111.0000 (2021).
Rob J Hyndman and George Athanasopoulos. 2018. Forecasting: principles and practice. Texxts
Rob Hyndman and Yeasmin Khandakar. 2008, Automatic time series forecasting: the forecast package for R. Journal of statistical software 27 (2008), 1-22.
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi. Pin- Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al. 2024. Time-llm: Time series forecasting by reprogramming large language models. ICLR (2024).
Harshavardhan Kamarthi, Lingkai Kong, Alexander Rodríguez, Chao Zhang, and B Aditya Prakash. 2021. When in doubt: Neural non-parametric uncertainty quantification for epidemic forecasting. Advances in Neural Information Processing Systems 34 (2021), 19796-19807.
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Ho Choi, and Jaegul Choo. 2021. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference Learning Representations.
Rahul Krishnan, Uri Shalit, and David Sontag. 2017. Structured inference networks for nonlinear state space models. Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 31.
Ananya Kumar, Aditi Raghunathan. Robbie Jones, Tengyu Ma, and Percy Liang. 2022, Fine-Tuning can Distort Pretrained Features and Underperform (2022).
ArXiv abs/2202.10054
Longyuan Li, Junchi Yan, Xiaokang Yang, and Yaohui Jin. 2021. Learning interpretable deep state space model for probabilistic time series forecasting, arXiv preprint arXiv:2102.0039> (2021).
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. 2017. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting, arXiv preprint arXiv: (2017).
Bryan Lim, Sercan Ô Ank, Nicolas Loeff, and Tomas Pfister. 2021. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting 37, (2021), 1748-1764.
Haoxin Liu, Harshavardhan Kamarthi, Lingkai Kong, Zhiyuan Zhao, Chao Zhang, and B Aditya Prakash. 2024a. Time-Series Forecasting for Learning. In Forty-first International Conference on Machine Learning.
Generalization Using Invariant
Haoxin Liu, Shangqing Xu, Zhiyuan Zhao, Lingkai Kong, Harshavardhan Kamarthi, Aditya B Sasanur, Megha Sharma, Jiaming Cui, Qingsong Wen, Chao Zhang, and Aditya Prakash. 2024c. Time-MMD: A New Multi- Domain Multimodal Dataset for Time Series Analysis. Advances in Neural Information Processing Systems (2024).
Haoxin Liu, Zhiyuan Zhao, Jindong Wang, Harshavardhan Kamarthi, and B Aditya Prakash. 2024d. LSTPrompt: Large Language Models as Zero-Shot Time Series Forecasters by Long-Short-Temm Prompting. In The 62nd Annual Meeting of the Association for Computational Linguistics. https //openreview net/ htte//ooummiiii WSStNcooo
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li. Weiyao Lin, Alex X Liu, and Schahram Dustdar. 2021. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting, In International conference on learning representations.
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. 2023. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.066 (2023).
12

---

Yong Liu, Guo Qin, Xiangdong Huang, Jianmin Wang, and Mingsheng Long, 2024b. AutoTimes: Au- toregressive Time Series Forecasters via Large Language Models. arXiv preprint arXiv:2402.02370 (2024).
Spyros Makridakis and Michele Hibon. 2000. The M3- MB-Competition: results, conclusions and implications. International journal of forecasting 16, (2000), 451-476.
Mike A Merrill and Tim Althoff. 2022. Self-supervised Pretraining and Transfer Learning Enable Flu and COVID-19 Predictions in Small Mobile Sensing Datasets. arXiv preprint 37X7 (2022).
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2022. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. arXiv preprint arXiv:2211.14730 (2022).
Boris Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. 2019. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting, arXiv preprint arXiv:1905.10437 (2019).
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. 2018. Deep state space models for time series forecasting. Advances in neural information processing systems 31 (2018).
Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Biloš, Hena Ghonia, Nadhir Vincent Hassen, Anderson Schneider, et al. 2023. Lag-llamat Towards foundation models for time series forecasting, arXiv preprint mrXh::2310.08278 (2023).
Nicholas G. Reich, Logan C. Brooks, Spencer J. Fox, Sasikiran Kandula, Craig J. McGowan, Evan Moore, Dave Osthus, Evan L. Ray, Abhinav Tushar, Teresa K. Yamana, Matthew Biggerstaff, Michael A. Johansson, Roni Rosenfeld, and Jeffrey Shaman. 2019. A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States. Proceedings of the National Academy of Sciences of the United States of America 116, 8 (2019), 3146-3154. https://doi. org. 10. 1073/pnas 1812594116
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. 2020. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting 36, (2020), 1181-1191.
Zezhi Shao, Zhao Zhang, Fei Wang, and Yongjun Xu. 2022. Pre-training enhanced spatial-temporal graph neural network for multivariate time series forecasting, In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 1567-1577.
Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. 2021. Unsupervised representation learning for time series with temporal neighborhood coding. arXiv preprint arXiv:2106.000 00750 (2021).
Willem G van Panhuis, Anne Cross, and Donald S Burke. 2018. Project Tycho 2.0: repository to improve the integration and reuse of data for global population health. Journal of the American Medical Informatics Association 25, 12 (2018), 1608-1617.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems (2017).
Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. 2022. Mien: Mclti-scale local and global context modeling for long-term series forecasting. In The Eleventh International Conference on Learning Representations.
Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. 2024. Unified training of universal time series forecasting transformers. arXiv preprint arXiv:2402.02592 (2024).
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. 2023. Timesnet: Temporal 2d- variation modeling for general time series analysis. ICLR (2023).
13

---

Selene Yue Xu and CU Berkely. 2014. Stock price forecasting using information from Yahoo finance and Google trend. UC Brekley (2014).
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang. Congrui Huang, Yunhai Tong, and Bixiong Xu. 2022. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligncee, Vol. 36. 8980-8987.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. 2023. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference artificial intelligence, Vol. 37.
11121-11128.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. 2021. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2114 2124.
Chuxu Zhang, Dongjin Song, Yuncong Chen, Xinyang Feng, Cristian Lumezanu, Wei Cheng, Jingchao Ni, Bo Zong, Haifeng Chen, and Nitesh V Chawla. 2019. A deep neural network for unsupervised anomaly detection and diagnosis multivariate time series data. Proceedings of the AAAI conference artificial intelligence, Vol. 33. 1409-1416.
Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. 2022. Self-supervised contrastive pre-training for time series via freefreqyyncy consistency. arXiv preprint arXiv:2206.088 08496 (2022).
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI conference on artificial intelligence, Vol. 35, 11106-11115.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning. PMLR, 27268-27286.
Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. 2023. One fits all: Power general time series analysis by pretrained Im. Advances in neural information processing systems 36 (2023), 43322- 43355.
14

---

Appendix for Large Pre-trained time series models for cross- domain Time series analysis tasks
A Hyperparameters
The model is run Intel Xeon CPU with 64 cores and 128 GB RAM. We use single A100 GPU with 80GB memory. For GRU we use a single hidden layer of hidden units. Dimension v is also 50. The transformer architecture consists of 10 layers with 8 attention heads each. For forecasting tasks, we train a separate decoder module with 4 more layers during fine-tuning whereas for classification aggregate the embeddings {er [e of the last transformer layer and feed them into single linear layer that provides logits for all classes. The SSL pre-training was done till convergence via early stopping with patience of 1000 epochs. We observed that LPTM takes 5000-8000 epochs to finish pre-training which takes around 3-4 hours. (Note that pre-training is a one-time step and downstream fine-tuning takes much less time and epochs). For both pre-training and fine-tuning, used the Adam optimizer with a learning rate of 0.001. The hyperparameters are tuned sparingly for both LPTM and baselines from their default settings. For RANDMASK, found the optimal 0.4, and for LASTMASK 0.2 was optimal.
B
Data efficiency
Table 4: Average training time (minutes) till convergence for LPTM and neural baselines. LPTM-TB shows the time taken by LPTM to reach performance of top baseline (in benchmarks where LPTM outperforms it). Since some baselines specific to forecasting classification and do not beat the of-t in few benchmarks we designate these cells in the table as NA.
Model
Flu-US
ETT2
PEM-Bays
NY-Bike
Nasdaq
M3
BasicMotions
Eigen
Informer
27.3
25.5
45.1
49.7
27.1
17.5
Autoformer
19.5
49.5
55.2
18.5
45.1
11.9
19.7
N-HITS
15.4
36.1
49.3
47.5
PatchTST
12.9
36.4
45.7
49.4
NAg
MICN
17.6
39.7
41.1
33.9
TimesNet
15.4
19.7
37.4
46.3
24.1
36.5
11.5
STEP
25.4
52.7
74.3
52.8
NA
EpiFNP
22.5
41.1
39.1
97.6
ColaGNN
34.7
33.6
53.1
47.6
32.1
72.2
TARNet
NA
NA
NA
NA
TS2Vec
29.3
41.9
41.9
29.8
67.4
13.2
TS-TCC
21.7
46.3
44.3
25.3
55.8
LPTM
12.2
41.9
37.5
LPTM-TB
NA
29.6
32.9
23.7
15

---

Fic-Us
Flu-Japan
R17
RIB
LPTM
2.5
Best Bapeliig inc
3000
Best Basciin
Best Baselinc
2500
2.00
1.5
2000
1.0
1.0
0.5-
0.5
0.4
0.6
5.8
0.2
0.4
0.6
0/
1.0
02
3.4
0.6
0.4
0.6
0.8
1.0
Training
Thiinin
PEM- Bays
NY-Bike
Ny-Taxi
Nasdag
LPTM
Basefnu
LPT
Bawelin
LPTM
Best Baseline
0.4
UMN
Base
RMSE
RICE
0.3
3-
10
0.2
0.4
0.5
0.2
0.4
0.6
6.6
0.4
0.6
0.8
1.5
0.4
0.6
0.8
of Trainiing Data
%2 of Training Data
No lreiiin Dala
54 of Th ming Daale
Motionss
RoreDetertion
FingarMovements
2.5
0.5
0.7
R015E
0.6
1.0
0.4
mrining
training Data
training
triningg Dada
PEMS-SF
RocketSparts
FigenWorms
0.9
0.9
0.8
2.88
0.8
UM
Best Base
Best Baseiie
0.4
0.6
38
0.2
0.4
0.5
1.0
074
0.6
0.8
1.0
Thining
Taining
Figure 4: Performance of LPTM and best baseline with varying fractions of training data, In most cases LPTM significantly outperforms baselines with lower amount of data.
16

---

C Effect of SSL hyperparameter Y
FLU4U5
PEM-Bays
Listloss
0.033
FARCHARK LASTIMAR
24
RMROPACE
0.923
L2
3.3
LASTRASE
0.339
0.503
2.5
2.5
0-2
0.4
06.3
Figure 5: Effect of performannceRMMSE) for different benchmarks
17

---

Table 5: Std. dev across 10 runs
Model
Flu-US
Flu-Japan
ETTI
ETT2
PEM-Bays
NY-Bike
NY-Taxi
Nasdaq
M4
AutoARIMA
0.043180
23.434077
0.014220
0.035201
0.056066
0.044572
0.151146
0.025181
0.024060
Informer
0.044072
23.918021
0.014514
0,035927
0,045492
0.154268
0.025701
0.024557
Autoformer
0.042310
22.961906
0.013934
0.034491
0.054936
0.043674
0.148101
0.024673
0.023575
PatcTTS
0.043320
23.509948
0.014266
0.035314
0.056248
0.044716
0.151636
0.025262
0.024138
N-HITS
0.048238
26. 26.178970
0.015886
0.039324
0.062633
0,049793
0.168850
0,028130
0.026878
MICN
0.030538
16.573161
0.010057
0.024895
0.039651
0,031522
0.106894
0.017808
0.017016
TimesNet
0.021264
11,539913
0.007003
0.017334
0.027609
0.021949
0.074431
0.012400
0.011848
LLM-Time
0.040279
21.859627
0,032836
0.052299
0.041577
0.140991
0.023489
0.022443
TimesFM
0.034938
18.961321
0.011506
0.028482
0.045365
0.036065
0.122298
0.020375
0.019468
Lag-LLAMA
0.028790
15.624783
0.009481
0.023470
0.037382
0.029719
0.100777
0,016789
0.016042
Chronos
0,040037
21.728648
0.013185
0.032639
0.051986
0.041328
0.140146
0,023348
0.022309
STEP
0.028283
15.349319
0.009314
0.023056
0.036723
0.029195
0.099001
0.016493
0.015759
EpiFNP
0.026098
14.163743
0.008595
0,021275
0.033887
0,026940
0,091354
0.015219
0.014542
ColaGNN
0.038447
20. 865267
0.012661
0.031342
0.049920
0.039686
0.134578
0.022421
0.021422
TS2Vec
0.032028
17.381828
0.010548
0.026109
0.041586
0,033061
0.112110
0.018677
0.017846
SimMTM
0,053732
29.160111
0.017695
0.043802
0.069767
0.055464
0.188081
0,031334
0.029939
TS-TCC
0.044399
24.095686
0.014622
0.036194
0.057649
0.045830
0,155414
0.025892
0.024739
LPTM
0.048677
26.417645
0.016031
0.039682
0.063204
0.050247
0.199390
0.028387
0.027123
DPtM--nSegmmett
0.045970
24.948198
0.015139
0.037475
0.059689
0,047452
0.160912
0.026808
0.025614
LPTM-NoPreTrain
0.058905
31.968143
0.019399
0,048020
0.076484
0.060804
0.206190
0.034351
0.032822
LPPMMNoo.inProb
0.025968
14.092985
0.008552
0.021169
0.033718
0.026805
0.090898
0.015143
0.014469
LPTM-Only RandMask
0.039517
21 21446657
0.013014
0.032214
0.051310
0.040791
0.138324
0.023045
0.022019
LasssOkkyyastMask
0.058067
315133655
0.019123
0.047337
0.075397
0.059940
0,203258
0.033863
0.032355
Model
Score
AutoARIMA
21.388899
Informer
14.055556
Autoformer
12.000000
PatchTST
6.833333
N-HITS
11,388889
MICN
7.277778
TimesNet
11.11111
LLM-Time
12.777778
TimesFM
12.277778
Lag-LLAMA
17.444444
Chronos
8.888889
STEP
9.111111
EpiFNP
14.666667
ColaGNN
16.500000
TS2Vee
19.111111
SimMTM
15.611111
TS-TCC
18.111111
LPTM
2,500000
LPTM-N OPTM-NoSegroent
12.055556
LPTM-NoPteTrrai
10.444444
inPoin
6.222222
JPTM-OnlyRandMM
7,555556
3.666667
Table Mean rank models in Table
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions listed in the Introduction are clearly expanded in Methodol- ogy and Results
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Conclusions discuss limitiations such as not applicable to probabilistic and multivariate forecasts and potential extensions as future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well- -ell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't seknowleddge in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [NA]
Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Ys]]
Justification: We provide model architecture and training details in methodology, link the anonymized repo and hyperparameters in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Data and code available in httppp//ggtttuoo.cm///aaaraaaa/a
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
hubblic//uilass/ddeeebbmm.sss wtt.:/swwmmiiiooooooooooooo. for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
guwdes/ddeeeee CooeeSumieeeeeeeeeeeee iss onyylicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Specific details on hyperparameters are in Appendix A. Training details are in Section 3.3. Experiment setup, dataset splits are Sec
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Ys]]
Justification:
We perform multiple (10) runs of each experiment and report the average scores. We highlight the instances where LPTM in statistically significantly better than all baselines.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer:
[Yes]
Justification: We mention the compute resources used in Appendix and training time in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We went through and confirm the paper follows all the statements the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Our work a general purpose time-series model that can utilized towards wide range of applications. Care should be taken to assess the ethics of the decision taken by the predictions, clean the data for any sensitive information and ensure fairness and positive utility in the resulting outcome.
22

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. dep.lymepl of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [No]
Justification: Since this work is general-purposs purposs foundational work on time-series and we do not focus on any potentially sensitive applications, providing safeguards for release of our work is not applicable.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets in code either created by authors derived from open-source works and are cited appropriately.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
23

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code is the only original asset which released as an anonymous repo linked in the paper. We will make public release of this acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing research.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work doesn't 't deal with research requiring IRB review.
Guidelines:

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "6HQCzUeDka", "reviewer": "Reviewer_bKzz", "summary": "This paper proposes a novel contribution to pretrained time series models for forecasting and classification by paying attention to the fact that currently several transformer models take time series segmentations of the same size, regardless of the particular characteristics of the time series in consideration. For instance, time series that have yearly frequency or minute frequency might require different segmentation lengths, or it might be that dynamics are more complex in certain time intervals requiring a more detailed segmentation. Based on this observation the authors proposed a model that can find a suitable segmentation schema that later on allows to observe where are the time intervals where more complex dynamics are shown.\n\nThe authors perform several experiments and claim empirically that the proposed approach is at least competitive to the state of the art.", "questions": "Question: \n\n* Eq-1: is the GRU applied entry-wise to the time series? Does this imply that we apply $GRU_1$ to each entry of $y$ (which has $t$ entries), and then the resulting $t$ values constitute the hidden embeddings?\n* Eq. 2: what is $z_i$? So far we have talked about $z^{(i)}$.\n* Missing closing parenthesis in fig 1: $S(y^{(1...T)$\n* Eq-1: the larger values of S(i,j) the better? Does it mean that the correlation between $z_i$ and $z_j$ is high, or that $z_i$ and $z_j$ are related somehow? \n* Eq-3: index $k$ is never used in this definition of output embeddings.\n* Eq-5: as pointed out by the authors, the selection of segments is not differentiable and hence it can not be directly integrated to the loss function. Does it mean that the segments are updated every 10 batches? This means that the loss will not be continuous, and hence it will be unclear if there is progress or not in terms of the training loss. Is this correct? I guess here what nevertheless can hint at improvement is the test loss.\n* In line 225: why are time series with missing values removed? is the proposed model able to handle missing values?\n* The authors claim that their model is a pretrained model. What datasets were used to pretrain the model? Are all datasets used as well for evaluation in Table 1? If yes, then the comparison is not fair. Several of the pretrained models considered might have not been exposed to those datasets in pretraining, giving an unfair advantage to the proposed approach. Further, doing pretraining in such a small amount of datasets further gives more advantage to the proposed model, as the larger the datasets potentially gives a smaller amount of exposure to each dataset.", "limitations": "The authors have acknowledged limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors study a clearly interesting problem: how to provide a suitable segmentation scheme for time series so that different time regions are segmented in different ways, depending on their complexity and amount of information. The motivation for this is well stated by the authors, leading to a novel approach to achieve this. \n\nThe authors further set up this in an Self-supervised learning setting, and consider multiple datasets to pretrain their model and further provide several evaluations. This is interesting because depending on the field/topic/area of time series a different segmentation scheme might be more suitable.", "weaknesses": "Some of the main limitations are as follows:\n- The proposed framework is not differentiable. The authors have acknowledged this in the paper and propose a workaround for this, basically to update the segmentation scores every 10 batches. Yet, this poses challenges like the interpretation of the training loss, and discontinuities in the test loss.\n- It is unclear if the proposed approach is able to handle missing values. If not, is there anyway to overcome this? Missing values are very often present in practice and having a sound way to handle them is relevant.\n- It is unclear if the current evaluation is fair. The authors present a corpus of datasets for which they pretrained the proposed model, but it is unclear which datasets where hold-out from pretraining. This is relevant as several of the pretrained models considered might have not been exposed to these datasets, which gives an unfair advantage to the proposed model. Further, since the amount of pretraining datasets is rather limited, there is the possibility that the proposed model is overly focused on these datasets, whereas other models, like (Ansari 2024) and (Woo 2024) were trained in a larger corpus of datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments. We address them as follows:\n\n**The proposed framework is not differentiable..**\n\nAs stated in the paper we did not observe any instability in the training.\nYes, the segmentation of time-series is a discrete operation. LPTM tackles this challenge via a novel differentiable scoring module which learns the best segmentation strategy based on pre-trained dataset.\n\nWe note that the frequency of training every 10 batches is key to providing stable training since changing the segmentation module every iteration will change the SSL objective function too frequently leading to instability. Similar strategies are used in many papers that encounter bilevel optimization such as for discrete operations such as graph learning [1] and reinforcement learning over discrete action spaces [2].\n\n**Handling missing values**\n\nHandling missing values is not in the scope of our current work. However, LPTM could be extended to handle missing values using a similar method to random masking task by masking the segments associated with missing values. One issue to tackle would be to selectively mask part of the segment. \n\n**Regarding pre-trained data**\n\nFor pre-trainied models such as Lag-LLAMA and TimesFM, we start with the weights of the models publicly available and pre-train on the same datasets we use to pre-train LPTM. Therefore, all baselines are pre-trained with at least as much data as LPTM is to provide a fair comparison. In fact, the other pre-trained models such as TimesFM and Chronos have already been pre-trained on much larger pre-trained datasets. In spite of this, LPTM achieves SOTA performance. For fine-tuning, all models are fine-tined on the same train-test splits.\n\nFurther, we use all the datasets described in lines 222-244 for pre-training. Only the unseen test split was used for evaluation on Tables 1 and 2.\n\n**is the GRU applied entry-wise to the time series?**\n\nYes, for each $y^{i}$ we get an embedding $z^{(i)}$ from the GRU.\n\n**what is z_i?**\n\nThis is a typo, we meant $z^{(i)}$.\n\n**Value of s(i,j)**\n\nYes, larger is the value of $s(i,j)$ more important the segmentation module believes the segment between i and j to be. The intuition for Eq 2 is that since the GRU captures the information at timesteps i and j in context of entire time-series, the segment score function uses this information to derive the segment score.\n\n**Index k in Eq 3**\n\nk is just the indexing variable to aggregate the output embeddings of the self-attention for time-stamps from i to j. We will clarify this.\n\n\n**Regarding Eq 5**\n\nDue to the discrete problem of finding the optimal set of segments, the actual objective function during SSL is not continuous. However, we overcome this by using a bi-level optimization approach and periodically optimize the segmentation by learning on SSL tasks. This provides a more stable approach to training when we encounter discrete objectives similar to such strategies used in many papers that encounter bilevel optimization such as for graph learning [1] and reinforcement learning [2].\n\n**Missing values removed (line 225)**\n\nMany time-series in Project Tycho are very sparse with lots of missing values. We remove these time-series since LPTM and other baselines cannot handle them. However, as noted earlier, extending LPTM to time-series with missing values is an interesting and important challenge for future research.\n\n### References\n\n[1] Hu, Minyang, et al. \"Learning Continuous Graph Structure with Bilevel Programming for Graph Neural Networks.\" IJCAI. 2022.\n\n[2] Haarnoja, Tuomas, et al. \"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.\" International conference on machine learning. PMLR, 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DVwkuUpQiR", "reviewer": "Reviewer_7ZoZ", "summary": "The paper proposes Large Pre-trained Time-series Models (LPTM), a novel method designed to improve the efficiency and performance of time-series analysis across multiple domains. \nThe key contribution is an adaptive segmentation module that automatically identifies optimal segmentation strategies for diverse datasets during pre-training. \nThis approach aims to overcome the limitations of fixed-length segmentation, which may not adequately capture the temporal patterns of heterogeneous time-series data. \nLPTM demonstrates superior forecasting and classification performance, requiring up to 40% less data and 50% less training time compared to state-of-the-art models.", "questions": "Q1 (cr. W1). Please provide intuitions and empirical evidences on why full self-attention-based architectures work for images while an explicit segmentation module is required for time series.\n\nQ2 (cr. W2). Please discuss the convergence stability, especially the initial convergence stability and the influence of random initialization, as well as the influence of the adaptive segmentation modules on the pre-training speed.\n\nQ3 (cr. W3). Please discuss the influence of fine tuning on the adaptive segmentation, e.g., which the current framework can make a large change to the existing segmentation. \nExperiments with and without fine-tuning the adaptive segmentation would be interesting to report.\n\nQ4. It would be interesting if the authors may show some case studies on the adaptive segmentation results, i.e., whether and how much the adaptive segmentation results conform to the source domain of the dataset, whether some periodic patterns can be well preserved after the adaptive segmentation module.", "limitations": "L1. There is a lack of explanability and interpretability in the adaptive segmentation results.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1. This paper focuses on the time series segmentation problem.\nAs the basic semantic unit in time series is not as clear as in text, a proper segmentation is a promising direction towards better series modeling.\n\nS2. The proposed segmentation method is adaptively calculated over each specific input series.\n\nS3. The experiments are extensive.", "weaknesses": "W1. Although time series has a weaker semantic structure than natural language, it is closer connection to images.\nIn both time series and images, a semantic unit, e.g., a small item or a texture in an image, can have different lengths and scales.\nThis raises a challenge against the main motivation: why a full self-attention-based architecture works for images (e.g., ViT), why for time series the segmentation needs to be explicitly done?\nIt would be interesting if the authors can further discuss this problem and provide their intuitions.\n\nW2. The introduction of the adaptive segmentation module seems to bring instability in the initial model training, as well as requiring longer training time (although the authors propose to backpropagate the gradients every 10 batches).\nSpecifically, the loss function for segmentation is a hard loss based on the selected subset of best segments.\nHowever, the parameters seem to be randomly initialized, which could provide highly random \"best\" segments.\nHence, the convergence stability and the training time with and without the dynamic segmentation modules should be discussed.\n\nW3. The dynamic segmentation modules seem not to be fine-tuned with specific attention.\nHowever, as the author(s) mentioned, different datasets could have very different best segmentation.\nHence, it would be interesting to discuss why this is sufficient and provide theoretical or empirical evidences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and we address them as below:\n\n**Intuition on segmentation**\n\nWe note that even for vision models such as ConvNets and ViT, the images are ingested as fixed sized patches (such as 16 x 16 patches) which have some similarity to segments in time-series. Moreover, we argue that there is much more  diverse heterogeneity in time-series compared to images since even images of different domains have important properties such as edges, shapes that are useful to capture by the model.\n\nHowever, for time-series, each domain can have a wide range of patterns specific to the domain along with cross-domain information. Further, these patterns could differ across time. Therefore, segmentation that adapts to these patterns across time can better capture these semantic structures and handle cross-domain downstream tasks.\n\n**Stability of adaptive segmentation**\n\nFirst, we did not observe instability in training.  We evaluated our model over multiple runs with different random initializations and we didn't experience large variation in stability or performance\n\nSince the overall objective of finding optimal segments as well as pre-train weights of the backbone model on SSL tasks has discrete components we use a bi-level optimization to simultaneously learn both objectives. We update the loss for segmentation every 10 epochs to make the training stable for SSL. This kind of training strategy is used in other bi-level optimizations encountered in discrete tasks such as graph structure learning and reinforcement learning on discrete action space[1,2].\n\n\n**Fine-tuning adaptive segmentation**\n\nWe fine-tuned adaptive segmentation to provide optimal segmentation for each domain dataset since the patterns observed across different domains is very heterogenous. \nFor example, while we need to capture monthly or yearly patterns for demand datasets, behavioral datasets need to encode different kinds of patterns occurring in order of seconds or minutes. \n\nTraining a single segmentation strategy across all domains therefore would be highly suboptimal. In fact, empirically, we observed that training a single segmentation strategy across all domains lead to unstable training due to which we didn’t explicitly include it as a baseline.\n\n**Interpretability of segmentation**\n\nWe also show some visualizations of the segmentations for different domains in Appendix Section D. We can get some intuitive understanding from these visualizations such as LPTM capturing smaller segments near epidemic peak to encode fine-grained variations as well as segment lengths correlating with variance of time-series in general.\n\n### References\n\n[1] Hu, Minyang, et al. \"Learning Continuous Graph Structure with Bilevel Programming for Graph Neural Networks.\" IJCAI. 2022.\n\n[2] Haarnoja, Tuomas, et al. \"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.\" International conference on machine learning. PMLR, 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B49KiiD3dO", "reviewer": "Reviewer_RuJ6", "summary": "The paper introduces a new approach for creating pre-trained models for time-series data, similar to those used in language and vision tasks. The authors propose a model called Large Pre-trained Time-series Models (LPTM), which includes an innovative adaptive segmentation module to handle diverse time-series data from multiple domains.\n\nKey contributions include:\n\n- Developing a framework for pre-training time-series models on multi-domain datasets, using a novel adaptive segmentation module to tokenize inputs effectively. This is achieved via a self-supervised learning objective.\n- Demonstrating that LPTM performs as well or better than state-of-the-art domain-specific models when fine-tuned for various time-series tasks, such as forecasting and classification, with less training data and compute time.\n- Proving that LPTM achieves superior results in both zero-shot and fine-tuned settings across diverse domains like epidemiology, energy, and economics, requiring up to 40% less data and 50% less training time compared to existing methods.", "questions": "- Can the authors add error bars (in the appendix possibly) for their experiments? They mention that they already run 10 experiments per setup so these should be readily available and would give a good idea of the robustness of the findings.\n- Can the authors ensure that code to reproduce their experiments is available as stated? \n- Can the authors run the same experiments on the \"standard\" long-term forecasting datasets, as listed in the weaknesses section?\n\nNote: I feel the paper is definitely interesting and makes valid contributions. Addressing my questions, in particular the one about the long-term forecasting datasets would be a strong argument for me to raise my score.\n\nEdit: I've read the rebuttal provided by the authors, and since my open questions have been addressed I'm raising my score to 7.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper has the following strengths:\n\n- Well-written, clear, easy to follow. Algorithm is a nice plus.\n- Baseline choice reasonable: most recent methods are considered.\n- Experimental results good, when considered on the set of datasets chosen (more points on that in the weaknesses section).", "weaknesses": "- It's a bit hard to get a good feel for the relative advantage of the proposed method. In table 2, the approach is clearly better, but we are left to infer that from that fact that it is commonly second or first in the rankings. Could the authors maybe add some for of aggregate metric, e.g. the average rank across datasets of a given method?\n- Despite mentioning code is available, the link does not work (subscript 3 on page 7, time of access 2024-07-12, and previously): \"The repository is expired\".\n- For a paper dealing in large part with forecasting, I was surprised by the absence of almost all of the classical long-term forecasting datasets used by other papers: traffic, electricity, weather, illness... Given that these are by far the most heavily studied ones in the literature, including them (as proposed in the questions section). While I don't find it a critical (but still important) concern, I strongly advise the authors to consider adding them as it will help avoid concerns other readers might have about cherry-picking of results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for comments and suggestions. We address them as follows:\n\n**Aggregate rank metric**\n\nWe thank the reviewer for the suggestion. We will add the average rank of each model as:\n| Model                 | Score       |\n|-----------------------|-------------|\n| AutoARIMA             | 21.388889   |\n| Informer              | 14.055556   |\n| Autoformer            | 12.000000   |\n| PatchTST              | 6.833333    |\n| N-HITS                | 11.388889   |\n| MICN                  | 7.277778    |\n| TimesNet              | 11.111111   |\n| LLM-Time              | 12.777778   |\n| TimesFM               | 12.277778   |\n| Lag-LLAMA             | 17.444444   |\n| Chronos               | 13.888889   |\n| STEP                  | 9.111111    |\n| EpiFNP                | 14.666667   |\n| ColaGNN               | 16.500000   |\n| TS2Vec                | 19.111111   |\n| SimMTM                | 15.611111   |\n| TS-TCC                | 18.111111   |\n| LPTM                  | 2.500000    |\n| LPTM-NoSegment        | 12.055556   |\n| LPTM-NoPreTrain       | 10.444444   |\n| LPTM-NoLinProb        | 6.222222    |\n| LPTM-OnlyRandMask     | 7.555556    |\n| LPTM-OnlyLastMask     | 3.666667    |\n\nLPTM has the best average rank of 2.5 with significant lead over second best model.\n\n**Code link**\n\nSorry. We didn’t realize the link was expired during the review process. The code can be found in the LPTM.zip file at https://osf.io/vy3sc/?view_only=c10508d340ba468287c984496cf57be1\n\n**Long-term forecasting benchmarks**\n\nWe have benchmarked on ETT and M4, two popular long-term forecasting benchmarks in the paper. We have added additional benchmarks on ILI, Electricity and Exchange datasets (similar setup to [1]):\n\n| Dataset/Model | AutoARIMA | PatchTST | LLM-Time | TimesFM | Lag-Llama | Chronos | SimMTM | LPTM |\n|---------------|-----------|----------|----------|---------|-----------|---------|--------|------|\n| Electricity   | 0.6       | 0.48     | 0.93     | 0.51    | 0.63      | 0.82    | 0.81   | 0.74 |\n| Exchange      | 1.23      | 0.94     | 1.72     | 0.86    | 1.51      | 1.54    | 0.93   | 0.91 |\n| ILI           | 1.95      | 0.97     | 1.11     | 1.83    | 2.11      | 1.85    | 0.84   | 0.96 |\n| Traffic       | 2.14      | 1.66     | 1.94     | 1.85    | 2.56      | 1.98    | 1.84   | 1.52 |\n\n**Error bars**\n\nWe have added the std. Dev across 10 runs in the additional PDF.\n\n### References\n\n[1] Nie, Yuqi, et al. \"A time series is worth 64 words: Long-term forecasting with transformers.\" arXiv preprint arXiv:2211.14730 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bhc8k0BGln", "reviewer": "Reviewer_fxq9", "summary": "Training large time series (TS) models is often limited by the scarce data available for a specific application. Existing pretraining methods use a simplistic tokenization scheme where the TS is cut up into equally sized parts, independent of its content. The newly proposed method *Large Pre-trained Time-series Models*, therefore, adaptively segments the input time series into (potentially) overlapping tokens depending on the TS itself. It shows very good forecasting performance in zero-shot and finetuning settings. It can also be used for classification.", "questions": "1. The problem statement (l. 96-98) only considers in-domain applications. Is it also interesting to consider domain generalization settings?\n2. The term \"Large Pre-trained Time-series Models (LPTM)\" (l. 48-51) is extremely generic and better describes the emerging field than a single method. Would it make sense to focus more on the \"Adaptive Segmentation\" contribution in the title and exposition?\n3. Why did you choose Eq. (2) to be that way? Are all parameters learned?\n4. The scaling of the method is not discussed sufficiently; could you comment on that? There are potentially a lot of tokens -- in the worst case, there are as many as input tokens. Even more, the number of evaluations of $s(i,j)$ scales quadratically in the input length. While the heuristic for selecting a good set of segments is defined well, a discussion of why it is sensible is missing. See also question 7.2.\n5. Why did you use masked reconstruction as the two pretraining tasks? Could you explain why this is more desirable than alternatives, like contrastive methods?\n6. The interplay of training the segmentation strategy and the encoder simultaneously requires a more nuanced discussion. To what extent can this cause something like a \"mode collapse\" (as in Generative Adversarial Networks), where the $s(i,j)$ always chooses the same segments since they were found to be beneficial at some point and stops \"exploring\" others? This should (1) be discussed in more depth and (2) may be a significant limitation of the method.\n7. This leads to the experimental evaluation.\n\t1. Following (6), can you provide insights into the relative convergence speeds of the scorer and the encoder/decoder? How good is the score function $s$ at predicting the final encoder loss?\n\t2. How many segments are there typically? A histogram would help judge the typical number of tokens resulting from the adaptive segmentation.\n\t3. L. 281: \"2x to over 10x smaller than other pre-trained time-series models\" -> Where do you provide that comparison of model sizes?\n\t4. Section 5 only discusses (almost) exclusively the forecasting setup. Why were the specific classification datasets chosen? Section 6 mentions 35 datasets (UCI has more time series classification datasets), but Table 4 only contains 32. Were three datasets removed at some point?\n\t5. L. 305: \"We observe that LPTM has highest mean rank\" -> Please provide that rank in (or alongside) Table 4.\n\t6. The caption of Table 4 talks about statistical significance. What are the test's results? Which significance test did you even perform? How were results aggregated (if it was the arithmetic mean, what is their std. dev.)?\n\nMinor comments:\n- The paper would benefit from a language pass.\n- Improvements to formatting: References are formatted confusingly, e.g., in lines 22, 23f, 36f, etc. This problem occurs throughout the paper. Table 4 in the Appendix is too wide. REVIN -> RevIN. Formatting of LASTMASK, etc., is inconsistent (l. 163 vs. l. 335), ...\n- The example in line 40ff could be given more bluntly and convincingly by discussing the same event in different sample rates.\n- L. 91, what is R? The same as in l. 152?\n- L. 89, isn't $\\mathcal{D}_\\text{pre}$ the union over the individual datasets?\n- The abbreviation SSL has never been introduced.\n- Red and green are difficult to distinguish for people with common color deficiencies. Therefore, Figure 1 could be improved with a different color palette.\n- Table 3 should mention that it is about finetuning. Are the baselines trained from scratch or merely finetuned as well?\n- The authors should check whether they intentionally want to cite many preprints (e.g., from Arxiv) or their published variants.\n- The capitalization of the title is inconsistent.", "limitations": "Depending on the answers to the questions above, possible limitations mentioned could be made more transparent. For example, the insights into the newly induced biases are currently limited.\n\nThe answer to section 2. in the checklist is neither sufficient nor truthful. For example, \"multivariate\" is never mentioned in Section 7.\n\nThe discussion of the societal impact could also consider a possible data leakage from one private application (e.g., in the medical domain) to another one. Even a rather mundane problem, like a feasible membership inference attack, could be problematic in privacy-sensitive scenarios.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The relevance of the problem and motivation for adaptive segmentation is convincing.\n- The method of adaptive segmentation is an interesting solution to the issue.\n- LPTM is compared against a plethora of appropriate and challenging baselines.\n- It shows promising empirical results.", "weaknesses": "- I am under the impression that this paper may have found a strong method yet does not sufficiently investigate *why* it works. The interplay between learning the scoring function and training the encoder is not very clear. See below.\n- A lot of experimental claims are not adequately substantiated. It is claimed in question 6 of the checklist that error bars are provided and that statistical significance tests are performed, yet I did not find them. See below.\n- The overall presentation (language and formatting) should be improved.\n- The provided implementation is not accessible. (Error: \"The repository is expired\") In the current state, results are not reproducible since key hyperparameters are missing. The authors claim in question 6 of the checklist that they state how hyperparameters were chose, yet I could not find it in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We will address them as follows:\n\n**Domain-generalization settings**\n\nWe wish to emphasize that we generalize to a wide range of domains but we require to learn segmentation module for each of the domains. \nGeneralizing to unseen domains is an important research question to tackle for general pre-trained model\n\n**Title**\n\nWe chose LPTM as the title since our innovation using adaptive segmentation for cross-domain pre-training enables state-of-art performance across multiple datasets with heterogenous patterns. We are happy to add to adaptive segmentation to the title.\n\nMoreover, other pre-trained baselines also have generic names such as TimesFM (Time-series foundational model).\n\n**Eq 2**\n\nThe intuition for Eq 2 is that since the GRU captures the information at timesteps i and j in context of entire time-series, the segment score function uses this information encoded in z^(i) and z^(j) to derive the segment score. The exact form of the equation was chosen empirically and is similar in form to Bahdanau attention function. All the parameters (W_1, W_2 and b) are learnable.\n\n**Scaling and number of tokens**\n\nActually, we found that adaptive segmentation doesn’t provide any significant contribution to overall overhead of the model since the transformer backbone consumes the bulk of the compute time. While the entire adaptive segmentation module’s runtime is quadratic in length of the time-series, Empirically we found that the average size of the segments across datasets is around 5.2 with some domains such as ETT having larger average lengths (13.4) and some domains such as behavioral datasets with lower average lengths (3.1).\n\nRegarding the heuristic, solving for the optimal set of segments that provide the maximum score as well as cover the entire length of the input is a non-trivial optimization problem whose complexity could be upto exponential in number of time-steps. Therefore, we use a simple heuristic of eliminating the lowest valued segments till we cannot cover the entire time-series. (l 12-14).\n\n\n**Choice of SSL tasks**\n\nRandon masking and lask token predictions are popular SSL pre-training methods that are widely adopted in LLMs and some time-series methods like PatchTST. We also point out that using these simple tasks already provides state-of-art results.\nWhile we can use more sophisticated methods, the focus of our work is on using adaptive segmentation to enable multi-domain pre-trained time-series model. Indeed LPTM is very flexible to adapt other SSL loss functions to improve performance could be a interesting extension of our work. \n\n**Stability of training and mode collapse**\n\nWe did not observe instability or mode collapse during pre-training. We found that the training of the model was stable across different domains and datasets. This was observed across multiple runs of the model as well. The frequency of updating the segmentation loss relative to SSL loss is key in ensuring the stability of training. \n\nThe overall objective of self-supervised pre-training involves a discrete objective of choosing an optimal segmentation strategy along with the prediction of masked segments. Similar to other works in graph learning[1] and reinforcement learning on discrete action space [2] which involve learning over discrete structures that cannot be trivially integrated we use a bilevel optimization approach, and periodically optimize the segmentation by learning on SSL tasks.\n\nImproving the efficiency and stability of this approach and more tightly coupling the two objectives could be an important research direction to improve model performance.\n\n**Convergence of scorer and encoder**\n\nIt is not straightforward to compare the SSL loss with the loss for score function since they indicate different objectives. We found that the SSL loss may slightly perturb when there is an update to score function loss but generally follows a relatively smooth downward trend to convergence. The scoring function loss, while trending lower over time, doesn't converge to zero and hence is not a predictor of SSL loss, Eq 5 only forces the scoring function to be trained to update in the direction of decreasing SSL loss.\n\n**Model Sizes**\n\nLPTM model size is around 100M which is 2x smaller than than TimesFM, the smallest of the pre-trained baselines and is around 10x smaller than Chronos which has around 1B parameters.\n\n**Classification tasks**\n\nWhile we focused on forecasting for most of the experiments similar to other general pre-trained time-series models, we also showcased the flexibility of LPTM to perform classification due to our architectural choice.\nTable 4 contains only 32 datasets and this is a typo. We added the other 3 datasets in the additional pdf.\n\n**Mean rank of Models**\n\nWe post the mean ranks in the additional PDF. LPTM has the best average rank of 2.5 and is significantly better than other models.\n\n**Significance test**\n\nWe used the standard t-test to determine statistical significance. We have added the standard deviations across 10 runs in the additional pdf. \n\n**Regarding Limitations**\n\nWe thank the reviewer for the suggestions. We will add the discussion on multivariate datasets and issues regarding data privacy to discussions.\n\n**Code link**\n\nSorry. We didn’t realize the link was expired during the review process. The code can be found in the LPTM.zip file at https://osf.io/vy3sc/?view_only=c10508d340ba468287c984496cf57be1", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Tables for the standard deviation of RMSE across 10 runs, mean rank of the models and 3 additional classification tasks.

Meta-review: The submission describes an approach called Large Pre-trained Time-series Models (LPTM) that improves time-series forecasting and classification by introducing an adaptive segmentation module. This module automatically identifies optimal segmentation strategies for different datasets, addressing the limitations of fixed-length segmentation methods. The paper demonstrates that LPTM achieves strong performance across multiple domains with reduced data and training time

The paper received generally positive feedback from reviewers, with a few areas for improvement noted. 

**Positive Points:**
* The reviewers appreciated the innovative adaptive segmentation approach, which provides a flexible and effective way to handle diverse time-series data across multiple domains.
* Experiments against a wide range of baselines were seen as a strength.
* The clarity and thoroughness of the paper's presentation, particularly in terms of the problem's relevance and the method's motivation, were commended.

**Negative Points:**
* Several reviewers pointed out the lack of detailed investigation into the interplay between the adaptive segmentation module and the encoder, particularly in understanding why the method works as well as it does.
* Concerns were raised regarding the presentation and reproducibility, including issues with language, formatting, and an expired link to the implementation code.
* The scalability and computational overhead of the adaptive segmentation module were not sufficiently discussed, particularly in relation to the potential increase in tokens and the quadratic scaling of the method.

The LPTM method represents an advance in time-series analysis. While the paper could benefit from a deeper exploration of certain technical aspects and improved clarity in its presentation, these issues do not detract from the overall quality and impact of the work. Given its technical merit and practical relevance, this paper is well-suited for presentation at NeurIPS.
Decision: Accept (poster)