Large Pre-trained time series models for cross-domain Time series analysis tasks

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 16:27:06

Abstract: Large pre-trained models have been vital in recent advancements in domains like language and vision, making model training for individual downstream tasks more efficient and provide superior performance. However, tackling time-series analysis tasks usually involves designing and training a separate model from scratch leveraging training data and domain expertise specific to the task. We tackle a significant challenge for pre-training a foundational time-series model from multi-domain time-series datasets: extracting semantically useful tokenized inputs to the model across heterogeneous time-series from different domains. We propose Large Pre-trained Time-series Models (LPTM) that introduces a novel method of adaptive segmentation that automatically identifies optimal dataset-specific segmentation strategy during pre-training. This enables LPTM to perform similar to or better than domain-specific state-of-art model when fine-tuned to different downstream time-series analysis tasks and under zero-shot settings. LPTM achieves superior forecasting and time-series classification results taking up to 40% less data and 50% less training time compared to state-of-art baselines.

Main paper: 
 introduction: 
 Time-series analysis tasks involve important well-studied problems involving time-series datasets such as forecasting (Hyndman & Athanasopoulos, 2018) and classification (Chowdhury et al., 2022) with applications in wide-ranging domains such as retail, meteorology, economics, and health. Recent works (Chen et al., 2021; Wang et al., 2022; Zeng et al., 2023) have shown the efficacy of purely data-driven deep learning models in learning complex domain-specific properties of the time series over traditional statistic and mechanistic models across many domains.However, coming up with a model for a specific application or time-series analysis task is usually non-trivial. Most state-of-art neural models are known to be data-hungry and require substantial training data from the same domain on which we deploy to train. This can be prohibitive in many real-world applications. While we have access to a large amount of time-series datasets from other tasks and domains, that contain useful background patterns and information, time-series models typically cannot leverage them to improve their performance.In contrast, for many language and vision tasks, we use pre-trained models trained on a larger pre-training dataset (Qiu et al., 2020; Du et al., 2022; Gunasekar et al., 2023) . These pre-trained models are then fine-tuned to the downstream task. There are two important benefits of pre-trained models. First, the pre-trained weights are good initialization for faster and more effective training. They require less training resources and data and produce superior performance. Moreover, pretrained models learn useful underlying structures and patterns from larger pre-trained datasets such as common syntactic and semantic knowledge in the case of language and the ability to recognize useful patterns in the case of vision. Initiating training from these pre-trained models usually results in faster training and better performance. compared to training the model from scratch on task-specific training data. Therefore, we tackle the goal of building a unified pre-trained models for time-series that are pretrained on datasets from multiple domains and can be applied to a wide range of downstream time-series analysis tasks across all domains. However, there are important challenges intrinsic to time-series that makes pre-training non-trivial. Most neural sequential models input time-series values for each time-step separately. However, unlike text data, each individual time stamp may not provide enough semantic meaning about local temporal patterns of the time series. To tackle this, Nie et al. (2022) proposed to segment the time series and input each segment as individual tokens to their transformer-based model and showed superior performance to more complex transformer-based architectures. However, in the case of pre-training with multiple domains, each dataset in pre-train datasets are derived from different domains with different set of underlying generative dynamics, sampling rate, noise, etc. Using uniform segment sizes similar to Nie et al. (2022) for all datasets would be suboptimal. For example, among two datasets, a dataset with a higher sampling rate may require longer segments than those with lower sampling rates to capture similar patterns in the model. Further, the optimal segment size used for the same time-series may vary with time. For, time intervals that are smoother with less complex dynamics, using longer segment sizes may suffice whereas intervals where time-series have more complex and multiple temporal patterns may require finer-grained segmentation.We tackle these challenges and propose Large Pre-trained Time-series Models (LPTM), a novel method for generating pre-trained models for time-series data across multiple domains. LPTM uses a simple transformer-based architecture and leverages a self-supervised pre-training to simultaneously train on multiple datasets from different domains. We utilize simple self-supervised tasks based on masking tokens input to the transformer and learning to reconstruct the masked tokens. However, we input segments of time-series as tokens to the transformer. To overcome the challenges associated with segmentation on diverse datasets discussed above, we propose a novel adaptive segmentation module that segments the time-series of each domain based on how well it performs on self-supervised pre-training. The segmentation module uses a novel scoring mechanism for the segmentation strategy used by the model on input time-series for a domain based on the SSL (self-supervised learning) loss and optimize the segmentation strategy to lower the SSL loss. We show that LPTM can be fine-tuned to a variety of forecasting and classification tasks in varied domains such as epidemiology, energy, traffic, economics, retail, and behavioral datasets. We also show that LPTM can provide performance on par with state-of-art models with lesser training data during fine-tuning as well as with fewer training steps showcasing the efficiency of our pre-trained framework. Our main contributions can be summarized as follows:1. Multi-domain Pre-trained time-series model We propose a framework for generating large pre-trained models for time-series that are trained on multiple datasets across varied domains. LPTM is an important step towards general pre-trained models for time-series similar to LLMs for text and vision. 
adaptive segmentation for cross-domain pre-training: 
 To optimally extract semantically useful information from time-series of different domains with varied dynamics and sampling rates for pre-training, we propose a novel adaptive segmentation module that learns segmentation strategy for each domain based on losses from self-supervised learning tasks.3. State-of-art and efficient performance in diverse downstream time-series tasks We evaluate LPTM on downstream forecasting and classification tasks from multiple domains and observe that LPTM consistently provides performance similar to or better than previous state-of-art models usually using lesser training steps and compute time. We also observe that LPTM typically requires less than 80% of training data used by state-of-art baselines to provide similar performance.Time-series analysis tasks Our pre-trained model can be used for many time-series tasks including forecasting and classification from multiple benchmarks and domains. For a given downstream task let D T be the time-series dataset consisting of time series y 1...T . A time-series analysis task's goal is to predict important properties of the time-series. For example, the forecasting task involves predicting the future values y T +1...T +K whereas classification involves predicting the class label of the input time-series based on labeled training data.Self-supervised pre-training on multi-domain datasets The goal of our work is to learn useful knowledge and patterns from time-series datasets from time-series from different domains. This is in contrast to previous works which typically train the models only on time-series from current downstream tasks.Formally, we have access to time-series datasets from K domains where the datasets of domain k is denoted asD ′ k = {D ′ k,i } N (k)i=1 where N (k) is the number of datasets in domain k. Examples of these domains include epidemiology, energy forecasting, macroeconomics, traffic prediction, etc. The entire set of heterogenous multi-domain pre-train dataset is denoted asD pre = {D ′ 1 , D ′ 2 , . . . , D ′ K }.In order to effectively pre-train LPTM on D pre we formulate the problem as a set of self-supervised learning tasks T pre = {T i } R i=1 on the set of pre-training datasets D pre . During pre-training, we sample (D ′ k,i , k), a dataset and its domain label from D pre and train the model on each of the self-supervised learning tasks in T pre . The tasks in T pre are self-supervised and do not require additional labels or other ground truth. These tasks transform the input data and train the model to recover the original input or important properties or parts of the input. Therefore, our problem can be formally stated as: Given a heterogeneous set of multi-domain datasets D pre and their domain labels, we train a model leveraging SSL tasks T pre that learns important patterns and knowledge that can be leveraged on fine-tuning the model to any time-series analysis task on any novel dataset from any of the domains d ∈ {1, 2, . . . , K}. Most of the parameters θ pre of the pre-trained model are trained over all the datasets and tasks. However, we use a separate segmentation module for each dataset domains to capture varied sizes of segments that differ across datasets. These segments are used as tokens for a transformer model that shares the parameters across all the tasks. For each of the pre-trained task as well as downstream tasks we append a final linear layer on the output embeddings of the transformer to generate the final prediction. Note that during fine-tuning on downstream tasks we update the parameters of all the modules of LPTM.We use ETT electricity datasets (ETT1 and ETT2) collected from (Zhou et al., 2021) at 1 hour intervals over 2 years. We use the default 12/4/4 train/val/test split and use the train split for pre-training as well. 
segmentation module: 
 Due to their ability to model long-range temporal relations as well as scale up to learn from large datasets, transformers (Vaswani et al., 2017) segments and use each of the segments as tokens to the transformer model. However, different pre-trained datasets may have varied temporal scales, periodicity and other temporal dynamics that cannot be encompassed by a single uniform segmentation strategy. For example, epidemic time-series are usually observed at weekly scale and may have characteristic properties like seasonality, peaks and sudden outbreaks that should be captured by segmentation. Economic time-series, in contrast, are typically captured every quarter and are more monotone with sudden anomalies and changes in data distribution. Moreover, using a uniform segmentation may not be ideal for time series that have multi-scale trends with some time-stamps having denser temporal information requiring finer-graned segmentation than others. Therefore, our goal is to identify an independent segmentation strategy for each domain of time-series dataset.For a given input time-series y (1...t) , we pass it through a GRU to get hidden embeddings {z (i) } t i=1 that models the temporal patterns of the input:EQUATIONWe then introduce a segment score function s that provides a scalar score for any subsequence of the input time-series:s(i, j) = v T tanh (W 1 z i + W 1 z j + b) .(2) The score s(i, j) for a subsequence from time-stamp i to j denotes how good the given segment is for the dataset.In next step, we sample subset S(y (1...t) ) of subsequences over the time-series that a) covers the entire input time-series, b) has a high score function value. While retrieving the optimal S(y (1...t) ) is an interesting combinatorial optimization problem, we generate S(y (1...t) ) using a simple process as follows: for each i ∈ {1, 2, . . . , t -1}, we denote h(i) = arg max j∈{i+1...,t-1} s(i, j) as the best segment starting from time-step i. Then we generate the set of segments Ŝ(y(1...t) ) = {(i, h(i))} t-1 i=1 .In order to reduce the number of segments, we iteratively remove the lowest-scoring segments until we cannot remove any more segments without having time-steps not being covered by any segments in the set. The final set of segments after pruning is denoted as S(y (1...t) ).To generate the token embeddings for each segment (i, j), we pass the embeddings {z (i) , z (i+1) , . . . , z (j) } through a self-attention layer used in transformers and aggregate the output embeddings. Additionally, we concatenate the following features to the token embedding of each segment token:• Positional encoding of the starting time-step of the segment pos(i) defined as:EQUATION)where D is the dimensions of output embedding of self-attention over {e i , e i+1 , . . . , e j }. • Positional encoding of the length of the segment pos(j -i) • The time-series values of segment are passed though a single layer of transformer encoder and aggregated to a fixed length embedding of dimension D.These features allow the transformer additional information from the segment directly derived from values of time-series. The final output of the segmentation module is a sequence {e i } R i=1 where R is the size of S(y (1...t) ) and sequence is arranged based on the ascending order of the first time-stamp of each segment.Pre-training on a wide range of heterogeneous datasets from multiple domains helps LPTM learn from useful patterns and latent knowledge across these domains that can be generalized to range downstream tasks on multiple domains. We propose two general self-supervised learning tasks motivated by pre-trained language models to enable LPTM to learn from all pre-trained datasets. We leverage a transformer model and use the segment token embeddings of the segmentation module. The two pre-training SSL tasks are Random Masking (RANDMASK) and Last token masking (LASTMASK). RANDMASK allows the model to extrapolate and interpolate masked segments of the input time-series. RANDMASK has also been explored for representation learning in previous works (Zerveas et al., 2021; Nie et al., 2022) but they are applied on the same dataset as that used for training unlike our data and task-agnostic pre-training setup. Formally, we mask each input segment token with a probability of γ and decode the values of time-series of the masked segments from the output embeddings of the transformer. We use a simple GRU with a single hidden layer on the transfer's output embedding to decode the values of the segment and use mean-squared error as the loss. LASTMASK is similar to RANDMASK except we mask last γ fraction of the segments. This allows the model to forecast the future values of the time-series, a very important task in many time-series domains.The values of the time-series of each dataset can vary widely based on the application the the target value observed in the time-series. Therefore, as part of pre-processing we first normalize the time-series of each dataset of pre-train datasets independently. Moreover, the data distribution and the magnitude of the time-series can vary across time. We use reversible instance normalization (REVIN) layer Kim et al. (2021) . REVIN performs instance normalization on the input time-series and reverses the normalization of the output values. The normalization step is part of the neural model and gradients are calculated over the normalization and reverse normalization layers.Training the score function We use the loss from the SSL tasks to also train the score function of the segmentation module. Since there is no direct gradient flow between the score function and the final predictions, due to the discrete nature of choosing the segments, we match the aggregated scores of all the chosen segments in S(y (1...t) ) to the negative logarithm of the total MSE loss of both SSL tasks:EQUATIONwhere L SSL is the total loss of both SSL tasks. We also backpropagate over L g once every 10 batches. This is to stabilize training since changing the segmentation strategy for every batch leads to unstable and inefficient training.Linear-probing and fine-tuning Kumar et al. (2022) showed that fine-tuning all the parameters of the pre-trained model for a specific downstream task can perform worse than just fine-tuning only the last layer (linear probing), especially for downstream tasks that are out-of-distribution to pre-trained data. To alleviate this, based on the recommendation from Kumar et al. (2022) , we perform a two-stage fine-tuning process: we first perform linear probing followed by fine-tuning all the parameters.We use 2 datasets related to traffic speed prediction. PEMS-Bays and METR-LA (Li et al., 2017) are datasets of traffic speed at various spots collected by the Los Angeles Metropolitan Transportation Authority and California Transportation Agencies over 4-5 months. 
datasets: 
 We derive pre-train time-series datasets from multiple domains:1. Epidemics: We use a large number of epidemic time-series aggregated by Project Tycho (van Panhuis et al., 2018) . from 1888 to 2021 for different diseases collected at state and city levels in the US. We remove time series with missing data and use time series for 11 diseases of very diverse epidemic dynamics such as seasonality, biology, geography, etc.: Hepatitis A, measles, mumps, pertussis, polio, rubella, smallpox, diphtheria, influenza, typhoid and Cryptosporidiosis (Crypto.).We use bike and taxi demand datasets from New York City collected from April to June 2016 sampled every 30 minutes. We all but the last 5 days of data for training and pre-training.We test the pre-trained LPTM trained on datasets discussed in §4.1 on multiple forecasting and time-series classification tasks. We perform forecasting on the influenza incidence time series in US and Japan. Specifically, we use the aggregated and normalized counts of outpatients exhibiting influenza-like symptoms released weekly by CDC1 . For influenza in Japan, we use influenza-affected patient counts collected by NIID2 . We forecast up to 4 weeks ahead over the period of 2004 to 2019 flu seasons using a similar setup as Flusight competitions Reich et al. (2019) .We also perform electricity forecasting on the ETT1 and ETT2 datasets using the train/test split mentioned previously. The last 10% of PEM-Bays dataset is used for traffic forecasting up to 1 hour ahead and the last 5 days of New York demand datasets for demand forecasting up to 120 minutes in the future. We also perform forecasting on the Nasdaq dataset for up to 5 days ahead and M3 time-series for 1 month ahead. We use 6 of the sensor datasets from Asuncion & Newman (2007) for time-series classification tasks. We use an 80-20 train-test split similar to Chowdhury et al. (2022) .We compared LPTM's performance in a wide range of time-series tasks against seven state of art general forecasting baselines as well as domain-specific baselines. We compared with (1) Informer Zhou et al. (2021) and (2) Autoformer Chen et al. (2021) , two state-of-the-art transformerbased forecasting models. We also compare against the recent model ( 3) MICN (Wang et al., 2022) which uses multiple convolutional layers to capture multi-scale patterns and outperform transformerbased models. We also compared against best models for individual tasks for each domain. For influenza forecasting, we compared against previous state-of-art models (4) EpiFNP Kamarthi et al. ( 2021) and ( 5) ColaGNN Deng et al. (2020) respectively. We also compare against (6) STEP Shao et al. (2022) that leverages Graph Neural Networks for forecasting and provides the best performance for demand forecasting, traffic prediction, and stock prediction benchmarks among the baselines by automatically modeling sparse relations between multiple features of the time-series.For classification tasks on behavioral datasets, we compare against the state-of-art performance of ( 7) TARNet Chowdhury et al. (2022) .In order to test the efficacy of our multi-domain pre-training method, we also compare it against two other state-of-art self-supervised methods for time-series. These prior SSL methods (Yue et al., 2022; Tonekaboni et al., 2021; Eldele et al., 2021; Nie et al., 2022) have shown to improve downstream performance by enabling better representation learning. However, the SSL pre-training is only done on the same dataset used for training for the downstream task and does not cater to pre-training on multiple heterogenous datasets from varied domains, unlike LPTM. Therefore, we also compare LPTM against previous works on self-supervised representation learning on time-series: TS2Vec (Yue et al., 2022) and TS-TCC (Eldele et al., 2021) . 
stock forecasting:: 
 We also collect the time-series of daily stock prices of Nasdaq and S&P 500 index using yfinance package (yfi) from July 2014 to June 2019. We train and pre-train using the first 800 trading days and use the last 400 for testing.6. M3 competition time-series: We also used the 3003 time-series of M3 forecasting competition (Makridakis & Hibon, 2000) which contains time-series from multiple domains including demographics, finance, and macroeconomics.7. Motion and behavioral sensor datasets: We use the set of sensor datasets extracted from UEA archive (Bagnall et al., 2018) and UCI Machine learning repository (Asuncion & Newman, 2007) similar to (Chowdhury et al., 2022) .The code for implementation of LPTM and datasets are provided at anonymized link3 and hyperparameters are discussed in the Appendix.We summarize the forecasting performance using RMSE scores in Table 1 . LPTM is either the first or a close second best-performing model in all the benchmarks in spite of comparing our domain-agnostic method against baselines designed specifically for the given domains. LPTM beats the previous state-of-art domain-specific baselines in five of the benchmarks and comes second in four more. Moreover, LPTM improves upon the state-of-art on electricity forecasting, traffic forecasting, and M3 datasets. Further, we observe that LPTM is better than other transformer-based state-of-art general time-series forecasting models as well as SSL methods which underperform all other baselines in most cases. This, therefore, shows the importance of our modeling choices to be capable of learning from diverse time-series datasets to provide performance that is similar to or better than previous state-of-art in most downstream tasks. We evaluate LPTM and baselines on the classification of sensor and behavioral datasets from (Asuncion & Newman, 2007) . We report the F1 scores in Table 2 . We observe that LPTM outperforms the previous state-of-art model, TARNet (Chowdhury et al., 2022) in 3 datasets and is a close second best model in others.A significant advantage of leveraging pre-trained models in the case of vision and language models is that we do not require a large amount of training data for fine-tuning to a specific task. In fact, in many cases, we require very few examples (Brown et al., 2020) to fine-tune the model.We evaluate the efficacy of LPTM to train with a smaller fraction of task-specific training data. For each time-series analysis task, we fine-tune the model using only k% of training data for different values of k. The k% chosen is generated by using on the first k% of the timestamps' values. We do not choose a random sample to prevent data mixing from the rejected portion of training data. We also performed the similar experiment on the best baseline for each task and compare data efficiency of baseline with LPTM. The comparison plots are shown in Figure 2 . With lesser data, the performance of the baseline is much worse whereas LPTM typically requires much less data to provide similar performance to when we have access to the full dataset. This shows the importance of pre-training to quickly ramp up the performance of the model with much less data, a problem we encounter is many real-world settings such as when we need to deploy a forecasting model on novel applications such as a new pandemic with sparse data availability.Another important advantage of pre-trained models is that they require much less training time and resources to fine-tune to a downstream task compared to time required for pre-training or even training from scratch. We compare the training (or fine-tuning) time of LPTM with baselines on benchmarks from different domains. We also measure the avergae time required by LPTM to reach the performance of best baseline in cases where we eventually outperform them.The training times are summarized in Table 3 . First, we observe that the time taken by LPTM to reach the performance of best best-performing baseline (LPTM-TB) is significantly smaller than the time taken by any other baselines. Further, even in cases where LPTM doesn't outperform the best baseline, it typically converges much faster. This shows that LPTM requires fewer training steps and therefore less compute time to fine-tune to any downstream task.We finally study the impact of our various technical contributions to LPTM by performing an ablation study. Specifically, we formulate the following variants of LPTM to study the impact of our important modeling choices:• LPTM-NoSegment: We remove the novel segmentation module and directly encode each time-step as a separate token. • LPTM-NoPreTrain: We do not perform any pre-training and instead directly learn from scratch for each downstream task.• LPTM-NoLinProb: Instead of the two-step fine-tuning procedure discussed in §3.4, where we first fine-tune only the last layer (linear-probing) followed by fine-tuning all parameters of the model, we skip the linear-probing.The performance of the ablation variants for forecasting and classification tasks are also shown in Tables 1 and 2 respectively. We observe that the ablation variants' performances are significantly worse than the variants, underperforming some of the baselines. The worst performing variant is usually LPTM-NoSegment, showing the importance of deriving good time-series segments to improve representation learning of time-series for each dataset. 
conclusion: 
 We make a significant contribution towards general pre-trained models for time-series analysis tasks replicating the success of large pre-trained models in language and vision domains. We introduce LPTM, a general pre-trained model that provides state-of-art performance on a wide range of forecasting classification tasks from varied domains and applications. LPTM provides similar performance to state-of-art domain-specific models in applications such as epidemiology, energy, traffic, and economics and significantly beats state-of-art in widely used traffic prediction and M3 datasets. We also observe that LPTM required significantly lesser training data during fine-tuning to reach optimal performance compared to other baselines in most benchmarks. LPTM is also more efficient by requiring much less training steps (20-50% lesser) to attain similar performance as domain-specific models.Our work mainly focuses on the important challenge of providing semantically meaningful inputs to the model that caters to learning time-series segmentation strategies specific to each domain. This is crucial when pre-training on diverse datasets, a key challenge for time-series data. The underlying model architecture is a straightforward transformer encoder that uses well-known masking techniques for self-supervised pre-training. Therefore, our method can be extended to leverage novel time-series model architectures and SSL methods. Extending our methods to provide calibrated forecasts that provide reliable uncertainty measures is also another important direction of research.Since our model can be applied to any generic time-series analysis tasks including those in critical domains such as public health, medicine, economics, etc., important steps need to be taken to address potential misuse of the our methods such as testing for fairness, data quality issues, ethical implications of predictions, etc. 
 
 Appendix: https://gis.cdc.gov/grasp/fluview/fluportaldashboard.htmlhttps://www.niid.go.jp/niid/en/idwr-e.htmlhttps://anonymous.4open.science/r/SegmentTS-6145/ 


Decision: The submission describes an approach called Large Pre-trained Time-series Models (LPTM) that improves time-series forecasting and classification by introducing an adaptive segmentation module. This module automatically identifies optimal segmentation strategies for different datasets, addressing the limitations of fixed-length segmentation methods. The paper demonstrates that LPTM achieves strong performance across multiple domains with reduced data and training time

The paper received generally positive feedback from reviewers, with a few areas for improvement noted. 

**Positive Points:**
* The reviewers appreciated the innovative adaptive segmentation approach, which provides a flexible and effective way to handle diverse time-series data across multiple domains.
* Experiments against a wide range of baselines were seen as a strength.
* The clarity and thoroughness of the paper's presentation, particularly in terms of the problem's relevance and the method's motivation, were commended.

**Negative Points:**
* Several reviewers pointed out the lack of detailed investigation into the interplay between the adaptive segmentation module and the encoder, particularly in understanding why the method works as well as it does.
* Concerns were raised regarding the presentation and reproducibility, including issues with language, formatting, and an expired link to the implementation code.
* The scalability and computational overhead of the adaptive segmentation module were not sufficiently discussed, particularly in relation to the potential increase in tokens and the quadratic scaling of the method.

The LPTM method represents an advance in time-series analysis. While the paper could benefit from a deeper exploration of certain technical aspects and improved clarity in its presentation, these issues do not detract from the overall quality and impact of the work. Given its technical merit and practical relevance, this paper is well-suited for presentation at NeurIPS.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel contribution to pretrained time series models for forecasting and classification by paying attention to the fact that currently several transformer models take time series segmentations of the same size, regardless of the particular characteristics of the time series in consideration. For instance, time series that have yearly frequency or minute frequency might require different segmentation lengths, or it might be that dynamics are more complex in certain time intervals requiring a more detailed segmentation. Based on this observation the authors proposed a model that can find a suitable segmentation schema that later on allows to observe where are the time intervals where more complex dynamics are shown.

The authors perform several experiments and claim empirically that the proposed approach is at least competitive to the state of the art.
strength_weakness: strength_weakness: Some of the main limitations are as follows:
- The proposed framework is not differentiable. The authors have acknowledged this in the paper and propose a workaround for this, basically to update the segmentation scores every 10 batches. Yet, this poses challenges like the interpretation of the training loss, and discontinuities in the test loss.
- It is unclear if the proposed approach is able to handle missing values. If not, is there anyway to overcome this? Missing values are very often present in practice and having a sound way to handle them is relevant.
- It is unclear if the current evaluation is fair. The authors present a corpus of datasets for which they pretrained the proposed model, but it is unclear which datasets where hold-out from pretraining. This is relevant as several of the pretrained models considered might have not been exposed to these datasets, which gives an unfair advantage to the proposed model. Further, since the amount of pretraining datasets is rather limited, there is the possibility that the proposed model is overly focused on these datasets, whereas other models, like (Ansari 2024) and (Woo 2024) were trained in a larger corpus of datasets.
questions: questions: Question: 

* Eq-1: is the GRU applied entry-wise to the time series? Does this imply that we apply $GRU_1$ to each entry of $y$ (which has $t$ entries), and then the resulting $t$ values constitute the hidden embeddings?
* Eq. 2: what is $z_i$? So far we have talked about $z^{(i)}$.
* Missing closing parenthesis in fig 1: $S(y^{(1...T)$
* Eq-1: the larger values of S(i,j) the better? Does it mean that the correlation between $z_i$ and $z_j$ is high, or that $z_i$ and $z_j$ are related somehow? 
* Eq-3: index $k$ is never used in this definition of output embeddings.
* Eq-5: as pointed out by the authors, the selection of segments is not differentiable and hence it can not be directly integrated to the loss function. Does it mean that the segments are updated every 10 batches? This means that the loss will not be continuous, and hence it will be unclear if there is progress or not in terms of the training loss. Is this correct? I guess here what nevertheless can hint at improvement is the test loss.
* In line 225: why are time series with missing values removed? is the proposed model able to handle missing values?
* The authors claim that their model is a pretrained model. What datasets were used to pretrain the model? Are all datasets used as well for evaluation in Table 1? If yes, then the comparison is not fair. Several of the pretrained models considered might have not been exposed to those datasets in pretraining, giving an unfair advantage to the proposed approach. Further, doing pretraining in such a small amount of datasets further gives more advantage to the proposed model, as the larger the datasets potentially gives a smaller amount of exposure to each dataset.
limitations: limitations: The authors have acknowledged limitations.
paper_summary: paper_summary: The paper proposes Large Pre-trained Time-series Models (LPTM), a novel method designed to improve the efficiency and performance of time-series analysis across multiple domains. 
The key contribution is an adaptive segmentation module that automatically identifies optimal segmentation strategies for diverse datasets during pre-training. 
This approach aims to overcome the limitations of fixed-length segmentation, which may not adequately capture the temporal patterns of heterogeneous time-series data. 
LPTM demonstrates superior forecasting and classification performance, requiring up to 40% less data and 50% less training time compared to state-of-the-art models.
strength_weakness: strength_weakness: W1. Although time series has a weaker semantic structure than natural language, it is closer connection to images.
In both time series and images, a semantic unit, e.g., a small item or a texture in an image, can have different lengths and scales.
This raises a challenge against the main motivation: why a full self-attention-based architecture works for images (e.g., ViT), why for time series the segmentation needs to be explicitly done?
It would be interesting if the authors can further discuss this problem and provide their intuitions.

W2. The introduction of the adaptive segmentation module seems to bring instability in the initial model training, as well as requiring longer training time (although the authors propose to backpropagate the gradients every 10 batches).
Specifically, the loss function for segmentation is a hard loss based on the selected subset of best segments.
However, the parameters seem to be randomly initialized, which could provide highly random "best" segments.
Hence, the convergence stability and the training time with and without the dynamic segmentation modules should be discussed.

W3. The dynamic segmentation modules seem not to be fine-tuned with specific attention.
However, as the author(s) mentioned, different datasets could have very different best segmentation.
Hence, it would be interesting to discuss why this is sufficient and provide theoretical or empirical evidences.
questions: questions: Q1 (cr. W1). Please provide intuitions and empirical evidences on why full self-attention-based architectures work for images while an explicit segmentation module is required for time series.

Q2 (cr. W2). Please discuss the convergence stability, especially the initial convergence stability and the influence of random initialization, as well as the influence of the adaptive segmentation modules on the pre-training speed.

Q3 (cr. W3). Please discuss the influence of fine tuning on the adaptive segmentation, e.g., which the current framework can make a large change to the existing segmentation. 
Experiments with and without fine-tuning the adaptive segmentation would be interesting to report.

Q4. It would be interesting if the authors may show some case studies on the adaptive segmentation results, i.e., whether and how much the adaptive segmentation results conform to the source domain of the dataset, whether some periodic patterns can be well preserved after the adaptive segmentation module.
limitations: limitations: L1. There is a lack of explanability and interpretability in the adaptive segmentation results.
paper_summary: paper_summary: The paper introduces a new approach for creating pre-trained models for time-series data, similar to those used in language and vision tasks. The authors propose a model called Large Pre-trained Time-series Models (LPTM), which includes an innovative adaptive segmentation module to handle diverse time-series data from multiple domains.

Key contributions include:

- Developing a framework for pre-training time-series models on multi-domain datasets, using a novel adaptive segmentation module to tokenize inputs effectively. This is achieved via a self-supervised learning objective.
- Demonstrating that LPTM performs as well or better than state-of-the-art domain-specific models when fine-tuned for various time-series tasks, such as forecasting and classification, with less training data and compute time.
- Proving that LPTM achieves superior results in both zero-shot and fine-tuned settings across diverse domains like epidemiology, energy, and economics, requiring up to 40% less data and 50% less training time compared to existing methods.
strength_weakness: strength_weakness: - It's a bit hard to get a good feel for the relative advantage of the proposed method. In table 2, the approach is clearly better, but we are left to infer that from that fact that it is commonly second or first in the rankings. Could the authors maybe add some for of aggregate metric, e.g. the average rank across datasets of a given method?
- Despite mentioning code is available, the link does not work (subscript 3 on page 7, time of access 2024-07-12, and previously): "The repository is expired".
- For a paper dealing in large part with forecasting, I was surprised by the absence of almost all of the classical long-term forecasting datasets used by other papers: traffic, electricity, weather, illness... Given that these are by far the most heavily studied ones in the literature, including them (as proposed in the questions section). While I don't find it a critical (but still important) concern, I strongly advise the authors to consider adding them as it will help avoid concerns other readers might have about cherry-picking of results.
questions: questions: - Can the authors add error bars (in the appendix possibly) for their experiments? They mention that they already run 10 experiments per setup so these should be readily available and would give a good idea of the robustness of the findings.
- Can the authors ensure that code to reproduce their experiments is available as stated? 
- Can the authors run the same experiments on the "standard" long-term forecasting datasets, as listed in the weaknesses section?

Note: I feel the paper is definitely interesting and makes valid contributions. Addressing my questions, in particular the one about the long-term forecasting datasets would be a strong argument for me to raise my score.

Edit: I've read the rebuttal provided by the authors, and since my open questions have been addressed I'm raising my score to 7.
limitations: limitations: Yes
paper_summary: paper_summary: Training large time series (TS) models is often limited by the scarce data available for a specific application. Existing pretraining methods use a simplistic tokenization scheme where the TS is cut up into equally sized parts, independent of its content. The newly proposed method *Large Pre-trained Time-series Models*, therefore, adaptively segments the input time series into (potentially) overlapping tokens depending on the TS itself. It shows very good forecasting performance in zero-shot and finetuning settings. It can also be used for classification.
strength_weakness: strength_weakness: - I am under the impression that this paper may have found a strong method yet does not sufficiently investigate *why* it works. The interplay between learning the scoring function and training the encoder is not very clear. See below.
- A lot of experimental claims are not adequately substantiated. It is claimed in question 6 of the checklist that error bars are provided and that statistical significance tests are performed, yet I did not find them. See below.
- The overall presentation (language and formatting) should be improved.
- The provided implementation is not accessible. (Error: "The repository is expired") In the current state, results are not reproducible since key hyperparameters are missing. The authors claim in question 6 of the checklist that they state how hyperparameters were chose, yet I could not find it in the paper.
questions: questions: 1. The problem statement (l. 96-98) only considers in-domain applications. Is it also interesting to consider domain generalization settings?
2. The term "Large Pre-trained Time-series Models (LPTM)" (l. 48-51) is extremely generic and better describes the emerging field than a single method. Would it make sense to focus more on the "Adaptive Segmentation" contribution in the title and exposition?
3. Why did you choose Eq. (2) to be that way? Are all parameters learned?
4. The scaling of the method is not discussed sufficiently; could you comment on that? There are potentially a lot of tokens -- in the worst case, there are as many as input tokens. Even more, the number of evaluations of $s(i,j)$ scales quadratically in the input length. While the heuristic for selecting a good set of segments is defined well, a discussion of why it is sensible is missing. See also question 7.2.
5. Why did you use masked reconstruction as the two pretraining tasks? Could you explain why this is more desirable than alternatives, like contrastive methods?
6. The interplay of training the segmentation strategy and the encoder simultaneously requires a more nuanced discussion. To what extent can this cause something like a "mode collapse" (as in Generative Adversarial Networks), where the $s(i,j)$ always chooses the same segments since they were found to be beneficial at some point and stops "exploring" others? This should (1) be discussed in more depth and (2) may be a significant limitation of the method.
7. This leads to the experimental evaluation.
	1. Following (6), can you provide insights into the relative convergence speeds of the scorer and the encoder/decoder? How good is the score function $s$ at predicting the final encoder loss?
	2. How many segments are there typically? A histogram would help judge the typical number of tokens resulting from the adaptive segmentation.
	3. L. 281: "2x to over 10x smaller than other pre-trained time-series models" -> Where do you provide that comparison of model sizes?
	4. Section 5 only discusses (almost) exclusively the forecasting setup. Why were the specific classification datasets chosen? Section 6 mentions 35 datasets (UCI has more time series classification datasets), but Table 4 only contains 32. Were three datasets removed at some point?
	5. L. 305: "We observe that LPTM has highest mean rank" -> Please provide that rank in (or alongside) Table 4.
	6. The caption of Table 4 talks about statistical significance. What are the test's results? Which significance test did you even perform? How were results aggregated (if it was the arithmetic mean, what is their std. dev.)?

Minor comments:
- The paper would benefit from a language pass.
- Improvements to formatting: References are formatted confusingly, e.g., in lines 22, 23f, 36f, etc. This problem occurs throughout the paper. Table 4 in the Appendix is too wide. REVIN -> RevIN. Formatting of LASTMASK, etc., is inconsistent (l. 163 vs. l. 335), ...
- The example in line 40ff could be given more bluntly and convincingly by discussing the same event in different sample rates.
- L. 91, what is R? The same as in l. 152?
- L. 89, isn't $\mathcal{D}_\text{pre}$ the union over the individual datasets?
- The abbreviation SSL has never been introduced.
- Red and green are difficult to distinguish for people with common color deficiencies. Therefore, Figure 1 could be improved with a different color palette.
- Table 3 should mention that it is about finetuning. Are the baselines trained from scratch or merely finetuned as well?
- The authors should check whether they intentionally want to cite many preprints (e.g., from Arxiv) or their published variants.
- The capitalization of the title is inconsistent.
limitations: limitations: Depending on the answers to the questions above, possible limitations mentioned could be made more transparent. For example, the insights into the newly induced biases are currently limited.

The answer to section 2. in the checklist is neither sufficient nor truthful. For example, "multivariate" is never mentioned in Section 7.

The discussion of the societal impact could also consider a possible data leakage from one private application (e.g., in the medical domain) to another one. Even a rather mundane problem, like a feasible membership inference attack, could be problematic in privacy-sensitive scenarios.

Comments: title: Response to Reviewer
Thank you
Response to rebuttal
comment: Thanks to the authors for responding to my concerns. I remain positive about this work, and would like to raise my score to 6.

Just to clarify a bit about my question Q3, what I would expect are more details on fine-tuning in LPTM. For example, how easily it can adapt to different domains, how quickly it can converge, and how much improvement it can provide. I think this is interesting because I already acknowledged the importance of fine-tuning in dynamic segmentation, and cross-domain is a main topic for this paper. I look forward to the future versions.
I would like to thank the authors for taking the time for these replies and for addressing my concerns.

I have updated my rating to 5: Borderline accept.
We thank the reviewer for acknowledging our response and are glad to have addressed their concerns. We are grateful for increasing their scores.
We again thank the reviewer for their valuable comments that helped improve our work. We hope we have addressed their questions and concerns. 

Since the discussion period is soon ending, we hope the reviewer can acknowledge our response. We would also gladly answer any further questions or clarifications.
We again thank the reviewer for their valuable comments that helped improve our work. We hope we have addressed their questions and concerns. 

Since the discussion period is soon ending, we hope the reviewer can acknowledge our response. We would also gladly answer any further questions or clarifications.
Thank you for your insightful responses. I have no further major questions. I hope the above discussion and improvements to the language and general presentation make it into the camera-ready version.

**I raised my score from 3 to 5.**

As a side note, providing Critical Difference diagrams for the classification results would improve the rigor of the analysis.
We are glad that our response has addressed your comments and strengthened your case for the paper. We are thankful for the increase in score.

We would be glad to answer any more of your questions.
I thank the authors for responding to my comments and questions. I feel that the added points strenghten the case for accepting the paper, and have raise the score to 7 as a result.