Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series
Authors: Vijay Ekambaram
Arindam Jati
Pankaj Dayama
Sumanta Mukherjee
Nam H Nguyen
Wesley M. Gifford
Chandra Reddy
Jayant Kalagnanam
Venue: NeurIPS 2024 poster (2024)
Keywords: time series
foundation models
pretrained models
forecasting
time-series
time
tsfm
light-weight
forecasters
Primary area: deep_learning_architectures
TL;DR: Fast and Light-weight Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series

Abstract: Large pre-trained models excel in zero/few-shot learning for language and vision tasks but face challenges in multivariate time series (TS) forecasting due to diverse data characteristics. Consequently, recent research efforts have focused on developing pre-trained TS forecasting models. These models, whether built from scratch or adapted from large language models (LLMs), excel in zero/few-shot forecasting tasks. However, they are limited by slow performance, high computational demands, and neglect of cross-channel and exogenous correlations. To address this, we introduce Tiny Time Mixers (TTM), a compact model (starting from 1M parameters) with effective transfer learning capabilities, trained exclusively on public TS datasets. TTM, based on the light-weight TSMixer architecture, incorporates innovations like adaptive patching, diverse resolution sampling, and resolution prefix tuning to handle pre-training on varied dataset resolutions with minimal model capacity. Additionally, it employs multi-level modeling to capture channel correlations and infuse exogenous signals during fine-tuning. TTM outperforms existing popular benchmarks in zero/few-shot forecasting  by (4-40\%), while reducing computational requirements significantly. Moreover, TTMs are lightweight and can be executed even on CPU-only machines, enhancing usability and fostering wider adoption in resource-constrained environments. The model weights for reproducibility and research use are available at https://huggingface.co/ibm/ttm-research-r2/, while enterprise-use weights under the Apache license can be accessed as follows: the initial TTM-Q variant at https://huggingface.co/ibm-granite/granite-timeseries-ttm-r1, and the latest variants (TTM-B, TTM-E, TTM-A) weights are available at https://huggingface.co/ibm-granite/granite-timeseries-ttm-r2. The source code for the TTM model along with the usage scripts are available at https://github.com/ibm-granite/granite-tsfm/tree/main/tsfm_public/models/tinytimemixer

Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-shot Forecasting of Multivariate Time Series
Vijay Ekambaram
Arindam Jati
Pankaj Dayama
IBM Research Bangalore, India vijaye128 in. ibm. com
IBM Research Bangalore, India
IBM Research
Bangalore, India
arindam, jat 8imm. com
panka pankajday jadayaaan ddaymmal iin. ibm. com
Sumanta Mukherjee
Nam H. Nguyen
Wesley M. Gifford
IBM Research Bangalore, India sumanm030 in. 1.mm com
IBM Research
IBM Research
Yorktown Heights, NY, USA
Yorktown Heights, NY, USA
nnguyenêus. ibm. com
wmgi ingifforduu.. ibm com.
Chandra Reddy
Jayant Kalagnanam
IBM Research Yorktown Heights, NY, USA creddy8us, wim.ccoom com
IBM Research Yorktown Heights, NY, USA
jayant i.m. com
Abstract
Large pre-trained models excel in zero/few-shot learning for language and vision tasks but face challenges in multivariate time series (TS) forecasting due diverse data characteristics. Consequently, recent research efforts have focused on develop- ing pre-trained TS forecasting models. These models, whether built from scratch or adapted from large language models (LLMs), excel in zero/few-shot forecast- ing tasks. However, they are limited by slow performance, high computational demands, and neglect of cross-channel and exogenous correlations. To address this, we introduce Tiny Time Mixers (TTM), a compact model (starting from 1M pa- rameters) with effective transfer learning capabilities, trained exclusively public TS datasets. TTM, based on the light-weight TSMixer architecture, incorporates innovations like adaptive patching, diverse resolution sampling, and resolution prefix tuning to handle pre-training on varied dataset resolutions with minimal model capacity. Additionally, it employs multi-level modeling capture channel correlations and infuse exogenous signals during fine-tuning. TTM outperforms existing popular benchmarks in zero/few-shor forecasting by (4-40%). while reduc- ing computational requirements significantly. Moreover, TTMs are lightweight and be executed even CPU-only machines, enhancing usability and fostering wider adoption resource-constrainee environments. The model weights for reproducibility and research
available here, while enterprise-use weights under the Apache license can be accessed as follows: the initial TTMM here) and the latest variants (TTMg, TTMg, TTM,) weights available here The source code for the TTM model along with the usage scripts are available here
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Multivariate time-series (TS) forecasting entails predicting future values for multiple interrelated time series based on their historical values. The channels being forecast are called target variables, while those influencing the forecasts are termed exogenous variables. This field has seen significant advancements through the application of statistical and machine learning (ML) methods across various domains such as weather, traffic, retail, and energy.
Related Work: Recent advances in multivariate forecasting have been marked by the advent of Transforrerr-base [31] approaches, exemplified by models like PatchT'T [22]. Autoformer [38], and FEDFormer [45). These models have demonstrated notable improvements over traditional statistical and ML methods. Furthermore, architectures based on MLP-Mixer [30], such as TSMixer [6] and TimeMixer B3, have emerged as efficient Transformer alternatives, boasting 2-3X reduced compute requirements with no accuracy compromise compared to their Transformer counterparts.
Recently, there has been substantial interest in the research community to build general pre-trained or foundation models (FMs) for 'S forecasting that can successfully transfer the learning to unseen target TS dataset, similar to the successes in NLP and vi- sion tasks. However, pre-training in the TS domain is particularly challenging due to the limited pub- lic availability of the datasets and the diverse nature across applications. Early in 2024, this interest cul- minated in the release of several "large" and "mas- sive" TS pra-trained models for forecasting, gener- ating considerable excitement in the research com- munity. Among these releases were Moment 10) 3 TimesFM 3 Chronos _ Moirai [35 and Lag- laaaa (26 S that successfully established strong bench- marks in zero-shot forecasting. In addition, there has been a trend towards leveraging pre-trained large lan- guage models (LLMs) for TS forecasting, treating the forecasting task as a form of cross-domain transfer learning. These universal cross-transie approaches, exemplified by recent works such as LLMTime [II), TiMe-LLM [15) and GPT4TS 46). exhibit promising outcomes in zero/few-shot forecasting scenar- ios. However, most of these "large" TS pre-trained models demand extremely high computational resources, given their scale ranges from several hundred million to billions of parameters. Given the recent surge in popularity of "small" language models I 29 39 that address practical resource and cost constraints in woold industrial settings, this work considers the following question: Can "tiny" pre-trained models succeed in the TS domain too? If so, can they outperform the zeroffew-shot forecasting results of "large" " TS pre-trained models demanding significant computational resources and runtime? Surprisingly, as we demonstrate in this work, the answer is yes.
Chrunees 32% L (6M, 2200s
Claouito 17% (20M, 20.0s)
CPU Inference Time (6)
10P
Marinai (SUIM 10s)
Lag-Lluma 40%
10F
(2/MM, (7s)
Moinal, 4%% (14M, 1.4x)
Momuas 3% (3IMM 1.fe)
10
FimePM 15% 220M, (ias)
GPTITS [MIM, (.216)
TTM, (ours) (IM 107*s)
10
10P
100
10
Model Size (M)
Figure 1: Size, time, and accuracy overview of TTM2, vs. open-sourced pre-trained TS hench- marke] We plot each model based on its model size and per batch CPU inference time. The X% mentioned for each baseline indicates that the base- line's forecast is X% less accurate compared to TTM's forecast. Full details in Tables 5-
Toward this, we propose Multi-level Tiny Time Mixers (TTM), significantly smaller pre-trained model (starting from I million (M) parameters) for effective zero/few multivariate forecasting. In particular, TTM supports channel correlations and exogenous signals, which are critical and practical business requirements in the context of multivariate forecasting, features lacking in many existing TS pretrained models. TTM is based on the light-weight TSMixer architecture that uses MLPMixer blocks interleaved with simple gated attention as alternatives to the quadratic time- consuming self-attention blocks in Transformers, which makes TTM pre-training and fine-tuning extremely fast. TTM is pre-trained using multiple public datasets (~1 billion (B) samples) from the Monash and LibCity data repositories. Note that the datasets exhibit considerable diversity in characteristics, such as different domains, temporal resolution B (ranging from seconds to days),
""Caanell" refers to individual dimension in multivariate data (i.c., multivariate or multichannel). 'Work done concurrently with this research.
'Time-LLM and LLMTime are excluded here as we couldn't run them on CPUs, but their accuracy compared later in experiments.
"Resolution refers to the sampling rate of the input time series (e.g., hourly, 10 minutes, 15 minutes, etc.)

---

lengths, and numbers of channels. Pre-training on such heterogeneous datasets using extremely small models requires specialized architectural advancements. Hence, TTM proposes the following enhancements to the TSMixer architecture for resource-constrainet pre-training/finetnnii adaptive patching (AP) across layers. considering the varied suitability of patch lengths for different datasets, (ii) diverse resolution sampling (DRS) to augment the data for increasing coverage across different resolutions, (iii) resolution prefix tuning (RPT) to explicitly embed resolution information modeling while training on diverse datasets.
(i)
in the first patch, facilitating cnolution-conditionn Moreover, our approach leverages multi-level modeling, where TTMs are first pre-trained a Srannel-independeer way, and then fine- with channel mixing to incorporate correlations across targets and exogenous channels in the target domain.
Outline of TTM's key capabilities: (1) Amidst the prevalence of "large" pre-trained models demanding significant compute and training time, our work the first to demonstrate the power of transfer learning using "tiny" TS pre-trained models for zero/few-sho forecasting. (2) Pre-training tiny models on heterogeneous multi-resolution datasets with extremely limited model capacity is challenging. Towards this, propose various architectural and training enhancements, such as AP, DRS, and RPT for robust and resource-constrained defined above). (3) TTM employs a multi-level modeling strategy to explicitly model channel correlations, and incorporate exogenous signals - a crucial capability lacking in most of the existing pre-trainee mre-trained models. (4) Through extensive evaluation of zero/few-shot forecasting on 11 datasets, we establish that TTM models, with sizes as small as IM parameters, consistently outperform the forecasts of "large" TS pretrained models while offering significant computational benefits. Figure [ highlights that TTM outperforms popular benchmarks in all three primary dimensions: size, runtime, and accuracy. (5) Given their compact size, zero-shot inference and fine-tuning of TTM models can be easily executed with just one GPU or in CPU-only environments. This greatly enhances the practical adoption and extended reach of our pre-trained models with ease of use.
cee-training/ine-tuniin
workflows (as
[d's]] Reverse Instance Normaizztion
Patsh Mergo
Fleverse Inshence Normsdzaller
[<xexb]]
TSMdaa Bloce (n.
TSMiler Block mane
Fired farecaads for begats, P. and
Optiond Exogencus Miserr
Forillon
T Tixi
[xxl|
Forecast Urear
TIM Decotder
111x nXMi
[
(chanee inkppeniettriiiig
Skmn TTM Desodhrr
[X
[X
Lovel=2
[ex2xx
......r [[Exn'
Paich Parl
15Mar flackk
Patch Me go
(n?2 HV2)
darrd spteraa
mane
ToMoow with Strdect
Unear Hoad
TTM Backsons
TTM Rackkoon (Frwers)
(channel- independent)
Merge
Panthing a
[uuzerg]
[ox
M.
TEMber Blook 1o74, HiM)
Tirgel
Instunes
Norma laation
[ox]]]
[X11XM
Level
n-1
Purch Partison
Contral
Exog
[11 x|]
Rencarion
Univaratic ioott
xe
Prdlo
series
Espard
Houry
50
18:m0
Mulharito ino scries
Foodt Boc Pir
Tuntig 12
M
foanshg
(c) Exogenous mixer for 4 3. The green contrallecogenous charnels denote that forecasts coming out of the TTM head are replaced by their vallees forecast horizon.
resolution Pinraiingg
Embeosing Layer
Enboodding
Data, camples
Targl
Layer
Workfow
F Finelune Workllow
15. 13min 10min
Bon Joken
jex of
1M learnable
leamable paremeters
DCV,
[TTMp4s)
(TTM
(b) backbone having levels (i.o.. and TTM blocks in each level (l.e., M 2)
(a) Components and Workflows
Figure2 2: TTM overview (a) Refer to Sections and 3
(b) Refer to Section 3.1
(c) Refer to Section 3.2
2 TTM Components
Let X E Rex be multivariate time series of length si and number of channels c. The forecasting
task can be formally defined as predicting the future values Y E & xfi given the history/context X. Here, I'l denotes the forecast length/horizon, and e' denotes number of forecast channels, where e' S c. The predictions from the model are denoted by Y E Re' In a general multivariate forecasting task, each channel falls into one of the following categories: (a) Target variables (mandatory): channels for which forecasts are required, (b) Exogenous variables (optional): channels influencing the targets, with known estimated values throughout the forecast horizon.
2.1 Multi-level Modeling: TTM follows a multi-level architecture consisting of four key compo- nents (see FigureDaa) (1) The TTM backbone is assembled using building blocks derived from the efficient TSMixer architecture (6). TSMixer is based on MLP blocks interleaved with gated
3

---

attention, that enable the mixing of features within patches, across patches and channels, surpassing existing Transformer-bassee TS approaches with minimal computational requirements. Since TSMixer was not designed to handle multi-resolution data with limited capacity, we introduce various novel enhancements to as explained later. (2) TTM decoder follows the same backbone architecture but is considerably smaller in size, approximately 10-20% of the size of the backbone, (3) Forecast head consists of a linear head designed to produce the forecast output, and (4) Optional Exogenous mixer serves to fuse exogenous data into the forecasting process. The TTM decoder and forecast head together constitute the TTM head, whose weights get updated during the fine-tuning process. This multi-leved model refactoring is required to dynamically change the working behavior of various components based on the workflow type, as explained in Section 3 In addition to the above primary components, there is also preprocessing component as explained next.
2.2 Preprocessing: As shown in Figure 2 a) with colorless blocks, the historical time series X first normalized per instance to have zero mean and unit standard deviation for each channel, to tackle any possible distribution shifts [2 6). This process is reversed at the end before computing the loss. The normalized data X is subsequently patched, X, E Rexxxpi, into n oveeraaaaa windows, each of length pl, and then passed the TTM backbone. Patching, as introduced in [22), has proven to be highly valuable for forecasting. Its effectiveness lies in preserving local semantic information, accommodating longer history, and reducing computation.
TTM Methodology
3.1
Pre-training Workflow: TTM operates in two stages: pre-training and fine-tuning (Fig- ure (a). the pre-training stage, we train the model on large collection of diverse public datasets. Since the primary focus of TTM is forecasting, pre pre-training is modeled with direct forecasting objective. TTM is first pre-trained in univariate fashion with independent channels on all the existing datasets. Due to varied channel counts in pre-training datasets, modeling multivariate corre- lations is challenging here; it is addressed later during the fine-tuning stage. Multivariate pre-training datasets are initially transformed into independent univariate TS (X1, XN) E Rc(=1)xxl, These are rre-proceseed (Section 2.2 and subsequently fed into the TTM backbone for multi-resolution pre-training. The output of the backbone X E [e(==))xxXff is passed through the decoder and forecast head to produce the forecast Y E [(==))xf! fl which is then reversc-normalized to return to the original scale. We pre-train the TTM with mean squared error (MSE) loss calculated over the forecast horizon: L = - Y 12.. Thus for given input context length si and forecast length fl, we get a pre- trained model capturing the common temporal forecasting dynamics and seasonal pattemss as observed in the overall pre-training data.
3.1.1 Multi-Resolution Pre- Pre-training via TTM Backbone: In TTM, our goal is to create models that are extremely small yet capable of generalizing well across wide range of diverse datasets with varying resolutions. This is a significant challenge because the models can easily under-fit due to their small size. To tackle these challenges of resource-constrained pre-training, we introduce the following enhancements to the TSMixer backbone.
Adaptive patching (AP): The TTM backbone is crafted with an adaptive patching architecture where different layers of the backbone operate at varying patch lengths and numbers of patches. Since each dataset in the pre-training corpora may perform optimally at a specific patch length, this approach greatly aids in generalization when the pretraining datasets with different resolutions are introduced. Moreover, it helps in scenarios when the availability of the pre-training data is limited as adaptive patching quickly generalizes the model across different granularities. As shown in Figure 2(b). the patched data X, E Rexxxpl is passed through a embedding layer to project it to the patch hidden dimension, X E [Rexxxhf_ If the resolution prefix tuning module is activated (as explained later), the resolution prefix is concatenated with X,, For notational simplicity, we denote the concatenated tensor with X,, as well. The TTM backbone consists of L levels, each comprising M TTM blocks with identical patch configurations. The first block in the first level receives X The first TTM
block in the i-th level, i = 2...., L., receives the processed data x(i E Rexxxxf from the previous block. Each TTM block is further comprised of a patch partition block, a vanilla TSMixer block, and patch merging block. The patch partition block at level i increases the number of

---

patches by factor of K; and reduces the patch dimension size by the same factor by reshaping X x E R x K))(/())))((/ where K; 2((--)) Figure 2)) shows the TTM backbone for L 3 and M 2. Note that, we set hf = m 24 for some integer m. Then, TSMixer is applied to the adapted data X Finally, the output from TSMixer is again reshaped to its original shape (i.e., [=xxx*/) in the patch merging block. In subsequent layers, for each increment in level i, the number of patches is halved and the patch dimension doubled. This enables better generalization for small models as we pre-train across multiple datasets. The idea of adaptive patching popular and very successful in the vision domain (e.g., Swin Transformers (0] and we successfully apply it to the TS domain to resolve multi-resolution issues in modelling diverse TS datasets. Note that adaptive patching is enabled only in the backbone and not in the decoder, which is designed to be very lightweight.
Augmentation via diverse resolution sampling (DRS): A significant challenge in TS pre-training datasets is the scarcity of public datasets with diverse resolutions. Generally, high-resolution datasets will account for larger fraction of the samples given their finer sampling resolution. Without adjustment to the training strategy, this can lead to model that is biased toward the finer resolution data. To overcome this, different strategies are applied to high-resolution datasets to balance the volume of samples at lower resolutions and lead to more uniform coverage. Strategies used include: 1) averaging & samples in sequential, non -on-ovrlaapinng windows to produce lower resolution dataset; and 2) conventional decimation where only every kth sample is retained. In both cases, the integer k is chosen to achieve the desired resolution from the resolution of the base dataset. For example, from 4-second resolution dataset, we derive multiple datasets at minutely (k 15) and hourly resolutions (k 900). Note that the original high-resolution dataset remains within the pool of pre-training datasets. This methodology increases the number of datasets for each resolution which greatly improves the model performance.
Resolution prefix tuning (RPT): This technique explicitly learns and incorporates a new patch embedding learnable prefix into the input data based on the input resolution (see TigureZtt and Table 8). Similar to the concept of prefix tuning [16), this approach provides an explicit signal to the model about the resolution for coolutioonconditionn modeling. First, we map every resolution to a unique integer, which is then passed through an embedding layer to project it to the hidden dimension, h.f. Subsequently, we expand the embedding across all channels to have a representation of shape x 1 x hf. This resolution -ased learnable embedding is particularly beneficial in quickly modeling huge volumes of diverse resolution datasets with limited modelling capacity, as the model can easily decouple the data from different resolutions for resolution-conditione modeling. In addition, RPT also helps in scenarios when the context length (sl) is short. In these scenarios, automatically detecting the resolution becomes challenge for the model. Hence, by explicitly fusing the resolution information as a prefix, we can enhance the model's s ability to learn effectively across resolutions without increasing its size.
3.2 Fine-tuning Workflow: In the fine-tuning workflow, we work with data from the targer domain that has no overlap with the pre-training datasets. We have three options here: (a) In zero-shot forecasting, we directly use the pre-trained model to evaluate on the test part of the target data; (b) In few-shot forecasting, we utilize only a tiny portion (5-10%) of the train part of the target data to quickly update the pre-trained weights of the 7TM head, and subsequently evaluate it on the test part; (c) In full-shot forecasting, we fine-tune the pre-trained weights of the TTM head on the entire train part of the target data, and then evaluate on the test part.
The backbone is frozen during fine-tuning, and still operates in channel-indepentee univariate fashion. However, the slim decoder in the TTM Head can be fine-tuned utilizing channel mixing or channel independence for multivariate or univariate target data, respectively. pure multivariate modeling is needed, then the channel-mixer block in all the TSMixer components (see Figure 2bb) in the decoder is enabled to explicitly capture the cross-channel correlations. The forecast head and reverse normalization perform similar operations as in the pre-training stage. The fine-tuning also optimizes the forecasting objective with MSE loss. This thoughtful multi-level design choice ensures that our backbone excels in channel-independent pre-training, enabling effective temporal correlation modeling across diverse datasets. Simultaneously, the decoder handles gagee--dta-speecine tasks like coorelaioo modeling and fine-tuning, In addition. if the target data has exogenous variables, then an exogenous mixer block is applied to the actual forecasts as explained next.
5

---

Exogenous Mixer Block: As described in Section 2 the future values of the exogenous channels are known in advance. Let the forecast from the forecast head be Y E Rex Let the channels "0.".., Fe , denote the target variables and 2c'+1,''' a denote all exogenous variables with their future values known. First, we replace the forecast values for the exogenous channels with the true future values (Y) and transpose it: Y [o.... . "e'+1, , yel E Riixe Next, to learn inter-channel lagged correlations, we patch Y. into a series of overlapped windows (i.c., patching with stride= 1) to create a new tensor: Ye.p E R where A 2 + with being the context length to incorporate on either side of time poin Subsequently, we pass Y.Pp through a vanilla TSMixer block with channel mixing enabled. Thus, the lagged dependency of the forecasts for the target channels on the exogenous channels is seamlessly learned. Finally, we attach a linear head to produce the forecasts for the target channels which is then reshaped as Y E 12 gge'xi, Thus, TTM easily handles exogenous infusion which is a practical requirement in any industrial forecasting problem. Figure ((c) depicts this procedure.
4 Experiments and Results
Pre-training employs a subset of ~ samples from Monash (9] and Libeity [2] data collection. We specifically exclude a few datasets (like yearly, monthly) as they do not possess sufficient length for the long-term forecasting task. Moreover, we remove all the datasets that we utilize for evaluation (i.e., Weather, Electricity, and Traffic). For zero/few-shot evaluation we consider seven public datasets (D1): ETTHI, ETTH2, ETTMI, ETTM2, Weather, Electricity, and Traffic as popularly used in most prior state-of-the-art (SOTA) works 44 22]. Since these datasets do not contain any exogenous variables nor exhibit cross-channel correlation benefits, we incorporate four other datasets (D2) for separately validating the efficacy of the decoder channel mixing and exogenous mixer module: bike sharing (BS) 77]. carbon capture plant (CC) [13] and 2 more datasets, Application (APP) and Service (SER), from Business and IT observability domain 27 24]. For full details, refer to the Appendix [ We use mean squared error (MSE) as the standard error metric. In addition, we use the following relative improvement metrics: (i) forecast improvement percentage (f-imp( 96) which refers to the MSE (%) improvement of TTM over the considered baseline, averaged across all datasets, and (ii) size improvement metric (s-imp(X)) is calculated as the ratio of the baseline model size to the TTM model size (i.e., total parameters).
4.1 Datasets & Metrics: :
4.2 SOTA Benchmarks: We benchmark 6 TTM with 24 of the latest open-sourced SOTA
forecasting models categorized as follows: (a) TS pre-trained models: Lag-Llama (26).
TimesFM 31. Moirai (35), Chronos (2) and Moment [10). (b) LLM-based TS pre-trained models:
GPT4TS 46), LLMTime M.. Time-LLM (15), Tni [8]
(c) Self-supervised pre-trained
models: SimMTM 5 M,-i (7), TST 42), LaST 34 TF-C 43 CoST (36] and Ts2 (4)
(d) Other architectures: PatchTST 222 TSMixer 6. TimeMixer
33 i'Transfommer [19 DLin-
ear [1 and TimesNet
37 FEDFormer
45 and Autoformer 38
4.3 TTM Model Details: We pre-train three primary variants of TTM as follows: (i) TTM-Base
(TTMg): 1M parameter model trained with context length, sl 512 and patch length, pl 64, (ii) TTM-Enhanced (TTM)): 4M parameter model trained with sl = 1024 and pl 128. (iii) TTM-Advanced (TTM)): 5M parameter model trained with s! 1536 and pl 128. These TTMs are pre-trained using the IB pre-training dataset, which takes only 24-30 hours with 6 A100 GPUs, a notably faster time compared to existing counterparts which often take days to weeks. Additionally, for secondary studies, we utilize Quick TTM (TTMg). variant trained on a smaller subset of the Monash dataset (~250 million samples), requiring only 4-6 hours for pre-training.
Although, a TTM model needs to be pre-trained for specific forecast length (FL), we provide two forecast length adaption (FLA) techniques (explained in Section (47) that enable a pre-trained TTM to work across different FLs. Users can either build a direct pre-trained model (from one of the above variants) targeting& a specific FL. or use the FLA techniques adapt an existing TTM model their
'This needs padding Y. with zeros length on both sides. 'For all tables. we highlight the best and second best models with bold and underline, respectively. We denote TTM's improvement and degradation w.r.f. a baseline with 1 and I respectively.

---

application setting. Primary results are reported using the direct approach, and a detailed ablation study provided to compare the effectiveness of various FLA techniques. In the direct approach. model parameter size varies across FL.s and we report the average parameter size in the result tables. TTM fine-tuning and inferencing are highly efficient and fast, requiring only GPU or even CPU execution. All model hyperparameters are chosen based on validation performance, and final test results are reported. Refer to Appendix D for detailed model specifications and hyper-parametest
Dara
TTMg
gTMg
LTM,
Moirais
Moirala
Moiraiz
TimesFM
ETTMI
ETTH
ETTH2
0.345
0.394
0.335
0.404
0.38
0.362
0,333
0.4
0.448
0.341
0.4
0.434
0.346
0.354
0.51
0.403
0.479
Model
GPL TIM
(ms)
Param: (M)
MEM
(GB)
TIME
(5)
ETTM2
0.281
0.271
0.252
0.3
0.272
0.276
0,334
TTSG
4,7
0.8
0.06
0.81
Weather
0.237
0.238
0.231
0.242
0.238
0.26
Chroneso
1385
2340
Electricity
0.205
0.194
0.192
0,233
0.188
0,188
(2024)
(2510)
(267X)
(259KX)
Size
1M
4M
5M
14M
9IM
311M
200M
Chroaos,
TIMa
TTMe
f-ämp (8) (nmpp())
Fimpp 33 X-impiX/
6%
7%
14X
4X
1%
1%
191X
23X
44%
6%
T
3IIXX
78X
T
15%
16%
200X
50X
Chronesy
(2024)
(2024)
(8820)
(96X) (58X)
1.86
46
(683X)
(260KX
6
(20030
2349
TTM,
f-imp s-imptX)
10%
3X
1
44
18X
9%
62X
19%
7 40X
Chromesy
1389
2504
(2024)
(10X)
(23X)
(256K2)
Table 1: Zero-shol Forecast improvement (f- imp) and model size improvement (s (x-imp) TTM over Mairai (ICMI'24) and TimesFM (ICML'24). MSE averaged acmss F7. E [96, 192, 336,723) 720}.
GPT4TS
NNoutP 23:
(3]
(185X)
(36X)
(26X)
Electricity and Weather results for TimesFM are not reported as its used by TimesFM for pretraining, Similarly, Traffic was used in pro-training for both Moirai and
Lag-Llaaa
(2024)
(366X)
1619
(3X)
2.4
(3X)
0.2
(36300)
37,5
Morris
2015
14
0.1
1.4
TimesM. Full table Appendix [22
OUML '20
(4XX)
(18X)
(2X)
(14X)
Morrus
T
M1M,
TTM,
TTM,
Chronosy
Chronosy
Chroonss
Chrones;
Lag-Ilama
OCML (34)
(33X)
(1070%)
ETTHI
0.204
0.227
0.214
0.311
0.302
012552
0.266
0.334
GCML '24)
Motraia
(72X)
335
(114X)
(17%)
(421X)
ETTH2
0.131
0.151 0.162
0.177
0,16
0.164
0.155
0.168
Moment L
88
348
1.4
ETTMI
0.206
0.486
OCML '24)
(19X)
(4350)
(133X)
(144X)
ETTM2
0.124
0.128
0.117
0,206
0.174
0.19
0.187
0.308
TimesFM
Weather
0.039
0.032
0.043
0.043
0.046
0.03
0.033
0.126
'30
(33%)
Elearricity
0.335
0.351
0.349
0.423
0.377
0.344
0.339
0.393
Traffic
0.246
0.24
0.244
0,291
0.3
Tahle 3: Computational improvement
Size
IM
4M
SM
RMI
46MT
201M
709M
3M
of TTM wist. ctisting TS pre-trained mod- els. Inference time per-barch in GPU and CPU. lotal parameters (Params), and imum GPU memory usage (MEM) are reported, aX indicates the scaling factor
TTMa
Fimpp
x-imptX)
17%
201X
118%
709X
400
T3x
TTMr
Fimmp())
immp(XX)
30%
2X
24%
1
12X
15%
1 500
16%
1 177X
37%
1X
TTM,
(k x-Imp(X)
28%
2X
22%
9X
12%
40X
13%
142X
37%
2X
Table 2: Zero-shot improstttmprooeemett (f-imp) model size-improvement (5-imp) of TIM over Chronos and Lag -lama over the last test-window. Since Chronos and Lag-Ilama recummend/repont results with forecast lengths, use (24, 48, 60, 96, 1923. Mean MSE across FLs reported. Full table the Appendix F2
for TTM's improvement. Set-up details are the Appendix D.3
4.4 TTM's Zero-shot Performance and Inference Cost: Recently, popular pre-trained models like TimesFM, Moirai, Chornos, Lag-Ilama, and LLMTime have gained traction for their zero-shot (ZS) forecasting capabilities. Among these, Chornos, Lag-Ilama, and LLMTime suffer from lengthy ZS inference time, posing practical challenges for testing across all sliding windows of the test set. To address this, LLMTime suggests using the last test window for benchmarking, a practice we also adopt for comparing with this set of SOTA models. On the other hand, TimesFM and Moirai exhibit comparatively faster ZS inference speeds, enabling testing across all sliding windows of the test set. Table presents a comparison of TTM performance with Moirai and TimesFM. Despite having significantly fewer parameters, the variants of TTM consistently outperform most benchmark variants. Notably, TTM,, which is 3-62X smaller than all Moirai variants and 40X smaller than TimesFM, outperforms the Moirai variants by 4-10% and TimesFM by 19%. Even TTMg, with just 1M parameters, outperforms most benchmarks by a considerable margin, highlighting the effectiveness of TTM. Moreover, as depicted in Appendix F.4. TTM rero-shot results consistently outperform the full-shot results of popular architectures in short context length settings. Likewise, Table 2 presents a comparison of TTM performance with Chronos and Lag-Ilama on the last test-window set. As indicated, TTMg which is 8-709X smaller than Chronos, outperforms it by 17-32%. Likewise TTMp, which is 2-3X smaller than Lag-Ilama, outperforms it by 40%. In addition, TTM also outperforms the massive LLMTime and UniTime by over 25% reported in Appendix [F3 Table 3 presents the inference time per batch and maximum GPU memory requirement of different TS pre-trained models. Notably, TTM exhibits the lowest inference time and memory usage among them.
4.5 TTM's Few-shot and Full-shot Head Probing Performance: In operational deployments, users typically leverage a small set of target data for fine-tuning to enhance the model performance. In this regard, TTM provides highly efficient quick fine-tuning process, enabling users to enhance forecasting accuracy swiftly by training only the model head. GPT4TS and Time-ILMM are two -fate-of-the-ar pre-trained models that present results for shot training. As demonstrated in
7

---

Table 4 TTMg surpasses GPT4TS by 15% and Time-LLM by 10% in the few-shot 5% setting, where only 5% of the train data is used for fine-tuning, In addition. we also report the Few-shot 5% results of several popular SOTA architectures in T where TTM demonstrates superior performance. This underscores the significance of TTM's pre-trained weights, which substantially contribute to its effectiveness in data-constrained scenarios. Likewise, TTM also excels in few-shot cross-transfer learning tasks outperforming popular SOTAs (including SimMTM (5) as shown in the Appendix F.6
Alternatively, if the train split of the complete target dataset is available, head probing using the entire dataset becomes feasible. This involves fine-tuning the model head using all available data while keeping the backbone weights unchanged. Recently, the Moment [10] model has achieved the SOTA results in head probing as compared GPT4TS and Time-LLM, However, as indicated in Table TTM further outperforms the results reported by Moment by 3-4%. In addition. TTM head probing results are very competitive as compared to the full end-to-end training of popular architectures as depicted in Appendix F.7 Hence, TTM, with its significantly reduced model size and the absence of compute-intensiv components like self-attention, enables quick fine-tuning of models compared to the cumbersome process required by the massive Transformer models. Note that Moment is excluded from the comparison of zero/few-shot forecasting results because it does not report them.
Pre-truined models fine-tuned 5% data
Other popular architectures trained on 3% data
TTM
TTM,
TTML
GFT4TS
Time-LLM
PaccTTT
TSMaxr
TimeMMier Transtommrr
TimesNet
Dlinear
Data
(Ours)
(Ours)
(Ours)
(NeurIPS 23)
(ICLR 24)
(ICLR '23)
(KDD '23
(ICL '24)
(ICLR '24)
[ICLR '23)
(AAAI '23)
ETTHI
0.383
0.385
0.386
0.682
0.627
0.695
0.635
1,088
0.756
0.926
0.75
ETTH2
0.324
0.318
0.114
0.401
0.382
0.439
0.385
0.508
0.437
0.464
0.828
ETTMI
0.378
0.425
0.578
0.568
FTTM2
0.272
0.268
0.253
0.308
0:274
0.314
0.297
0.34
0.309
0.344
0.399
Weather
0.234
0.24
0,229
0,263
0.261
0.27
0,268
0.317
0.297
0.298
0.264
Electricity
0.183
0.207
0.178
0.177
0.176
0.197
0.239
0.202
0.402
0.177
Traffic
D.433
0.437
0.49
0,434
0.423
0,418
0.435
0.503
0.452
0.867
0.451
Size
4M
84M
TTM 5-mmpl (6) s-imp(X)
15%
84X
10%
T 7KX1 T
17% 1
15%
31% 1
22%
40%
23%
TTMr
f-imp! s-ImptX)
13%
21X
8%
1.7KX
13%
20%
40%
21%
TTM,
f-imp(() (6) s. (X)
15%
17X
11% L4KX
17%
15%
32%
22%
41%
23%
Table 4; Few-shot 5%, MSE averaged across FL E (90, 192, 336, 720), models are trained with 5% train data (Appendix R.S.
TTMg
TTMg
TTM, (Ours)
Moment (ICML '24)
GPTATS (NeurIPS '23)
Time-IIIM (ICLR '24)
Data
(Ours)
(Ours)
0.348
0,406
0.402
0,486
Models
BS
APP
SFR
ETTH2
0.33
0,338
0.327
0,346
0.346
0,342
TTMg
0.635
0.261
0.073
0..14
18%
ETTMI
0.355
0.35
0.349
0.354
0.41
PatcTTST
0.735
0.267
0.060
0.119
15%
ETIM2
0,257
0.252
0.264
0,266
0.275
0.273
TSMixer-CC
0.651
0.284
0.053
0.6
15%
Weather
0.234
0.239
0,233
0.234
TSMixer-CM CM
0.303 0.069
0.118
20%
Electricity
0.164
0.161
0.16
0.174
0.172
TSMiser
0.664
0.267
0.066
0.1
17%
Traftic
0.399
0.398
0,385
0.419
GPT4TS
0.254 0.075
Size
IM
4M
5M
348M
84M
7B
TTMp-CM
0.582
0.250 0.042
0.114
TTME
Fiimpt (6)
s-impe))
3%
348X 1 4%
84X
9%
7000X
TTMg
f-lmp: %6) s-imp(X)
3%
87X
4%
21X
9%
1750X
Table 6: Efffect of decoder mixing and exog. fu-
TTM,
f-imp! 98) s-imp(X)
4%
70X
6%
17X
10%
1400X
sion. MSE results are reported using (sl, fI) with values of (512, 96) for BS dataset and (96, 24) for other D2 datasets. fimp% of TTMg CM werr others are provided.
Table 5: Full-shot head probing: Finetuning the pre trained model heads on with backbone weights frozen.
averaged across EL 96, 720 reported in [0]. Time LLM resulis for large datavets are not reported in [I0 due to computational challenges (Appendivfe7)
4.6 TTM's Effectiveness in Cross-channel and Exogenous Modeling: Since the datasets (D1) used in previous experiments do not have exogenous variables, we evaluate the effectiveness of TTM on other datasets (D2, as explained Section 4.T) to quantify its benefits. Since these datasets are already very small, we used their full data for fine-tuning. Table|Bsshows the performance of the pre-trained TTMg model fine-tuned on the target data with exogenous mixer module and decoder channel-mixing enabled (TTM-CM). We compare TTM-CM with plain TTM finetuning and other primary SOTAS (PatchTST, TSMixer variants, and GPT4TS) trained from scratch. Specifically, we compare with TSMixer with channel-mixing enabled (TSMixer-CM and TSMixer with cross-channel reconciliation head (TSMixer-CC) 6 as they are the latest SOTAS in channel- correlation modelling. From Table 6 margin (15-44%), thus, demonstrating the power of TTM in capturing cnter-channel correlations.
that TTM-CM outperforms all the competitive models with significant
4.7 Ablation Studies: The impacts of various techniques used in TTM are analyzed here. Pre-training data (Quality Vs Quantity): Figure 3 demonstrates the vital role of both pretraining
8

---

data and diverse resolution sampling (DRS), Initially, the zero-shot results were unsatisfactory when pre-training TTM with the smaller Monash dataset (i.e., PT(M)). To improve performance, we introduced the DRS technique on the Monash data to increase diversity and coverage (250M PT samples). This significantly improved the results by 37%. In addition, extending the dataset size from 250M to IB further improved the results by 6%. These experiments highlight that while the quantity of pre-training data is significant, the quality of the data, especially in terms of resolution diversity and coverage, is even more crucial for improving the model performance.
Effect of Resolution Prefix Tuning (RPT) and Adaptive Patching(AP): RPT enhances forecast performance, especially with large and diverse pretraining (PT) data. Adding learnable resolution prefix token allows models easily decouple weights across different resolutions, leading to a 3% improvement in IB PT data setup (Table 7. RPT is also beneficial for very short context length scenarios, improving performance by 8% (Appendix F.9.. On the other hand, AP generally improves the forecasting performance across all set-ups, but the impact is consistently high in less PT data settings (3% boost). Further details are Appendix F8
0.511 37% IMP
Less PT Data (250M)
More
(IB)
0.322
6% IMP
TTM, FROM FL.6
TIV, DIRECT
w/o AP
W//
Wio RPT
MSE
00.303
TTM, PRUMING FRO FL 723
ETTHT
0.369
0.365
D.366
0.364
0.283
0.285
0.277
FTTMI
0.446
0.413
0.341
0.322
ET
0.191
0.187
0.18
0.171
Weather
0.159
0.154
0.153
0.158
PT(M)
PT(M) +
PT(F)HDRS 1B
Electricity
0.179
0.169
0.178
0.166
DRS
FL 720
Traffic
0.521
0.518
0.528
0.514
250M samples samples
IMP (%)
3%
3%
Figure 4: FL adaptation: impact of adapting TIM, (FL 720) and TIM, (FZ 96) to all other FLs. MSE averaged acmss
Table Impact AP and RIT: Impucts of adaptive patching (AP) less pre-training (PT) data setting (i... TTMg), and resolution prefix tuning (RPT) more pre training (PT) data setting (i..., TTMg). Zero-shot results on FL 96 reported. [ww": with, 'wlo": 'without'. I
Figure 3: Impact of pre-training data (PT) and diverse resolution sampling (DRS) technique. PTTM): pretraining with only Monash data. PT(F): full pretraining data used. Average MSE zero-shot re- sults across FL 96, 192 reported.
darasets reported for FÝ. E (96, 192, 336, 720}. Best viewed in color.
Forecast Length Adaptations (FLA): Given FL. we can either pre-train a Direct TTM tailored for the specific FL or adapt existing TTMs trained differ- ent FLs. Two possible adaptations are: (i) Pruning:
Weather (5-11
Weather 15-2
Weather 15-3
Traffic 1s 10
Take TTM trained FL' where FL' >FL, and prune
Traffic 15 20
Truffic 55-30
it to the required FL (e.g.. TTM (FL = 720) pruned
Eacenoty 19-1)
Eecenony (922
other reduced FL E {96, 192, 336]). (ii) Recur- sive: Take TTM trained on FL', where FL' <FL and do recursive prediction (of length FL') till we reach the required FL (c.g., Extend TTM (FL- = -96) re- cursively to greater FL E {192, 336, 720]). Figure compares these techniques. For shorter adaptation (96 to 192), recursive predictions yield the best perfor- mance and match the direct forecast results. However, for wider adaptations (336-96 or 720-96), the prun- ing approach gives more stable and closer results to the direct forecasts. Hence, using these approaches, TTM models can be easily adapted to various Fl.s based on user requirements.
Excoidty (5:3)
meaco
weath
Instaty
eneeetay
wortingiday
wentfferrt
stemm
Figure 5: (a) TTM embedding projections across
3 datasets and 3 segments within datasets. (b)
Cross-channel attention based explanation.
4.8 TTM Model Insights & Explainability: Fig- ure
5 illlustrates the TTM embeddings from various datasets (weather, traffic, and electricity) using PCA projection, each represented by different color. From each dataset, three distant, non-overlapping. fixed-length time segments (S-1, S-2, S-3) are selected, each depicted with unique marker shape. The visualization uses the first and second principal components of the TTM embeddings. The inset image focuses on the weather dataset alone, revealing a deeper structure learned by the TTM architecture. The cyclic orbits in the embeddings reflect the seasonal patterns in the data. Both

---

hourly datasets (traffic and electricity) form concentric orbits due to similar seasonal patterns, while the weather data, with its distinct seasonal pattern, shows cyclic orbits in a different aub-dimension. In addition, the cross-channel attention from the fine-tuned model's channel mixing layers reveals feature importance across channels. As shown in Figure the model focuses on channels like weath- ersit, season, holiday, and temperature to predict bike-rental counts. These attention model weights correlate with the general data characteristics where bike rental demands are heavily influenced by weather and holidays, providing explanation for the fine-tuned model predictions. More details are in Appendix G
4.9 Discussion on TTM Design choices: In this section, we intuitively explain the important design choices of TTM that greatly enhance its forecasting accuracy and transfer learning capabilities despite its extremely small model capacity:
All existing pre-trained models use a very high volume of pretraining data (for example, TimesFM used 300B and Moirai used 27B time-points), hence they naturally require massive model sizes. However, as shown in Figure we observe that "limited" pretraining data with "high resolution diversity" greatly helps time-series model generalization, as opposed to simply increasing the pretraining data size. This is an important observation and finding that resolution diversity in pretraining data is very crucial for time-series FMs. Based on these findings, we proceed with a well-reduced dataset (1B samples) with high resolution diversity which naturally reduces our model size compared counterparts needing pre- train with several hundred billion time-series. We introduce a high diversity in our data via Diverse Resolution Sampling technique (DRS) which our counterparts fail to do.
Secondly, we opted for TSMixer-based models instead of transformer- models, which further reduced the model size drastically. The TSMixer architecture has successfully estab- lished in the past that interleaving simple gated attentions with mixing components across patches, channels. and features greatly enhances forecasting accuracies with very limited model capacity, as the quadratic time- complexxy of self-attentions can be entirely avoided. Following TSMixer, several other mixer architectures [33] (24 have been published. re- iterating the power of these simple architectures. Thus, avoiding complex transformer architectures further reduced our model size significantly.
In addition, we further increased the modeling power of TSMixer without drastically increasing its size by introducing several innovative components, such as adaptive patching, diverse resolution sampling, and resolution prefix tuming. These enhancements are crucial for effectively handling large pre-training across datasets with varying resolutions, all while keeping the model capacity very minimal.
Finally, framing the pre-training objective as a direct forecasting task demonstrates improved zero-shot performance compared to the traditional masking-based pre-training approaches. We hypothesize that this method enables the model to effectively learn complex non- linear mappings between the fixed context and forecast windows during pre-training that generalizes well to unseen datasets.
mappings
Conclusions and Future Work
We propose TTM, an extremely lightweight pre-trained model for multivariate time-series forecasting. Unlike existing large models, TTM is significantly smaller and faster, with efficient pre-training and fine-tuning workflows. Results show that TTM is highly effective in pre- pre-training on heterogeneous datasets despite its limited model capacity. It achieves state-of-the-an results in zero/few-shor forecasting, offering significant computational efficiency while capturing cross-channe relationships and exogenous variables critical features lacking in popular methods. Additionally, TTM supports both CPU and GPU deployments, greatly enhancing its adoption and ease of use. Moving forward, we plan to generalize our approach to support other downstream tasks beyond forecasting.
References
[1] Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re-
10

---

port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.142119 2024.
[2] Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, al. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.0744 2024.
[3] Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. International Conference on Machine Learning (ICML), 2023.
[4] Jacob Devlin, Ming-Wor Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1 810.04805, 2018.
[5] Jiaxiang Dong, Haixu Wu, Haoran Zhang, Li Zhang, Jianmin Wang, and Mingsheng Long. Simmtm: A simple pre-training framework for masked time-series modeling. Advances in Neural Information Processing Systems, 2023.
[6] Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. Tsmixer: Lightweight mip-mixer model for multivariate time series forecasting. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, page 459-469 New York, NY, USA, 2023.
[7] Hadi Fanace- Bike Sharing Dataset. UCI Machine Learning Repository, 2013. DOI:
https://oo.org//02244////CC
[8] Azul Garza and Max Mergenthaler-Cansee Timegpt-1, 1, 2023.
[9] Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, and Pablo Montero-Manso. Monash time series forecasting archive. In Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
[10] Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. International Conference on Machine Learning (ICML), 2024.
[11] Nate Gruver, Mare Anton Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[12] R.J. Hyndman and G. Athanasopoulos, editors. Forecasting: principles and practice. OTexts: Melbourne, Australia, 2021. [OText comp://.ccoo//ooooo Fpp3
[13] Kevin Maik Jablonka, Charithea Charalambous, Eva Sanchez Fernandez, Georg Wiechers, Juliana Monteiro, Peter Moser, Berend Smit, and Susana Garcia. Machine learning for industrial processes: Forecasting amine emissions from a carbon capture plant. Science Advances, ([)xeadc)956, 2023.
[14] Arindam Jati, Vijay Ekambaram, Shaonli Pal, Brian Quanz, Wesley M. Gifford, Pavithra Harsha, Stuart Siegel, Sumanta Mukherjee, and Chandra Narayanaswami. Hierarchical proxy modeling for improved hpo in time series forecasting. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, page 891-900, New York, NY, USA, 2023.
[15] Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin- Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-LLM: Time series forecasting by reprogramming large language models. The Twelfth International Conference on Learning Representations, 2024.
[16] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 4592-4597, Online, August 2021.
11

---

[17] Zhe Li, Zhongwen Rao, Lujia Pan, Pengyun Wang, and Zenglin Xu. Ti-mae: Sulf-super masked time series autoencoders. arXiv preprint arXiv:2301.08871, 2023.
[18] Xu Liu, Junfeng Hu, Yuan Li, Shizhe Diao, Yuxuan Liang, Bryan Hooi, and Roger Zimmermann. Unitime: A language-empowerrs unified model for cross- dros--domain time series forecasting. In Proceedings of the ACM Web Conference 2024, 2024.
[19] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting, 2024.
[20] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[21] Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. M5 accuracy com- petition: Results, findings, and conclusions. International Journal of Forecasting, 2022.
[22] Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In ICLR, 2023.
[23] Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. International Conference on Learning Representations, 2020.
[24] Santosh Palaskar, Vijay Ekambaram, Arindam Jati, Neelamadhav Gantayat, Avirup Saha, Seema Nagar, Nam Nguyen, Pankaj Dayama, Renuka Sindhgatta, Prateeti Mohapatra, et al. Automixer for improved multivariate time-series forecasting on business and it observability data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22962-22968, 2024.
[25] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training, 2018.
[26] Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos. Rishika Bhagwatkar, Marin Biloš, Hena Ghonia, Nadhir Vincent Hassen, Anderson Schneider, al. Lag-llam:: Towards foundation models for time series forecasting. arXiv preprint arXiv:2310.08278. 2023.
[27] BizITOps Dataset Repository.
Robot.Shop 2023.
[28] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1181-119,, 2020.
[29] Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07//88 2020.
[30] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp- mixer: An mll- architecture for vision. Advances in Neural Information Processing Systems, 34:24261-24272. 2021.
[31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems, volume 30, 2017.
[32] Jingyuan Wang, Jiawei Jiang, Wenjun Jiang, Chao Li, and Wayne Xin Zhao. Libeity: An open library for traffic prediction. In Proceedings of the 29th International Conference on Advances in Geogruphic Information Systems, SIGSPATIAL 211 page 145-148, New York, NY, USA,
2021.
12

---

[33] Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y Zhang, and JUN ZHOU Timemixer: Decomposable multiscale mixing for time series forecasting, In International Conference on Learning Representations (ICIRR) 2024.
[34] Zhiyuan Wang, Xovee Xu, Weifeng Zhang, Goce Trajcevski, Ting Zhong, and Fan Zhou. Learning latent seasonal-trend representations for time series forecasting. Advances in Neural Information Processing Systems, 35:38775-38788 2022.
[35] Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. International Conference on Machine Learning (ICML), 2024.
[36] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. CoST: Con- trastive learning of disentangled seasonal- representations for time series forecasting, In International Conference on Learning Representations, 2022.
[37] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations, 2022.
[38] Haixu Wu, Jiehui Xu, Jianmin Wang. and Mingsheng Long. Autoformer: Decomposition transformers with Auto-Correlation for long-term series forecasting. In Advances in Neural Information Processing Systems, 2021.
[39] Zhengqing Yuan, Zhaoxu Li, and Lichao Sun. Tinygpt-v: Efficient multimodal large language model via small backbones. arXiv preprint arXiv:2312.16862, 2023.
[40] Zhihan Yue, Yujing Wang. Juanyong Duan, Tianmeng Yang. Congrui Huang. Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8980-8987, 2022.
[41] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205. 13504, 2022.
[42] George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer- based framework for multivariate time series representation learning. In Proceedings the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 21 2114-2, 2021.
[43] Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via frne-frey consistency. Advances in Neural Information Processing Systems, 35:3988 2022.
[44] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty-Fifth AAAI Conference Artificial Intelligence, volume 35, pages 11106-11115, 2021.
[45] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning, 2022.
[46] Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, and Rong Jin. One Fits All: Power general time series analysis by pretrained Im. In NewrIPS, 2023.
13

---

Appendix
A TSMixer Background
We employed TSMixer 66 as a building block for the proposed TTM model due to its otate-of-the-art performance, faster execution, and significantly lower memory usage. However, explained in the main paper, vanilla TSMixer cannot be trained on multiple diverse datasets. Therefore, necessitated the incorporation of the proposed novel components. In this section, we provide a high-level overview of the TSMixer model for simpler and quicker understanding by the readers.
TSMixer is a lightweight alternative to transformer-based time series models, with no compromise on forecast accuracy. TSMixer adopts some well-established pre-processing steps from the literature, such as normalization and patching. Additionally, it offers the flexibility of enabling or disabling channel mixing. Channel mixing has been found to be beneficial in handling multivariate datasets with tross-channe correlations. For the main learning process, TSMixer employs a series of MLP- Mixer 30 blocks that perform inter-patch, intra-patch, and inter-channel mixing operations. mixing operation in TSMixer ensures learning correlations across a specific dimension. For example, inter-channel mixing enables it to learn cross-channel correlations. the experiments, we employed three different flavors of the TSMixer model: TSMixer vanilla (referred as TSMixer throughout the text), TSMixer with cross-channel mixing enabled (TSMixer-CM) and TSMixer with cross-channel reconciliation head (TSMixer-CC). We request the readers to refer to 6 for further details about these variants.
B Literature Survey
B.1 Multivariate Time Series Forecasting: Statistical approaches for time series forecasting, such SARIMAX and Exponential Smoothing, generally generate forecasts independently for each time series 12]. These methods are essentially univariate and do not build single model by learning from multiple time series. On the other hand, more advanced models, built upon machine/deep learning techniques, including LightGBM- have the capability learn from multiple time series. However, these models still follow univariate approaches, thus ignoring any potential cross-channel correlations.
models [4], N-BEATS [23). and DeepAR (28).
Advanced multivariate forecasting models mostly involve deep neural networks, specifically the transformer [31 architecture. A series of transformer- based model have been proposed in the last few years including Informer [44), Autoformer (38), and FEDFormer [45). Although these models outperformed all the prior arts, the DLinear 41 model showed that an embarrassingly simple
linear model can beat these models by following few empirically established steps like time series decomposition, normalization, and
modeling.
PatchTST [22 showed that transformers can be effective for forecasting if the input time series is patched or segregated in multiple windows, and subsequently, modeled by a transformer. The patching operation helps preserve local semantic information, accommodates longer history, and reduces computation time. The PStchTST model outperformed all prior transformer-bas based models and the DLinear model.
Although PSt reinstated faith in transformers for time series modeling, transformer-based models are generally resource-intensive. with slow execution and a high memory footprint. The recently proposed TSMixer model (6] addresses these challenges effectively. TSMixer, built on the MLPMixer architecture [30), stands out for its exceptional speed and lightweight design. It has attained state-of-the-ar (SOTA) performance on benchmark datasets, demonstrating a 2-3X reduction in both execution time and memory usage.
Recently, several new Transformer- and Mixer-based architectures have been proposed. The iTrans- former model [19] applies attention and MLP modules to the inverted dimension. Instead of operating on the temporal tokens, these operations are applied to the variate tokens, resulting in "variate- unmixed representations". This approach is claimed to enhance generalization across different channels and improve the use of arbitrary context lengths. The TimeMixer model [33] leverages the observation that time series exhibit unique patterns at different sampling scales. By utilizing
14

---

different MLPMixer blocks, it aims to capture both microscopic and macroscopic information to produce more accurate forecasts. Similarly, the recent TimesNet model [37 disentangles the complex multi-periodicity in a time series into intra-period and inter-period variations. It then learns time series representations using an Inception block, enhancing the model's ability to capture intricate patterns in the data.
B.2 Pre-trained Models for Time Series: One major drawback of all the above models is that they need to be trained in-domain. Hence, none of these models can be transferred to out-of-domain data with zero or minimal training. This approach has been found to be extremely beneficial in the natural language processing (NLP) domain with the invention of BERT 4 and GPT 225 models.
However, this is an extremely challenging task in the time series domain because of the unavailability of publicly accessible large pre-training corpora. There are multiple independent time series datasets, but, unlike in NLP, these datasets differ significantly in important characteristics such as the domain of the data (c.g., retail, sensor data, traffic, etc.), the number of channels, temporal resolution, and length. This makes it hard to train a single model on all the datasets together.
Hence, a few prior works have focused experimenting with same-dataset self-supervised learning for time series [7 34 36 40). These methods learn a time series representation from the train split of dataset, build a forecaster
top of the learned representation the same data, and then evaluate it on the test split of the same dataset. Although these approaches have demonstrated promising results, they do not provide evidence of the transfer capability of the model between datasets.
Subsequently, models such as SimMTM 5S and TF-C [3 have demonstrated the transfer capabilities of their models between pairs of datasets. These pairs are carefully chosen so that the source (the dataset where the model is pre-trained) and target (the dataset where the model is fine-tuned and tested) datasets share some matching properties. For instance, SimMTM showcased its few-shot capability by selecting ETTH2 as the source data and ETTHI as the target data. Both ETTHI and ETTH2 are collected from Electricity Transformers at two stations, denoting data from similar domain. TF-C demonstrated the transferability of the model across four different (source, target) pairs, such as (ECG, EMG) and (FD-A, FD-B), where domain -imaiitiimilarity exists in both the source and target datasets.
To overcome this limitation, the time series research community is increasingly focused on developing General Pre-Traine Pre-Trained (GPT) or Foundation Models (FM) for time-series forecasting, capable of effectively transferring knowledge to new target TS datasets. This growing interest led to the release of several "large" and "massive" pre-trained time-series models for forecasting in early 2024, generating significant excitement among researchers. Notable releases include Moment [I0], TimesFM (3) Chronos (2), Moirai (35), and Lla-alama (26), all of which set strong benchmarks in zero-shot forecasting. The Moment 110 model tre-tsains Transformer encoder model in univariate way on a collected set of diverse "Time Series Pilc". Moment is pre- pre-trained with mask reconstruction objective, and it can be fine-tuned decoder-style attention model (with causal self-attention) in univariate fashion on a large collection of real world and synthetic datasets. The Chronos [2] model tokenizes the input time series, and feed the tokens into a large langugae model (specifically the T5 model). Chronos is pre-trained in a univariate fashion. During inference, Chronos regressegsys samples tokens and map them the numerical values via dequantization. Chronos is trained on a large corpora of time series including synthetic data for better generalization. The Moirai (35] model pre-trains a Transformer encoder on a massive collection of "LOTSA" dataset (27B time points). Moirai masks the forecast horizon of each target channel and performs mask reconstruction. The flattening operation of all channels in a multivariate time series enables Moirai to pre-train on "any-variate" settings. The Lag-Llama (26] model pre-trains a decoder-only Transformer model that utilizs the time series lags as covariates. Lag-Llama is pre-trained on large collection of diverse time series datasets in univariate fashion. All the above models are open-sourced and used in our experiments for comparison. However, closed-source models such as TimeGPT 8 are not included due to their inaccessibility.
downstream forecasting task. The TimesFM 33] pre- pre-trains
B.3 Pre-trained LLMs for Time Series: Parallel to the above trend of general pre-trained TS models, there has been notable increase in the adoption of pre-trained large language models (LLMs) for time series tasks. These models are approached as cross poos--dmain transfer learning problems. The LLMTime model [ feeds the time series values as text representations and demonstrates promising
15

---

Source
Datasel
Resolution
kaggle web, traliic dataset without missing values
duily
nn5 daily dataset without missing values solar 10 minutes, _ datasct + Downsample australion, electricity demand dataset Downsample solar 4. secunds, dataset + Downsample wind seconds, dataset + Downsample us births, dataset
daily
10 mins, 30 mins, bourly
30 mins, hourly, daily
4 seconds, 10 mins. 15 mins, 30 mins, hourly
seconds, mins, 15 mins, 30 mins, hourly daily duily
Monash
sangeenday dataset
sunspot, dataset without, missing_ values australian weather dataset
daily
daily
cup_ 2018 datasel without missing values
hourly
bitcoin, dataset without, missing values
daily
wind farms, minutely dataset without, missing values Downsample
minutely, 10 mins, mins, 30 mins, hourly
london smart meters datasct without missing values Downsample
30 mins, hourly, daily
PEMSO3 + Dowasample
5 mins, 10 mins, 15 mins. 30 mins, bourly mins, mins, mins, mins, bourly 5 mins, 10 mins, 15 mins, 30 mins, hourly mins, 10 mins, mins, 30 mins, bourly
PEMS04 Downsamphe
PEMS07 Downsample
PEMS08 Downsample
LibCity
PEMS BAY Downsample
mins, 10 mins. mins, 30 mins, boutly mins, 10 mins. mins, 30 mins, hourly mins, 10 mins, mins, 30 mins, hourly
LOS 1OOP Downsample
1.0P SEATTIE Downsample
SZ. TAXI + Downsumple
15 mins, 30 mins mins, 30 mins, bourly
Q Q-TRAFFIC Downsample
Table 8: List of pre-training datasets. A dataset with "+ Downsample" denotes that the proposed Diversity Resolution Sampling (DRS) has been applied on that dataset to generate new diverse datasets at frequeneies lower than the original frequency of the data. Please note that, these pre-training datasets have no overlap with the evaluation datasets. Specifically, the australian mustralian_elcctriittyy. demand dataset and australian weather dataset used in pre-training are completely different (w.zz location, measured variables, lype, resolution, length, etc.) from the standard Electricity (ECL) and Weather dataset used in the evaluation. Please note that, the last three datasets in the Libcity section have been excluded from the pre- pre-training process for the model releases intended for enterprise use
performance in a zero-shot setting. The GPT4TS model [46] adopts a pre-trained LLM like GPT and fine-tunes only the imput embedding layer, normalization layers, and output layer. Specifically, it does not alter the self-attention weights and feed ford-fo layers. The Vime-LLM [15] model proposed a reprogramming framework, where they reuse existing LLMs for forecasting while keeping the LLM backbone intact. The overall approach to building pre-trained model for time series from LLMs is promising, but it does not model cross-channel correlations observed in many multivariate time series datasets. Moreover, these LLMs are very large and exhibit slow execution and a large memory footprint.
C Datasets
C.1 List of Pre-training Datasets: Pre-training employs a subset of ~ ~BB samples from Monash (9] and Libcity (5] data collection, where Monash results in ~200 samples and LibCity accounts for the rest. In this estimate, one sample denotes a pair of training windows: X E IR 1xxs and Y xt We employ subset of the datasets available the Monash forecasting data repository 9 available at wuthttps::// worecast.ttooooo ingdata org/ Since our primary focus in this study is long term forecasting with forecast length ranging from 96 720, is not possible to use yearly, monthly, quarterly, weekly datasets due to their short lengths. Hence, we skip a few datasets of short lengths. The Monash datasets used are available under a Creative Commons Attribution 4.0 International license. For LibCity, we employ all datasets released by the Moirai authors
35 available at (ex- cept the Rotterdam dataset which was not available during experimentation). The LibCity datasets at the above link were released under an Apache 2.0 license. The final list of all pre-training datasets is shown in Table g Please note that, the last three datasets in the Libcity section have been excluded from the pre-training process for the model releases intended for enterprise--uee
C.1.1 Temporal cross-validation: Temporal cross-validation used to chronologically split all the time series into train and validation parts. During pre-training, moving windowing technique used to create (X, Y) pairs of lengths al and fl respectively. Please note that, these pre-training datasets have no overlap with the evaluation datasets. Specifically, the aus-
16

---

Total
# Target variables
#Exog. variables
Set
Dataset
Resolution
Length
Source
#Channels
ETTHI
hour
17420
ETTH2
hour
17420
7
7
38
ETTM1
15 minutes
69680
D1
ETTM2
15 minutes
69680
Not Applicable
Weather
10 minutes
52696
21
38
ECL
hour
26304
321
321
38
Trallic
hour
17544
862
862
38
BS
hour
17379
14
11
77
CC
2 minutes
5409
8
2
13
APP
10 seconds
8834
39
35
17
SER 10 seconds
8835
107
72
35
27
Table 9: Details of the evaluation datasets.
used in pre-training are pletely different (w.rt location, measured variables, type, resolution, length, etc.) from the standard Electricity (ECL) and Weather dataset used in the evaluation.
ciliian_neettttttyy..mo diletticitt_demaand__ demamdddddd.eem _atasded__tttt---a and australian australian_weather
C.2 List of Evaluation Datasets: Table illustrates various characteristics of the eleven evalua- tion datasets. Below, we present the details.
Set D1: For zero/few/full--deee evaluation, we utilize seven multivariate time series datasets that have consistently been employed in the literature. Below, we offer brief overview of these datasets.
ETT datasets: The four ETT datasets 44 (ETTHI, ETTH2, ETTMI, ETTM2) contain multivariate time series data collected from electrical transformers two stations. ETTH and ETTH2 are collected at an hourly interval, while ETTMI and ETTM2 are collected every 15 minutes. All four datasets have channels.
2. Weather: The weather dataset consists of 21 channels, which serve weather indicators. It is collected at 10-minute intervals at the Max Planck Institute of Biogcochemistry weather station.
3. Electricity (ECL): The Electricity dataset, also known as the ECL dataset, comprises the hourly electricity consumption data of 321 clients.
4. Traffic: This dataset records the hourly rates road occupancy on the San Francisco Freeways using 862 sensors.
We used the datasets provided in the repository of the Autoformer paper [8 For all the D1 datasets, we execute the same train/validde crin//alidation/test splitting as was performed the literature 44
38
6].
Set D2: To assess the effectiveness of the proposed TTM model in extracting information from exogenous channels, we conduct evaluations on four additional datasets that are known to contain exogenous or control variables.
1. Bike Sharing (BS): The Bike Sharing dataset 7 documents the hourly rental counts of bikes from the Capital Bikeshare system in Washington D.C., USA, spanning the years 2011 to 2012. Rental counts are typically associated with environmental and seasonal conditions. Consequently, this 14-channel dataset encompasses various weather-related features. Our goal was to forecast all three rental counts: "casual". "registered", and "ent" (total count). As the remaining 11 features are consistently available at all future time points, they are treated as exogenous variables in experiment.
2. Carbon Capture Plant (CC): The Carbon Capture Plant data 11 records the emission profiles of "2-amino-2 -eethll---roppameee
(AMP) and "piperazine" (Pz) collected at every 2 minutes interval. We utilize the 8-channel dataset made available in the official repos- itory 13 Among the remaining 6 channels, the following serve as control variables:
Available at sihttps:: :ttpss:///gitttt wttp://tttw.//ttottt........e

---

"FI-19" "FI-19", "TI-3", "FI-II", "TI-1213"]. The remaining 1 variable is treated as a conditional variable (as neither target variable nor available during the forecast period to consider it as exogenous). For additional details, please refer to the supplementary materials of [3].
3. Service (SER):
This dataset pertains to the cloud-based "Stan's Robot Shop" application, managed by Instana. It simulates a user's e-commerce experience, encompassing site access to shipping, utilizing load generator. Intermittent fault injection introduces diverse IT events. The dataset provides business KPIs for services (e.g., payment, catalog) and IT events tracked by Instana. Sampling occurs every 10 seconds due to high traffic and event frequency. For our experiments, all business KPIs are treated as target variables and IT events are treated as exogenous variables and the goal of our forecasting is predict the business KPIs given the IT events.
4. Application (APP): This dataset is similar to the SER data, but captures KPIs for the entire application instead of capturing at the service level. Even in this case, all business KPIs are treated as target variables and IT events are treated as exogenous variables and the goal of our forecasting is to predict the business KPIs given the IT events.
D TTM Model Hyper-parameterr and Baselines
D.1 Pretraining: Pre-training is performed in distributed fashion with 50 CPUs and 6 NVIDIA
A100 GPUs. Standard model configurations are as follows: patch length pl = 64 (when sl is 5 512), 128 (when s! is 1024 1536) and 8 (when s! is 96); stride pl (i.e., non-overlapping patches), number of patches n = sl/pl, number of levels in backbone L. = 3, number of TTM blocks per level M 2, number of decoder layers 2, batch size b 4500, number of epochs ep 20, and dropout do = 0.4. PatchhiSMiier--peecfffc hyperparameters include feature scaler fs = 3, hidden feature size h. = f pl, expansion feature size ef hff 2. Please note that h and n will change across TTM blocks based on the adaptive patching strategy. Resolution prefix tuning is enabled by default on all variants other than TTMg- Decoder channel-mixing and exogenous mixer blocks are disabled during pre-training and enabled during fine-tuning based on the dataset requirement.
D.2 Fine-tuning: Most model parameters remain the same from pretraining except the following parameters. Head dropout is changed during finetuning based the target dataset used (0.7 for smaller ETT datasets and 0.2 for other datasets). Likewise, the batch size is set to 8 for Traffic, 32 for Electricity, and 64 for all other datasets. Moreover, decoder channel-mixing and exogenous mixer block are enabled for datasets that need cross-channel modelling (i.e. D2 datasets). Unlike pre- training, fine-tuning executed in just A A100 GPU as it is fast process. All these hyper-parameters are selected based on the validation performance, and the final test results are reported in the paper.
D.3 Computational Benefits of TTM over existing models Setup details: Table compares the computational benefits of TTM over existing TS- TS-pretrained models and reports the following metrics: (i) GPU Inference Time per batch (in milliseconds (ms)), (ii) CPU Inference Time per batch (in seconds (s)), (iii) Max GPU Memory used during inference (in GB), (iv) Params: Total parameters of the models (in Millions). Experiments are conducted using si = 512, fl= and batch size = 32 in one A100 80GB GPU, 16 cores with 256GB memory. GPU not enabled while capturing the CPU time. Since many pre-trained models process data in purely univariate fashion, while TTM processes data in a multi-variate fashion, experiment so that, the number of samples per batch remains the same across all models for a fair comparison. In addition, we used small batch size of 32 for this experiment, as many pre-trained models (like Chronosy, were encountering out-of- off-mmoryy (OOM) errors with high batch sizes. For this experiment, we set the number of probabilistic samples probabilistic algorithms (such as Chronos or Lag-Llama) to compute their fastest possible runtime. Note, that for forecast accuracy comparison, we set the number of samples to 100 for Lag-Llama and 20 for Chronos as suggested in their open-source code examples. All the baselines algorithms were evaluated using their open-sourced inference A APIs as detailed Section D.4| Please note that the computational benefits of TTM will further amplify if we use higher batch sizes or high number of channels as our models are extremely small and can process multiple channels at the same time using the channel-indeeenderr approach (22).
set the number of channels
for this
(i.e., rum__ s samples
1) for
18

---

Results
Link to the
Category
Baselime
Used in Table
Generated From
implementation
Moirai 35
Zero-shot in Table
and
Table and 22
X
N/A un:2ts
Implemeniation
(a) TS pre-trained
Zero-shot in Table and
Runtime in Table
Official implementation
timesfm
SOTA models
Zero-shot Table umn
Runtime in Table
Official implementarion
chronos chronos-Corecast
Zero-shot Tabler
Lag-Llama
26
Ollicial implementation
lag-11as Big-1lama
Runtime in Tabll
Moment
10
Table and 77
lleadpprobing
Table 2 of IM
N/A
Moment
10
Runtime in Taboe
Official Implomentation
moment
GPT4TS 46
Few-shot 5% in Table and
Table 12 of (46)
N/A
GPT4TS
46
Runtime in Table
Official Implementation
Cae Pits Fils-All All
Exog.
Tanc
LLM-based
Full Shot lead probing
pre-trate vre-trained models
GPT4TS
46
Table
5
Table of TT
N/A
Results availalle in Repository
LLMTiim
Zemo shot Table
20
1mttime
Time-LLM
Füll-shor Head-probing
Table
10
N/A
Table 5
and
Uni
18
Zero shot Tante 21
Table of 18
N/A
SimMTM
(b) Self-
TST FR
supervised
Directly reported
pre-trane pre-trained
LaST 34
Table
from SimMTM paper
S
N/A
models
TF-C
CoST 16
TS2Vec (0)
PatchTST [2
Taken from 46
N/A
TSMixer
Ollicial Implementation
PatchTsxixer
(d) Other SOTA
TincMMier 33
Tahle 16
Official Implementation
Architectures
iltanslormer [19
Oflicial Implementation
1136 Serres-Libran
TiieesNet Ai
NIA
Dincar [4]
Taken from
N/A
FEDFormer 45
Autoformer
Full-shot end2ead Table
Taken
Infommer 44
Table 10: Implementation details for the baseline algorithms.
D.4
Baseline Implementation Details:
report the implementation details for all the baselines
in To
E
Sample Zero-shot Visualizations
visualizes the zero-shot forecasts of TTM across different datasets illustrating the power of TTM to capture complex trends and seasonal patterns.
Figure. O
F Full Results Tables
Here, we present the complete versions of various tables in the main paper. These full versions essentially include the test results for multiple forecast lengths (fl) across all datasets. Occasionally, these results are averaged across forecast lengths conserve space in the main paper.
F1 Full table for all TTM variants: Table 11 and Table 12 captures the fine- tiniigraieed results of all TTM variants (i.e. TTMg, TTMg, TTMgand (TMMgnddTMM)) on the D1 data benchmark set.
Full table for zero-shot experiment: Table show the sliding window zero-shot results for all forecast lengths across all DI datasets, and compares TTM variants with Moirai variants and TimesFM. Table T5l depicts the last-window zero-shot results for all forecast lengths across all of datasets, and compares TTM with Chronos and Lag-Llama.
19

---

FL
1TM.
TTMg
TTMA
96
0.365
0.364
0.363
0.359
Traffic Flectricity Weather ETTM2 ETTMI ETTH2 ETTHI
192
0.393
0.386
0.393
0.389
336
0.415
0.404
0,406
0.405
720
0,538
0.424
0.452
0.448
96
0.285
0.277
0.271
0.264
192
0.341
0.334
0.324
0.321
336
0.383
0.362
0.357
0.351
720
0.441
0.408
0.388
0.395
96
0.413
0.322
0.327
0.318
192
0.476
0.376
0.377
0.354
336
0.553
0.407
0.395
0.376
720
0.737
0.439
0.419
0.398
96
0.187
0.171
0.178
0.169
192
0.261
0.238
0,238
0.223
336
0.323
0.304
0.29
0.276
720
0.436
0.41
0.379
0.342
96
0.154
0.158
0.166
0.159
192
0.203
0,206
0.214
0.203
336
0,256
0.256
0.254
0.247
720
0.329
0.328
0.319
0.314
96
0.169
0.166
0.157
0.152
192
0.196
0.191
0.174
0.179
336
0.209
0.207
0.195
0.193
720
0.264
0,255
0.25
0.243
96
0.518
0,514
0,476
0,462
192
0.548
0,544
0.5
0.491
336
0.55
0,575
0.51
0,509
720
0.605
0.622
0.571
0,547
Model Size
1M
IM
4M
5M
Table 11:
Zero-shot results all TTM variants on D1 data benchmark across sliding test windows
(standard test protocol).
F.3 Other zero-shot comparisons: LLMTime [1]] reported the test performance only on the last windows of the test datasets (instead of sliding windows) for horizons 96 and 192 due to computational reasons. We recreate the same experimental setup for TTM, and depict the comparative results in
Table We observe 26-36% improvement across all three variants of TTM with tremendous (70,000 to 14,000) reduction in model sizes. Another zero-shot comparison with the UniTime [18] model is shown in Table In this comparison, TTM outperforms UniTime by 29-31%.
F44 TTM Zero-shot rs. SOTA Full-shot (short context setting): Table we compare the zero-shot results of TTM variants with full-shot end-2-end training of popular TS architectures like iTransformer, PatchTST etc.. The full-shot SOTA algorithms were trained in short-context length setting (sl 96) the train split of each target dataset, and these results are obtained from the Moirai paper 35 where the authors draw similar comparison. TTM was tested in zero-shot setting without any training the target datasets. We also provide the zero-shot results of Moirai variants and TimesFM from Table for reference purpose. We can see that the zero-shot performance of all variants of TTM outperforms the full-shot performance of SOTA models, even though the latter trained on the target datasets. This underscores the strength of the pre-trained TTM model.
F.5 Full table for 5% shw-shot experiment: Table forecast lengths across all D1 datasets.
shows the 5% few-shot results for all
E6 TTM vs. Cross-transfer models: Table 22 draws a comparative analysis of TTMg with SimMTM, Ti-MAE, TST, LaST, TF-C. CoST, and TS2Vec models different few-shot settings (10% to 100% availability of training data) on ETTHI dataset. The baseline models are trained on ETTH2 data, and tested on ETTH1 data, thus demonstrating their transferability across datasets having similar characteristics. The baseline numbers are taken from 5.. TTMg outperform all of them (including the recent SOTA SimMTM) by significant margin. This highlights the usefulness of the pre-trained TTM weights and their ability to adapt to target domain with few-shot fine-tuning.
20

---

FL
TTM
1TMg
TTMg
TTMA
96
0.366
0.364
0.363
0.359
Traffic Flectricity Weather ETTM2 ETTMI ETTH2 ETTHI
192
0.391
0.387
0.393
0.394
336
0.421
0.399
0.398
0.406
720
96
0.282
0.277
0.271
0.267
192
0.338
0.334
0.325
0.321
336
0.383
0.361
0.357
0.354
720
96
0.359
0.313
0.326
0.317
192
0.402
0.357
0.371
0.355
336
0.424
0.395
0.396
0.374
720
0.575
0.437
0,418
0.399
96
0.174
0.171
0.178
0.17
192
0.23
0.237
0.222
336
0.299
0.293
0,284
0.274
720
0.407
0.393
0.373
0.345
96
0.152
0.154
0.162
0.155
192
0.198
0,203
0.215
0.201
336
0.252
0.262
0.244
720
0.326
0.327
0.319
0.316
96
0.142
0.146
0.15
0.141
192
0.162
0.164
0.171
0.16
336
0.184
0.185
0.202
0.179
720
0.228
0,236
0.303
0.24
96
0.401
0.411
0,411
0.469
192
0.425
0.42
0.44
0.488
336
0.437
0.468
0.46
0,512
720
Model Size
1M
IM
4M
5M
Table Few-shot results all TTM variants on D1 data benchmark across all sliding test windows (standard test protocol).
F.7 Full table for full-shot head-probing experiment: Head Probing (HP) involves finetuning the pre-trane pre-trained model heads on full data with backbone weights frozen. Table 17 compares the full-shot head-probing results of TTM with Moment, Time-LLM and GPT4TS. Table 18 and Table compares the TTM Head probing results with the Full-shot End-To-End Training results of popular time-series architectures. It important to note that end-to-end training updates the backbone weights, whereas head probing does not. TTM head probing results either superior highly competitive with other popular state-of-the-an methods that are trained end-to-end on the target data.
F88 Full table: Impact of Adaptive Patching (AP): Table shows the full table for studying
the impact of adaptive patching across different amounts of pre-trained data settings. In both the settings, AP helps TTM to produce more accurate forecasts. However, the impact of AP is greater in the setting with a lesser amount of pre-training data, where there is more need to model at multiple granularities to compensate for the data size.
F.9 Full table: Impact of Resolution Prefix Tuning (RPT): Table presents a comprehensive analysis of the impact of RPT on TTM. RPT generally enhances forecast performance, particularly when the pretraining (PT) data abundant and diverse. In this scenario, incorporating a learnable resolution prefix token significantly benefits the models by allowing them to decouple the weights across resolutions effectively. Conversely, in setups with limited PT data where diversity challenges are minimal, RPT has a reduced impact. Additionally, we can see that RPT helps in scenarios when context length short. Table 25 shows the impact (sl = 96). We report the zero-shot results for fl = 24 in the table. In these scenarios, automatically detecting the resolution becomes challenge for the model. Hence, by explicitly fusing the resolution information prefix, we can enhance the model's ability learn effectively across resolutions.
RPT in shorter context length setting
21

---

FL
TTMp
TTMg
ТТМ4
Moirais
Moirais
Moiraiz
TimesFM
96
0.364
0.363
0.359
0.375
0.384
0.38
0.421
192
0.386
0.393
0.389
0.399
0.425
0.44
0.472
336
0.404
0.406
0.405
0.412
0,456
0.514
0.51
720
0.424
0.452
0.448
0.413
0.47
0.705
0.514
96
0.277
0.271
0.264
(1.281
0.277
0.287
0.326
Electricity Weather ETTM2 ETTMI ETTH2
192
0.334
0.324
0.321
0.34
0.34
0.347
0.4
336
0.362
0.357
0.351
0.362
0.371
0.377
0.434
720
0,408
0.388
0.395
0.38
0.394
0,404
0.451
96
0.322
0.327
0.318
0.404
0.335
0.353
0.357
192
0.376
0.377
0.354
0.435
0.366
0.376
0.411
336
0.407
0.395
0.376
0.462
0.391
0.399
0.441
720
0.439
0.419
0.398
0.49
0,434
0.432
0.507
96
0.171
0.178
0.169
0.205
0.195
0.189
0.205
192
0.238
0.238
0.223
0.261
0.247
0.247
0.293
336
0.304
0.29
0.276
0.319
0.291
0.295
0,366
720
0.41
0.379
0.342
0.415
0.355
0.372
0.472
96
0.158
0.166
0.159
0.173
0.167
0.177
192
0.206
0.214
0.203
0.216
0.209
0.219
336
0.256
0.254
0.247
0.26
0.256
0.277
720
0.328
0.319
0.314
0.32
0.321
0.365
96
0.166
0.157
0.152
0.205
0.158
0.152
192
0.191
0.174
0.179
0.22
0.174
0.171
336
0.207
0.195
0.193
0.236
0.191
0.192
720
0.255
0.25
0.243
0.27
0.229
0.236
Table 13: Zero-shot results TTM over Moirai (ICMI:24) and TimesFM (ICMI.24). Flectricity and Weather results for TimesFM are not reported as they were used by TimesFM for pretraining, Similarly, Traffic data was used in both Moirai and TimesFM pre-training, hence, skipped in this comparison.
Models
Training
length
ITM,
ITM,
MTM
Moumi
Maira,
Morri
Transfoneer
TimreeNe
TST [Cosstommer
Dlinsii
SCTNeT
FFDFoneer
BTTHI
0.394
0.404
01
0.4
0.444
0.31
03479
0.454
0,455
0,450
0.3.3
0.541
0.4.5
0.747
0..4
ETIH2
0.345
0.335
0.33
0,34
0.346
0,354
0.403
0.383
0,414
0.387
0.942
0.611
0.559
0.954
0.437
ETIMI
0.386
0.38
0.362
0.445
0.382
0.30
0.429
0.407
0.4
0.387
0.5113
0.419
0.403
0.486
0.448
ETTM2
0.281
0.27
0.252
0.3
0.272
0.276
0.334
0.288
0.29]
0.281
0.757
0.358
0.35
0.571
0.305
0.242
0.238
0.259
0.265
0.193
0.244
6%
13%
4%
7%
34%
22%
15%
37%
13%
TTM, Fdueptt)
7%
1%
6%
16%
6%
89
8%
34%
23%
16%
39%
15%
TIM. (dmpp)))
10%
4%
9%
19%
9%
11%
11%
36%
26%
19%
40%
17%
Table 14: Zero-shot suecast-Improvemmen (Himp) of TTM Moirai (ICML'24), TimesFM (ICML/24) and other popular architectures (fall shot trained with vontext length 96). MSE averaged across Fla: [96, 192, 336, 720). Electricity and Weather results for TimesFM reported they used TimesFM for pretraining. Similarly. Traffic data was used both Moiral and TimesMM pre traininnnng Full-shot and Moirai results reported from (35), TimesHM results were generated from their released code.
G Model Insights and Explanation
G.1 Dataset preparation for TTM embedding analysis: To understand the representation
obtained from TTM encoder, we have carried out controlled analysis, using 3 datasets with varying observation frequency, viz., (1) weather (10 min), (2) electricity (1 hour), and (3) traffic (1 hour). We have selected three temporally distinct non-overlapping windows of length 1024 from each dataset (Figure 7 The selection criteria these segments are distinct mean and standard deviation measures. From each of these segment, 512 context length windows are extracted in rolling window fashion. Embedding vector from the encoder is collected at backbone output. Each of these representation tensors flattened, and Principal Component Analysis (PCA) is carried out on the whole dataset. The project on the first two principle components is used to obtain the figure 5
The channel mixing block in the decoder of TTM consists of gated attention block that produces an attention weight for each feature across channels. We have considered the mean attention weight across features and data samples to derive the feature contribu- tion. We have used the model finetuned on the Bikesharing dataset for this purpose. Bikesharing data
G.2 Channel Attention Map:
22

---

FL.
T
1TMg
TTM,
Chronosy
Chronoss
Chronosg
Chronos;
Lag-Ilama
24
0.195
0.243
0.217
0.207
0.217
0.163
0.161
0.372
E7TH2 BTTH!
48
0.193
0.244
0.226
0.33
0.277
0.228
0.226
0.376
0.209
0.212
0.2
0.295
0.331
0.303
0.311
0.321
96
0,194
0.209
0.207
0.424
0.383
0.272
0.298
0.299
192
0,231
0.227
0.221
0.299
0,303
0.295
0.335
0.303
24
0.12
0.149
0.158
0.1.3
0.12
0.0866
0.065
0.216
0.197
0.186
0.194
0.139
0.187
60
0.166
0.175
0.193
0.217
0.187
0.195
0.2
0.182
96
0.111
0.13
0.155
0.197
0.156
0.22
0.207
0.135
192
0.105
0. 0.123
0. 107
0.151
0.131
0.125
0.162
1.1
0.247
0.187
0.285
0.196
0,284
0.247
0,207
ETTM2 ETTMI
0,161
0.241
0.182
0. 7.704
0,343
0.207
0.311
0.881
60
0,153
0.205
0.166
0,959
0,545
0,536
0.818
0.958
96
0.172
0,209
0.184
0.854
0.589
0,508
0,538
1.006
192
0.402
0,294
0.232
1.394
0.757
0,914
0.774
1.16
24
0.04
0.063
0.0G8
0.04
0.041
0.054
0.075
0.325
48
0.144
0.16
0.152
0.379
0.253
0.236
0.198
0.337
60
0.125
0.143
0.119
0.274
0,233
0.222
0.222
0,294
0.137
0.108
0.101
0.147
0.12
0.171
0.192
0,314
0.165
0.146
0.191
0.223
0.265
0,246
0,268
24
0.015
0.015
0.016
0.048
0.011
0.009
0.01
0.084
Weather
48
0.018
0.017
0,026
0.024
0.017
0.016
0.022
0.505
60
0.024
0.03
0.054
0.036
0.04
0.029
0.028
0,141
0.025
0.042
0.049
0.057
192
0.103
0.072
0.065
0.105
0.095
0.047
0.055
0.243
Electricity
24
0.409
0.421
0.433
0.463
0.429
0.42
0.409
0.384
48
0.28
0,308
0.304
0.338
0.313
0.289
0.278
0.351
0.26
0.287
0.279
0.346
0.301
0.279
0.273
0,363
0.231
0.243
0.238
0.352
0,294
0.231
0.227
0.317
192
0.495
0.495
0,491
0.616
0,548
0.503
0.507
0.551
24
0.231
0,232
0.239
0.324
0.3
0.288
0,301
0,237
Traffic
48
0.193
0.183
0.187
0.244
0.232
0,213
0.226
0.192
0.191
0.258
0.332
0.326
0.198
96
0.21
0.199
0.215
0.256
0.219
0.217
0.196
0.213
192
0.387
0.393
0.382
0.373
0.419
0.351
0.297
0.374
Table 15: Zero-shot results of TTM Chronos (2024) and Lag-Ilama (2024) Since Chronos and Lag- Ilama recommend/report results with shorter forecast lengths, we use different FLs E {24, 48, 60, 96, 192) in this experiment.
the last test-window.
includes exogenous variables, viz. temperature, humidity, wind speed, etc. We analyzed the mean attention of the model across these exogenous variables for the forecast of rental bike count (Figure S). As we observe, these attention weights highly correlate with the general data characteristics of his data, wherein- bike rentals are highly influenced by weather and holiday signals. Thus, TTM fine-tuning process is quick as well as explainable.
H Limitations and Future Work
TTM is currently focused solely forecasting tasks, similar other forecast pretraining models like Moirai (35), Chronos [2], and TimesFM 33. However, recent models such as Moment and GPT4TS taking initial steps expand their capabilities across multiple downstream tasks, including classification, regression, and anomaly detection. Inspired by these advancements, we plan to extend
TTM's functionality encompass broader range of downstream tasks.
Another limitation TTM is the need to train different models for different context length settings. architecture, TTM is sensitive to context lengths. Consequently. in
Due to its non- transformer-ben this paper, we introduce three variants of TTM, each optimized for different context length settings. Looking ahead, we aim enhance TTM's backbone automatically adapt to dynamically varying context lengths.
In addition, existing pre-trained models like lag-Ilama, Moirai support probabilistic forecasting while TTM currently supports only point forecasting. We plan to extend TTM with distribution heads to facilitate probabilistic forecasts in future work.
23

---

Duta
Pre-trained Models
Other popular architectures
FL
ITMg
TTMp
TTM,
GPT4TS
Time-LMM
PatchTST
TSMixer
TimeMixer
iTranfoommrr
TimesNet
Dlincar
96
0.364
0.363
0.359
0.343
0.483
0,557
0.554
0,899
0.674
0.892
0.547
192
0.387
0.393
0.394
0.748
0.629
0.711
0.673
0.942
0.757
0.94
0.72
336
0.399
0.398
0.406
0.754
0.768
0.816
0.678
1.423
0.838
0.945
0.984
720
Traffic Electricity Weather ETTM2 ETTMI ETTH2
96
0.277
0.271
0.267
0.376
0.336
0.401
0.348
0.356
0.382
0.409
0.442
192
0.334
0.325
0
0.418
0.406
0,452
0.419
0,549
0.445
0.483
0.617
336
0.361
0.357
0.354
0.408
0.405
0.464
0.389
0.619
0.483
0.499
1.424
720
96
0.313
0,326
0.317
0,386
0.316
0,399
0.361
0,315
0.437
0.606
0.332
192
0.357
0
0.355
0.44
0.45
0,441
0,411
0,535
0.49
0,681
0.358
336
0.395
0.396
0.374
0.485
0.45
0.499
0.467
0.621
0.563
0.786
0,402
720
0.437
0.418
0.397
0.577
0.483
0.767
0.677
0.64
0,78
0.796
0,511
96
0.171
0,178
0.17
0.199
0.174
0,206
0.2
0,231
0.217
0.22
0.236
192
0.23
0.237
0.222
0.256
0.215
0.264
0.265
0.313
0.266
0.311
0.306
336
0.293
0,284
0.274
0,318
0.273
0.334
0.314
0,356
0.322
0.338
0.38
720
0.393
0,373
0.345
0.46
0.433
0.454
0.41
0.46
0,43
0.509
0,154
0.162
0.155
0.175
0.172
0.171
0.188
0.187
0.211
0.207
0.184
192
0.203
11.115
0.201
0.227
0.224
0,23
0.234
0,324
0,269
0.272
0.228
336
0,252
0,262
0.244
0,286
0.282
0,294
0.287
0,307
0.316
0.313
0.279
720
0.527
0.319
0.316
0,366
0,366
0.384
0.365
0,451
0.393
0.4
0.364
96
0.146
0.15
0.141
0,143
0.147
0.145
0.147
0.555
0.157
0.315
0.15
192
0.164
0.171
0.16
0.159
0.158
0.163
0.172
0,219
0.181
0,318
0.163
336
0.185
0.201
0.179
0.179
0,178
0.175
0.19
0.273
0.219
0,34
0.175
720
0.236
0.303
0.24
0,233
0.224
0.219
0.28
0,309
0,249
0.635
0.219
96
0,411
0.411
0.469
0.419
0.414
0,404
0.408
0,463
0.43
0.854
0.427
192
0.42
0.44
0.488
0.434
0,419
0.412
0.421
0.548
0,445
0.894
0.447
336
0.468
0.46
0.512
0.449
0.437
0.439
0.477
0.498
0.481
0.853
0.478
720
Table 16: TTM Few-shot 5%
MSE reported across all the standard FL.s considered. TTM, Pre-trained baselines and other model architectures are trained with 5% train data.
Data
Head Probing of Pre- Trained Models
Pall-shot end 2 2-end Training with 312 context length
F1
TTM,
TTM,
TTM,
Moment GPT4TS) Time-LLM
TimeMixer
PRIGHTS
Timesher
TSMixer
FEDFomer
AAttofommer
Informer
0.37
0.354
0.436
0.454
0.523
0.445
0.447
0.521
0.506
0.514
1.181
ETTH2
720
96
0.269
0.59
0.273
0
40402
0.392
0.262
0.403
0.288
0.283
0.405
0.399
0.285
0.342
0.271
0.379
0.274
0.462
0.34
0.276
0.395
0.346
0.463
0.515
0.358
3.755
3.647
0.291
0.29
0.358
0.672
ETTM1
720
0.49
0.408
0.393
0.405
0.417
0.437
0.415
0.416
0.488
0.416
0.543
0.671
1.166
ETTM2
720 0.35
96
0.16d
0.347
0.155
0.369
0.158
0.363
0.17
0.173
0.378
0.366
0.181
0.359
0.164
0.362
0.165
0.108
0.187
0.364
0.358
0,202
0.121
0:433
0.255
0.365
3,379
Weather
0.147
0.172
720 0.323
0.324
0.318
0.315
0.326
0.31
0.314
0.365
0.316
0.403
0.419
1,059
Blectricity
720 0.2
96 0.129
0.193
0.129
0.128
0.191
0.158
0.211
0.206
0.139
0.129
0.194
0.129
0.197
0.168
0.22
0.129
0.186
0.246
0.193
0.254
0.201
0.274
0.373
Traflie
720
0.368
0.431
0.425
0.419
0.352
0.45
0.45
0.36
0.43
0.432
0.36
0.593
0.64
0.444
0.587
0.626
0.66
0.864
0.719
Table 17: Head Probing (HP) involves finetuning the pre-trained model heads on full data with backbone weights frozen. Head probing results of TTM are compared with the Head probing results of other Pre- trained models and also with the Full-shot end-te andtoo-end training of popular TS architectures. TTM's Head probing results consistently outperform other HP benchmarks the full end-to -nd training of popular TS architectures. MSE across FLs (96,700) are reported from (10). Time-LLM, results for large datasets are not reported in [10] due to computational issues. It is important to note that end-to-end training updates the backbone weights, whereas head probing does not.
also very competitive as compared to
24

---

Data
TTM Head Probing
Full-shot end-2-end Training with 512 context length
FL
11Mg
TTN
LTM
LTM,
TimeMixer
TSMixer
PaccTTT
TimesNet
Crosdformes
Dlinear
FEDForer
Autofoemer
Informer
0.375
0.36
0.362
0.363
0.361
0.368
0.37
0.334
0.118
033755
0.376
0.749
0.865
720
0.424
0.436
0.449
0.442
0.445
0.444
0.447
0.521
0.753
0.472
0.506
0,514
1
192
0.398
0.392
0.394
0.392
0.409
0.399
0.413
0.436
0.539
0.405
0.42
0.5
1,008
336
0.397
0.403
0.421
0.439
0.599
0:521
1.107
96
0.285
0.269
0.275
0.262
0.271
0.276
0.274
0.34
0.425
0.289
0.346
0:358
3.755
720
0.417
0.39
0.402
0.392
0.342
0.395
0.379
0.162
0.775
0.605
0.463
0.515
3,647
192
0.231
0
0:456
336
0.361
0.359
0.356
0.351
0.332
0.357
0.329
0.452
0.581
0.448
0.496
0.482
4.721
16
0,286
0.291
0.293
0.283
0.291
0.291
0.293
0.338
0.361
03.299
0.379
0.505
0.672
720
0.417
0.478
0.671
192
0.333
0.325
0.335
0.332
0.327
0.333
0,333
0.374
0.387
0.335
0.426
0.553
0.795
336
0.364
0.363
0.364
0.383
0.36
0.365
0,369
0.41
0.605
0.369
0.445
0.621
1.212
ETTM2
0.165
0.158
0.158
0.166
0.187
0.167
0.200
0.255
720
0.358
0.35
0.347
0.369
0.359
0.358
0,362
0.108
1,208
0.397
0,421
0.133
3,379
192
0.22
0.219
0.215
0.213
0.223
0.219
0.223
0.249
0.345
0.224
0.269
0.281
0.533
336
0.269
0.277
0,26
0.269
0.279
0.273
0.274
0.521
0.657
0.281
0.325
0.339
1.363
Electticity Weather
0,144
0.1.6
0.135
0.1-9
0.147
0.6
0.179
0.172
0.252
0.176
0.217
0,266
0.3
720
0.317
0.323
0.324
0.318
0.31
0.316
0,314
0.365
0.526
0.523
0.403
0.419
1,059
192
0.19
0.19
0.207
0.192
0.189
0.191
0.191
0.219
0.371
0.22
0.276
0.307
0.598
336
0.247
0.243
0.306
0.265
0.339
0.359
96
0.13
0.129
0.129
0.128
0.129
0.129
0.129
D. 0.168
0:15
0.14
0.193
0.201
0.274
720
0.202
0.2
0.193
0.191
0.191
0.186
0.197
0.22
0.251
0.203
0.216
0.254
0.373
199
0.148
0.153
0.201
0.222
336
0.164
0.163
0.161
0.162
0,161
0.158
0,163
0.198
0.182
0.169
0.214
0.213
0.3
96
0,367
0.368
0.372
0.352
0.36
0.356
0,36
0.593
0.514
0:41
0.587
0.613
0.719
0.424
0,432
0.626
0.66
192
0.387
0.403
0.365
0.359
0.375
0.377
0.379
0.617
0.549
0.123
0.604
0.616
0.696
336
0.414
0.395
0.379
0.375
0.385
0.385
0.392
0.629
0.53
0.436
0.621
0.622
0.777
Table 18: Full view: TTM Head probing Vs Full-shot End-To-End Training of popular time-series architectures reported for all FLs. Head Probing (HP) involves finetuning the pre-trained model heads full data with backbone weights frozen. TTM head probing results are either superior or highly competitive with other popular state-of of--hh--thhtrt methods that are trained end to- on the target data. It is important to note that end-to-em end training updates the backbone weights, whereas head probing does not.
Dala
TTM Frobing
Full-shot end-2-enn Training with length
Daia
TTMe
TIMs
TTM:
TTM.
TimeMixer
TSMMssrr
PaccTTT
TimesNer
CrossForner
Dlincar
HEDFormer
Autoformer
Informer
ETTH1
0.3%
0.397
0.402
0.402
0.411
0.408
0.413
0.495
0.6
0.423
0.44
0.4%
1.04
ETTH2
0.347
0.332
0.564
KTIM1
0.35
0.35
0.35
0.34
0.348
0.351
11353
11.4
0.314
0.357
0.44%
0.58%
D.961
EТTM2
0.253
0.252
0.245
0.252
0.256
0.254
0.256
0.291
0.621
0.267
0.304
0.327
1.41
Weather
0.224
0.225
0.224
0.246
0.309
Electricity
0.161
0.16
0.158
0.156
0.156
0.155
0.159
0.192
0.186
0.166
0.214
0.222
0.311
Traffic
0.4
0.399
0.385
0.37%
0.588
0.386
0.391
0.62
0.542
0.434
0.609
0.628
0.764
Table 19: Average view: TTM Head probing Vs Full-shot End-To-End Training of popular time-series architectures averaged across FLs 162,333636 Head Probing (HP) involves finetuning the pre-trained model heads on full data with backhone weights frozen. TTM head probing results are either superior or highly competitive with other popular sate-of--hhe-al methods that are trained end-to-end on the target data, It is important to note that and training updates the backbone weights, whereas head probing does not.
25

---

TTM-A Forecast
Ground truth
Weather
90
-250
10-
-1-225
ETTH1
My May
-03
ETTH2
-10-
ETTM1
-15
ETTM2
-20
-1.0
ECL
M
SIN/COS
Figure 6: Sample TTM Zero-shot Forecasts across datasets
26

---

FL
M
TIME
TTМ,
LLMTime
ETTM2
96
0,137
01188
0.101
0.167
192
0.175
0.165
0.146
0.198
Weather
96
0.034
0.025
0.053
0.107
192
0.103
0.072
0.065
0.062
96
0.231
0.243
0,238
0.609
Electricity
192
0.495
0.495
0.491
0.96
Traffic
96
0.21
0.199
0.215
0.34
192
0.387
0.393
0.382
0.526
Model Size
1M
4M
70B
TTMp
f-umpt s-impiX)
26%
70000X
TTMg
f-impl %6) s-im((X)
36%
17500X
TTM,
F-mpl Se) s-imp(X)
36%
14000X
Table 20: ILM--Time Vs TTM: Zeroshot MSE reported last test window set. Results reported in LLMTime [1] used for this comparison.
TTMg
TTMg
TTM,
96
0.277
0.271
0.264
0.306
Electricity Weather ETTH2
192
0.334
0.324
0.321
0.389
0.357
720
0.408
0,388
0.395
0.434
96
0.158
0.166
0.159
0.21
192
0,206
0.214
0.264
336
0.256
0.254
0.247
0.326
720
0.328
0,319
0.314
0.402
0.166
0.157
0,152
0.409
192
0.191
0.174
0.179
0.41
336
0.207
0.195
0,193
0.439
0.25
0.243
TTMg r-imp( %
29%
TTMg f-imp(% %%)
30%
TTM, (-imp(%) %))
31%
Table 21: TTM vs UniTime MSE Improvement (f-imp) in zero-shot setting using full diding-window test Results reported in UniTime [8] used for this comparison.
10%
25%
30%
75%
100%
IMP
TTMg
0.422 0.421 0.413
0::02
0.398
SimMIM
0.535
0.466
TiMAE
0.660
0.594
0.55
0.475
0.466
24%
0.783 0.641 0.525
0.516
28%
LaST
0.645 0.610 0.540000.55000
0.479
0.443
23%
0.736 0.733
43%
CoST
0.784 0.624 0.540
4.4
0.428
26%
TS2Vec
0.655 0.631 0.599 0.577 0.517
31%
Table 22: Cross transfer learning MSE improvement (IMP) seff pre-training methods in various few-shot settings
Less PT Data
More PT Data
(250M samples)
(IB samples)
Less PT Data
More PT Data
Data
W/o RPT
w7 RPT
W/o RPT
RPT
(50M samples)
(1Bsamples)
0.365
0.364
Data
w/ AP
M/o AP
AP
ETTH2
0.285
0.28
0.285
0.277
ETTH1
0.369
0.365
ETTMI
0.413
0.384
0.341
0.322
ETTH2
0.283
0.285
0.275
0.277
ETTM2
0.187
0.194
0.18
0.171
ETTMI
0.446
0.322
Weather
0.154
0.153
0.158
ETTM2
0.191
0.187
0.172
0.171
Electricity
0.169
0.175
0.178
0.166
0.158
Truffic
0.518
0.518
0.528
0.514
Electricity
0.179
0.169
0.174
0.166
T-impi %1
0%%
Traffic
0.514
(-imp( %c)
3%
1.5%
Table 24: Impact of Resolution Prefix Tuning (RPT) in training (FT)) and more pre-training (PI)) data setting. Zero-shot results on FL 96 reported. RPT generally enhances the forecust performance espccially when the volume and diversity in the pretraining (PT) are high.
Table 23: Impact of Adaptive Patching(AP)
pre training (PT)) and more pre-traiinng (PT) data setting. Zero-sbot results on reported. generally improves forecasting accuracy acroes hoth setups, hut the impact more when PT data is less as AP enables modelling different resolutions different layers of the model. [wW': with. 'wlo': "without'.]
setling. adding learnable resolution prefix token greatly helps, it enables the models casily decouple the weights across resolations, However, in loss PT sctup where the challenges diversity modelling observed. RPT does not have much impaci. l'w": with, "wlo": "without", :]
27

---

TTMp SL
wio RPT
w/ RPT
BTTHI
0.373
0.358
BTTH2
0,180
0.179
0.559
ETTM2
0.127
0.108
Electricity Traflic
0.208
0.201
0.740
IMP (6)
8%
Table 25: Impuct
in context setting Zero shot results on FL. 24 reported. RPT helps in scenarios when context length (af) short. matically detecting the resolation becomes challenge for the model. Hence, by explicilly fusing the resolution information as prefix, enhunce the model's ability to learn effectively ncross resolutions. ['w':: with, 'waa:; 'withour", -]
scenarios, auto-
500
Nam
4.0
5000
0.075
0.000
800
Figure 7: Data segments TTM embedding analysis.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. The main claims made in the abstract and introduction (Section
accurately reflect the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Appendix Section H explains the limitations of this work and future directions.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
29

---

Answer: [NA]
Justification: Not applicable, as this work is grounded more on large-scale experimentation and empirical analysis.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed information on the full experimental setup, model hyper- parameters and dataset details. We also provide information on how each result of every baseline reported. Refer to Section D
D.4 for more details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
30

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Yes. The source code and model weight links for reproducibility are shared in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https://nips.. cc/
public/ gwwec///u/eeee/eeee/ CCdde ss ciooooolli onPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https /nniss. cc http///blccc/cceeee guw//uuiisssseeeee Code SeabbmsssioPooiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The relevant details are provided in Section D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The proposed approach and many of associated baseline papers do not report error bars as experiments fall under Foundation Models, which are computation- ally very expensive to pre-train for multiple seeds. However, TTM is compared with other ofte-off-the-ne ahe models across multiple settings (4 different forecast lengths and 3 different variants), wherein, TTM outperforms the baselines consistently in all these experiments, to give substantial evidence for our claims.
Guidelines:
The answer NA means that the paper does not include experiments.
31

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The experimental section has all the relevant details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttppss//eerrrs...cc///lll////p
Answer: [Yes]
Justification:
We have reviewed the details and conform the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[NA]
Justification: Not Applicable, as our work does not directly relate to the societal impacts in the ecosystem.
32

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Not applicable to our work, as no such misuse has been reported.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we followed the licensing and terms of use very carefully while architect- ing our design. All assets used this work will be credited properly.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
33

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperssw woperswithhoos com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, all source code files and associated scripts are well documented and will further be improved before open-source release.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Not applicable our work because human participants involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Not applicable to our work because no human participants were involved.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
34

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "hvQxUzlPQX", "reviewer": "Reviewer_LiJn", "summary": "This paper proposes a novel time-series pre-trained model TTM that instead of trying to over-parameterize the model, TTM tries to under-parameterize the model for better generalization ability. The model architecture is simple, straightforward, and easy to understand, coming along with advanced training strategies, such as adaptive patching, diverse resolution sampling, etc.\n\nEmpirical evaluations show the benefit of TTM in both state-of-the-art forecasting performance with other supervised, zero-shot, and pre-trained methods, as well as super-efficiency in training.", "questions": "1. Following the point about weakness, could the author intuitively explain why such few parameters can work well for time-series forecasting tasks?\n\n2. As the model is under-parameterized, my worry about TTM is that TTM can be a model of garbage in and garbage out (take it easy, I am saying this is bad, I am just very curious). Here is my concern:\n\nLet us make a comparison here, the proposed TTM is like a linear regression, very simple, while other over-parameterized models are like a polynomial/kernel regression. We all know that linear regression has less capability than polynomial regression for interpretations. However, for extrapolations, if the extrapolation data still follows the distribution of interpolation, then we can still expect polynomials to work better than linear. However, the reality is that extrapolation data does not follow the distribution of interpolation for most cases in time series (non-stationary problem), so in this case, both linear and polynomials can make mistakes. However because linear is less 'aggressive' in predicting trend changes due to the capability limit itself, it makes less wrong predictions than polynomials in many cases.\n\nThat is, given the fact that the datasets used in this work, such as ETT, are highly non-stationary, i.e., data distribution can change heavily as time goes. I might want to see two things:\n\nFirst, I want to see whether TTM can still show trend information in long-term forecasting. I wonder if the outperformance is because TTM is making 'conservative' and simple predictions that sacrifice the trend information because of the under-parameterization than other pre-trained models.\n\nSecond, I want to see some evaluations on some very easy stationary signals, such as sin/cos wave, and see that TTM can still work better than other pre-trained models.\n\n3. Does the training process involve using any synthetic data, such as simple sin/cos waves, which can be common practices in training other pre-trained models?", "limitations": "Please refer to the Weaknesses and Questions", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The idea of using an under-parameterized pre-trained model (1M parameters in TTM compared to millions or billions of parameters in other pre-trained time-series models) for better generalization ability seems to be novel and promising.\n\n2. The training strategies are clearly presented, the evaluations are comprehensive, and the results are impressive.", "weaknesses": "1. While the idea of using much fewer model parameters can work as presented in this paper, it is counter-intuitive as the current popular pre-trained models tend to be much larger than the proposed TTM. Beyond the empirical results, the paper does not go into deep intuitions of why this manner can work well in generalization, making the idea less convincing.\n\n2. Presentation may be improved, currently the presentation of figures and tables is too crowded.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you reviewer for the constructive feedback. Please find our response below:\n\n**Q1: Could the author intuitively explain why such few parameters can work well for time-series forecasting tasks?**\n\nThere are three important design choices of TTM that greatly enhance its forecasting accuracy despite its extremely small model capacity:\n\n1. All existing pre-trained models use a very high volume of pretraining data (TimesFM used ~300B and Moirai used 27B), hence they naturally require massive model sizes. However, as shown in Figure 3, Section 4.7, we observe that “limited” pretraining data with “high resolution diversity” greatly helps in time-series model generalization, as opposed to simply increasing the pretraining data size. This is an important observation and finding for the community that resolution diversity in pretraining data is very crucial for time-series FMs. Based on these findings, we proceed with a well-reduced dataset (1B) with high resolution diversity which naturally reduces our model size compared to counterparts needing to pretrain with several hundred billion time-series. We introduce a high diversity in our data via Diverse Resolution Sampling technique (DRS) which our counterparts fail to do.\n\n2. Secondly, we opted for TSMixer-based models instead of transformer-based models, which further reduced the model size drastically. The TSMixer architecture has successfully established in the past that interleaving simple gated attentions with mixing components across patches, channels, and features greatly enhances forecasting accuracies with very limited model capacity, as the quadratic time-complexity of self-attentions can be entirely avoided. After TSMixer, several other mixer architectures have been published, reiterating the power of these simple architectures. Thus, avoiding complex transformer architectures further reduced our model size significantly.\n\n3. Finally, we further increased the modeling power of TSMixer without drastically increasing its size by introducing several innovative components, such as adaptive patching/multi-scale patching, diverse resolution sampling, and resolution prefix tuning. These enhancements are crucial for effectively handling large pre-training across datasets with varying resolutions, all while keeping the model capacity very minimal.\n\nThrough these three innovative design choices, we managed to keep TTM as small as possible while outperforming state-of-the-art accuracies. \n\n\n**Q2: Does TTM predict the trends and seasonality well or it’s just doing conservative or simple predictions ?**\n\nYes. Thanks for raising this important concern. Kindly refer to the pdf attached in the common rebuttal section where we have shared several zero-shot forecasting samples of TTM on various datasets, showcasing its ability to model trends and complex seasonal patterns (both real-world and synthetic sin-cos as per the request). \n\n\n**Q3: Does the training process involve using any synthetic data ?**\n\nIn the current pre-trained models, we do not use any synthetic data. However, we augment more data via Diverse Resolution Sampling (DRS) technique to create different resolutions of existing datasets,  which greatly enhances the model performance (as shown in Figure 3, Section 4.7).\n\n\n**Q4: Presentation may be improved, currently the presentation of figures and tables is too crowded.**\n\nThanks for the feedback. We will address this in the final manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RdR2bNLzsY", "reviewer": "Reviewer_Csq6", "summary": "The paper introduces Tiny Time Mixers (TTMs), a series of pre-trained models designed for efficient zero/few-shot multivariate time series forecasting. TTMs are built on the lightweight TSMixer architecture and incorporate innovations such as adaptive patching, diverse resolution sampling, and resolution prefix tuning to handle varied dataset resolutions with minimal model capacity. These models outperform existing benchmarks in accuracy while significantly reducing computational requirements. The empirical studies demonstrate superior performance across multiple tasks and datasets.", "questions": "1. Given that the pre-training datasets may include time series with varying input lengths, how do TTMs manage these differences during training? Additionally, how do TTMs handle time series with input lengths not encountered before during inference?\n2. Considering that resolution prefix embeddings are learned discretely, how do TTMs accommodate resolutions that were not seen during training when encountered during inference?\n3. What criteria were used to determine the patch size for lagged exogenous features in the TTMs?\n4. The training and evaluation of TTMs primarily focus on long time series with hundreds of input steps. Does this focus impact the performance of TTMs when applied to shorter time series, such as those with fewer than 30 time steps?", "limitations": "The limitations regarding the restricted number of downstream tasks are discussed in the appendix.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper introduces innovative solutions to overcome the limitations of large pre-trained models in time series forecasting, offering techniques that could potentially be adapted to enhance other time series forecasting models as well.\n2. The empirical studies are robust, thoroughly assessing the model's accuracy and efficiency across multiple benchmark datasets.\n3. The experimental results are impressive, demonstrating enhanced accuracy and substantially lower computational demands compared to existing methods.\n4. The ablation studies are thorough, providing a detailed analysis of the impact of different pre-training datasets and the effectiveness of the proposed training techniques.", "weaknesses": "I did not see significant weaknesses that need to be addressed in this paper. As a minor suggestion, given that there is another mixer architecture known as TSMixer[1], it would be beneficial to include a footnote or a mention in the appendix to clarify that the TSMixer referenced in this work differs from the other one. The clarification will help avoid potential confusion and ensure the distinctiveness of the models is clearly understood.\n\n[1] Chen, Si-An, et al. \"Tsmixer: An all-mlp architecture for time series forecasting.\" arXiv preprint arXiv:2303.06053 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you reviewer for the constructive feedback. Please find our response below\n\n**Q1: Adding clarification note on the TSMixer architecture**\n\nThank you for the feedback. Sure. We will clarify the TSmixer used in this paper as compared to the other one.\n\n**Q2: Given that the pre-training datasets may include time series with varying input lengths, how do TTMs manage these differences during training?**\n\nDuring pretraining - we do a sliding window on every pretraining dataset to get them converted into several windows based on the TTM’s context and forecast lengths so that we can train the model with all the windows. In cases, when even a single window cannot be created as the training data is extremely small, then these datasets are skipped during pretraining. Appendix Table 8 lists the datasets which were NOT skipped as part of this filtering step. \n\n**Q3: How do TTMs handle time series with input lengths not encountered before during inference?**\n\nDuring inference, if the user-provided input length is greater than the TTM-configured length (K), we use the last-K time-points as the input context. On the other hand, if the user-provided length is shorter than the configured TTM length, we have 2 options: if minor adjustments are needed - then we can prepend zeros to virtually extend the length. However, for major adjustments - it is preferable to quickly pre-train another TTM with the required shorter length for enhanced accuracy.  Note that, pre-training shorter context length TTM is computationally inexpensive and can be quickly trained in a matter of few hours. Even, pretraining TTM on very long context lengths can be achieved in less than a day, a notably faster time as compared to existing counterparts which often take several days to weeks. As part of this paper, we will release and open-source a few pre-trained TTMs with different forecasting contexts that can widely cover most of the common enterprise use-cases across industries. \n\n\n**Q4: Considering that resolution prefix embeddings are learned discretely, how do TTMs accommodate resolutions that were not seen during training when encountered during inference?**\n\nIf we encounter unseen resolutions, we recommend the user to either use Out-of-Vocabulary (OOV) token configured as part of the pre-trained model to accommodate these unseen resolutions or use the TTM model pre-trained without the Resolution Prefix Tuning (RPT) module. \n\n**Q5: What criteria were used to determine the patch size for lagged exogenous features in the TTMs?**\n\nPatch size for lagged exogenous features is mostly a hyperparameter and depends on the target data-set characteristics.  Please note that, Exogenous Mixer block is introduced only during the finetuning phase (on the target-data) where these parameters can easily be configured based on the target data characteristics. In general - at least a patch length of 3 or more is suggested so that we have at least one or more forward and backward lags across all channels for effective inter-channel correlation modelling of exogenous signals.\n\n**Q6: The training and evaluation of TTMs primarily focus on long time series with hundreds of input steps. Does this focus impact the performance of TTMs when applied to shorter time series, such as those with fewer than 30 time steps?**\n\nIn general - TTM is pre-trained with higher context length (512, 1024 and 1536) as longer context naturally gives more information for the model to learn better and enable transfer learning. However, TTM also performs well with shorter context length too. For Example. APP, SER and CC data in our benchmarks require a shorter context length of 96 and TTM pretrained with context length 96 works pretty well and outperforms other benchmarks by a good margin (Table 6).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KuaWpTesIG", "reviewer": "Reviewer_qDvN", "summary": "This paper introduces Tiny Time Mixers (TTM), a new compact pre-trained model for efficient zero/few-shot multivariate time series forecasting. TTM is based on the lightweight TSMixer [1] architecture and incorporates several innovations, which enable effective pre-training on varied dataset resolutions with minimal model capacity. Besides, TTM also employs multi-level modeling to capture channel correlations and fuses exogenous data into the forecasting process during fine-tuning.\nThe authors comprehensively evaluate TTM on multiple datasets and compare their performance with other state-of-the-art models. The results highlight TTMs' superior accuracy, computational efficiency, and lightweight nature.", "questions": "see weakness", "limitations": "see weakness", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Enhanced Zero/Few-shot Forecasting Performance**: TTMs demonstrate a substantial improvement in zero/few-shot forecasting capabilities, outperforming existing benchmarks by 4-40%. This advancement is particularly notable as it shows that smaller models can achieve high accuracy without the extensive computational resources typically required by larger models.\n2. **Specialized Techniques for Pre-training and Fine-tuning**: The paper presents novel techniques such as adaptive patching, diverse resolution sampling, and resolution prefix tuning. These innovations enable robust pre-training on heterogeneous datasets and facilitate effective fine-tuning, allowing TTMs to capture channel correlations and integrate exogenous signals, which is critical for accurate multivariate forecasting.\n3. **Innovation in Model Architecture**: The paper introduces Tiny Time Mixers (TTMs), a compact pre-trained model architecture tailored for multivariate time series forecasting. Starting from just 1 million parameters, TTMs offer a lightweight and efficient alternative to larger, more computationally intensive models. This addresses the need for fast and resource-friendly forecasting tools.", "weaknesses": "1. The base model TSMixer used for pre-training is not newly proposed.\n2. TTM necessitates to train different models for different context length and forecast length, posing some new weaknesses compared to Transformer-based pre-training models. While this paper presents forecast length adaptation strategy to handle the fixed forecast length problem, performance loss still emerges when there is a disparity between the model's forecast length and the actual forecast length. \n3. There are not sufficient experiments to prove the scaling law on pre-training models. The paper's results show that the TTM performs best with 5M parameters. Will the TTM perform better with a larger number of parameters? Showing the tradeoff between performance and cost along with parameters size increasing will facilitate understanding of the model and help choose the most suitable model size.\n4. Some parts of TTM lack ablation studies to sufficiently prove their effectiveness, such as decomposing the exogenous mixer module and decoder channel-mixing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you reviewer for the constructive feedback. Please find our response below\n\n**Q1: The base model TSMixer used for pre-training is not newly proposed.**\n\nYes, you are correct that TTM builds on the TSMixer foundation. However, TSMixer doesn’t discuss the techniques required to construct a pre-trained model with effective transfer learning capabilities. To achieve this, we have introduced several innovative components on top of TSMixer, such as adaptive patching, diverse resolution sampling, and resolution prefix tuning. These enhancements are crucial for effectively handling of pre-training across datasets with varying resolutions, while maintaining minimal model size. Section 4.7 details the positive impact of these techniques through comprehensive ablation studies. With the help of these novel components and the learned pre-trained weights, we outperform TSMixer by 15% in few-shot setting, as highlighted in Table 4. \n\n**Q2: TTM necessitates to train different models for different context length and forecast length, posing some new weaknesses compared to Transformer-based pre-training models. While this paper presents forecast length adaptation strategy to handle the fixed forecast length problem, performance loss still emerges when there is a disparity between the model's forecast length and the actual forecast length.**\n\nRecent advancements in language and vision have seen a shift towards adopting small, focused foundation models over large, all-purpose ones (SLMs Vs LLMs) considering the ease and effectiveness for production deployments and high task-specific accuracy of small focused models.   We have applied a similar strategy to time-series foundation models by designing TTMs that are small, focused, and tailored for specific forecasting contexts. These models are particularly well-suited for production enterprise applications, which often require minimal GPU resources for deployment and enable rapid finetuning without the risk of overfitting, challenges that are more pronounced with massive transformer models. \n\nIn addition,  pretraining TTM is computationally inexpensive and can be quickly trained in less than a day, a notably faster time as compared to existing counterparts which often take several days to weeks. Hence pre-training multiple TTM has no practical challenges or limitations and can easily be achieved.  We would like to highlight that, as part of this paper, we also plan to release and open-source a few pre-trained TTMs with different forecasting contexts that can widely cover most of the common enterprise use-cases. \n\nIn addition, we also support several forecast length adaptation techniques to adapt a pre-trained model for different forecast lengths with extremely minimal accuracy impact. Particularly, the pruning technique has found to be very effective (Ex. only 0.8% drop in MSE when more than 50% pruning is applied for the scenario of pruning forecast length from 720 to 336.) Please see Figure 4 for the details.\n\n**Q3: There are not sufficient experiments to prove the scaling law on pre-training models. The paper's results show that the TTM performs best with 5M parameters. Will the TTM perform better with a larger number of parameters? Showing the tradeoff between performance and cost along with parameters size increasing will facilitate understanding of the model and help choose the most suitable model size.**\n\nThe scaling laws of TTMs are influenced by the amount of pretraining data used. When training the TTM on Monash data alone (~250M samples), model accuracy saturates beyond 1M parameters (this represents the TTM-Quick model referred in the paper). \nHowever, by expanding the pretraining data to 1B samples by integrating Monash with additional data sources, accuracy improvements continued with increased model size, reaching saturation around 5M parameters (the TTM-Advance Model referred to in the paper). Further increasing the model size beyond 5M did not provide additional benefits for the 1B pretraining dataset. We will include these findings in a new section on scaling laws covering various model sizes. Thank you for bringing up this important aspect.\n\n**Q4: Some parts of TTM lack ablation studies to sufficiently prove their effectiveness, such as decomposing the exogenous mixer module and decoder channel-mixing.**\n\nIn general, the Exogenous Mixer and Decoder Channel-Mixing components are designed to be used together. The Exogenous Mixer captures channel correlations in the forecasts, while the Decoder Channel-Mixing captures these correlations in the past context. Using both components together allows the model to learn channel correlations from both forecasts and past contexts, providing a comprehensive view.  Running one without other will have information loss in the channel-correlation. Therefore, in the ablation study, we reported them as a single component.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0ituibwVfU", "reviewer": "Reviewer_TN2b", "summary": "It presents a lightweight pretrained model for TS with strong performance.", "questions": "In the weakness", "limitations": "not applicable", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, this is a strong paper with extensive experimental work.", "weaknesses": "Adaptive patching is a well-conceived design, although I am unclear why it is termed \"adaptive\" when the design appears to be fixed and pre-set. Perhaps \"multiscale patching\" would be a more accurate descriptor. Additionally, the writing should more clearly distinguish between the designs used for pretraining and those used solely for finetuning.\n\nFor the full-shot head probing, it is important to include comparisons with state-of-the-art end-to-end methods, as readers are likely interested in such comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you reviewer for the constructive feedback. Please find our response below\n\n**Q1: Adaptive patching is a well-conceived design, although I am unclear why it is termed \"adaptive\" when the design appears to be fixed and pre-set. Perhaps \"multiscale patching\" would be a more accurate descriptor.** \n\nYes, we agree with the suggestion to rename it to multiscale patching. We initially used the term “adaptive patching” to indicate that the patch length changes across layers. However, since this change is not runtime adaptation, renaming it to multi-scale patching is indeed more accurate. \n\n**Q2: Additionally, the writing should more clearly distinguish between the designs used for pretraining and those used solely for finetuning.**\n\nSure.  All the multi-variate and exogenous related components are part of the fine-tuning flow, while the rest of the components are used in both pretraining and finetuning.  We will clearly distinguish between the designs used for pretraining and those used solely for finetuning in the revised manuscript. \n\n**Q3: For the full-shot head probing, it is important to include comparisons with state-of-the-art end-to-end methods, as readers are likely interested in such comparisons.**\n\nDue to space constraints, we couldn’t add the full end-to-end methods in the main result section. However, these results are available in the appendix (Table 14) where TTM with Head probing (HP) consistently outperform other HP benchmarks and are also superior or very competitive as compared to the full end-to-end training of popular TS architectures. This demonstrates that  TTM with simple head probing is both highly effective and extremely lightweight, as it avoids the need to update the backbone weights. We will move this appendix table to the main result section in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you reviewers for your time, effort, and valuable feedback on our paper. We have clarified your queries in the respective sections. We also extend our gratitude to the Area Chairs  and all the PC members for investing their valuable time throughout the review process. 

**Short summary:** In 2024, the landscape of time-series forecasting has been dominated by large pre-trained models. Large Transformer-based pre-trained models like TimesFM, Moirai and Moment published at ICML 2024, have garnered significant attention within the time-series community. However, these models are massive, often requiring several hundred millions of parameters, and they face challenges in supporting faster runtime, quick fine-tuning, and integrating exogenous data. In contrast, our study introduces several innovative design modifications and enhancements to traditional model architectures, resulting in an exceptionally small model starting from just 1 million parameters that outperform the existing state-of-the-art models by a notable margin. Moreover, our model offers several other practical advantages such as faster inference, fine-tuning, explainability, exogenous data infusion, and compatibility with CPU deployments—features that are highly valued in industrial forecasting but often lacking in current SOTA models.  TTM seeds the first effort towards building light-weight pre-trained models for time-series forecasting and we believe that our model will inspire numerous exciting research endeavours in this area.

We have clarified all the reviewer’s queries below in the respective sections. Thank you.

Meta-review: The paper presents TTMs, small, pre-trained time-series models. The architecture is based on TSMixer but with a number of improvements to enable the models to transfer well (both zero-shot and finetuning). The performance, both in terms of forecasting quality and latency appear impressive. The reviewers unanimously recommend acceptance, and I agree with this assessment. Congratulations to the authors!
Decision: Accept (poster)