UniTS: A Unified Multi-Task Time Series Model
Authors: Shanghua Gao
Teddy Koker
Owen Queen
Thomas Hartvigsen
Theodoros Tsiligkaridis
Marinka Zitnik
Venue: NeurIPS 2024 poster (2024)
Keywords: Time Series Forecasting
Time Series Classification
Time Series Imputation
Time Series Anomaly Detection
Prompt Learning
Pretraining
Few-Shot Learning
Unified Model
Multi-task
Task Tokenization
Primary area: deep_learning_architectures
TL;DR: UniTS is a unified multi-task time series model that can process predictive and generative tasks across time series domains.

Abstract: Although pre-trained transformers and reprogrammed text-based LLMs have shown strong performance on time series tasks, the best-performing architectures vary widely across tasks, with most models narrowly focused on specific areas, such as time series forecasting. Unifying predictive and generative time series tasks within a single model remains challenging. We introduce UniTS, a unified multi-task time series model that utilizes task tokenization to integrate predictive and generative tasks into a single framework. UniTS employs a modified transformer block to capture universal time series representations, enabling transferability from a heterogeneous, multi-domain pre-training dataset—characterized by diverse dynamic patterns, sampling rates, and temporal scales—to a wide range of downstream datasets with varied task specifications and data domains. Tested on 38 datasets across human activity sensors, healthcare, engineering, and finance, UniTS achieves superior performance compared to 12 forecasting models, 20 classification models, 18 anomaly detection models, and 16 imputation models, including adapted text-based LLMs. UniTS also demonstrates strong few-shot and prompt capabilities when applied to new domains and tasks. In single-task settings, UniTS outperforms competitive task-specialized time series models. Code and datasets are available at https://github.com/mims-harvard/UniTS.

UnITS: A Unified Multi-Tasie Time Series Model
Shanghua Gao
Teddy Koker MIT Lincoln Laboratory
Harvard University
Laanghua_gaa@hmg harvard. .dd
maixoer@mit.com
Owen Queen
Thomas Hartvigsen
Theodoros Tsiligkaridis MIT Lincoln Laboratory ttsili@11 mit.. edu
Harvard University
University Virginia
qwen_queenOhhs. harvard. edu
hartviggees&iirginia. edu
Marinka Zitnik Harvard University
marinkaChms. wwwpwhhw.........e.ee.. www.aaadd edu
Abstract
Although pre-trained transformers and reprogrammed text-based LLMs have shown strong performance on time series tasks, the hest-performing architectures vary widely across tasks, with most models narrowly focused on specific areas, such as time series forecasting. Unifying predictive and generative time series tasks within single model remains challenging. We introduce UNITS, unified multi- task time series model that utilizes task tokenization to integrate predictive and generative tasks into a single framework. UNITS employs a modified transformer block to capture universal time series representations, enabling transferability from a heterogeneous, multi-domain pre-training dataset- characttize by di- dynamic patterns, sampling rates, and temporal scales downstream datasets with varied task specifications and data domains. Tested on 38 datasets across human activity sensors, healthcare, engineering, and finance, UNITS achieves superior performance compared to 12 forecasting models, 20 classification models, 18 anomaly detection models, and 16 imputation models, including adapted text- based LLMs. UNITS also demonstrates strong few-shot and prompt capabilities when applied to new domains and tasks. In single-task settings, UNITS outperforms competitive task-specialized time series models. Code and datasets available at hhttts: htttp:://githut com/mims hwtersahaaarrddddd
wide range of
Introduction
Foundation models, particularly large language models (LLMs), have transformed deep learning by enabling a single pre-trained model to support multiple tasks, eliminating the need for task- specific models. Language and vision models (9 101
92 50 32 can be adapted to new tasks
with minimal additional training through approaches such as multi-task learning [25], few-shot learning [108] (86), and prompting [66]. Beyond language and vision, there is growing need for similarly versatile models in time series that can accommodate data from diverse domains-includin medicine (34). engineering [102) and science 48 and support wide range of tasks, such as forecasting, classification, imputation, and anomaly detection.
Developing multi-task time series models that unify predictive and generative tasks under single framework remains an open challenge. Time series datasets temporal scales, sampling rates, and dynamic patterns, making them complex manage [24 78
span multiple domains and exhibit varied
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Existing models often fall short in adaptability, as they either struggle to handle samples with varying 67 14 or treat each variable as independent, overlooking important interdependencies [82] Time series tasks are also highly diverse, encompassing distinct objectives and specifications across generative and predictive tasks. For example, generative forecasting tasks aim to produce future values within a time series, while predictive tasks may involve making discrete predictions for entire samples. Additionally, task requirements can vary significantly even within the same task type; for instance, generative tasks may involve different forecast lengths, and predictive tasks may feature multiple classification categories. As a result, time series models have mainly remained task-specific, with unique architectures typically designed and trained from scratch for
numbers of variables T12
forecasting [67 82 119), classification [30] [131, or other specialized tasks [16
[12]
Recent efforts
to pre-train unified models [36] [22 or adapt LLMs for time series [18] 12
129
47
97
100] still
heavily depend on extensive fine-tuning or the addition of task- and dataset-specific modules. Some models have explored generative pre-training transformers specifically for time series forecasting [I0
118. 47 28 reporting strong results but focusing exclusively on forecasting without addressing other types of time series tasks. Consequently, these approaches require users to design and train new modules for each task or limit their application to a single type of tasks. To achieve a versatile, unified time series model-akin to foundational models in vision and language that operate across unified task spaces-a model must accommodate both generative and predictive tasks. Such a unified model would leverage a single set of weights for multiple tasks, removing the need to develop task-specific models from scratch. This approach would support a broad range of tasks and facilitate rapid adaptation to new datasets.
Present work. To address these challenges, we introduce UNITS, a unified multi-task time se- ries model capable of handling a broad spec- trum of time series tasks. We rigorously com- pare UNITS against 12 forecasting methods, 20 classification methods, 18 anomaly detec- tion methods, and 16 imputation methods, in- cluding transformer-basee L.MM-based RNN- based, and traditional approaches, to highlight UNITS's generalizability to new tasks. This ca- pability is achieved through the following model design: 1) Task tokenization: UNITS encodes task specifications into unified token repre- sentation, enabling universal task specification without post-hoc architectural modifications. 2) Unified time series architecture: UNITS pro- cesses heterogeneous time series data with vary- ing numbers of variables and sequence lengths without altering its network structure. To accomplish this, UNITS employs self-attention across time and variable dimensions to adapt to diverse temporal dynamics. We introduce dynamic linear operator to model complex relationships between data points along the time dimension and a module reduce interference in the feature space of hetero- geneous data. 3) Suppori for generative and predictive tasks: The combination of universal task specification and a unified time series architecture allows UNITS to share weights across tasks by co-training on multiple datasets. We use a masked reconstruction pre-training approach, enabling UNITS to be jointly optimized for generative and predictive tasks.
Partly Cloudy Healhy Clsoficction
Forcoasting
Impactation
Anomaly Detection
UniTS
Ta Wtop
time
G craraee
Volictoo
Domain: Weather
38 Dacansads
Domain FDG
Figure 1: UNITS is unified multi-task time series model for predictive and generative tasks.
In the single-task setting, where models are trained individually for each dataset, UNITS outper- forms task-specialized time series models and repurposed LLMs across forecasting, classification, anomaly detection, and imputation. In a challenging multi- domti, multi-task setting, we find that a single searrd-weight UNITS model successfully handles 38 tasks, demonstrating its versatility as a multi-task time series model. UNITS surpasses top baselines that rely on data- and task-specific modules, achieving the highest average performance across tasks and excelling in 27 out of 38 tasks. Additionally, UNITSS supports prompt-based learning and direct multi-step forecasting with flexible sequence lengths, capabilities not offered by models using task- and data-specific heads. In direct multi-step forecasting, UNITS outperforms the strongest baseline (which uses a sliding-window approach) by 10.5%. UNITS can also adapt to new tasks through parameter-efficient prompting, achieving results comparable to its fully fnnet counterpart. For example, across 20 forecasting datasets, prompted UNITS slightly outperforms the fully fine-tuned model, reducing MAE from 0.381
2

---

to 0.376. Furthermore, UNITS demonstrates effective few-shot transfer. successfully addressing tasks like imputation, anomaly detection, and out-of-domain forecasting and classification without requiring specialized modules. For instance, UNITS improves on the strongest baseline by 12.4% in MSE on imputation and 2.3% in F1-score on anomaly detection. UNITS paves the way toward unified time series models, offering strong performance and adaptability across tasks and domains.
2 Related Work
Traditional time series modeling. Time series analysis has been extensively explored in both the statistics and machine learning communities for many years [5
[03
[23 [8 g0]. Numerous neural
architectures have been developed for specific time series tasks such as forecasting [14] 65 68
67 1071, classification 115 70], anomaly detection 225
56 16]. and imputation [17 49
Task-specific models are typically trained via supervised learning on individual datasets, necessitating specialized modules. For example, classification model requires classification head with specific number of classes, while data processing modules must handle a predetermined number of variables. In contrast, UNITS aims to unify various tasks into a universal task specification, enabling the handling of diverse data with a single, unified network architecture. This approach facilitates training a multi-task model capable of addressing multiple time series tasks.
General time series modeling, Foundation models, including language models 9 101 and vision models 62 55], are trained on broad data scale to address diverse tasks with no or minimal additional training [8]. Recent studies in time series analysis have sought to develop models with similar capabilities. This includes developing novel architectures
capture diverse time series signals. For instance, TimesNet [12 uses multiple frequency- brauueney-bbseed features obtained through Fourier transform capture complex time series signals. There have been several efforts to reprogram LLMs for time series tasks [21] [19 47 [0]. Models such as GPT4TS [129] and Time-LLM [47] adapt LLMs by fine-tuning their embedding layers or aligning time series samples with LLsed text prototypes (e.g.. GPT-2 19)) Unlike these models, UNITS is trained exclusively on time series data rather than relying on LLM architectures. Another approach, Lag-Llama 190). pre-trains a model on time series data from multiple domains specifically for forecasting tasks. Similarly, the Moment model [36] is pre-trained on a diverse range of time series data. However, these approaches still require task-specific modules and tuning for each task. In contrast, our UNITS model supports generative and predictive tasks without requiring extensive task-specific model adjustments.
Prompt learning. Prompt learning has emerged as an efficient method for task adaptation in large models 555 88 121
13 42 Some approaches construct prompts directly in the model's s input domain, such text prompts for LLMs (2). Other methods involve tuning soft token inputs to frozen language models (58). In time series, PromptCast [118] and LLMTime [81] convert time series data into prompts for LLMs to facilitate forecasting. TEMPO [TO] is another prompt-based approach that uses learned set of prompts for LLL--aased forecasting applications, while GPT4MTS [46] integrates both textual and numerical data to fine-tune LLMs for forecasting. In contrast, UNITS is trained exclusively on time series data, climinating the need for computationally expensive pre-trained LLMs. Moreover, the universal task tokenization enables frozen UNITS to adapt to new tasks beyond forecasting, such as classification and imputation. Further discussion of related work can be found in Appendix A
3 Problem Formulation
Notation. We are given a set of multi-domain datasets D = (D,li 1,....,, .n.)
where each dataset D. can have a varying number of time series samples; samples can be of varying time lengths and have varying numbers of sensors/variables. Each dataset is described as D, = (2,, V.). where X; denotes time series samples and V/ specifies a task defined on X, Let X' and y be collections, defined as X' [i = -1....,,, and y = = 1..... .n.. respectively. A time series sample in datasets is denoted as x E R where t and v are the length of the time series sample and the number of variables, respectively. We use time dimension and variable dimension to indicate the row and column dimensions in x. Di contains four common time series tasks: forecasting, classification, anomaly detection, and imputation. Further, each task type can be instantiated in numerous ways, e.g., forecasting over different time lengths and classification with varying numbers of classes. We use F(X, 0) to denote multi-task task model trained on X. See Table 12 for notation details.
3

---

Forcaasing
Glass
31-
3/ Cleartozton
UnITS
Unpatiiliy
UNITS
Disance Monsing
Healhy
a) Generative Tasks
b) Predictive Tasks
Types of
Proorpt tnken
Semple token
MHBA
Time
Varidbie MHHEA
Dynemic FPN
GEN tokon
CLS
ToweY
GLS takon
UniTS Architecture
Figure 2: a) UNITS for forecasting; imput is tokenized, and GEN tokens are un un-pacchiiied to infer the forecast horizon. b) UNITS for classification; CLS token is used to represent class information and then compared to class tokens to get prediction class. c) Architecture UNITS model.
Desiderata for a unified multi-task time series model. Unlike specialized time series models designed and separately trained for each specific dataset D., unified time series model F(X, 0) is a single model with weights that are shared across all types of tasks and satisfies the following three desiderata: 1) Heterogeneous time series: To process time series from all sources, the model F must be agnostic with any imput samples in X, given the heterogeneity in time series lengths and variable counts in time series samples from various sources. 2) Universal task specification: For easy multi-task support and swift adaption to new tasks, the model F should adopt a universal task specification F(X,0) 0) Y applicable across all type of tasks y. 3) One shared model: Sharing weights across tasks enables the unified model to handle multiple tasks simultaneously. contrasts with existing methods that typically train separate models on task-specific datasets, often involving elaborately tuned training parameters.
To realize the above desiderata, UNITS supports multi-task, prompt-based, and few-shot learning. Multi-task learning: UNITS specifies single model F(X, 0) > y for tasks y defined on datasets X. Multi-tast learning showcases the flexibility of the model learn across time series domains and tasks. Prompt learning: By leveraging prompt tokens, UNITS supports prompt learning, Prompting (XX 0), token) - VM across tasks while keeping the model frozen. Additionally, UNITS can be trained a single-task manner, following the same setup as used by many existing models. Other settings described in Appendix C.1
UNITS Model
UNITS is multi-task model with unified network architecture. It uses token-based format to describe tasks and time series from different domains. We introduce a novel approach with three distinct token types: sample, prompt, and task tokens, each serving unique purpose in time series analysis. The input time series sample tokenized into sample tokens. Prompt tokens provide essential context for the task, guiding the model to accomplish the user-specified task. Task tokens (GEN and CLS) are combined with other tokens and used for generative and predictive tasks. UNITS then converts task tokens into task predictions produce the final model output. Unlike transformers such as PatchTST [2]. UNITS introduces new token types: sample tokens allow for modeling of multivariate time series, prompt tokens enable efficient multi-task and prompt learning [I01], and task tokens unify predictive and generative tasks into one format.
4.1 Prompting UNITS with Unified Time Series Data Tokens
We introduce how to unified tokens unify different task types and data for inference. Tokens on different network layers have the same shape, so we omit the layer index for simplicity.
Sample tokens. We divide time series input sample E Rixe into patches along the time dimension using a non-overlapping patch size of k. linear layer projects each patch into an embedding vector of length d, obtaining sample tokens Zx E sxxxxx where t/k. Since and vary across time

---

series data domains, we keep the variable and time dimension in tokens. 2x are then added with learnable positional embeddings.
Prompt tokens. Prompt tokens 2p E Rexexd are defined as learnable embeddings, where is the number of tokens. In multi-task setting, each dataset has its own of prompt tokens. These tokens incorporate the specific context related to the data and the task the model needs to complete. For each sample in the dataset, these prompt tokens are appended to the sample tokens and sent to the network to provide context information about the current sample. For prompt learning, with the pre-trained model weights being frozen, UNITS adapts to new tasks by utilizing prompt tokens learned with the prompt tuning. Prompt learning is more efficient than tuning new data/task-specific heads and achieves comparable performance to full model fine-tuning, as shown by few-shot learning experiments new tasks (Tables 4 and 5) and new datasets (Table 3
Task tokens. In Figure 2bb, we categorize task tokens into two types: 1) GEN (Generation) tokens used in forecasting, imputation, and anomaly detection, and 2) CLS (Classification) tokens, which are used for classification tasks (in given task, the number of CLS tokens corresponds the number of classes in the task). Task tokens define general format for representing tasks and support flexible adaptation to new tasks. For tasks involving forecasting, in Figure 2p. the GEN token Zm RR. lxxd
is replicated f-times based on desired forecasting length to get 2m E IR [xxxx These tokens 2m, are then concatenated with the sample and prompt tokens and fed into the UNITS network:
Fore CA(Zp; Ex, 2m E [R(++s+)xxxdd
(I)
where CA is the concatenation operation along the time dimension. At the output of the model, embedding vectors with length in 2m are unpatchified to patches with size e to obtain the forecasting
sample &, i.e. & = Proj (m E R [fxe)xxr This approach allows the UNITS model to perform direct
multi-step forecasting [99
76 119 over arbitrary time lengths, as illustrated in Figure 3 For
classification, in Figure 2b,
CLS token Ze E Blxexd concatenated along the time dimension with
the prompt and sample tokens, resulting in:
Zpred = CA(Zp, Ex, Ze) E F (g(+*+1))xx
(2)
which is then fed into the model. We define class embeddings Zg E Rexexd for each of classes in the task. These class embeddings are either trained or generated by averaging CLS tokens of training samples in each class. Finally, the class for sample is predicted by finding the class embedding vector in ze that is the closest to the CLS token z, from the model output:
Class = argmin ||ze
Ze, 112, i E (),
(3)
For imputation, missing values are imputed using the GEN tokens. For anomaly detection, the model takes time series sample containing any number of potentially anomalous values, generates the output sample by reading out the sample tokens, and then determines anomalous values based on the reconstruction error between the input sample and the generated sample. Details using tokens for imputation and anomaly detection are in Appendix C.2 achieve their functions.
All tokens and embeddings are trained to
4.2 Unified Network Architecture in UNITS
Time series samples can have varying numbers of variables, temporal dynamics, and time lengths across different domains and types of tasks. UNITS uses modified transformer architecture [104] to handle heterogeneous multi-domain data with varying dynamics and the number of variables (Figure 2b.. In the following, we describe key modules of UNITS architecture. Note that UNITS can also be used with other backbones, such as Mamba (38)
Time and variable self-attention. We use a two-way self-attention to both variable and time dimensions. This approach contrasts with previous methods that apply self-attention to either time [82] or variable dimension (67). but not to both dimensions. Time and variable self-attention effectively handle time series samples with various numbers of variables and different time lengths t. DyLinear. We modify the transformer block by adding dynamic operator (DyLinear) into the feed-forward network layer (FFN). This modification enables the FFN to capture dependencies between tokens. In contrast to the standard FFN, which processes embedding vectors on point-wise basis, DyLinear uses weight interpolation to accommodate varying time lengths. Given a sequence of
5

---

sample tokens Z E Rluxd, DyLinear interpolates weights w E R to accommodate varying time lengths as follows:
DyLinear(z,, w) = WiterpZe E R d, Winterp = Interp( w) E I2 xii
(4)
where Interp is a bi-linear interpolation to resize w from shape Wout X Win to lout x In to match the input and output length. DyLinear captures dependency patterns across time series samples, which leads to improved performance on generative tasks (Table (3).
Gating module. We add gating module after each layer to mitigate interference in the latent repre- sentation space caused by multi-domain and multi-task datasets F This module dynamically re-scales features in layer-wise latent spaces and promotes the stability of latent representations.
Generative and predictive towers. We design a shared GEN tower (Haww) and CLS tower ( (HaL)) for transferring GEN/CLS tokens generate time series samples and classification classes, introduced in Section 4.1 Unlike existing works that use standalone, task-specific heads for individual datasets, our approach leverages GEN tower and CLS tower for all generative and predictive tasks, respectively, ensuring a more unified and efficient model architecture.
The UNITS architecture includes the backbone network composed of N modified transformer blocks described above, a CLS tower, and a GEN tower. Implementation details are in Appendix Ablations in Appendix [F] verify the effectiveness of this architecture.
4.3 UNITS Model Training
Unified masked reconstruction pre-training. To enhance UNITS'S abilities to 1) learn general features applicable to both generative and predictive tasks and 2) efficiently adapt to downstream tasks via prompt learning, we introduce unified mask reconstruction pre-training scheme. leverages the semantics of both prompt and CLS tokens (Section 4.1) for masked reconstruction pre-training, therefore learning representations for both generative and predictive capabilities. This is distinct
from pre-training strategies that use either generative [82] 20 26 [54] or predictive [72 109 117
29 124, (7) approach. Unlike these approaches that pre-train only the model backbone, our strategy pre-trains all components of UNITS, including the backbone and GEN/CLS towers (Section 4.2).
enabling prompt and zero-shot learning over a frozen pre-trained model. For each time-series sample x, handful of sample tokens get masked and replaced with GEN tokens. These masked sample tokens is then concatenated with prompt tokens and CLS tokens, sent to the UNITS backbone network. In the unified pre-training loss, tokens from the backbone network output are sent the CLS/GEN towers to reconstruct the input sample x, formulating as follows:
(5) LMSE the MSE loss predict the full sample x. For the left side of the loss, prompt token Zp is sent along with sample token Zx to GEN tower HςEN to help with the reconstruction. For the right side of the loss, to leverage the semantics of the CLS token and train the CLS tower HeLS for predictive tasks, Zpred (Eq. 2 from the model output is processed by the CLS tower HaLs to get classification-reltted embedding vectors Pred = HaLs (PPrea), and another GEN tower Haen takes in âpred and 2x to predict the full sample. HAEX is only used for pre-training and will be removed for downstream tasks. This unified pre-training strategy involves pre-training both tokens, the backbone network, and the GEN/CLS towers for both generative and predictive abilities.
Lpeetrain LMSE (Zp 2x), x) + LMSE MaaSE((Aeze (Hegg (HcLs((Prec) Zx), x).
Training UNITS models. We implement and evaluate two UNITS models, each trained in a different regime. We start with pre-trained UNITS that is optimized using self-supervised Lpretain in Eq. and trained across a collection of multi multi-domain datasets. Given a self-supervised pre- tre-trained UNITS whose weights are frozen, consider fine-tuned model where only tokens for predictive or generative tasks are fine-tuned (denoted as UNITS- PMMT in Experiments). We also consider a standard multi-task supervised learning regime, where a single UNITS model is trained from scratch to simultaneously perform many tasks (denoted as UNITS-SUP in Experiments). These two regimes use multi-task setup, where a single model is trained and tested on multiple tasks and datasets. During multi-task training, we sample batches of time series samples and aggregate dataset-centric loss values: Local = E
= (D:), where L; is the loss of batch i, X is the weight for each loss, and / denotes the number of batches. We follow [112] and use the MSE loss for forecasting and cross-entropy loss for classification. For fair comparison with models trained in single-task manner, we follow the experimental setup of [112 67 and benchmark UNITS in single-task setting (denoted as UN in Experiments), where the model is trained separately on each dataset/task.

---

Forecasting URITS-ST lTrnasfommnr
RLiseu 10
PuccTTT
Crossformer
TIDE
Temesser
DLinew
SCINer
FEDGomer
Stationay
Auteformer
36 datasets
(Ouns)
D7
82
ITC
63
[6
60
[EC)
Metric
MSE MAE
MSE
MSE
MAE
MSE
MAP
MAE
MSF
MAE
ETTml
0.377
0.4977.407
0.410
h.414 0.407
0.887
0.403/5113
0:496
0:499
44190.400
406
0.400 0.407
0.885 0.41
0.452
0.456
0:517
ETTm2
0.275 0.323 0.288 0.332
K.256
0.327
0.281 126
0.757 0.600
0.358
0.404 (,29] 0.353 0.350 0:401
0.571 0.533 805 0 549 0.306 3.34 0.327 0.971
ETTb]
0.483
0.424 0.454 0.447
0.446
0.434
0.469 0.454 0.522 0.522
0.544 (1:507 [1458 0.450 0.456 0.452
0.347 0.647 0.440 0:460 0.537 (.496 0.457
ETTb2
0.366
0.395
0.574 0.39%
0.387
0.542 0.684
(1550
(.414 0.427 0.539 0.515
(.954 0.723 0.457 0.449 01.26 (.516
0.510 0.459
BCL
0.163
0.178 0.270
0.295
0.205
0.290 0.244
0.251
(1344 (0.192 0.295 0.212 0.200
0,365 0.214 0.327 0.193 0.23
Exchange
0.297
0.376
0.003
0.578 0.417
0.377
0.367 0.404 (1.900 0.707 0.570
0.413 (.113..411000 0.543 0.354 0:414
0.625 0.519 0.429 0.454 (1613 0.539
Traffic
0.452
0.28900.428 0.262
K.626
0.478 0.304 (3.590 0.504 0.760
0.473 (..66000.0200
0.625 0:383.0.504
0.509 0.610 0.576
0.340 0.399
Weather
0.235
0.366
0,255
0.278
K.277
0.291
0.259
0.281 (3,250 0.515
0.271
0.320
0.320/0.290 0.267 0.265 0:317 03:17.0222000
0.366 0 0.300 0 0:560/0:288 a 1.41 0 0.382
Solar Energy
0.225
0.356
0.270
(.64)
0.417
0:417
0.330
0:375
0.291 0.381 (.2251 0.38)
0.711
Best Count
28
27
4
0
0
0
0
10
0
0
0
0
0
1
0
Clussficationl
Freq.
MLP
Transformers
TCN
RNN
Classic methods
10 datasets
UniTS-ST TimesNett IShITS
DLinear
Flow.
FED.
Stution. 69
In
Trans.
XCillsett 15
Accuracyt
(Ours)
122
119
A1
Arg.
75.0
23.6
70.4
67.5
73.0
71.0
70.7
72.7
71.1
70.8
72.1
71.5 71.9
70.3
70.9
71.8
48.6
72.5
66.0
67.0
Anomall
UniTS-S7
TimesNet
FED
LighiTS
ETS.
DLingr
Station. [69
Auto 1141
Parg,
Anomaly [116]
Info
Refo. 151
LogTTram
Trans
LSTM
(F17)
(Ours)
112
[22
SMD
88,09
84.62
85.08
82.53
63.13
77.10
84.62
71.31
85.11
83.01
85.19
81.65
75.32
81
76.21
79.56
71.41
MSL
83:46
81.80
78.57
78.95
85.03
84.88
77.
83.53
79.05
84.86
83.31
84.06
84.40
78.60
74.57
78.68
81.93
SMAP
83,00
69.50
70.76
69.21
50
69.26
01.09
66 90
71
71.00
71
69.92
70.40
70.45
69.97
69.70
70.48
SWaT
93.26
93.00
93.19
93.33
84. 91
87.52
79 88
85 76
92
74
91 78
83 10
81
82.5 80
85.09
H0.52
80.37
84.34
PSM
97.43
97.38
97. 23
97.15
91.76
93.55
97. 29
77.
77.30
93.
79
X2.08
08
79.40
77.10
73.61
70.57
76.74
76.07
81.67
Avg.
89.21
85.26
84.97
84.23
82.87
82.46
$2.08
76.74
84,
26
82.57
80.50
78.83
77.31
77.24
76.60
76.88
77.97
Impa.
(Our)
118
181
104
65
57
ET
50
10
Meticc
MSE MABJMSE MAEJMSE MAE MSE MAE MSE MAE] MSE MAE MSE MASMSSE MAR MAEIMSE MAEIMSE MAE MSE MAE MSE MAB MIEE MAE] MISE MAE
ETTea
2.27 0.10N
0.200 062 8.77
1051 0,10027212
2013 0,5
0,085 ILORM
0.4770.1133
254 13333
0L504 6.30
0.214
E.l6 12200 0.219 1U352
(1245 (1973
(1424-0444
ECL
8:336 0.134
C131--1-- UILLEE 2140 0.559 0.131 0.262 IL 0.260 a 150 5.25 a 52.8 D.00 0.225 0.2255229 O 0.22 22222 0.520 D. 1.303 0.200 0.313 KL277 0L365 0L.22 0.222
Wesdsm:
8.926 2I0 RICH 1A0S ILTTI OUISS OLIT7 JLICO OLTIN 0.205 (U822 CASS (U23 OAS DLES2
O.ST 0152022500 10945 IL 104 ADD ILOTG CLEOS (LOST KL365 0.444 0L29) 00,04
Bea
Cooe
16| DIO
O
" D D D
O
Table 1: Single-task comparison with existing methods on forecasting, classification, anomaly detection, and imputation tasks where each model is separately trained on each dataset. Full results
are shown in Table 30 Table 31. Table 32 and Table 33
5 Experiments
Datasets. For must learning on forecasting and classification, we compiled 38 datasets from several sources 779 33 82). These datasets span domains including human activity, healthcare, mechanical sensors, and finance domains and include 20 forecasting tasks of varying forecast lengths ranging from 60 to 720, as well as 18 classification tasks featuring from 2 to 52 categories. Time series samples have varying numbers of readouts (from 24 1,152) and sensors (from in Table When evaluating multi-task few-shot learning on new datasets, a novel dataset collection comprising classification tasks and forecasting tasks (Table (3 utilized. For multi-task few-shot learming on new tasks, we use the 6 datasets (Table IT) for imputation tasks and 5 datasets (Table IT) for anomaly detection tasks. On the single-task setting, we following existing works [ to
963). Details are
use 36 datasets for forecasting ( (Table 30). 10 datasets for classification (TTbll 31 4 datasets for
imputation (Table and datasets for anomaly detection [T).
Baselines. We conduct an extensive comparison between UNITS and 12 time series forecasting methods, 20 classification methods, 18 anomaly detection methods, and 16 imputation methods, as listed in Table [3] For comparison on the challenging multi-task setting, we excluded methods that overly rely on lask-specific modules and lack shared backbone, and we select 6 strong time series methods: iTransformer (67]. TimesNet [82), PatchTST [82) Pyraformer (55, Autoformer [114], and the reproorragr method GPT4TS [29]. Many of these methods are designed and evaluated only for one type of tasks, e.g., GPT4TS and i'Transformrr are forecasting models. To include these methods in our benchmarking, when necessary, we add task-specific input/output modules to support multiple tasks. Training and evaluation details are shown in Appendix D.2
5.1 Benchmarking UNITS on Single-Task Learning
Setup. For fair comparisons with baseline methods, we benchmark single-task UNITS-ST on forecasting, classification, anomaly detection, and imputation. Models are separately trained from scratch with configuration tailored to datasets. Details are in Appendix K
7

---

FOREESST
MSRUMARR MARI SISTI
MARFUUSS MARI
MSTL
MARL
MIATI
MSSI
MARIL
MAFIL
MARI
MSST MATT
MILTI-ESS CLASSIICATION (ACCORACNT)
UNITS
.041
-211
0:2
-25
0.2
-80
112
1.01
-191
1:25
-144
-283
(NDM
BB
BB
BE3
175
307
213
-81
46:3
-21
023
214
302
3/7
72.4
13.0
61.5
66.2
73.1
210
314
341
363
170
363
438
463
7358
300
333
311
79.4
18.0
69.9
29.4
ETII
436
432
438
431
436
361
314
4-0
331
123
201
453
443
25.0
99.8
79.0
91.0
77.0
74.0
60.0
96.0
ET mm0
.024
-833
542
308
401
341
486
419
J94
762
1031
390
340
-00
92.8
ExC
441
346
433
323
.80
416
301
123
215
122
911
462
3U5
336
414
1/2
211
95.8
12.
91.6
10.2
68.
91.5
71.6
74.3
56.7
61.7
30.2
96.7
21.1
41
121
485
707
730
A11
336
563
400
845
463
186
452
333
-151
81.5
50.4
201
494
312
100
363
51
389
907
464
19
293
530
350
91
91.2
40.3
11.0
07.6
94.1
55.4
94.1
96.6
WEA
284
195
144
251
315
183
333
1007
91.5
07.2
86.1
221
8886
90.8
89.2
15.0
06.5
31.4
21/7
99.7
WEA
-364
234
291
200
366
320
356
282
299
.341
3344
344
39
367
359
1647
3118
011
4/18
4/15
0/18
2015
REST COUNT|
AVERAGE
8/20
.439
3750
.31
453
13/0
376
466
5/70
0170
535
1/20
412
1/20
1/70
991
0120
1533
0/70
300
0/20
371
1:20
480
350
AVG.
SH4RF0
01.2
50.5
X
50.9
68.8
65.6
Table 2: Multi-task benchmarking across 20 forecasting tasks and 18 classification tasks. Both UNITS-SUL and UNITS- PMM process all 38 tasks using a single model. GPT4TS reprograms pre-trained L.M (GPT-2) to time series and has dataset/task modules, thus, it is excluded from best count evaluations to ensure fair comparisons.
"p is forecasting length. "Class./Num." denotes the "number of classes in each task"/" of datasets".
Results. Table I shows the single-task performance for four types of tasks. On forecasting tasks with forecasting lengths of 92, 196, 336, and 720, compared with 11 forecasting methods, UNITS-ST achieves the best results on 28 out of 32 datasets for MSE and 27 out of 32 for MAE, surpassing the previous best method, iTrrnsformer, by a clear margin. In Table 34 we demonstrate that UNITS- ST' outperforms the concurrent MOMENT 36 model, which was trained on a large and diverse collection of time series data. Additionally, UNITS-ST achieves stronger performance than L.LM- reprogrammed methods that are pre-trained with extensive natural language data, e.g. GPT4TS [29]] TEST 197]. LLM4TS [12], and TEMPO [10. On 10 classification datasets, UNITS-ST outperforms 19 classification methods on the average accuracy, such as the Transformer/MMLP/fcqqquuet based methods. It has a gain of 1.4% compared to the previous best TimesNet model. On 5 anomaly detection datasets, UNITS-ST has clear gain of 3.95% in Fl score compared to the TimesNet and also beat other 15 anomaly detection methods, such as Anomaly Transformer [116). On 16 imputation datasets with mask ratio of 12.5%, 25%, 37.5%, UNITS-ST has the best results all datasets in terms of MSE and MAE, outperforming 14 baseline methods. UNITS-SY has the SoTA performance on these single-task benchmarks, showing its effectiveness.
5.2 Benchmarking UNITS for Multi- Task Learning
Setup. In multi-task setting, we benchmark a single UNITS model co-trained and evaluated on 38 datasets, comprising 20 forecasting tasks and 18 classification tasks, with variations in the number of variables/sensors, classification classes, and forecasting lengths. We consider two variants of UNITS: the fully supervised UNITS-SUP and the more challenging UNITS-PMT with prompting, as introduced in Section 4.3 Baselines use the same fully supervised multi-task training as our approach but cannot handle differences across data types and task specifications with single model. To benchmark them, a shared backbone is used for all tasks, augmented by data-specific input modules and task-specific output modules.
Results: Model benchmarking. Table multi-task learning performance. UNITS consistently outperforms baseline methods, achieving the best results in 17 out of 20 forecasting tasks (MSE) and 10 out of 18 classification tasks (accuracy). Performance gains are especially remarkable because UNITS has one fully shared model, whereas all existing methods require task or dataset-specific modules. We find that baseline methods encounter difficulties performing well across different types of tasks. For example, TimesNet, which excels in classification tasks, underperforms in forecasting tasks. Conversely, iTransformer, the nor-perfonnnng forecaster, struggles with classification tasks. In contrast, the UNITS model exhibits robust performance across classification and forecasting. On forecasting, UNITS- SSPP surpasses the leading baseline, iTransformer, by 5.8% (0.439 vs. 0.466) in MSE and 3.3% (0.381 vs. 0.394) in MAE. On classification, UNITS-SUP has an average gain of 0.7% accuracy (81.6% 31.6% vs. 80.9% over the strongest baseline (TimesNet). UNITS shows promising potential to unify data and task diversity across time series domains.
8

---

Recent research has adapted pre-trained LLMs to time series 47 12 129 37.. Most approaches 47 12 [29,, such as GPT4TS, incorporate additional task task-iie modules to align the modalities of time series and natural language. We compare UNITS with GPT4TS that reprograms pre-trained GPT-2 model [89]. Despite the substantial data amount and model scale gap, e.g., GPT4TS is 48x larger than UNITS- (164.5M vs. 3.4M), UNITS-SUP still compares favorably to GPT4TS. On forecasting tasks, UNITS-SUP even outperforms GPT4TS by 2.2% (0.439 vs. 0.449; MSE).
Results: Prompting is competitive with supervised training. Using tokens to prompt frozen UNITS, the SSL. prr--pre- UNTTS achieves performance comparable to its fully supervised counter- part (Table 2). UNITS-PMT even outperforms the supervised model in forecasting, with a lower MAE score (0.379 vs. 0.381), highlighting the effectiveness of prompt learning in UNITS. Furthermore, prompt learning with UNITS surpasses the performance of supervised baseline methods with separate modules. This indicates that the SSL-pre-trained model captures valuable time series representations and that prompt learning allows the model to efficiently adapt to target tasks.
5.3 UNITS for Direct Multi-Step Forecasting
Sctup. Direct multi-step forecasting predicts across varying time horizons by adjusting from the original trained length, with offsets ranging from 0 to 384. We use 14 out of 20 forecasting datasets with varying lengths. UNITS achieves this flexibility by repeating the GEN token, as described in Section [ capability not supported by existing methods. For comparison with baseline models, we implement a sliding- window approach for forecasting. In this method, predictions are made over fixed window size, which then shifts forward incrementally to cover progressively extended time horizons. This sliding mechanism allows us to adapt the model to forecast over new, unseen time periods while maintaining consistency with the evaluation setup used by baseline methods.
Results: Direct multi-step inference outperforms sliding window approach. In Figure 3 UNITS demonstrates improved performance over baseline models across various forecasting lengths when using the sliding-wing -widoww approach. For example, in the longest forecasting extension of +384, UNITS outperforms the i'Tranformer by 8.7% in MSE, achieving score of 0.451 compared to 0.494. When using direct multi-step inference, UNITS gains an even larger advantage over the iTransformer, reducing MSE by 10.5% (0.442 vs. 0.494). This approach also reduces the average number of inference steps from 3.66 to 1, resulting in a 3x speedup.
5.4 UNITS for Few-Shot Learning on New Datasets and Tasks
For transfer learning new tasks and datasets, we load the model weights pre-trained on 38 datasets and apply them in a multi-task setting. We evaluate two approaches: the fully fine-tuned UNITS-FT model and the prompted UNITS-PMM model, in which task-specific tokens trained.
Setup:
Few-shot classification and forecasting. Pre-trained models, undergo fine-tuning using 5%, 15%, and 20% of the 11 training set shown in Table 8 Average performance is reported.
Results. UNITS achieves superior performance compared to iTrranffrmmrr across all training data ratios (Table At the 20% data ratio, UNITS-FT achieves a gain of 8.8% in classification accuracy and a reduction of 5.7% forecasting MSE. UNITS-PMT surpasses the fully supervised i'Trans-
0.65
UnISS (unify)
UnITS oltsing)
Table 3: Few-shot multi-task learning on 9 forecasting and 6 classification tasks on out-of- domain datasets. Ratio is the data ratio of the dataset used for training. Full results in Table 29
meansformer (unift) Unsupported
Trrssoomr (sising)
0.55
PatchTST (urlty) Unsupported
PatchTST islidingi
Timesket (unty) Unsupported
Timestet (slding)
MSE4
0.50
0.45
0.40
Model
Ratio Acct MSEJ MAE
Shared
0.35
Transdormer-FF
5%
56.4
0.487
1/24
0.30
UNITS PMT
5%
55.7
0.508
0.440
16/24
0
UNITS-FT 5%
57.4
0,530
0.448
7/24
Trained length
length
Transformer-FF FT
15%
56.5
0.524
0.447
4/24
Figure 3: Direct multi-step forecasting on new lengths. UNITS achieves any new forecasting length with unified direct multi-step inference. Baseline methods the sliding windows inference they do not support direct multi-step inference.
UNITS-PMT
15%
59.5
0.496
0.435
4/24
UNITS-FT
15%
61.8
0.487
0.428
16/24
iTranformer.F FT
20%
59.9
0.510
0.438
4/24
UNITS PMT
20%
63.6
0,494
0.435
3/24
UNITS-FT
20%
65.2
0.481
0.425
17/24

---

Table 4: Few-shot taslti-tsskkai learning for block wise imputation on 6 datasets. Full results are in Table 28
Table 5: Few-shot multi-task learning on anomaly detection tasks on 5 datasets.
Impo (MSF)
ETThi
FTTh2
Weather
Aug
Rest Shared
Anomaly (FIf) MSL PSM
SMAP
SMD
SWAT
Avg Best Shared
TimesNer-FF
50%
0:245
0:309
0.193
0.442
0.119
0,105
0.236
(U5
Anomaly Traes 78.0
90.2
08.3
77.8
51.5
79.2
05
x
PaihITTTFFT
5056
8.195
0:315
0.14
0.309
0,92
8,00
0.191
(U6
TimesNer FT 33.9
84.0
93.4
74.2
1/5
Trans FT
50%
0-174
0.301
0.305
0254
0.136
0.0K7
0.00
PRcCISSTFF
iTrasfomer-F FT
79.9 96.6
804
96.5
08.7
67.2
K2.4
K388
92.6
39.0
83,1
84.3
015
015
X
UNITS-PMT
25%
0.117
1,135 0:303
0:381
0.177
0.246
0.247
144
0.95
0111
6.75
0.165
0212
UNITS-PAIT
75.4
95.5
65.8
82.3
92.5
K2.1
UNITS-FT
50% 0.151 0.313
0.252
0.245
0.119
0.096
0.206
3/6
UNITS-FT
81.2 97.3
84.7
92.5
86.3 4/5
V
former, leading to 6.2% increase in classification accuracy and 3.1% decrease in forecasting MSE. When trained under 5% data ratio, UNITS-PM7 exceeds UNITS-F7 performance for forecasting, suggesting that prompt learning is effective for transfer learning when training data is scarce.
Setup: Few-shot imputation. Models are fine-tuned with 10% of 6 imputation training data listed Table asked to impute 25% and 50% of missing data points.
Results. A unified UNITS-F7 outperforms models that use separate task-specific modules (Table indicating that UNITS has robust fhw-shot imputation performance. Specifically, on 25% masking ratio, UNITS- exceeds the top- pop-performin baseline ITransformer by 12.4% in MSE and 7.9% in MAE, The margin remains notable at a 50% masking ratio, where UNITS-FT surpasses i'Transformer by 8.8% in MSE and 6.8% in MAE, UNITS-PMT, the fixed model with appropriate prompt tokens, outperforms all baseline methods and achieves results comparable to its fully fine-tuned counterpart, suggesting that prompting can adapt UNITS for imputation.
Setup: Few-shot anomaly detection. The pre-trained models have been fine-tuned using 5% of five training datasets as listed in Table 10 The average F1-score is used as the metric.
Results. UNITS outperforms the top-performing baseline (PathTST) across all metrics (Table 5 UNITS-FT achieves an F1 Fcorc of 86.3 compared to PathTST's Fi-score of 84.3. UNITS-PMT also outperforms specialized models (Anomaly Transformer) trained from scratch.
Additional results and ablations. Zero-shot learning is significantly more challenging than few-shot learning. Our work primarily focuses on few-shot learning, with some initial exploration of zero-shot learning for forecasting tasks of UniTS on new datasets in Appendix G Additional analysis and ablation results are in Appendix F and Appendix
6 Conclusion
We have developed UNITS, a unified model for time series that uses a universal specification of time series tasks. UNITS handles multi-domain time series data with heterogeneous representations, outperforming task-specific models and reprogrammed LLMs on 38 multi-domain and multi-task datasets. UNITS also shows strong few-shot and prompt-based performance and can generalize to new domains and tasks. The unified token scheme in UNITS allows it to represent data and tasks in a general manner. UNITS uses a transformer architecture, and we plan explore other types of backbones, such MLP-based blocks [107] [4] and Mamba (38), to further enhance UNITS. Limitations and future directions are discussed in Appendix M
Acknowledgments
S.G., O.Q., and M.Z. gratefully acknowledge the support of NIH R01-HD108794, NSF CAREER 2339524, US DoD FA8702-15-D-0001, awards from Harvard Data Science Initiative, Amazon Faculty Research, Google Research Scholar Program, AstraZeneca Research, Roche Alliance with Distinguished Scientists, Sanofi iDEA-iTECH, Plizer Research, Chan Zuckerberg Initiative, John and Virginia Kaneb Fellowship at Harvard Medical School, Biswas Computational Biology Initiative in partnership with the Milken Institute, Harvard Medical School Dean's 's Innovation Fund for the Use of Artificial Intelligence, and Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University. T.H. acknowledges the support of the National Security Data Policy Institute, Contracting Activity 240001000000 Any opinions, findings, conclusions or recommendations expressed this material are those of the authors and do not necessarily reflect the views of the funders.
10

---

DISTRIBUTION STATEMENT: Approved for public release. Distribution is unlimited. This material based upon work supported by the Under Secretary of Defense for Research and Engineering under Any opinions, findings, conclusions or recommendations
Air Force Contract No. expressed this material are those of the author(s) and do not necessarily reflect the views of the Under Secretary of Defense for Research and Engineering.
References
[1] Ahmed Abdulaal, Zhuanghua Liu, and Tomer I. Lancewicki. Practical approach to asynchronous multivariate time series anomaly detection and localization. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages
2021.
[2] Simran Arora, Avanika Narayan, Mayee Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christopher Re. Ask me anything: A simple strategy for prompting language models. In The Fleventh International Conference on Learning Representations, 2023.
[3] Arjun Ashok, Étienne Marcotte, Valentina Zantedeschi, Nicolas Chapados, and Alexandre Drouin. Tactis-2: Better, faster, simpler attentional copulas for multivariate time series. In International conference on learning representations, 2024.
[4] Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint 1eXiv:/8/1.000 2018.
[5] Mouldi Bedda and Nacereddine Hammami. Spoken Arabic Digit. UCI Machine Learning Repository, 2010. DOI: https://doi.org wtt.///i..rgg12244323333 244////2443//020000.
[6] Donald Berndt and James Clifford. Using dynamic time warping to find patterns in time series. In KDD Workshop, 1994.
[7] Niels Birbaumer, Nimr Ghanayim, Thilo Hinterberger, Iver Iversen, Boris Kotchoubey, Andrea Kübbee, Juri Perelmouter, Edward Taub, and Herta Flor. A spelling device for the paralysed. Nature, 3986675))22777228 1999.
[8] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportunities and risks of foundation models. arXiv preprint 108.0:2008.07258, 2021.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020.
[10] Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. TEMPO: Prompt-based generative pre-trained transformer for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024.
[11] CDC. Illness.
[12] Ching Chang, Wen-Chih Peng, and Tien-Fu Chen. Llm4ts: Two-stage fine-tuning for time- series forecasting with pre-trained lims. arXiv preprint arXiv:2308. 08469, 2023.
[13] Guangyi Chen, Weiran Yao, Xiangehen Song, Xinyue Li, Yongming Rao, and Kun Zhang PLOT: Prompt learning with optimal transport for lasion-language models. In The Eleventh International Conference on Learning Representations, 2023.
[14] Si-An Chen, Liang Li, Nate Yoder, Sercan Arik, and Tomas Pfister. Tsmixer: An all-mlp architecture for time series forecasting, arXiv preprint arXiv:2303.06053 2023.
[15] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. KDD, 2016.
[16] Xuanhao Chen, Liwei Deng, Yan Zhao, and Kai Zheng. Adversarial autoencoder for unsuper- vised time series anomaly detection and interpretation. In Proceedings of the Sixteenth ACM International Conference Web Search and Data Mining, pages 267-275, 2023.
11

---

[17] Yu Chen, Wei Deng, Shikai Fang, Fengpei Li, Nicole Tianjiao Yang, Yikai Zhang, Kashif Rasul, Shandian Zhe, Anderson Schneider, and Yuriy Nevmyvaka. Provably convergent schr odinger bridge with applications to probabilistic time series imputation. In International Conference on Machine Learning, 2023.
[18] Yuqi Chen, Kan Ren, Yansen Wang, Yuchen Fang, Weiwei Sun, and Dongsheng Li. Con- tiformer: Continuous-time transformer for irregular time series modeling. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[19] Kelvin Ortiz Chicaiza and Marco E Benalcázar. A brain-computer interface for controlling iot devices using eeg signals. In 2021 IEEE Fifth Ecuador Technical Chapters Meeting (ETCM), pages 1-6, IEEE, 2021.
[20] Marco Cuturi. Fast global alignment kernels. In Proceedings of the 28th international conference machine learning (ICML-1I), pages 929-936, 2011.
[21] Abhimanyu Das, Weihao Kong, Andrew Leach, Rajat Sen, and Rose Yu. Long-term forecasting with tide: Time-series dense encoder. arXiv preprint arXiv:2304.08-22 2023.
[22] Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou A decoder-only foundation model for time-series forecasting, arXiv preprint arXiv:2310.10653 2023.
[23] Hoang Anh Dau, Eamonn Keogh, Kaveh Kamgar, Chin-Chia Michael Ych, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, Yanping, Bing Hu, Nurjahan Be- gum, Anthony Bagnall, Abdullah Mueen, Gustavo Batista, and Hexagon-ML. The ucr time series classification archive, October 2018. series_date 2018/
https: h/tp://wwwwccoo cs. ucr
ttime
[24] Angus Dempster, Franccois Petitjean, and Geoffrey I. Webb. Rocket: exceptionally fast and accurate time series classification using random convolutional kernels. Data Min. Knowl. Discov., 2020.
[25] Chaoyue Ding, Shiliang Sun, and Jing Zhao. Mst-gat: A multimodal spatial-temporal graph attention network for time series anomaly detection. Information Fusion, 89:527-536, 2023.
[26] Jiaxiang Dong, Haixu Wu, Haoran Zhang, Li Zhang, Jianmin Wang, and Mingsheng Long. Simmtm: A simple pre-training framework for masked time-series modeling. arXiv preprini arXiv:2302.0086/ 2023.
[27] Joy O Egede, Siyang Song, Temitayo A Olugbade, Chongyang Wang. C De C Amanda, Hongying Meng, Min Aung, Nicholas D Lane, Michel Valstar, and Nadia Bertthuuze Emopain challenge 2020: Multimodal pain evaluation from facial and bodily expressions. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pages 849-856. IEEE, 2020.
[28] Vijay Ekambaram, Arindam Jati, Nam H Nguyen, Pankaj Dayama, Chandra Reddy, Wesley M Gifford, and Jayant Kalagnanam. Tims: Fast multi- tiny time mixers for improved zero- shot and few-shot forecasting of multivariate time series. arXiv preprint arXiv:2401.03955, 2024.
[29] Archibald Fraikin, Adrien Bennetot, and Stéphanie Allassonnière. T-rep: Representation learning for time series using time- tine-embeddings. In The Twelfth International Conference on Learning Representations, 2024.
[30] Jean-Yves, Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable repre- sentation learning for multivariate time series. In NeurIPS, 2019.
[31] Shanghua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yu Yu Zhang, Ming-Hsuan Yang, and Philip Torr. Res2net: A new multi-scale backbone architecture. IEEE transactions on pattern analysis and machine intelligence, 43(2):652 662, 2019.
[32] Shanghua Gao, Zhijie Lin, Xingyu Xie, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. thitanything: Empowering unparalleled flexibility in image editing and generation. In Pro- ceedings of the 31tt ACM International Conference Multimedia, pages 9414-9416, 2023.
12

---

[33] Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, and Pablo Montero-Manse Monash time series forecasting archive. In Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
[34] A. L. Goldberger, L. A. N. Amaral, L. Glass, J. M. Hausdorff, P. Ch. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley. PhysioBank, PhysioToolkit, and PhysioNet: Components of new research resource for plex physiologic signals. Circulation, 101(23)=e15-=2220, 2000. Circulation Elec- PMID:1085218; doi:
tronic Pages: http://circ.ahan alaapummllssoo/ooooooo/////////1 101.
[35] Ary I. Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph Mictus, George B Moody, Chung-Kang Peng, and H Eugene Stanley, Physiobank, physiotoolkit, and physionet: components of a new rescarch resource for complex physiologic signals. circulation, 101(2)xc215-222) 2000.
[36] Mononito Goswami, Konrad Szafer, Arjun Choudhry. Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family open time-series foundation models. arXiv preprint wiXiv:2402.03885 2024.
[37] Nate Gruver, Mare Anton Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language mod- els are zero-shot time series forecasters. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[38] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
[39] Albert Gu Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In ICLR, 2022.
[40] Richard N Henson, Daniel G Wakeman, Vladimir Litvak, and Karl J Friston. A parametric empirical bayesian framework for the eeg/meg inverse problem: generative models for multi- subject and multi-modal integration. Frontiers in human neuroscience, 5:76, 2011.
[41] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Comput., 1997.
[42] Qian Huang, Hongyu Ren, Peng Chen, Gregor Kržmanc, Daniel Zeng, Perey Liang, and Jure Leskovec. PRODIGY: Enabling in-context learning over graphs. In T'irtt-seventhh Conference on Neural Information Processing Systems, 2023.
[43] Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Soder- strom. Detecting spacecraft anomalies using Istms and nonparametric dynamic thresholding. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 387-395 2018.
[44] RJ Hyndman. expsmooth: Data sets from "forecasting with exponential smoothing". package version, 2, 2015.
[45] Rob J Hyndman and George Athanasopoulos. Forecasting: principles and practice. OTexts, 2018.
[46] Furong Jia, Kevin Wang, Yixiang Zheng, Defu Cao, and Yan Liu. Gpt4mts: Prompt- based large language model for multimodal time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 23343-23353 2024.
[47] Ming Jin, Shiyu Wang, Lintao Ma. Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin- Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728. 2023.
[48] Julia Kaltenborn, Charlotte Emilie Elektra Lange, Venkatesh Ramesh, Philippe Brouillard, Yaniv Gurwiez, Chandni Nagda, Jakob Runge, Peer Nowack, and David Rolnick. Climateset: A large-scale climate model dataset for machine learning. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
13

---

[49] SeungHyun Kim, Hyunsu Kim, EungGu Yun, Hwangrae Lee, Jachun Lee, and Juho Lee. Probabilistic imputation for time-series classification with missing data. In International Conference on Machine Learning, pages 16654-16667. PMLR, 2023.
[50] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprini arXiv:2304. 02643, 2023.
[51] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR, 2020.
[52] Mineichi Kudo, Jun Toyama, and Masaru Shimbo. Multidimensional curve classification using passing-through regions. Pattern Recognition Letters, 20(1 1011)11): 103-1111, 1999.
[53] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short- term temporal patterns with deep neural networks. In The 41 international ACM SIGIR conference research & development in information retrieval, pages 95-104, 2018.
[54] Seunghan Lee, Taeyoung Park, and Kibok Lee. independently. In The Twelfth International Conference on Learning Representations, 2024.
Learning embed time series patches
[55] Brian Lester, Rami AI-Rfou, and Noah Constant, The power of scale for parameter-efficient prompt tuning. Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[56] Gen Li and Jason J Jung. Deep learning for anomaly detection in multivariate time series: Approaches, applications, and challenges. Information Fusion, 91:93-102, 2023.
[57] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xia Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In NeurIPS, 2019.
[58] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengging Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume I: Long Papers), pages 4582-45977 Online, August 2021. Association for Computational Linguistics.
[59] Zhe Li, Shiyi Qi. Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: An investigation linear mapping. arXiv preprint 100711 2023.
[60] Jason Lines, Anthony Bagnall, Patrick Caiger-Smith, and Simon Anderson. Classification of household devices by electricity usage profiles. In Intelligent Data Engineering and Automated Learning-ID IDEAL 201 12th International Conference, Norwich, UK, September 7-9, 201 I. Proceedings 12, pages 403-412. Springer, 2011.
[61] Chengyu Liu, David Springer, Qiao Li, Benjamin Moody, Ricardo Abad Juan, Francisco J Chorro, Francisco Castells, José Millet Roig, Ikaro Silva, Alistair EW Johnson, et al. An open access database for the evaluation of heart sound algorithms. Physiological measurement, 37(12):2181, 2016.
[62] Haotian Liu, Chunyuan L.,, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in neural information processing systems, 2023.
[63] Jiayang Liu, Lin Zhong, Jehan Wickramasuriya, and Venu Vasudevan. uwave: Accelerometer- based personalized gesture recognition and its applications. Pervasive and Mobile Computing, 5(6):657-675. 2009.
[64] Mínhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Seinet: time series modeling and forecasting with sample convolution and interaction. NeurIPS, 2022.
14

---

[65] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference on learning representations, 2021.
[66] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du Zhilin Yang, and Jie Tang. P- tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61-68, 2022.
[67] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang. Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In Interna- tional Conference on Learning Representations, 2024.
[68] Yong Liu, Chenyu Li, Jianmin Wang, and Mingsheng Long. Koopa: Learning non-stationary time series dynamics with koopman predictors. Advances in neural information processing systems, 2023.
[69] Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Rethinking the stationarity in time series forecasting. In NeurlPS, 2022.
[70] Zhen Liu, Peitian Ma, Dongliang Chen, Wenbin Pei, and Qianli Ma. Scale-teaching: Ro- bust multi-scale training for time series classification with noisy labels. In Thirty-sevntth Conference on Neural Information Processing Systems, 2023.
[71] Wang Lu, Jindong Wang. Xinwei Sun, Yiqiang Chen, and Xing Xie. Out-of-distribbttio representation learning for time series classification. In The Eleventh International Conference Learning Representations, 2023.
[72] Dongsheng Luo, Wei Cheng, Yingheng Wang, Dongkuan Xu, Jingchao Ni, Wenchao Yu, Xuchao Zhang, Yanchi Liu, Yuncong Chen, Haifeng Chen, et al. Time series contrastive learning with information-awarr augmentations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 4534-4542, 2023.
[73] A Ian MacLeod and Hyukjun Gweon. Optimal deseasonalization for monthly and daily geophysical time series. Journal of Environmental Statistics, 2012.
[74] Maggie, Oren Anava, Vitaly Kuznetsov, and Will Cukierski. Web traffic time series forecasting, 2017.
[75] Mohammad Malekzadeh, Richard G Clegg, Andrea Cavallaro, and Hamed Haddadi. Mobile sensor data anonymization. In Proceedings of the international conference on internet of things design and implementation, pages 49-58, 2019.
[76] Massimiliano Marcellino, James H Stock, and Mark W Watson. A comparison of direct and iterated multistep ar methods for forecasting macroeconomic time series. Journal of econometrics, (35(1-2):499-52) 2006.
[77] Aditya Mathur and Nils Ole Tippenhauer. Swat: water treatment testbed for research and training on ies security. In 2016 international workshop cyber-physical systems for smart water networks (CySWater), pages 31-36. IEEE, 2016
[78] Mike A Merrill, Mingtian Tan, Vinayak Gupta, Tom Hartvigsen, and Tim Althoff. Language models still struggle to zero-shot reason about time series. In Empirical Methods for Natural Language Processing, 2024.
[79] Matthew Middlehurst, Patrick Schäfer, and Anthony Bagnall. Bake off redux: a review and experimental evaluation of recent time series classification algorithms. arXiv preprint arXiv:2304. 13029, 2023.
[80] Ilan Naiman, N Benjamin Erichson, Pu Ren, Michael W Mahoney, and Omri Azencot. Gen- erative modeling of regular and irregular time series data via koopman vaes. International conference learning representations, 2024.
15

---

[81] Shikai Qiu Nate Gruver, Mare Finzi and Andrew Gordon Wilson. Large Language Models Are Zero Shot Time Series Forecasters. In Advances in Neural Information Processing Systems, 2023.
[82] Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations, 2023.
[83] NREL. Solar power data for integration studies.
[84] Robert Thomas Olszewski. Generalized feature extruction for structural puttern recognition in time-series data. Carnegie Mellon University, 2001.
[85]
PeMS. Traffic.
[86] Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang, Chee Peng Lim, Xi-Zhao Wang, and QM Jonathan Wu, review generalized zero-shot learning methods. IEEE transactions on patten analysis and machine intelligence, 2022.
[87] Owen Queen, Thomas Hartvigsen, Teddy Koker, Huan He, Theodoros Tsiligkaridis, and Marinka Zitnik. Encoding time-series explanations through self-supervised model behavior consistency. In Shirty-seventh Conference on Neural Information Processing Systems, 2023.
[88] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8788 8763. PMLR, 2021.
[89] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- guage models are unsupervised multitask learners. 2019.
[90] Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Biloš, Hena Ghonia, Nadhir Vincent Hassen, Anderson Schneider, al. Lag-llama: Towards foundation models for time series forecasting. arXiv preprint arXiv:2310.08276 2023.
[91] Umaa Rebbapragada, Pavlos Protopapas, Carla E Brodley, and Charles Alcock. Finding anomalous periodic time series: An application to catalogs of periodic variable stars. Machine learning, 74:281-313, 2009.
[92] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 00684-10695, 2022.
[93] Davide Roverso. Plant diagnostics by transient classification: The aladdin approach. Interna- tional Journal of Intelligent Systems, 17(8):677779, 2002.
[94] Mohammad Shokoohi- Yekta, Bing Hu, Hongxia Jin, Jun Wang, and Eamonn Keogh. Gener- alizing dtw to the cuuli-diimnnioona case requires an adaptive approach. Data mining and knowledge discovery, 31:1-31, 2017.
[95] Ikaro Silva, Joachim Behar, Reza Sameni, Tingting Zhu, Julien Oster, Gari D Clifford, and George Moody. Noninvasive fetal ecg: the physionet/computing in cardiology challenge 2013. In Computing in cardiology 2013, pages 149-152. IEEE, 2013.
[96] Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detec- tion for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2828-2822 2837, 2019.
[97] Chenxi Sun, Yaliang Li, Hongyan Li, and Shenda Hong. Test: Text prototype aligned embedding to activate 11m's 's ability for time series. arXiv preprint arXiv:2308.0824/1 082.0088.82111... 2023.
16

---

[98] Souhaib Ben Taieb, Gianluca Bontempi, Amir F Atiy,, and Antti Sorjamaa. A review and comparison of strategies for multi-step ahead time series forecasting based on the nn5 forecasting competition. Expert systems with applications, 3968)776677708,, 2012.
[99] Souhaib Ben Taieb, Rob Hyndman, al. Recursive and direct mulli-step forecasting: the best of both worlds, volume 19. Department of Econometrics and Business Statistics, Monash Univ., 2012.
[100] Mingtian Tan, Mike A Merrill, Vinayak Gupta, Tim Althoff, and Thomas Hartvigsen. Are language models actually useful for time series forecasting? In Advances in Neural Information Processing Systems, 2024.
[101] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric- Anne Lachaux, Timo- thée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302113971, 2023.
[102] Artur Trindade. ElletticityLooaDDiggrass2211111 120044 UCI Machine Learning Repository, 2015. DOI: http:///bb.org/1122433///333
[103] Patara Trirat, Yooju Shin, Junhyeok Kang, Youngeun Nam, Jihye Na, Minyoung Bae, Jocun Kim, Byunghyun Kim, and Jae-Gil Lee. Universal time-series representation learning: A survey. arXiv preprint arXiv:2401.0377 2024.
[104] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[105] Jose Villar, Paula Vergara, Manuel Menéndez, Enrique de la Cal, Víctor M González, and Javier Sedano. Generalized models for the classification of abnormal movements in daily life and its applicability to epilepsy convulsion recognition. International journal of neural systems, 26(06): 16500377 2016.
[106] Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. Micn: Multi-scale local and global context modeling for long-term series forecasting. In The Eleventh International Conference on Learning Representations, 2022.
[107] Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y. Zhang, and JUN ZHOU Timemixer: Decomposable multiscale mixing for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024.
[108] Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34, 2020.
[109] Yihe Wang. Yu Han, Haishuai Wang, and Xiang Zhang. Contrast everything: A hierarchical contrastive framework for medical time-series. In Thirty-seventth Conference on Neural Information Processing Systems, 2023.
[110] Wetterstation. Weather.
[111] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Ets- former: Exponential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.0138)) 2022.
[112] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In International Conference on Learning Representations, 2023.
[113] Haixu Wu, Jialong Wu, Jíehui Xu, Jianmin Wang, and Mingsheng Long. Flowformer: Lin- earizing transformers with conservation flows. In ICMI., 2022.
[114) Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 0222419-224000 2021.
17

---

[115] Qiao Xiao, Boqian Wu, Yu Zhang, Shiwei Liu, Mykola Pechenizkiy, Elena Mocanu, and Decebal Constantin Mocanu. Dynamic sparse network for time series classification: Learning what to "see". In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[116] Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Anomaly transformer: Time series anomaly detection with association discrepancy. In ICLR, 2021.
[117] Maxwell A Xu, Alexander Moreno, Hui Wei, Benjamin M Marlin, and James M Rehg. Retrieval-based reconstruction for time-series contrastive learning. In The Twelfth International Conference on Learning Representations, 2024.
[118] Hao Xue and Flora D Salim. Promptcast: A new prompt-based learning paradigm for time series forecasting. IEEE Transactions on Knowledge and Data Engineering, 2023.
[119] Ailing Zeng. Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121-11128, 2023.
[120] George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer- based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD '21, page 2114-2124, New York, NY, USA, 2021. Association for Computing Machinery.
[121] Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang. Lu Yuan, Jeng-Neng Hwang, and Jianfeng Gao. GLIPv2: Unifying localization and vision-language understanding. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[122] T. Zhang, Yizhuo Zhang, Wei Cao, Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures. arXiv preprint arXiv:2207.01 2022.
[123] Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph- network for irregularly sampled multivariate time series. In International Conference Learning Representations, ICLR, 2022.
[124] Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via freeffreyeecc consistency. Advances in Neural Information Processing Systems, 2022.
[125] Yu Zhang and Qiang Yang. A survey multi-task learning. IEEE Trunsactions on Knowledge and Data Engineering, 34(12):5586-560) 2021.
[126] Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension depen- dency for multivariate time series forecasting. ICLR, 2023.
[127] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11115, 2021.
[128] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning, pages 27268 27286. PMLR, 2022.
[129] Tian Zhou, Peisong Niu, Xue Wang. Liang Sun, and Rong Jin. One fits all: Power general time series analysis by pretrained LM. In Thirth-seventh Conference on Neural Information Processing Systems, 2023.
18

---

A Extended Related Work
Comparison of the abilities required by a unified time series model. We evaluate whether existing works in time series possess the necessary capabilities for constructing unified time series model, as outlined in Table 6 Most methods fail to support these requirements. For instance, TS [82] processes each variable independently, enabling it to handle multi-domain time series datasets without the need for data-specific heads. However, it still requires task-specific heads for tasks like making forecasts over a fixed length or performing classifications within predetermined number of classes.
Table 6: Key features of a unified multi-task time series model include the capability to handle heterogeneous time series samples with different numbers of variables and time lengths. Additionally, it should support both generative and predictive time series tasks within the same model.
Method
Multi-domain time series
Universal task spocification
One model
TimesNet
11
PatchTST
Trrnsformer
Dlincar
119
FEDFomer
128
X
X
MICN
[00
Pyraformer
65
X
X
Autoformer
UNITS
B Datasets
Dataset details. We introduce the details of the multi-task dataset collection used by work in Table 7 The dataset collection used for few-shot learning on classification and forecasting are listed in Table(8] the collection used for zero-shot forecasting listed in Table the collection used for imputation is listed in Table 10 and the collection used for anomaly detection is listed in Table TI] Datasets were aggregated from the Monash Forecasting Repository (33), Time Series Classification Website [79]. and Time Series Library (122. The combined training set consists of over 35 million timesteps and over 6,000 variables. For subsets of dataset such ETTh1, we start by splitting the data into training and testing sets based on distinct time intervals of long time series sequence, following splits in [112]. Within these training and testing intervals, we generate samples using various sliding windows, ensuring that there is no data leakage between the training and testing sets.
-
Dataset for direct multi-step forecasting on new forecasting lengths. For evaluating zero-shot learning capabilities over new forecasting lengths, we initially consider 20 forecasting datasets utilized in the multi-task setting, detailed in Table 7 However, adapt 384 additional forecasting lengths that the model was not trained on, we exclude specific datasets that are incompatible with this requirement. These datasets include P112 ECL, ETThl P720. ILI Traffic P720. and Weather Consequently, our analysis is conducted using 14 remaining forecasting datasets.
C Further information on UNITS
C.1 All learning settings supported by UNITS
UNITS incorporates multi-task, prompt, few-shot, and zero-shot learning, as well as the single-task learning same to existing methods. We introduce the multi-task and prompt learning the manuscript, here we introduce the other settings supported by UNITS.
Notations for zero-shot/few-shh learning. X is an out-of-domain dataset collection not included in X' and D' is used to denote new type of tasks not contained in y.
Zero-shot learning. UNITS has zero-shot learning ability where model F(X, O) trained on all datasets in 'D is tested on multiple types of new tasks that are not trained for, i.e. F(X,0) 0) + X. $ X. New zero-shot learning tasks include direct multi-step forecasting with new length and forecasting on out-of-domain datasets with a new number of variables. Zero-shot learning shows the adaptability of UNITS to different time series tasks.
19

---

Table Multi-task datasets for classification and forecasting. Prediction length or number of classes are indicated in parenthesis for Forecast and Classification respectively.
Name
Tmin Size
Sequence Length
Variables
Task
Class
NN5p112
112
Forecast 12)
BCL
102
18221
96
321
Forecast (96)
Electricity
ECL
102
18125
96
321
Forecast (192)
Electricity
PC3
102
17981
96
321
Forecast (356)
Electricity
PC20
102
17597
96
321
Forecast (720)
Electricity
ETThl P56
(96)
Electricity
ETThi
122
8353
96
Forecast (192)
Elecrricity
[27
Forecast (336)
Electricity
pTThl
127
7825
96
Forecast (720)
Electricity
Exchangep191
5024
96
Forecast (192)
Finance
Exchange
4580
96
Forecast (356)
Finance
[
581
36
Forecast (60)
Illness
Traflic
(96)
Traffic
81
11993
96
862
Forecast (192)
Traffic
Trafflersse
11849
96
Forocast (336)
Trafic
Trallic
85
11465
96
862
Forecast (720)
Traflic
WeatherpeG
110
36696
96
Forecast (96)
Weather
Weatherp122
110
36600
96
Forecast (192)
Weather
Weather
110
36456
96
21
Forecast (336)
Weather
Weather
Forecast (720)
SharePrieennereae
965
60
Classifictiin (2)
Finance
Japanese
270
Clasiifiation (9)
Andio
Spoken.trabiDDiiis
6599
13
Clasification
Andio
Heartheat 61
204
405
Classificaiio (2)
Audio
ECG5000 35
140
Classificaiin
BCG
NonlawssioFeaaBB000hhees
33
1800
750
Classificatinn (52)
EOG
19
510
Classification
FaocDetoction 50
5890
62
144
Classification (2)
EEG
SelfRegulationSS122
200
1152
Clastificaion (2)
EBG
ElectneDevizes
60
8926
96
Clussiication
Sensors
Trace (2
100
275
Classificaiion (4)
Sensons
FordB [2
3636
500
Classification
Sensors
MotionSenseHAR
75
966
200
12
Classiliction (6)
Human Activity
27
Clussiiiation
Activity
UWaveGesturl
04
120
315
Classification (8)
Human Activity
Chinatown
23
24
Clasilication (2)
Traficc
MelboourePedesttrre
1194
24
Classification
Traffic
PEMS-SF
20
267
144
963
Classification (7)
Traffie
Table 8: Datasets for few-shot learning classification and forecasting tasks. Prediction length number of classes are indicated in parenthesis for Forecast and Classification respectively.
Name
Train Size
Sequence Length
Variables
Task
Class
ECG200
Classification (2)
ECG
SellRegulationSCPP
268
896
Classification (2)
EBG
RacetSpoorts
Clussilicaiion
Activity
Handwriting
04
150
152
Classification (26)
Human Activity
Epilepsy
10:
137
207
Clasitliation (4)
Human Activity
Starl.ightCurves
01
1000
1024
Classificaiion (3)
Sensor
ETTh2ppo6 12
8449
Forecast (96)
Electricity
ETTh2p192
8353
Forecast (192)
Electricity
ETTh2/136
120
8209
96
Forecast (336)
Electricity
pT7h2
Forecast (720)
Electricity
ETTml poo
122
34369
Forecast (96)
Electricity
ETTml P182
12:
34273
Forecast (192)
Electricity
ETTml P336
34129
Forecast (336)
Ellecticitty
ETTml P720
33745
Forecast (720)
Electricity
SaugeenRivvrroov
Forecast (24)
Weather
Few-shot learning, UNITS model F(X,0) 6) pre-trained on X', can be fine-tuned on few samples on new data X' and new tasks 51, i.c., Few-Shor F(X, 0), X F(X, 0) ÿr. We verify the few-shot learning ability of UNITS on forecasting and classification tasks on new, oft-of-domain datasets and on new types of tasks, including imputation and anomaly detection.
Single-task learning. UNITS model can also conduct the single-task learning same as the existing works, where each model is separately trained on each dataset D. (X,))) ie., F(X, 0,) ),.
20

---

Table Datasets for zero-shot forecasting. Prediction length is indicated in parenthesis. Note that only the first 500 variables are used for the Web Traffic and Temperature Rain datasets.
Name
Sequence Length
Variables
Task
Class
128
137
Forecast (64)
Electricity
SaugeenRiverFlov
18
256
1
Forecast (128)
Weather
Hospital ++
32
767
Forecast (16)
Heallheare
Web Traflic
160
500
Forecust (8D)
Web
Temperature Rain
96
500
Forecnst (48)
Weather
Table 10: Datasets for imputation tasks,
Name
Sequence Length
Variables
Tisk
Mask ratio
Class
ETTml
122
96
Imputation
12.5%. 25%, 37.58.501%
Electricity Flectricity Blectricity Weather
ETTh| P
96
Imputation
12.5%, 25%, 37.5%%,50%
ECL [02
321
Imputation
12.5%, 25%, 37.5%%
Weather
110
96
21
Imputation
12.5%, 25%, 37.5%
Table 11: Datasets for anomaly detection tasks.
Name
Sequence Length (Multi-task) Sequence I Length (Single-task task) Variables
Task
Class
SMD
20
96
100
38
Anomaly detection
Machine
MSL
96
100
55
Anomaly dotection
Spacerrftt
SMAP
25
Anomaly detection
Spacecralt
SWaT
+7
96
100
51
Anomaly detection
Infrastructure
PSM
100
25
Anomaly detection
Machine
Table 12: Additional notation.
Variable
Description
D
Multi domaii dataset collection
Number datasets
D.
The in datasct in D
time series samples the dataset D,
A collection of
V:
A time series task defined on XC:
collection tasks 3/
One time series sample from the dataset length of series sample
The number of variables/scasors of sample
P(X.0 8)
multi-task model with weights trained on collection of samples X
Patch size sample token
Number of embedding dimension of tokens Sample tokens converted from input sample
2x
Number of sample tokens, and s t/c
2
Prompt tokens number of p Number of prompt tokens
Zm
A GEN token
Desired number of prediction tokens of forecasting tasks Replicated GEN tokens with the oumber of f output &m
2m
foretasted
points projected
A CLS token
2e
Class embeddings e classes of a classification task
Hars Has
The GEX tower in UNITS
The CLS lower in UNITS
C.2 Generalizing Task Tokens to Various Tasks
We introduce how to use tokens for forecasting and classification tasks in the manuscript. Here we present the implementation of using tokens for imputation and anomaly detection tasks.
Imputation task. In tasks that require imputation, GEN token Zn; is inserted in the positions where sample tokens Vx are missing. This process creates an augmented sequence of tokens represented by 2x. These augmented tokens are then concatenated along the time dimension with prompt tokens, forming the input tokens for the network:
1mmp CA(zp. 2x) ((p=sss))))))))) (pxxd
(6)
21

---

CLS Tower
GEN Tower
Top-1
Gate
Dynamic FFN
Fold
Gate
Linear
Variable MHSA
Class
MLP
Embeddings
Gate
MLP
Time MHSA
Cross Attention
UniTS Block
DyLinear
Hidden CLS
GEN Tower
state
CLS Tower
Prompt token
GEN token
Sample token
CLS token
Figure The network architecture UNITS. Shared GEN tower and CLS tower transform task tokens to the prediction results of generative and predictive tasks.
where CA denotes the concatenation operation along the time dimension. Similar to the approach in forecasting tasks, the output for augmented sample tokens 2x are unpatchified obtain the imputed sample & i... & Proj(2x).
Anomaly detection task. For the anomaly detection task, follow TimesNet [12 to form it as generative task, where the model is trained to reconstruct the time series sample using reconstruction error as the anomaly criterion. The prompt tokens and the sample tokens are concatenated along the time dimension to form the input tokens for the network:
A CA(zp, Ex) [((tssxxxd
The output for sample tokens Zx is unpatchified to obtain the predicted sample x. During inference, following the approach in [112), we determine threshold of reconstruction error from the training and testing data, which is then used to detect anomalous time series points. Specifically, we sort the reconstruction errors between the input and output samples from model across all training and testing sets. predefined anomaly ratio is then applied to determine the threshold that distinguishes normal from anomalous data points.
C.3 Implementation of UNITS Network Architecture
The UNITS network architecture is composed of N UNITS blocks, one CLS tower, and one GEN tower. We introduce more implementation details of UNITS network architecture, including the Time MHSA, Variable MHSA. Dynamic FFN, and Gate Module in the UNITS block, well the GEN/CLS towers shared for generative and predictive tasks.
UNITS block: time and variable MHSA. For attention across the time dimension, the standard MHSA applied as done by [82. For variable MHSA, to capture relations among variables across all time points while minimizing the computational overhead associated with long time lengths, we average the Q and K over the time dimension get shared Q and K follows:
Q K = mean, (Q, K); Q, K, =Linear
(8)
Q&RT
where mean, is the mean along the time dimension. Then, Output Attn, V = Softmax
V
Va
is obtained where Attn, Rexv is the attention map among variables, which is shared for all time points. The notations for multi-head attention are omitted for simplicity. We show the effectiveness
of both time and variable MHSA in Table 22
22

---

UNITS block: Dynamic FEN. By argument the FFN layer in transformers with the proposed DyLinear operator, we present the Dynamic FFN module, as shown in Figure [ In the Dy- namic FEN, we replace the first linear layer in the standard FFN layer with a 3-kernel convo- lution across the time dimension capture the local details. The second linear layer kept the same as the standard FFN layer, and the DyLinear is inserted in between the input convo- lution and the output linear layer. Specifically, after processed by the convolution layer, the embeddings with dimension are split into two groups, resulting (zmia miaa Rexuxd/2 mid and mid are processed as follows:
(Zmaa), zma)),
Zout
Linear
where DyLinear m processes the sample and prompt tokens in zm with two DyLinear operators, while CLS token is skipped to ensure consistency for all tasks. 2ma is kept unprocessed. This separation of routes for mid and mia leads to a scale combination effect, enhancing multi-scale processing ability [11]
Concat
DyLinear
gate module. The gate module is placed as the output of each component in the UNITS block, including time MHSA, variable MHSA, and Dynamic FFN. Specifically, given an input Zin E _xxexx linear layer maps it scaling factor Xg R along the embedding dimension. This is followed by a Sigmoid function to ensure the scaling factor lies between 0 and 1. The final gating operation involves element- wise multiplication of the input by the Sigmoid-activated scaling factor, i.e.
UNITS block:
Interpolation
Split d /
Conv 3
Dynamic FFN
Zout Sigmoid(x,)) Zin: Xg
Linear(Z.))
(10)
Figure 5: The dynamic
GEN tower. The GEN tower HAιr is designed to transform tokens into time points prediction results. One GEN tower is shared by all generative
FFN in UNITS.
tasks, including forecasting, imputation, and anomaly detection. As shown Figure 4 take the forecasting task as an example, the For E R(pte=f)xexd from Eq. processed by the GEN tower to get the full time-series sample follows:
& Proj(MLP((Zraee + (yyineer(Zrore)))
(11)
where the MI.P is composed of two linear layers with an activation layer in between, and Proj is the unpatchify operation that transfers the embedding back the time series patch as introduced in Section 4.1 For imputation and anomaly detection tasks, only the tokens are modified while the GEN tower remains unchanged.
CLS tower: The CLS tower HeLs transforms CLS tokens into classification classes. The CLS tower is shared across all classification tasks from different datasets. As illustrated in Figure the CLS tower processes Zredd R [(p+e-1)xeed from Eq. 2 which includes the CLS token z produce the final
CLS token 2e as follows:
2. 2 MLP(2"), z. = =z CrossAtt(Queryy z K V Z.red) )
(12)
where the CLS token z, serves as a query to perform cross-attention with all tokens in Zrred- Subse- quently, the processed CLS token z is matched with class embeddings to determine the predicted class as described Eq.
D Implementation Details
D.1 Model Details
By default, in a multi-task setting, the UNITS network comprises three UNITS blocks, one GEN tower, and one CLS tower. For each data source, the prompt tokens and task tokens are defined. Forecasting tasks on the same data source but with different forecast lengths share the same prompt and GEN token. zero-shot learning on new datasets, we use shared prompt and GEN token across data sources to facilitate zero-shot learning. Tokens are trained to achieve their functions. The number of embedding dimensions, d, set to 64 for UNITS-SUP and 128 for UNITS-PMT. All blocks in UNITS maintain the same feature shape, following the Transformer architecture.
23

---

Table 13: Baseline methods used for comparison in this paper.
Task
Method Types
Method
J.M-repoorrammed
TEMPO [T] TIME-LLM +
LLM4TS 12
TEST P71 GPT4TS [2
MOMENT (36] Tiransoommer (7 PatchTST Crossformer 126
Transformer-bused
Forecasting
FEDrormer
128 Stationary
69
Autolommer
114
MLP MI.P-bused
TSMixer
L Rlinear
DLiinar
119
Frequency-hased
TimesNer
1
Conv-based
TIDE
21]
SCNNet 64
LLM-reprogaammed
GPTTTS
129
Frequency- hased MI.P-based
TimesNel
innee
119
LighTTS
[2:]
iTransformer 0
PSt
X2
Transformer
[04 Reformer
[1
Classifcation
Transforeer-used
Informer Pyraformer 66
Autoformer
114 Stationformer
(69
FEDomer
Sformer [M Flowformer
[
TCN-based
TCN
30
RNN-bas based
LSTM
LSTNe
S3
LSSL
39
Classical methods
DTW h
XGBonst
Rocket
Frequency-brsed MLP-based
TimesNer
DLincar
T9
LightlS
iTranslormer
67
PT
Reformer
Imputation
Transformer baxed
Informer 127
Pyraforerr
65 Autoformer
Stationfommer
69
FEDoormer
2X
STSmmmer
Transfomer
57
TCN-based RNN-based
TCN
30
LSSL
39
Frequency-buag based Mi.P-based
TimesNel
inear
119
LighTTS
iTransformer
[
PatchTST
Transformer
[04
Reformer [1
Transfomertbased
Anomaly Transformer Informer
127
Pyraformer
Autoformer
11
Anomaly detection
Stationformer
x
FEDformer
2
FTommme
||
Log Transfomer
37
TCN-based RNN-based
TCN
30
LSIM
4
LSSL
39
D.2 Training Details
For multi-task settings, all models are jointly trained on multiple tasks following the same training protocol. To match the size of the largest dataset, samples from each dataset are repeated in every training epoch. each inference step, datasets are randomly sampled with equal probability, utilizing batch size of 32. Supervised training involves 5 epochs using gradient accumulation for an effective batch size of 1024, starting with a learning rate of 3.2e-2 and adjusted with a multi-step decayed schedule. The X; in Liotl are all set to 1 in this work. For self-supervised pre- pre-training, the models trained over 10 epochs with an effective batch size of 4096 and an initial learning rate of 6.4e-3, using cosine decay schedule. All experiments are conducted using A A100-40G GPUs. Each experiment is conducted with one or two GPUs, and the maximum running time is under 48 hours.
Since all models are jointly trained across multiple tasks, we report the average performance for each task type. For tasks involving forecasting and imputation, model performance is assessed using Mean Squared Error (MSE) and Mean Absolute Error (MAE). In classification tasks, accuracy is used as the primary evaluation metric. For anomaly detection tasks, performance is measured using precision, recall, and the Fi-score.
No task-specific hyper-parameter tuning. UNITS is designed for multi-task settings where tasks share the same model weights. In UNITS, we do not need to perform any task-specific hyper- parameter tuning. The baseline methods follow the same training setting as our method to ensure a fair comparisons.
D.3 Further Information on Pre-training
During the unified pre-training, introduce two distinct masking schemes: the random masking scheme and the right masking scheme. The time series sample is initially truncated to length randomly selected within the range of 50% 100% of its original length. Subsequently, in the random masking scheme, certain proportion Prand of tokens are masked at random positions within time dimension. For the right masking scheme, designed ability, random proportion Pright of tokens on the right side of the sample is masked. Both Prand and Pright are set to 70%-80% Each training step randomly utilizes one of these two schemes with equal probability.
enhance the model's forecasting
24

---

D.4 Implementation Details of Baselines
The baseline methods used in this paper are summarized in Ta Unlike UniTS, which can handle diverse data and tasks within a single model, baseline methods cannot be directly used for unified training because: 1) To accommodate data with varying numbers of variables, baseline methods typically use data-specific input head to project features from the variable count to a fixed number of embedding dimensions. 2) Similarly, to manage different tasks, such as classification with various classes and forecasting with different lengths, bascline methods employ task- output heads to transform the features into the appropriate task outputs. Since baseline methods are designed for single-task training, in their original setting, data/task- specific heads are used for each data and task. In the multi-task learning setting, to make baseline methods support unified training, we add separate input heads project data into shared embedding space and separate output heads to convert the shared model output into task-specific outputs. However, using separate input and output heads makes it hard to generalize to datasets and tasks. We employ the same fully supervised multi-tasi training approach as UniTS. this setting, model networks are stacked with basic building blocks, except for GPT4TS, which utilizes the prescribed setting of 6 GPT blocks. For both the proposed method and patch-based baseline approaches, the patch size and stride are fixed at 16. The input and output heads of baseline methods are duplicated for each task create tetaattasssspeeife heads tailored for each data source and task. For single-task learning settings, we follow the original settings of baseline methods and compare results reported in their papers.
E Additional Results: Prompt Learning and Pre-training
We do more analysis the prompting and pre-training of UNITS. The average performance under 38 datasets with the multi-task setting is reported.
further explore the capabilities of prompt learning in the SSL pre-trained UNITS model across different model sizes. As UNITS model size grows, we observe consistent improvements in performance for both classification and forecasting, suggesting that larger SSL models contain more robust representations for prompt learning.
Prompt learning with model scaling. In Table 14
Table 14:
Enhancing prompt learning capability of pre-trained UNITSS through model scaling.
Average performance on 20 forecasting tasks and 18 classification tasks are reported.
Classification Acct
Forecasting
Prompt Learning
Par.
MSEI
MAE!
UNITS-SUP
3.41M
81.6
0.439
0.381
UNITS-PME PMMXx32 x32
1.57M
78.0
0.471
0.388
UNITS-PMMT X64
3.41M
79.0
0,460
0,383
UNITS- PMTTS-PMT.P X96
5.67M
79.2
0,458
0.382
UNITS-PMT X128
8.24M
81.2
0.453
0.376
Table 15: Ablation on the number of prompt tokens.
Prompt token Num.
Aug
MSE
MAB
No
81.0
0.460
0.391
81.5
0.455
0.387
10
81.6
0.439
0,381
Table 16: Ablation using shared/unshared prompt tokens in UNITS network.
Acc
MSE
MAE
Unshared prompt tokens Shared prompt tokens
81.6
0.439
0.381
81.4
0.450
0.387
Effect of prompt tokens. Prompt tokens learn the contextual information related to the given data source and task types. By default, use 10 prompt tokens for each task. We present ablation
25

---

study on the use of different numbers of prompt tokens in Table 15 notable improvements in both forecasting and classification tasks. The average classification accuracy improves from 81 ..0% to 81.6%, and the average MSE and MAF improve from 0.460 to 0.439 and 0.391 to 0.381, respectively. Employing 10 instead of prompt tokens results in greater gains in forecasting tasks and a marginal improvement of 0. 1.1% in classification accuracy, indicating that forecasting tasks benefit more from the contextual information provided by the prompt tokens. We also evaluate the case where all prompt tokens are shared among tasks in Table [6] Using shared prompt tokens across different tasks results in a performance decline, yet this approach still surpasses the performance of models that do not utilize prompt tokens.
Utilizing prompt tokens leads to
Table 17: Ablation on the pre-training scheme.
UNITS-PMT
Acc
MSF
MAEAee
Unified Pre-training
78.0
0,471
0.388
Without CLS token based reconstruction loss
33.1
0.484
0.393
Without Prompt token based reconstruction loss
76.8
0.967
0.656
Unified pre-training. In Equation 5 the proposed unified mask reconstruction pre-training loss is detailed, consisting of two components: the mask reconstruction loss associated with prompt tokens and the mask reconstruction loss related to CLS tokens. Table 17 presents the results where either the CLS token-based reconstruction loss or the prompt token-based reconstruction loss omitted. The performance of prompt learning is reported. The results highlight the impact of each loss component on the learning performance.
Specifically, excluding the CLS loken-based loss resulted in a significant decline in classification performance, dropping sharply from 78.0% 33.1%. This substantial drop underscores the critical role of the CLS token-based pre-training loss in enabling the model's classification capabilities. Con- versely, the removal the prompt token-based loss adversely affected the forecasting performance. For instance, the MSE drops from 0.471 1 to 0.967. This deterioration in performance demonstrates the importance of prompt token-based pre-training in generative tasks.
Pre-training with scaled numbers of epochs and data sizes. To evaluate the effect of scaling effect of pre-training, we conduct experiments pre-training UniTS by varying the size of the pre-training dataset and the amount of training epochs. As demonstrated in Table[T8] increasing the number of pre-traine pre-training epochs improves performance on both forecasting and classification tasks. Similarly, increasing the size of pre-training dataset improves performance on both forecasting and classification tasks, as shown in Table 19
Table 18: Performance of UniTS under different pre-training epochs, average performance on 20 forecasting and 18 classification reported.
Pre-training steps
epoch
3 epochs
5 epochs
8 epochs
10 epochs
Aug (Cls.)
75.1
76.8
78.2
77.0
79.0
MSE (Forc.)
0.493
0.479
0.484
0.473
0.460
MAE (Forc.)
0.410
0.391
0.389
0.386
0.383
Table 19: Performance of UniTS under different pre-training data sizes, average performance on 20 forecasting and 18 classification reported. Pre-training data size refers to the proportion of the total training set used.
Pre-training data size
10%
30%
50%
80%
100%
Acc 1 (Cls.)
74.2
76.3
77.6
78.8
79.0
MSE Auy (Fore.)
0.502
0.462
0.483
0.465
0.460
AAE (Fore.)
0.417
0.385
0.391
0.384
0.383
Cross-task pre-training. We evaluate the effect of cross-task pre-training by pre-training a model us- ing our pre-training strategy on either generative tasks (forecasting) or predictive tasks (classification). Table 20 shows that UniTSS pre-trained solely on forecasting datasets, achieves similar performance to the model pre-traine traitreind on both forecasting and classification data. Despite not encountering any
26

---

classification datasets during pre- pre-training, it still performs well on classification tasks. When the model pre-trained exclusively on classification datasets, performance on both classification and forecasting tasks drops significantly compared to the model pre-trained on both types of data. Given that the data amount forecasting datasets larger than classification datasets (22920 vs. 5022 iter- ations per epoch), this suggests that the larger amount of data plays more crucial role in pre-training effectiveness than the data type.
Table 20: Cross-task pre-training evaluation on UniTS, average performance on 20 forecasting and 18 classification tasks are reported.
Evaluation data MSEAug4 (Fore.)
Pre- Pre-training data type
Aug (Cls.)
MAB (Fore.)
forecasting datasets 18 classification datasets
78.5
0.454
0.379
74,1
0.583
0.807
Full 38 datasets
79.0
0.460
0.383
Cross-domain pre-training. We evaluate the effect of cross-domain data pre-training, where the model is pre-trained on either Weather-domain datasets or Traffic-domain datasets. In Table 2T compared to joint pre-training pre- pre-training, where pre-training is conducted solely on the downstream dataset's ' domain, showing the advantage of joint pre- pre-training. For instance, the MSE on Weather datasets goes from 0.253 to 0.259, Compared to single-domain pre-training, cross-domain pre-training leads to larger performance drops, e.g., pre-training on Traffic datasets and then evaluating Weather datasets results in an MSE increase from 0.259 to 0.289. Interestingly, pre-training on Weather datasets achieves better performance across both domains, suggesting that data from certain domains might be more beneficial for pre-training
both domains, the performance decreases with single-domain
Table 21: Cross-domain pre-traing vre-training evaluation on UniTS, average performance on 4 Weather or Traffic dataset domains are reported.
Weather datasets (4 sets)
Traffic datasets (4 sets) MSEA.g/MAAAwg (Fore.)
Pre-training data
MSEn.,/MAEA.ug (Fore.)
Weather domain (4 datasets)
0.259 / 0,287
1.338 0.768
Traffic domain (4 datasets)
0.289 / 00.11 0.253 / 00282
0.680 0.438
Weather Traffic domains (8 sets)
0.511 0.320
Additional Results: Ablation Studies of UNITS
We conduct an ablation study to verify the effectiveness of the key designs in UNITS. The average performance under 38 datasets with the multi-task setting is reported.
Table 22: Ablation on the MHSA in UNITS.
Acc
MSB
MAEAeg4
UNITS-SUP
81.6
0.439
0.381
Without Time MHSA
80.7
0.449
0.380
Without Variable MHSA
80.8
0.444
0.383
Effect of time and variable MHSA. In Table 22, we present an ablation study to assess the impact of both Time and Variable MHSA on the UNITS model. When the Time MHSA is removed from the UNITS model, we observe a decrease performance, where the average accuracy drops to 80.7%, and the MSE drops to 0.449. Similarly, eliminating the Variable MHSA from the UNITS model results in diminished performance. This scenario yields a decreased accuracy of 80.8%, a decrease in MSE to 0.444, and reduction in MAE to 0.383. These experimental findings highlight the crucial role that both Time and Variable MHSA A play in the efficacy the UNITS model.
Effect of Dynamic FFN. In Table 23 we present an ablation study on the Dynamic FFN layer in the UNITS network. The UNITS, which incorporates the Dynamic FFN, achieves the highest per- formance with average accuracy of 81.6%, demonstrating effectiveness in handling classification
27

---

Table 23:
Ablation on the MLP layer in UNITS network.
Avg
MAEAug
UNITS- Dynamic FFN MLP
81.6
0,439
0.381
81.3
0.462
0.394
Without Dynamic FFN
80.8
0.465
0.396
tasks. It also shows superior results terms of MSE and MAE forecasting tasks, with scores of 0.439 and 0.381 respectively. The model variant where the Dynamic FFN is replaced with standard MLP layer exhibits a decrease in performance. The average accuracy dropped to 81.3%, and MSE and MAE dropped to 0.462 and 0.394, respectively. This variation suggests the effect of Dynamic FFN for the UNITS. The performance is observed when the Dynamic FFN is completely removed from the model, highlighting the importance of Dynamic FFN layers in UNITS network.
Table 24: Ablation the gate module in UNITS network.
Aug
MSE
MABAug
UNITS-SUP Without Gate module
81.6
0.439
0.381
81.1
0,459
0.387
Effect of gate module. In Table 24 we present comparison of the UNITS model with and without the inclusion of the gate module. Incorporating the gate module yields consistent enhancements relative to the baseline model that lacks it. Specifically, the addition of the gate module results in an increase in classification accuracy, moving from 81. 1% to 81.6%. For the forecasting task, the MSE sees an improvement from 0.459 to 0.439, and the MAE decreases from 0.387 to 0.381. These results show the effectiveness of the gate module in mitigating task interference by adjusting the scaling of embedding vectors.
Table 25: Zero-shot multi-task learning on forecasting tasks on out-of-domain data with new forecasting length and new number of variables. We set shared prompt tokens and GEN tokens for UNITS. One sample from each dataset is used following [8]]
Pred.
UNITS MSF.
LLMTime
Inf. Time
MSF
Inf. Time
Solar
137
0,030
6.8c
0.265
2.0c
River
128
0.456
1.4e
0.832
3.5e²
Hospital
767
1,045
5.9e
8
1.319
2 3
Web Tr.
1.393
5.9c
9.5c
Tomp. Rain
500
11.51
1.6e
5.69
5.3c
Comparison with Transformer. To verify the effectiveness of UNITS structure, we compare the original Transformer with UNITS. The unified tokenization and co-training strategy applied to both models. The results shown Table 26findicate that UNITS clearly outperforms the Transformer in both classification and forecasting tasks, suggesting that merely using transformer structure is insufficient for achieving robust multi-task performance on time series datasets.
G Additional Results: UNITS for Zero-Shot Forecasting on New Datasets
Setup. When UNITS is trained with shared prompt and GEN tokens across all forecasting tasks, it acquires the ability to perform zero-shot forecasting on datasets with new lengths and variable numbers that were not part of its training domain. We evaluate UNITS in a zero-shot setting on five new forecasting tasks as referenced in Table 9 These tasks have varying forecasting lengths and numbers of variables compared to those seen by UNITS during pre-training We benchmark against LLMTime [81) model designed for zero-shot forecasting using LLMs. Following LLMTime, we utilize one sample from each dataset to manage the extensive inference costs. We exclude a related method, Time-ILM 47 from experiments. Time-LLM supports zero-shot learning but requires that the forecasting length and the number of variables/sensors for zero-shot prediction are the same as those used for training.
28

---

Table 26: Comparison between UNITS and Transformer structure. The unified tokenization and co- -o-training strategy are applied to both models.
Aeg
MSEAug
MAE Aug
Transformer-networkk 80.2%
0.468
0.397
UNITS-network
81.6%
0.439
0.381
Table 27: Multi-task learning comparison with existing networks under 20 forecasting tasks and 18 classification tasks. UNITS handles all tasks with a unified model and no task-specific head. While baseline models have shared backbone but task-specific input/output heads for each dataset/task. Bold indicates best-performing model for that dataset while underline second-best.
CLASSIFICTTION
UNITS-UP UNITS-PM7
TTANNOORMEE
TitesNer
PAICHIST
PERAPOMMEN
ALTUNOISIER
GPT-TS
ACCURACYT
AccuRacyt
ACCURACET
ACCURACYT
AccURACY?
ACCURACYT
AcCURACT1
HEARTBEAT
0.639
0.654
0,668
0.727
0,659
0.727
0.717
0.698
JAPANESE VOWELS
0.922
0.903
0.959
0.976
0.941
0.854
0.941
0.946
PEMS SF
0.832
0.827
0.832
0.775
0.838
0.832
0.792
0.792
FEL ATIONSCP2
0.489
0.572
0.489
0.528
0.489
0.567
0.45
0.456
SPOKENARABICDIGITS
TWAVEGESSTURELLBRARY
0.822
0.853
0.822
0.844
0.819
0.722
0.422
0.819
ECG5000
0.928
0.933
0.926
0.914
0.93
0.896
0.808
0.882
0.889
0.865
0.214
0.217
0.897
BLINK
0.976
0,916
0.933
0.876
0.896
0.882
0.631
0.924
F'CEDETECTION
0.654
0.58
0.66
0.662
0.639
0.673
0.592
0.661
Fl DEVICES
0.622
0.624
0.573
0.495
0.595
0.654
0.561
0.629
TRACE
0.96
0.99
0.79
0.77
0.74
0.96
FORDB
0.759
0.78
0.727
0.689
0.614
0.553
0.664
0.777
MOTIONSENSEHAR
0.951
0.936
0.906
0.758
0.887
0.302
0.962
EMOPAIN
0.797
0.814
0.794
0.78
0.792
0.814
0.699
0.794
CHINATOWN
0.98
0.98
0.974
0.977
0.977
0.274
0.968
0.965
MEL PEDESSERRRR
0.876
0,839
0.893
0.957
0.804
0.523
0.75
0.94
SHARRPPICEINCREASE
0.618
0.638
0.619
0.65
0.68
0.631
0.615
0.637
BEST COUNT
3/18
7/18
0/18
4/18
3/18
4/18
0/18
2/18
AVERAGE SCORE
0.816
0.812
0.803 X
0.809 X
0.781
0.688
0.656
0.820
FULLY SHARED MODEL
X
Results. UNIT'S considerably surpasses LLM'Time across most the tested datasets, demonstrating superior performance in handling different forecasting lengths and variable numbers (Table 25). For example, UNITS achieves 45.2% improvement in MSE over LLMTime (0.456 vs. 0,832) on River. Remarkably, UNITS exhibits inference speed approximately 10 times faster than LLMTime.
H
Additional Results: Relation among Prompt Tokens
We calculate the similarity between prompt tokens across datasets, illustrated in Figure 7 Datasets within the same class, for instance, FaceDetection and SelfRegulationSCP22 which both consist of EEG data, demonstrate higher similarity. While some out-of-domain datasets still exhibit strong similarities, indicating that they share certain similar requirements.
To compare the difference among tokens before and after training, beyond similarity comparison, we show UMAP plots generated with the prompt tokens before and after training, in Figure and Figure 9 Before training, the prompt tokens from all datasets are dispersed. In contrast, the UMAP of prompt tokens after training reveals that tokens from the same datasets are clustered. However, some tokens from different datasets remain closely positioned, indicating that data from different domains share similar information.
I Additional Results: Classification Performance Stratified by Datasets
We present the performance of multi-task classification each dataset in Table 27
29

---

Table 28: Full results of few-shot multi-tase ttskk learning of block-wiss imputation tasks on 6 datasets.
Mask Ratio
ECL
ETThi
ETTh2
ETiml
ETTm2
Weather MSE
Best Count
Imputation
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MAE
MSE
MAE
Shared
TimesNet-F7
25%
0.245
0.339 0.369 0.403 0.193
0.292
0.442
0.418
0.119
0.229
0.106
0.152
0.246
0.305
0/12
50%
0.258 0,350 0.412 0,420
0.211
0.302
0.607
0.485
0.140
0.247
0.125
0.171
0.292
0.329
0/12
X
PachTSSTFFT
25%
0.195
0.297 0.315
0.361
0.147
0,251
0.309
0.193
0.122
0.191
0,260
0/12
50%
0,230
0.325 0.353
0,382
0.175
0.271
0.442 0.400
0.111
0.214
0.105
0.139
0.236
0,288
0/12
X
TTans-FT
0.174
0.275
0.359
0.185
0.293
0.254
0.319
0.113
0.227
0.087
0.127
0.186
0/12
50%
0.203
0,300
0.332
0.376
0.205
0.307
0.372
0.382
0.136
0.252
0.106
0,150
0.226
0.295
0/12
UNITS-PMT
25%
0.117
0.231 0.281
0.339
0.177
0,281
0.247
0.308
0.095
0.198
0.075
0.113
0.165
0,245
5/12
50%
0.135
0.248
0.325
0,365
0.246
0.351
0,343
0.364
0.131
0.237
0.093
0.139
0.212
0
4/12
V
UT
0.143
0.341
0.284
0.204
0.186
0.163
0.242
7/12
50%
0.161
0.273
0.113
0.361
0.252
0.322
0.295
0.334
0.119
0.223
0.096
0.135
0.206
0.275
8/12
J Additional Results: Direct Multi-step Forecasting on New Forecasting Lengths
Average inference steps comparison. In 6 we present a comparison of the average number of inference steps required by our direct multi-step inference method and the multi-step sliding window-based inference approach. Contrary to the direct multi-step inference, which is completed in a single step, the sliding window-based method necessitates multiple inference steps. Specifically, for the maximum extra inference length of 384, the sliding window-based approach demands, on average, 3.66 times more inference steps.
K Additional Results: Benchmarking in the Single-Task Regime
Setup. As we are the first work that focuses on time series multi-task learning with one model, to make
fair comparisons with existing time series methods, we compare them with the single-task setting.
In this setting, for each dataset, one model independently trained with tuned hyperparameters. Following existing works 2 67 [4 tune the following hyperparameters, including number of channels, patch size, number of layers, learning rate, and dropout ratio. The baseline methods for time series forecasting, classification, anomaly detection, and imputation, are listed in Table 13 We
following existing works [ 67 to use 36 commonly used datasets for forecasting (Table 30 10
Table 29: Full results of few-shot multi-task learning on 9 forecasting and 6 classification tasks on out-of-domain datasets. Ratio is the data ratio of the dataset used for training.
Classification (Acct)
5%
15%
20%
(6 datasets)
TTans FT UNITS PMT UNITS- F7 TTans FT UNITS PM7 UNITS F7 iTrans FT UNITS PMT UNITS F7
ECG200
0.780
0.790
0.790
0.810
0.760
0.820
0.810
0.820
0.820
Handwriting
0.054
0.044
0.061
0.098
0.089
0.080
0.118
0.087
0.081
SelfRogulationSCPI
0.928
0.816
0.758
0.679
0.648
0.672
0.771
0.676
0.737
RacketSports
0.375
0.316
0.487
0.546
0.474
0.618
0.546
0.539
0.586
Epilepsy
0.399
0,514
0.522
0.413
0.732
0.681
0.500
0.797
0.855
StarLightCurrvs
0.851
0.862
0.826
0.842
0.869
0.834
0.848
0.895
0,833
Average
0.564
0.557
0.574
0.565
0.595
0.618
0.599
0.636
0.652
Best Count
1/6
1/6
4/6
2/6
2/6
2/6
2/6
2/6
2/6
Forecast
5%
15%
20%
(9 datasots)
FTran--FT UNITS-PMT UNITS-FT iTransFFT UNITS-PMT UNITS-FT iTrans-FT UNITS-PMT UNITS-FT MSE MAE MSE MAE MSE MAB/ MSE MAB MSE MAB MSB MAE] MSE MAE MSE MAE
MSE MAE
ET ETTh2
D.554 0,500 0.397
0.406
0.414
0.419 0.440 0.390
0.404
0.400 0.409
0.418 0.40910.418
0.426 0.387
0.403
0.396 0.407
ET p192
0.440 0.438 0.385
0.399
0.390
0.401 0.398
0.410 0.390
0.403
0.376
0.393
0,395
0.407
0.394
0.406
0.378 395
ETTh2p336
0.478
0.434
0.434
0.436
0.436
0.431
0.438
0.425
ETTh2p772
0.483 0.480 0.438
0.451
0.431
0.444 0.438
0.453 0.442
0.422
0.427
0.444
0.431 0.449
0.428
0.448
0.424 0.442
RiverFlow
p21
1.141
0.514 1.111
0.504
1,160
0.521 1.067 0.467 1.074
0.489
1.096
0.501
1.056
0.462 1.884
0.494
1.078 0.495
ETTml poc
0.504
0.370
0.397
0.417 0.423
0.419
0.385
0.408
0.357
0.391
0.346
ET ETTmIppu P192
0.555 0.485 0.416
0.421
0.453
0443 0.464
0.439 0.402
0415
0.394
0.406
0.444
0.428
0.398
0.414
0.386 0.401
ETTml p:30s
0.567
0.496 0.467
0.451
0.509
0.465 00.492
0.457 0.446
0.441
0.425
0.425
0.471
0.445
0.442
0.439
0.417 0.421
ETTml P720
0.659
0.565
0.500
0.573
0.499
0.558
0.493
0.484
0.460
0.536
0.482
0.527
0.481 0.454
Average
0.598
0.487
0.508
0.440
0.530
0.448
(0.524
0.447
0.496
0.435
0.428
10.510
0.438
0.494
0.435
0.481
0.425
Best Count
0/9
0/9
8/9
7/9
1/9
2/9
1/9
1/9
1/9
1/9
7/9
7/9
1/9
1/9
1/9
0/9
7/9
8/9
30

---

Table 30: Full results of the single-task long-term tergteerm forecasting task where the model is separately trained on each dataset. The input time series sequence length is set to 96 to ensure fair comparisons. Baseline results are obtained from 677.
Models
UNITS-T
ITranatormer
RLinear
PuccTIST
Chossformer
TimesNet TB
DLiseer 11
SCINet
FEDiormer
Statinary +
Aunoformer
(Qurr)
620
LPS
Metric
MSE
MAR MSS
MAH
MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE
MAE
MSB
MA
MSB
MAE
MAE
96
0.310 0.351 (0.334 0. 0.168
0.355 0. 3767/ 329 367 0. 0.40 4-426 O. 364 0.3 0.38 0. 0.338 0.375 0.345 0.372 0. 0.438 0.4990..36 0. 7.308 o 505 0.475
192.
0.357
01.82 0.377 0.591
0.991 0.392 0. 1.367
185
0.450
451 O. 0.798 0 404 O. 0.374 0.387 0 n 589 0.49 0 0.500 (.476 0.44] (.459 0.444 o 5.5 0.406
336
0.392
0.399 0.410 0,532
0.515
0.4 0.185 0.445
720
0..47
0.439
(1.491
0.439
0..85
0.450
0.454
0.439 0,666
0.5M9 0.487 0.461 0.478 0.430
(0,474 U.453 0.599 0..50(0.543 0.440
0.585 0.536 (1071 0.501
Avr
(0.377
0.395
(0.407
0.410
[0.414
0.407
(0335 0.400
0.513
0.496 0:419 0.419 0,400 0.403
0.407
0.481 (0.448
0.452 0.481 0.456 0.588 0.517
E1Tm2
192
96
0.171
0:238
0.255
(0,880 0.264
0.250
10.182 O
0
0,175
0.24] 0.302 0:414
0.259
0,267
0.492 0.290
0.366
0.207 0.305 0.167 0.267 0.193
0.29210.286
0.362 0.399
0.377
0.445 0.266 0.328 0.280
0.287 0.19 0.254 0.255 0.339
0 0,340
336
0.299
0.542
(1:111 0.248
0,307
0.342
0..355
0.345 0,597 0.542 0.373 0.422 0.321 0.351 00.369 (L427 0.637 0.291 (..225 0.366 0.334 0.361 (0.339 0.392
720
0.363
01,995
(1412 0.407
(0:407
0.39% 0.401 4.400) 7,750 042 0,558 O 0.524 0.408 0.403 0.554
01.22
0.960 0.735 0.42) 01.415 (0.417 0.443 (0.433 01442
Avg
002755
0.523
(0,269
01.322
(0.266
0.327
0.327/0.281
(0223]
0.326 0.757
016 0.358 0.404 0.291 0.333 (0.500 01400 [0557 0.537/ 003 0.34%) 0.30 0.347 (0:327 0.571
96
(0.366
0.405
00.346
0.345
0.414
0.419
0.423
0:479
0.384
0.402
0L385
0.400/o.655 0.599 0.376 0.419
0.491
0,459
192
0.404
0.425
00.011
0.436
0.439
0.424
0.450
0.445 0.871
0.474
0,525
0.492
0.436
0.429 (0.177 0.432 0.719
0.631
0.120 0.448 00.534 0.504 0 0.482
536
0.405
0.482
01487
0.458
0.477
0.446
0,.503
466
0,570 1.54 0,565 0,515 0.491 0.469
0.461
0.759
(1.59)0.455
0.459 0.465 00.588 01535 (1521 0.406
730
0.437
500
0.653
0.836
0.600 o 0.507 0.643
0 0,512
Avg
00.403
11.444
(0.454
0.447
[0:446
0.434
0.4499
454 0,529 O 522, 0.541 O 507 0.458 0.450 0.455 0.452/0.747 01647/00440 0 0.00 0.537 0.44% 0.487
96
0.2233
0.537 (1.297 (1.49
(0.285
0.338
0. 902
348 0.745 584 0.400 O 440 O. 0,340 0.574 00.33 01388
0.621 (..5% 0.307 0.447 0.45% (0 346 0.388
192
0.367
01,399 (1.30) 01400
0,374
0.350
0,3388
0.400 0.877 056 0.528 0.509 0.412 0.414 (1477 11476
0.660
(1689
0.429 0.439 0.512 0 0.456 0.452
336
0.404
(1478
0.435
0.643
0.452 O 0.541
1.000
0 0.487
0
720
0.411
0.434
(0-127 0.445
0.120
0.440
0.443 0.446
I. 1.004 0.763 0.874 0,679 0.462 0-468 (0831 0.657
1,249 0,838
0.463 0.474 0.474 (0.522 0.560 0.515 0.5L
Avg
(0:366
0.295
(0.383
0.407
(0.374 0.796/0.367 0.407
O 0,550
(0559 0,515
0.723/0 437 0,449|0.526
0.450 0,459
96
0..32
0.228
(0. O.
(0.201 0.281 0 181 2,20 O. 0,219 O 314 O. 0.237 O 329 O. 16668 0 2.2222 O. 197 0 20..0.2.27777 0. 0.3 0. 0.306 (n 169 0.273 ( 1.101 0.
ECL
192
0.158
0
0.
0.274 0.251
0.236
0
O. 0.285 0.257 0.355 0.201 0.315 33:55.1188
222 0.
336
0..68
0.264
0.178
(1
0.2155 O 298 0.204 0.293 0.245 0.3 0.249 344 O. 0 300 0.209 0.501 0.269
0.369 0.214 0.329 0.200 0.304
0:233 0.338
720
(0.192 0.287
(12255
0.317
0.231 0.226 0.324 (.280 0.363 0.284 0.373 0,220 0.320 0.2445 0.333 0.299 0.290/0.246 0.355 (..22 0.321 (0.254 0.361 0.257
Avg
(0.163
0.258
(0.178
0.271
10.219 O 2229
0.299/0.200 (..205 0.290 0.264 O 0.334 0.251 0.344 O. 0.192 O 0.295 0.211 0.300 0.200000000000
0,365
0.327 0.193 0.296 (0.277 0.338
96
0.416
1..89// O
29510. 0.522
0.493 0.593 00 O 0.396 0.199/0
0.366
6.12 0.338
0,358
Trafitc
192
0.456
012272
00.477
0.276
0..03 O 360 0.056 0.296 0.
293 0,756
0.474 0.617 0.336 0.595 (L370 0.279 0.200 0.004
0.373 (.615 0.340 0.016 0.382
336
0.448
0.250
0.433
0.283
0.609
O. 0.369 0.482 0.304 0.558
0,305 0.762
0:477 0.629 0.336
0.373 0.797 0.508 0.621 0.283 0.6188 0.328 0.622 0.337
730
0.513
0.6647
0,589
0.719
0.6-2 0.350 113994 0.841 0.523 0.626 0.382
0,408
Avg
0.455
11229
(0.428
0.282
00.02
0.378) 0.483
0L304 0,550 300 760 0.473 0.620 0.336 0LO25 01383 0.00 (1209) 0.610 0.376/0.024 0.340 0.628 0.579
96
0.149
0.198
00.174
0.214
00.192
0.292 0.177 0.218
1.211/0.1E 0.155 230 0.202 0.261 72 0 220 0.196 0.255 0.221
0.200 0.2 0.296/0.175 0.223 0,266 0.336
Weather
192
0.200
0.343
11:211
0.254
0.240
0.271 0.225
0,259
D.206 O 2.7 0.242 0 292 0.219 0 0.261 0.257 0
0.261 a (1340 0.276 0.336 0.25 0.255 0.307 0.367
336
0.257
0.286
01299
0.242 0
0.297
0.272 0.287 0.335 0,280
01335
0,309
0.37M 0.339 0.380
0.33M
0.395
730
0.334
0.347
0.964
0 3.353 0.354 348 0.398 0118 0.351 O O. 36565 0 0.359 0.345 0.581 0.377 0.422 0.4 0.438 ( 414
0.410
0.419 0.438
Aug
0.235
01.26
252259
01277
(0.272
0 291 (0.250 0.281 (0.259
0.271 0.320 259 0.287 (o (.265 0.31 0.317/0.202 ( 0.363 309 0.360
Solar- Energy
96
0.188
0.225
112003
012337
10.322
0.339/
0.286
0,310 O 0.331 0.312 0 0.399 O. 0.250 0 0.292 1 0.290 (1578 0.23/ ( (1344
0.24 0.342 0.215 0.349) (1.884 0.711
192
0.229
0.390 0 156 0.
310
0.734
0.3
0.41
0. 120
308 0.280
380
0.380 0
0.833 0.692
336
0.360
0..97
0.290 0.315 0.750
0.73 0.368
0.430 0.519
0.353
0.415 0.304
0.380 0.282 01.76 002900 0.206
0.723
730
0.249
0.277
0.349
0.275
0.397 0. 356 0.280
0.317 0.
o 766
0.370 0.425 0.338
0.337 0..356 0.413 0.388 0.357
0.427 0.288 0.296 0 0.882 0.717
Avg
(0.225
0.254
233
a 0.362
Jo. o 356 100.000776611
0.641 0.630
0417
0.301 0.319 0.330 0.40
|0 0.375/0
0.38 (0.66) 0. 385 0.711
Rost Count
28
datasets for clasiiiicatinnTabll 31 4 datasets for imputation (Table 10), and datasets for anomaly detection (Table TT].
Forecasting. We compare the forecasting performance with the forecasting length of 96, 192, 336, and 720. To make fair comparisons with baseline methods under different look back windows, we have forecasting results in both fixed and optimal back windows. The full results for forecasting with a 96 look back window are shown in Table 30 The full results with optimal look back window ranging from 96 512 are shown Table 34
Classification. Following 112,, we use 10 multivariate datasets from the UEA dataset collection 4 The full results for classification are shown in T1
Imputation. Imputation aims to fill in the missing data points of the time series samples. We randomly mask data points of the time series samples with mask ratios of 12.5%. 25%, 37.5%, and 50%, and then make the model predict the missing points. The full results of the imputation task are shown in Table 33
Anomaly detection. Anomaly detection identifies the anomalous data points the time series samples. We present the complete results of anomaly detection in Table 32
31

---

Table 31: Full results for the single-task classification task. in the Transformers indicates the name of *former. We report the classification accuracy (%) as the result.
Classical methods
RNN
TCN
Transformers
MLP
Freq.
Datascts / Models
DTWXIBewoRodeet STMLSTNN LSSL
Trans,
Ro.
Pyra, Auto, Starion. FED,
DLincsrLighTTS NeUiSSS
g
151
11
53
99
30
104
51
12
65
69
128
1111
113
119
[22]
(Ours)
82.3
43.7
45.2
32.3
39.9
31.1
28.9
32.7
31.9
31.6
30.8
31.6
32.7
31.2
28.1
33.8
32.6
29.7
35.7
37.6
FaceDatcction
52.9
63.3
64.7
65.7
66.7
52.8
67.3
68.6
67.0
65.7
68.4
68.0
66.0
66.3
67.6
68.0
67.5
68.6
70.5
Handwriting
15.8
58.8
24.6
27.4
29.4
31.6
27.0
26.1
29.7
Hearthent
71.7
73.2
79.6
72.2
77.1
72.7
75.6
76.1
77.1
80.5
75.6
74.6
73.7
73.7
71.2
77.6
75.1
75.1
78.0
80.0
JopaneseVowels
94.9
86.5
96.2
79.7
98.1
98.4
98.9
98.7
97.8
98.9
98.4
96.2
99.2
98.4
95.9
98.9
96.2
96.2
98.4
97.8
PEMS-SF
71.1
98.3
75.1
39.9
86.7
86.1
68.8
82.1
82.7
81.5
83.2 82.7
87,3
80.9
86.0
83.8
75.1
88.4
89.6
93.1
SellRogahaionSCP]
77.7
84.6
84.0
90.8
84.6
90.4
89.4
87.3
89.8
91.8
93,9
SellRegahtiinSC22
53.9
48.9
53.3
46.6
52.8
52.2
55.6
53.9 56.7 53.3
53.3
50.6
57.2
54.4
55.0
56.1
50.5
51.1
57.2
61.1
SpokenArahicliigts
96.3
69.6
71.2
31.9
100.0
100.095.6
98.4 97.0100.0999.6
100.0
100.0
100.0100.09%
98.8
81.4
100.0
99.0
98.9
UWWeeGesturel Jhoary
90.3
75.9
94.4
41.2 87.8 85.9 88.4 85.6 85.6 85.6
83.4
85.9
87.5
85.3
85.0
86.6
82.1
80.3
85.3
87.8
Average Accuracy
67.00
66.0 72.5 48.6
71.8 70.9 70.3 71,9 71.5 72.1 70.8
71.1
72.7
70.7
71.0
73.0
67.5
70.4
73.6
75.0
Table 32: Full results for the anomaly detection task. The P, R and Fl represent the precision, recall and Fl-score (%) respectively. Fl-score is the harmonic mean precision and recall.
Datasets
SMD
MSL
SMAP
SWaT
PSM
Avg FI
Metries
RT
(%)
LSTM
41
78.52
65.47
71.41
78.04
86.22
81.93
91.06
57.49
70.48
78.06
91.72
84.34
69.24
99.53
81.67
77.97
Transformer
104
83.58
76.13
79.56
71.57
87.37
78.68
89.37
57.12
69.70
68.84
96.53 80.37
62.75
96.56
76.07
76.8K 88
LogTrans
57
83.46
70.13
76.21
73.05
87.37
79.57
89.15
57.59
69.97
68.67
97.32
80.52
63.06
98.00 76.74
76. 60
TCN
30
84.06
79.07
81.49
75.11
82.44
78.60
86.90
59.23 70.45
76.59
95.71 85.09
54.59
99.77 70.57
77.24
Reformer
S1
82.58 69.24 75.32
85.51
83.31
84.40
90.91
57.44 70.40
72.50
96.53 82.80
59.93
95.38 73.61
77.31
Informer
127
86.60 77.23 81.65
81.77
86.48
84.06
90.11
57. 69.92
70.29
96.75 81.43
64.27
96.33 77.10
78.83
Anomaly"
116
88.91 82.23 85.49
79.61
87.37
83.31
91.85
58.11 71.18
72.51
97.32 83 68.35
94.72 79.40
80.50
Pyraformer
65
85.61 80.61 83.04
83.81 85.93 84,86
92.54
57.71 71.09
87.92 96.00 91.78
71.67
96.02 82.08
82.57
Autoformer
11
88.06 82.35 85. 77.27 80.92 79.05
90.40
58.62 71.12
89.85
95.81 92.74
99.99999999
99.08 88.15 93.29
84.26
LSSL
39
78.51 65.32 71.31 77.55
88. 18 82.53
89.43
53.43 66 66.90
79.05
93.72 85.76 66.02
92.93 77.
76.74
Station.
69
88.33 81.21 84.62 68.55
89.14 77.50
89.37
59.02 71.09
68.03
96.75 79.88
97.82 96.76 97.29
82,08
Diine
119
83.62 71.52 77.10
84.34
85.42 84.88
92.32
55.41 69.26
80,91
955.0 87.52
98.28
89.26 93.55
82.46
RTSformer
79.23
85.13
84.93
85.03
55.75
90.02
84.91
99.31
85.28 91.76
82.87
LighTTS
122
87.10
78.42 82.53
82.40
75.78 78.95
92.58
55.27 69.21
91.98
94.72 93.33
98.37
95.97 97.15
84.23
FEDformer
128
87.95
82.39 85.08
77.14
80.07
78.57
90.47
10 70.76
90.17
96.42 93.19
97.31
97.16 97.23
84.97
TimesNet"
122
87.95
81.54 84.62 89.55 75.29 81.80 11.80
90.14
56.56 69.50
90.76
95.35 93.00
98.50
96.29 97.38
85.26
UNiTS-ST
Ours 89.32 86.90 88.09 89.91 77.68 83.46
93.37
76.02 83.80
92.37 94.17 93.26 98.62 96 28 97.43
89.21
For fair comparisons, we follow the settings of 122 only use reconstruction error for Anomaly Transformer.
TimesNet are reproduced from https::/gittuub.com/tuuuu fair comparisons.
Series--erreesee Librar to ensure
32

---

Unified
3.5
Sliding window
Average Inference Steps
3.0
2.5
2.0
1.5
1.0
128
160
192
224
256
288
320
352
384
Trained length + new length
Figure 6: The comparison of average inference steps between our direct multi-step inference and
multi-step sliding window-base -base inference for zero-shot forecasting on new lengths.
Table 33: Full results for the imputation task. We randomly mask 12.5%. 25%, 37.5% and 50% time points to compare the model performance under different missing degrees.
Models
UnITS-57
ETS
LighTS
DLame
FED.
Asto
Pyna
LoxTean
Ri
LSTM
ICN
LSSL
(Curs)
IIH
18
[-]
21
MABIMSE MAEIMSE
MAEEMS
MSE MAE] MAB
25R
220
2.206 o 00000000 tor 1 pas ce
0.55 0.533 raco nu0 ouz 1o2 o auso0
0219
0052 0.0 0,792 1o 0.199
ju 01244
Ang
1.019
01225 0.218 puoso 0.20s
covnnuu u o
ous44
201
8.168.114
Sageen 1780
207
00040
0.31
135
0.577
0.002
0.44 (0
Jox
00943
202
01.20
0.313 11201
0.30
0.201
0.21
125
0.513 8.57) (/.42
Blentieiyy
0.300
0.52
287
Ra15
8.19
0,20
2.707
0.14
12190.000
200000000000000
103 1.5 aust
37.55
0
0.709
3010 31500
0.109
(.55)) 0.97
Ang
[uBS 8.13
150
11 0.M0 100
101 01:20
5.155
315
sed
Weathr
12.5%
0084404411
0084
a
255
0
0.103
0004
0.00000
ouate
187
0.042
37.5%
6827
8045
0,100 0.121 RC57 0.11 0.000
0230
0.02
0.60
0240
0049000040..11 0.100.111
5039
0.00
0.09)
0.406 0.463
(0947
Avg
117 17 01es 01117 Boo2 0.110 1ovo 20 0100 wtw.e//../e/////ee
cuuuusssuuss 0.29 /oo40
Additional Results: Multi-task versus Single-task Learning
To verify the gap between multi-task and single-task learning under fair comparisons, we conduct a experiment to train the single-task models using the same hyper-paramend parameters as the multi-task co- training. As shown in Table multi-task learning achieves stronger performance on both forecasting and classification tasks. Interestingly, under the same hyper-parameters, some classification models fail to converge in the single-task setting, whereas the multi-task model does not have this issue, demonstrating the robustness of multi-task training.
33

---

Table 34: Full results of the long-term forecasting task where model is separately trained on each dataset. The input time series sequence length is set ranging from 96 to 512 to ensure fair comparisons. Baseline results are obtained from their original papers. "Extra Training Data" indicates whether the model uses training data beyond just time series data. "Multi-task Support" refers to whether the model can handle multiple tasks or is focused solely on single task. Gray color represents LLM-reprogrammed models that reprogram pre-trained LLMs to time series domain and needs
seasetttaas-sppciifee modules. For the best count, we only consider the purely time series models.
Models
UniTS-ST
(Ours)
MOMENT
TSMixer
TEMPO
TIME-LLM
47
LLM-TS
12
TEST [97
GPTITS
1
Metric
MSE
MAP
MSE
MAE|
MSE
MAE
MSE
MAB
MSB
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.2778
0.338
0.293
0.349
0.285
0.359
0.458
0.424
0.272
0.334
0.360
0.388
0.293
0.345
0.292
0.346
E1 F77ml
192
0.319
0.364
0.327
0.365
0.461
0.432
0.310
0.358
0.386
0.401
0.332
0.369
0.332
0.372
335
0.354
0.386
0.356
0.382
0.515
0.467
0.352
0.384
0.415
0.417
0.368
0.392
0.365
0.394
720
0.397
0.416
0.405
0.416
0.419
0.414
0.591
0.509
0.338
0.411
0.470
0.445
0.418
0.420
0.417
0.421
Avg
0.337
0.376
| 00,399
0,383
0.347
0.375
00,501
0.458
| 00.529
0.372
0,,408
0.413
00.353
0.382
0.3352
0.383
0.167
0.
0.258
0.181
0.269
0.163
0.252
0.185
0.267
[0.161
0.253
0.184
0.265
0.173
0.262
192
0.233
0.295
0.216
0.290
0.243
0.304
0.219
0. 0.293
0.240
0.501
0.229
0.301
335
0.270)
0.325
0.268
0.324
0.309
0.345
0.271
0.320
0.294
0.357
0.286
0.341
720
0.35%
0.360
0.366
0..888
01:400
0.422
0. 38666666
0.395
0.352
03.770
0,386
0. 3.393
0.378
0.401
Avg
0.254
0.315
0.224
0.329
0.267
0.322
0.281
0.328
0.225
0.3
0.276
0.304
|0.284
0.339
0.387
0,406
0.362
0.400
0.397
01,01
0.404
0.426
0.308
0.403
0.414
0,418
0.425
0.420
0.441
0.430
0.430
0.421
0.422
0.457
0.433
0.434
0.454
0.472
0:463
0.445
0.442
0.422
0.444
0.444
0.467
0,456
Avg
0.405
0.421
0.441
0.412
0.428
[04428
0.427
00:408
0.424
00404
0:418
104414
0:431
00,428
0.426
0:277
0.346
002288
0.345
10.274
0.341
0.300
0.353
0.268
0.37%
(0.760
0.352
0.275
0.338
0.2885
0.325
0.382
0.339
0.385
0.355
0.389
0.529
0.322
0.3 377
0.340
0.379
0.347
0.398
0:361
0.406
0.379
0.408
0.368
01358
0.529
0.380
01:007
0.373
0.420
0.403
0.439
0:445
0.470
0.409
0.440
0.372
0.420
0.383
0.425
0.388
0.223
0,406
0.441
Avg
0.331
0.387
0.334
0.392
0.355
0.401
0.361
0.398
[ 0.334
0.383
0.333
0,388
00:331
0,380
00,355
0.395
96
0,130
0.224
0.138
0.212
0.131
0.229
0.178
0.276
0.131
0.221
0.128
0.223
0.132
0:223
0.139
0.238
192
0.147
0.242
0.151
0.216
0.198
0.293
0.152
0.211
0.116
0.210
0. 0.158
0:251
0.153
0.251
0.160
0.260
0.161
0.261
0.209
0.309
0.160
0.248
0.163
0.258
0. 163
0.260
0.169
0.266
720
0.188
0.284
0.211
0,305
0.197
0.293
0.279
0.355
0.192
0.288
0.200
0. 199
(.2291
0.206
0.297
Avg
0.156
0.253
| 0.175
0.274
0.160
0.257
0.216
0.308
| 00.159
0.253
0.119
0.253
0. 0.163
0.253
00.167
0.263
96
0.370
0.255
0,391
0,282
0.376
0.264
0.476
0.343
0.362
0.244
0.372
0.259
0:407
0.282
0.388
0.282
192
0.390)
0.263
0.397
0.277
0.496
0.355
0.374
0.244
0.391
02265
0-423
0.287
0.407
0.290
336
0:415
0.26N
0.413
0.290
0.503
0.356
0.385
16277
0:4015
0235
0-430
0.296
0.412
0,294
720
0.461
0.326
0.450
0.310
01444
0.306
0.538
0.376
0.430
0.284
0:457
0 0.202
00 8.465
0.318
0.450
0.312
Avg.
0.409
0.278
| (0:21
0,296
0,408
0.284
00.503
0.358
| 0.388
0,264
0.401
0.273
00:431
0.295
00:414
0,295
Weater
192
0.185
0.140
0.192
0.237
0.154
0,209
0.145
0.191
0.242
0.108
0.211
0.254
0.254
0.268
0.147
0.189
0.234
0.201
0.147
0.191
0.238
0.195
0,150
0.198
0.246
0.202
0.162
0.204
0.248
0:212
3356
0.234
0.278
0.242
0.280
0.292
0.332
0.262
0.279
0.241
0.277
0.245
0.286
0.254
0.286
720
0.306
0.330
0.315
0,336
0.320
0.336
0.370
0.379
0.304
0.316
0,313
0.329
0.324
0.312
0.326
0.337
Avg.
0.216
0.259
00,235
0.273
0.225
0.264
0.282
0.316
0.226
0.25%
0,223
0.260
0.229
0.269
|02337 0.271
Best Count
21/28
19/28
0/28
0/28
7/28
9/28
Extra Training Data
Yes
Yes
Yes
Mulri-cak Suppeet
Na
No
Table 35: Compare UNITS trained by multi-task learning with that trained by single-task learning under same syper-parameters
UNITS
Aug (Classification)
MSE (Forecasting)
Multi-task
81.6%
0.439 65.3%
Single-task
0.464
M Limitations and Future Directions
The datasets collected by this work do not yet cover all available time series datasets, such as some of the univariate datasets in UCR dataset collections (23 and the more physiologic time series signals from PhysioNet [34]. We will explore using larger dataset collections to further improve UNITS.
UNITS primarily aims to unify predictive and generative tasks within a single multi-task model. We demonstrate this by showcasing its adaptability to new data and tasks through prompt learning and few-shot learning. While adapting to new time series data differs fundamentally from generalizing to entirely new data, we will further explore UNITS's generalization ability for zero-shot learning.
34

---

UWAvEGessturi lbbrary
AeaeeennraiiDDiitt Digits
Noninvasiive wosniiaaiieeeeeeaeeeeee
japa oppantenn bpunesellewed
balRegulationSCCC
MetiornSensHAA
Share
Mclbourne
SpekennArabice
Face
ElectricDevicess
Exchange
PEMS SE
Chinstown
Weather
ECG5000
EMOPaIn
NNS
ECL.
ET
Traffic
Blink
Trace
FaraB
11
NN5
Exchange
SharePricetncrease
ECL
ETTh1
IU
Traffic
PEMS-SF
WelboumePPdeeee
Chinatown
Weather
Heartbeat
JapaneseVowe
SpokenMrabicDigit
Wodinvasiveeate
ECC5000
ECGThorax]
Blink
FaceDDtecction
SelfkegulationSCP2
ElectricDevices
0.2
MotionSenseHAR
EMOPain
uwaveGeesturel.ibrgg
0.0
Finance
Electricity
Epidemiology
Traffic
Weather
Audio
ECG
EEG
Sensors
Human
Activity
Figure 7: The similarity of prompt tokens among datasets.
N Impact Statement
This paper focuses on analyzing time series sequences from various domains and introduces versatile machine-learning approach designed for this purpose. While our research has numerous potential societal impacts, we believe none require specific emphasis in this context.
35

---

UMAP projection of prompt tokens before training.
Heartbeat
Trace
Exchange
SharePrcainccmase
MotionSenseHAR
Spokensrabiccigits
Weather
Blink
SelfReeulltionSSCP
Electric
ECC5000
Traftic
NN5
PEMS-SF
RaceDetection
JapaneseVowels
MelboumePedsstrinn
UWBaeGissturalbbrray
-7
Ford8
Figure 8: UMAP of untrained prompt tokens in UNITS. This plot illustrates that there no significant organization (clustering) of prompt tokens prior to UNITS training.
UMAP projection of prompt tokens after training.
Identifler
www.waistturatbbrrayy
12
Blink
Chinatown
Sell SellReggultionSSCP
E
Trace
ElectricDevices
Farde
JapaneseVowels
PEMS-SF
Heartbeat
ShanePricetncrease
MelboumePedastriinn
MotionSenseHAR
FaceDeteccton
NNS
Traffic
Exchange
ECL
2.5
SpokenArabicDigtts
Figure 9: UMAP trained prompt tokens UNITS. Unlike Figure above, this plot illustrates the meaningful organization (clustering) of prompt tokens by dataset domain category when trained by UNITS.
36

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction cover the contributions and scope of the paper regarding building a unified time-series model.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section M we discuss the limitations and future work regarding the size of dataset collections and the exploration more advanced network architectures.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
37

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the implementation details for experiment settings in Sec- tion 5] Section [D] and Section K We also provide the source code and datasets at https://gitane //t.tbb/tttbb... www.coommmmmmmmm wwwmmmmsoomeooooooooooooooo..o.m hatp:////aaaaaaaaaaa///ttt harvard/UniTS
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
38

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the source code and datasets at httpp:://githbb com/ mims- mamas-havvar//Unii Instructions for downloading data and running experiments are provided inside."
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
wwwpc//wwgeeeeeeeeeeeeeeee Saweeees//ooSSbbiiiiaa SonroddeummsssiooPPe.. wwriiisssiiiiiyiyioo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the details for training and test settings in Section 5 Section and Section K Full details are shown in the provided code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We fixed seed in experiments to maintain low statistical variance. Additionally, we conduct experiments on diverse datasets to ensure the statistical significance of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
39

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computer resources we used in Section D.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We have read the NeurIPS Code of Ethics and adhere
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
40

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use open source datasets and codes based on their licenses. Most baselines are released under mit lisxxx. We use opsource datasets preprocessed xxx we did not crxx build new dataset this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
41

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
42

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
43


Reviews:
[{"review_id": "xLQPnY3zC3", "reviewer": "Reviewer_h6vS", "summary": "This work aims to present a pre-trained foundation model for time series. They proposed a model called UNITS, that performs multi-task learning (both generative and discriminative) on time series datasets. Specifically, they embed a transformer with prompt tokens and task tokens to perform prompt tuning or few-shot learning. The proposed method is pre-trained on multi-domain datasets, and showed transferability across multiple downstream datasets with diverse task specifications and domains. The authors performed experimental results that show UNITS performs favorably, or comparably against many state-of-the-art models, and can be used in few-shot and prompt tuning ways.", "questions": "See above.", "limitations": "Can be more comprehensive.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Overall, the work has strong experimental results.\n- Great engineering efforts to perform cross-dataset training on multiple tasks.", "weaknesses": "Major concern 1: Overall, the contributions from the paper are overclaimed. \n- The authors claim \"generalizability to new tasks\", yet (1) the proposed model is pre-trained on all datasets that are later experimented on; (2) for all available tasks, the authors perform prompt tuning on an aggregate dataset-centric loss value that includes all tasks that are later considered. If the model does not see a new dataset or new tasks during test time, why claim generalizability to new tasks?\n\nMajor concern 2: Lack of analytical or theoretical understanding of how multi-dataset training benefits performance.\n- Lack of ablation experiments. For example: How much does the amount of pre-training datasets help (in terms of training steps or amount of datasets)? More systematic evaluation of the interrelationship between pre-training tasks (beyond Table 17)? How does the performance vary when the number of prompt tuning parameters change? \n- Lack of better understanding of how to handle varying channel-wise relationships. One of the biggest challenges in building foundational model for time series is that while temporal dynamics may be generalizable, the channel-wise relationships differ significantly for each dataset. How does the proposed method address this issue through pre-training, or does it just see all possible channel-wise relationships, that are going to appear at test time, during the pre-training process?\n- Lack of understanding of the learned tokens. How do the sample tokens capture the dynamics of the data, can they be visualized or categorized? How do the task tokens or prompt tokens differ from other task tokens or prompt tokens after training beyond similarity comparison? Does the similarity comparison show the intrinsic dynamical properties of the datasets?\n\nMinor concern 1: In experimental results, the baseline numbers seem to be slightly lower than that are reported in other papers.\n\nMinor concern 2: Lack of methodological novelty. Due to the lack of ablations, it is challenging for me to dissect which parts of the proposed model are critical to the results. The multi-task parts seem to be the work that tries to show that the more datasets one uses to train their model, the better the performance will be, as long as one uses the right combination of existing methods.\n\nMinor concern 3: Can the authors compare the number of prompt tuning parameters against other baseline models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer h6vS Part I (Part I of IV)\n\nThank you for your detailed comments and valuable feedback. We appreciate your recognition of experimental results in our work. We have carefully addressed each of your concerns and have added new experiments and analyses based on your suggestions. \n\nWe hope that our response and new experiments address your concerns, and we kindly ask that you consider raising your score. If, after reviewing our responses, you still have concerns about our work, we would greatly appreciate further guidance on how we can improve it to achieve a better score in your review. Thank you again\\!\n\n---\n\n### Major concern 1.1: Generalizability to new tasks: the proposed model is pre-trained on all datasets that are later experimented on.\n\nWe'd like to clarify this is a misunderstanding. We are confident and have confirmed that there is no information leakage between the training set in pre-training and the testing set in fine-tuning. Furthermore, we consider the generalization to new tasks and generalization to new datasets that have never been seen during pre-training. Moveover, we even consider the generalization to new data domains (e.g., traffic, weather-\\>spacecraft, web). The details of these generalization settings are in the following:\n\n* **Few-shot learning on new data and tasks**: 1\\) The datasets for few-shot classification and forecasting tasks are not used during pre-training, demonstrating generalization to new datasets (Tables 3, 24, Page 8, 29). 2\\) For few-shot imputation, the imputation task is not performed during pre-training, demonstrating generalization to new tasks (Tables 4, 23, Page 9, 28). The ETTm1 dataset is not used for pre-training, ensuring generalization to new data and new tasks (Tables 23, Page 28). 3\\) For anomaly detection, both the task and datasets are not used during pre-training, ensuring generalization to new data and new tasks (Tables 5, Page 9).  \n* **Direct multi-step forecasting to varying time lengths**: UniTS generalizes to new forecasting lengths not seen during training, which demonstrates task generalization (Figure 3 of Figure PDF).  \n* **Zero-shot forecasting**: UniTS conducts zero-shot forecasting on new datasets that are only seen during testing (Tables 21, Page 26).  \n* **Generalization to new domains:** UniTS is evaluated not only on tasks and datasets that were never encountered during pre-training or fine-tuning, but our experiments also consider generalization to new domains. These include the domains of server machine data (SMD, PSM), spacecraft (MSL, SMAP), and infrastructure (SWaT) in anomaly detection tasks (Table 11, Page 20), as well as the domains of healthcare (Hospital) and web data (Web Traffic) for zero-shot forecasting (Table 9, Page 20), none of which are seen during pre-training.\n\n### Major concern 1.2: Why claim generalizability to new tasks?\n\nUniTS has two levels of generalizability, 1\\) **Zero-shot learning** where the model is presented with new datasets or new tasks at test time that it has not encountered during training. 2\\) **Few-shot/prompt learning** where a pre-trained model is adapted to a new task via fine-tuning using only a small part of a specific dataset. \n\n* For zero-shot learning, we benchmark against LLMTime, a model designed for zero-shot forecasting using LLM. Table below shows UniTS considerably surpasses LLMTime across most of the tested datasets, demonstrating superior performance in handling different forecasting lengths and variable numbers.  For example, UniTS achieves a considerable improvement in MSE over LLMTime (0.030 vs.0.265) on Solar. Remarkably, UniTS exhibits an inference speed approximately 10e6 times faster than LLMTime. We also show in Figure 3 of Figure PDF (Figure 3 of the manuscript, page 8\\) that UniTS can generalize to new forecasting lengths not seen during training.\n| Solar dataset | MSE | Infer. Time (seconds) |  | 5 datasets (Full results in Table 21) | Best count | Var. number | Pred length |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| LLM-Time | 0.265 | 2.0e3 | - | LLM-Time | 1/5 | 1 to 767 | 16 to 128 |\n| UniTS | **0.030** | **6.8e−3** | - | UniTS | **4/5** | 1 to 767 | 16 to 128 |\n\n* For few-shot learning, the table below shows UniTS clearly outperforms the best performing baselines iTransformer on forecasting, classification, imputation, and anomaly detection tasks. (Full comparison with other methods are shown in Tables 3,4,5,23,24.)\n| Method/Best count | Forecast  (9 datasets) | Classification  (6 datasets) | Imputation  (6 datasets) | Anomaly detection  (5 datasets) |\n| :---- | :---- | :---- | :---- | :---- |\n| iTrans. | 0/9 | 1/6 | 0/6 | 0/5 |\n| UniTS | **9/9** | **5/6** | **6/6** | **4/5** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2bEOadmzpi", "reviewer": "Reviewer_YrSc", "summary": "This paper proposes UNITS, a multi-task time series model that handles multiple predictive and generative tasks within a single model. UNITS uses the mask modeling pre-training framework. To handle multiple downstream tasks, two new sets of tokens are concatenated with data tokens: 1)prompt tokens indicating the task corresponding to each dataset and are used for future prompt learning; 2)task tokens to generate generative/discriminative outputs. A modified transformer block is also proposed. Experiments of single-task and multi-task learning on forecasting, anomaly detection, imputation, and classification demonstrate the effectiveness of the proposed model.", "questions": "1. In Equation 5, what is the role of task tokens $z_m$? Compared with the left side, the right side encourages $H_{cls}(z_{pred}) = z_p$. How can it be discriminative enough for classification?\n\n2. How is $\\lambda_i$ in line 270 be determined?\n\n3. Please compare UNITS trained by multi-task learning with that trained by single-task learning to confirm the necessity of unified multi-task training.", "limitations": "No potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The issue studied in this work, i.e. using one shared model to model data from various domains and deal with multiple downstream tasks is of great importance.\n\n2. The experiments conducted in this paper are comprehensive, including single-task, multi-task and few-shot learning, etc.", "weaknesses": "1. The writing needs improvement. 1) Section 4.3 requires a more detailed description with formal equations. Actually, it is hard to get the role of three kinds of tokens in pre-training; 2) Experiment setting in Section 5 also requires re-organization to clarify what experiments are conducted and which datasets are used for which tasks. Moreover, several datasets are generated from the same dataset with different sliding windows and thus should be addressed in the paper.\n\n2. The modified model structure is orthogonal to the tokens and training methods proposed in the paper, and they are not well-coupled. For instance, training can still be conducted using the original Transformer.\n\n3. Compared with traditional single-task single-dataset supervised training models (Table 25), the proposed multi-datasets multi-task pre-training model (Table 2) does not show a better performance. As training a simple model like DLinear for each dataset and task is efficient, this calls into question whether joint training across multiple domains of data can provide real benefits.\n\n4. Some details of some experimental setups are not rigorous: for example in Table 4 (few-shot learning for imputation), datasets ECL, ETTh1 and Weather are generated from the same datasets used for pre-training, which the model can get access to, so these settings can not be viewed as few-shot learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer YrSc Part I (Part I of III)\n\nThank you for your valuable feedback\\! We appreciate your recognition of the importance of multi-task learning on time series and the comprehensive experimental results in our work. We have carefully addressed each of your concerns, clarified our descriptions of the methodology and analyses, and added experiments based on your suggestions.\n\nWe hope our response answers all your questions, and we kindly ask you to consider raising your score. If, after reviewing our responses, you still have concerns about our work, please let us know, and we will work on addressing any additional questions to achieve a better score in your review. Thank you again\\!\n\n---\n\n###  W1: Writing needs improvement.\n\nThank you for the suggestions. We have made the following improvements to make the paper easier to read. We are also committed to further refining the writing in the camera-ready version of the paper to make it easy to follow.\n\n###  W1.1: More detailed description with formal equations in Section 4.3.\n\nIn Section 4.3 and Equation 5, we have explained the roles of prompt and task tokens during pre-training. For each time-series sample, a handful of sample tokens get masked and replaced with GEN tokens. This masked tokenized representation of the input time-series sample is then concatenated with prompt tokens and CLS tokens and fed into the UniTS model. The UniTS model involves three task towers used during pre-training:\n\n- The GEN tower processes the prompt tokens, GEN tokens, and time-series sample tokens from the model output. The GEN tokens are used to predict the masked sequences.  \n- The CLS tower processes the prompt tokens, CLS tokens, GEN tokens, and time-series sample tokens from the model output, where the CLS token is returned from the CLS tower.  \n- Another GEN tower processes the GEN tokens and sequence tokens from the model output, along with the CLS token from the CLS tower output. The GEN tokens are again used to predict masked time-series samples.\n\n###  W1.2: Experiment setting in Section 5\\.\n\nDue to the limited space in the paper, we have included detailed experiment setups in Table 7, 8, 9, 10, 11 (page 19),  Section B (page 18\\) and Section D (page 22\\) of the appendix to explain the experimental settings and datasets used in experiments. Following your suggestions, we have further revised the implementation details in the revision to clarify the experiment settings.\n\n###  W1.3: Several datasets are generated from the same dataset with different sliding windows.\n\nFor subsets of a dataset such as ETTh1, we start by splitting the data into training and testing sets based on distinct time intervals of a long time series sequence. Within these training and testing intervals, we generate samples using various sliding windows, ensuring that there is no data leakage between the training and testing sets. We have included an explanation in the appendix detailing how various sub-datasets are generated using different sliding windows.\n\n\n###  W2: The modified model structure is orthogonal to the tokens and training methods. Training can be conducted with the original transformer.\n\nThank you for this question. Our response has two parts:\n\n**First**, the UniTS model architecture is strongly coupled with our multi-task co-training strategy in the following sense:\n\n- The unified tokenization strategy in the UniTS model integrates predictive and generative tasks, making it sufficiently flexible to support the co-training of various types of tasks in a single model without substantial task-specific model components.  \n- The Variable/Time MHSA separately manages features with varying numbers of variables and time slots, enabling the model to co-train with data from various domains.  \n- The Dynamic FFN adapts its weight shapes to capture shared temporal representations across different data types, whereas the original transformer cannot capture this with the point-wise FFN.  \n- Gate module modulates feature scales to reduce interference in the latent representation space caused by multi-domain and multi-task datasets, which is critical to multi-task learning.  Traditional transformers do not account for data and task interference.\n\nWe carried out an ablation study of the UniTS model architecture below to show the effectiveness of each of the above-mentioned components in the UniTS model (Full results in Tables 15, 16, 17, 18, 19, and 20 of the appendix).\n\n|  | Acc↑ (Classification) | MSE↓ (Forecasting) |\n| :---- | :---- | :---- |\n| UniTS | 81.6 | 0.439 |\n| UniTS w/o Time MHSA | 80.7  | 0.449 |\n| UniTS w/o Variable MHSA | 80.8 | 0.444 |\n| UniTS w/o Dynamic FFN  | 80.8 | 0.465 |\n| UniTS w/o Gate module  | 81.1  | 0.459 |\n\n**Second**, following your suggestion, we added new experiments comparing the original Transformer with UniTS. The unified tokenization and co-training strategy were applied to both models. Results shown in the table below suggest that simply using a transformer structure is insufficient for strong multi-task performance on time series datasets.\n\n|  | Acc↑ (Classification) | MSE↓ (Forecasting) | MAE↓ (Forecasting) |\n| :---- | :---- | :---- | :---- |\n| Transformer-network | 80.2% | 0.468 | 0.397 |\n| **UniTS-network** | **81.6%** | **0.439** | **0.381** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P0CGhhyThq", "reviewer": "Reviewer_EsEH", "summary": "The authors propose a unified model trained over multiple datasets to solve multiple tasks such as forecasting, imputation, anomaly detection, and classification. In the paper, the authors demonstrate their model abilities through extensive empirical results comparing with a large variety of baseline methods and suggesting new benchmark to compare multi-task results.", "questions": "- Eq. 2 is slightly confusing, $\\textbf{z}_c$ is defined as an input to the model but at the same time is used for the final class choice. Isn’t it should be the output of the model compared to each $\\textbf{z}_e(i)$?\n\n\n- In Eq. 4, $W_{\\text{Interp}}$ should be defined as $l_{\\text{out}} \\times l_{\\text{in}}$ instead of $l_{\\text{in}} \\times l_{\\text{out}}$, otherwise the DyLinear operations does not compile in terms of dimensions.\n\n- The implementation details of the baseline methods should be extended. Specifically, it is hard to understand what exactly was done to train the models for the new benchmark compared to UniTS-PM and UniTS-SUP. As it does not support unified training, in the appendix part, the authors mention: “For multi-task learning settings, models are configured with 3 blocks…” “The input and output heads of baseline methods are duplicated for each task to create data/task-specific heads tailored for each data source and task.” How is it different from the original set-up of those baseline methods? what do the authors mean by models configured with 3 blocks?\n- How do you perform zero-shot? Now that you do not have the corresponding prompt tokens, what do you feed to the network instead of those tokens?\n- The bold and underline in Table 2 for the ILI row seem to be incorrect as GPT4TS outperforms the given results.", "limitations": "See the Weaknesses and Questions sections.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper focuses on a foundational model for time series, which is unified for multiple datasets and tasks. The authors propose novel architecture and method to overcome the challenges that come with the problem.\n\n- Although it is not the main focus of the paper, the proposed method achieves state-of-the-art results for training per-task per-dataset via UniTS-ST.", "weaknesses": "- The details for comparing with previous baseline methods are missing, and thus it is hard to assess how well the model performs in the unified set-up in contrast to previous methods. (see questions)\n\n\n- Although most parts of the paper are clearly written and described, it is hard to follow the prompt learning part of the methods’ methodology. What exactly are the prompt tokens? The authors perform an extensive ablation study and include the number of prompts as part of it but do not describe or give an example of such prompts. (Is it just weights?)\n\n\n- The empirical evidence lacks comparison to similar methods that employ unified training or prompt-based approaches, as in [1,2,3]. Although the author’s unified multi-task model is indeed interesting, it still remains unclear whether the given approach is at least comparable to other unified forecasting models that use similar benchmarks such as unified training, few-shot, zero-shot, etc.\n\n\n[1]  TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting\n\n[2]  UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecasting\n\n[3]  Time-LLM: Time Series Forecasting by Reprogramming Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer EsEH Part I (Part I of II)\nThank you for your valuable feedback\\! We appreciate your recognition of our novel architecture and methods, as well as the state-of-the-art results achieved in our work. We have carefully addressed each of your questions, expanded on implementation details, and provided additional explanations following your suggestions. \n\nWe hope our response addresses your concerns and kindly ask you to consider raising your score. If you still have reservations after reviewing our responses, we would greatly appreciate further guidance on how we can improve our work to achieve a better score in your review. Thank you again\\!\n\n---\n\n###  W1 and Q3: Implementation details of the baseline methods are missing.\n\nWe have revised the implementation details of the baseline methods for better understanding, here are the explanations per your two concerns:\n\n* **Make baselines support unified training**: Unlike UniTS, which can handle diverse data and tasks within a single model, baseline methods cannot be directly used for unified training because: 1\\) To accommodate data with varying numbers of variables, baseline methods typically use a data-specific input head to project features from the variable count to a fixed number of embedding dimensions. 2\\) Similarly, to manage different tasks, such as classification with various classes and forecasting with different lengths, baseline methods employ task-specific output heads to transform the features into the appropriate task outputs.   \n  Since baseline methods are designed for single-task training, in their original setting, data/task-specific heads are used for each data and task. To make baseline methods support unified training, we add separate input heads to project data into a shared embedding space and separate output heads to convert the shared model output into task-specific outputs. However, using separate input and output heads makes it hard to generalize to new datasets and tasks.  \n* **Meaning of 3 blocks**: Existing networks and UniTS are built by stacking multiple basic blocks, e.g. transformer blocks and our designed block. “Configured with 3 blocks” means the network is stacked with 3 blocks. For fair comparisons, we stack “3 blocks” for all methods.\n\n###  W2: Explanation of prompt learning methodology. \n\nThe prompt tokens are learnable embedding weights with the shape $p \\times v \\times d$, where $p$ is the number of prompt tokens, $v$ is the number of variables in the current data sample, and $d$ is the number of embedding channels. Each dataset has its own unique group of prompt tokens. For each sample in the dataset, these prompt tokens are appended to the tokenized sample and sent to the network to provide context information about the current sample. We have revised the manuscript to clearly describe the prompt tokens.\n\n###  W3: Comparison with similar methods such as TEMPO, UniTime, and Time-LLM.\n\nAs we discussed in the related work of the manuscript (page 3), the notable works you mentioned that utilize unified training or prompt-based approaches rely on LLMs pre-trained with extensive textual training data. We have compared the forecasting performance of these LLM-based methods, such as TEMPO, UniTime, and Time-LLM, in the table below (full results in Table 29 of the manuscript, page 32). Remarkably, even without utilizing the additional textual training data employed by other methods, UniTS demonstrates superior performance across most datasets. Additionally, unlike these methods that focus solely on forecasting, UniTS supports both generative tasks (e.g., forecasting, imputation, anomaly detection) and predictive tasks (e.g., classification). We have updated the manuscript to include a discussion of these notable works.\n\n|  | UniTS | TEMPO | UniTime | TIME-LLM |\n| :---- | :---- | :---- | :---- | :---- |\n| Additional data necessary beyond the training set | **No** | Yes | Yes | Yes |\n| Multi-task support | **Yes** | No | No | No |\n| ETTm1 | 0.337/0.376 | 0.501/0.458 | 0.385/0.399 | 0.329/0.372 |\n| ETTm2 | 0.254/0.315 | 0.281/0.328 | 0.293/0.334 | 0.251/0.314 |\n| ETTh1 | 0.405/0.426 | 0.428/0.427 | 0.442/0.448 | 0.408/0.424 |\n| ETTh2 | 0.331/0.387 | 0.361/0.398 | 0.378/0.403 | 0.334/0.383 |\n| ECL | 0.156/0.253 | 0.216/0.308 | 0.216/0.305 | 0.159/0.253 |\n| Traffic | 0.409/0.278 | 0.503/0.358 | \\- | 0.388/0.264 |\n| Weather | 0.216/0.259 | 0.282/0.316 | 0.253/0.276 | 0.226/0.258 |\n\n###  Q1: Explanation of Eq. 2.\n\nIn Eq. 2, $z_c$ represents the input to the model, while in Eq. 3,  $z_c$ represents the output of the model. Since the feature shapes remain unchanged after being processed by each block in UniTS, we initially omitted the layer index for simplicity. For better clarity and understanding, we have now revised the manuscript to include the layer index.\n\n###   Q2: Revise Eq.4,\n\nThank you for the reminder. We have addressed this in the latest revision.\n\n###  Q4: How do you perform zero-shot?\n\nWe conducted two zero-shot experiments where the data or tasks were not encountered during training: 1\\) zero-shot multi-task learning on forecasting tasks, and 2\\) direct multi-step forecasting to varying time lengths.\n\n* For zero-shot forecasting on new datasets, we ask the model to make predictions on these new datasets. During pre-training, we use shared prompt tokens and the GEN token across all datasets for generative tasks. This allows the model to use the shared prompt and GEN tokens to perform predictions on new data.  \n* For direct multi-step forecasting to varying time lengths, we have the model forecast for time lengths not seen during training. Thanks to our unified task tokenization, we can easily predict new lengths by concatenating a new number of GEN tokens as the model input.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rz4lyveMqH", "reviewer": "Reviewer_bSMF", "summary": "This paper presents UNITS, a multi-task time series model that uses task tokenization to express predictive and generative tasks within a single model. UNITS can process heterogeneous time series with diverse variables and lengths without modifying the network architecture. Experiments show that UNITS demonstrates effective few-shot and prompt learning capabilities when evaluated on new data domains and tasks.", "questions": "1. Is the experiment setting in Section 5.2 consistent with that in Section 5.1? Does GPT4TS also use the same fully supervised multi-task training as UNITS?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper  is well-written and easy to follow.\n2. The paper introduces a unified multi-task time series model that handles a broad range of time series tasks.\n3. The proposed architecture achieve state-of-the-art performance on various time series tasks and can be adapted to new tasks via parameter-efficient prompt learning.", "weaknesses": "Although the idea of a multi-task model is somewhat novel, it seems that the UNITS on Single-Task Learning outperforms Multi-Task Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer bSMF Part I (Part I of II)\nThank you for your helpful feedback\\! We appreciate your acknowledgment of the quality and novelty of our method and the state-of-the-art performance achieved by UniTS. Below, we address each of your concerns, provide further details, and present new experiments. \n\nWe hope our responses have addressed your concerns and kindly request you to consider raising your score. If you still have questions after reviewing our responses, we would greatly appreciate any further guidance on how we can improve our work to meet your expectations. Thank you again\\!\n\n---\n\n### W1: It seems UniTS on Single-Task Learning outperforms Multi-Task Learning.\n\nThank you for pointing this out. We address your concerns in two parts.\n\n**First**, **we have added new experiments to show that multi-task learning of UniTS outperforms single-task learning under the same hyperparameters.** Following common practice \\[ICLR2024itrans, ICLR2023patch, ICLR2023times\\], results of single-task training are obtained by training the model on each dataset by selecting optimal hyperparameters for each dataset. However, the multi-task model can only use a single set of hyperparameters that are the same across all tasks/datasets. To ensure fair comparisons and to answer your question, we conducted a new experiment, where we trained the single-task models using the same hyperparameters as the multi-task co-training. Results show in the table below, multi-task learning achieves stronger performance on both forecasting and classification tasks. Interestingly, under the same hyperparameters, some classification models fail to converge in the single-task setting, whereas the multi-task model does not have this issue, demonstrating the robustness of multi-task training.\n| UniTS | Acc↑ (Classification) | MSE↓ (Forecasting) |\n| :---- | :---- | :---- |\n| **Multi-task** (same hyperparameters) | 81.6% | 0.439 |\n| **Single-task** (same hyperparameters) | 65.3% | 0.464 |\n\n**Second, we show the advantage of multi-task learning over single-task learning**. The advantages of multi-task learning in UniTS over single-task learning are: 1\\) enhanced performance without requiring task-specific hyperparameter tuning, and 2\\) improved generalization capabilities, including few-shot and zero-shot learning.\n\n- **Multi-task UniTS outperforms hyperparameters-tuned single-task DLinear:** We also compare the multi-task performance of UniTS with a popular single-task model DLinear, as summarized in the table below (full results in Tables 1 and 2, page 7 and 8). Despite DLinear being able to optimize and select dataset-specific hyperparameters for each dataset, UniTS, co-trained in a multi-task setting, consistently outperforms DLinear. Specifically, across 32 experiments (rows in the table below) across 16 datasets, we find that UniTS outperforms DLinear in 27 out of 32 experiments, showing that multi-task learning across data domains is an effective approach.\n| Best count on 16 datasets using MSE/MAE metrics | UniTS (Multi-task) | DLinear (Single-task) |\n| :---- | :---- | :---- |\n| One-model | **Yes** | No |\n| Best Count | **27/32** | 5/32 |\n\n - **Multi-task UniTS introduces new abilities beyond single-task models**. Using a single task model is insufficient to handle tasks that require strong generalization ability, such as few-shot learning and zero-shot learning.  \n    - For few-shot learning, the table below shows UniTS clearly outperforms the best performing single-task model iTransformer, on forecasting, classification, imputation, and anomaly detection tasks. (Full comparisons with other methods are shown in Tables 3,4,5,23,24.)\n| Method/Best count | Forecasting  (9 datasets) | Classification  (6 datasets) | Imputation  (6 datasets) | Anomaly detection  (5 datasets) |\n| :---- | :---- | :---- | :---- | :---- |\n| iTransformer | 0/9 | 1/6 | 0/6 | 0/5 |\n| UniTS | **9/9** | **5/6** | **6/6** | **4/5** |\n    - For zero-shot learning, the table below shows UniTS considerably surpasses LLMTime, a model designed for zero-shot forecasting using LLM, across most of the tested datasets, demonstrating superior performance in handling different forecasting lengths and variable numbers.  For example, UniTS achieves a considerable improvement in MSE over LLMTime (0.030 vs.0.265) on Solar. Remarkably, UniTS exhibits an inference speed approximately 10e6 times faster than LLMTime. We also show in Figure 3 of Figure PDF (Figure 3 of the manuscript, page 8\\) that UniTS can generalize to new forecasting lengths not seen during training.\n| Solar dataset | MSE | Infer. Time (seconds) |  | 5 datasets (Full results in Table 21) | Best count | Var. number | Pred length |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| LLM-Time | 0.265 | 2.0e3 | - | LLM-Time | 1/5 | 1 to 767 | 16 to 128 |\n| UniTS | **0.030** | **6.8e−3** | - | UniTS | **4/5** | 1 to 767 | 16 to 128 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you to all the reviewers for thoughtful and insightful feedback\! We appreciate Reviewers acknowledging our contributions. Reviewers emphasized the importance of the issue studied, noting that **“using one shared model to model data from various domains and deal with multiple downstream tasks is of great importance”** \[YrSc\]. Reviewers recognize the novelty of our approach, stating that the **“authors propose novel architecture and methods to overcome the challenges that come with the problem”** \[EsEH\]. Reviewers highlight that our work **“has strong experimental results”** \[h6vS\] and appreciate the comprehensive nature of our experiments, which include **“single-task, multi-task, and few-shot learning”** \[YrSc\]. Furthermore, reviewers commend our **“great engineering efforts to perform cross-dataset training on multiple tasks”** \[h6vS\]. Additionally, reviewers acknowledge the achievement of **“state-of-the-art performance on various time series tasks”** while being adaptable to new tasks through **“parameter-efficient prompt learning”** \[bSMF\]. Even though it is not the primary focus, our method achieves **“state-of-the-art results for training per-task per-dataset via UniTS-ST”** \[EsEH\]. They also praise the clarity and readability of the paper, noting that it is **“well-written and easy to follow”** \[bSMF\].


Reviewers also raised several key points, which we address in our point-by-point responses to each reviewer. Here, in the general response, we briefly highlight new experiments added in this rebuttal and address a few important points raised by reviewers.

In response to reviewers’ comments, we conducted seven new groups of experiments to achieve the following:

* Study the pre-training strategy of UniTS with varying amounts of data and training steps.  
* Systematically evaluate the interrelationship between pre-training tasks, including cross-task and cross-domain pre-training.  
* Analyze the learned tokens by generating new UMAP plots of prompt tokens before and after training.  
* Compare prompt tuning parameters of UniTS against strong baseline methods.  
* Compare UniTS architecture to original Transformer architecture.  
* Compare single-task models with multi-task models using the same hyperparameters.  
* Compare UniTS with similar methods, such as UniTime.

These new experiments and analyses, together with initial results across 38 datasets (spanning human activity sensors, healthcare, engineering, and finance domains), 12 forecasting models, 20 classification models, 18 anomaly detection models, and 16 imputation models, have strengthened our confidence in the effectiveness of UniTS. We address key points raised by the reviewers:

* We demonstrate the advantages of multi-task learning over single-task learning, particularly in terms of generalizability to new tasks through zero-shot and few-shot learning \[h6vS, YrSc, bSMF\].  
* We provide detailed explanations and show the effectiveness of the proposed unified tokenization and UniTS network structure \[h6vS, YrSc, EsEH\].  
* We clarify the implementation details and experimental settings of our method and baselines, and we add more experiments to offer a thorough analysis of UniTS \[YrSc, EsEH, YrSc, bSMF\].

### General Notes to Reviewers:  
We abbreviate weaknesses by “W\#”  and questions by “Q\#.” We are not allowed to update the revised manuscript during the rebuttal phase, but we assure the reviewers that comments have been incorporated into our revisions. Figure PDF refers to the one-page PDF we upload to the open review system.

---

*If you feel we have not sufficiently addressed your concerns to motivate increasing your score, we would love to hear from you further on what points of concern remain and how we can improve the work in your eyes. Thank you again\!*

Meta-review: This paper proposes a unified pre-trained model for time series data called UNITS, designed to handle multiple predictive and generative tasks across different datasets. The model employs a transformer architecture enhanced with prompt and task tokens, enabling it to perform prompt tuning and few-shot learning. UNITS is pre-trained on multiple datasets from various domains and is evaluated on its transferability across different downstream tasks, demonstrating comparable or superior performance to state-of-the-art models.

**Strengths:**
- Strong experimental results, showing effective cross-dataset training on multiple tasks.
- The issue of handling diverse domains and multiple tasks within a single model is of great importance.
- UNITS achieves state-of-the-art results in several tasks and can adapt to new tasks via prompt learning.
- The paper is generally well-written and easy to follow.

**Weaknesses:**
- Overclaimed contributions, particularly regarding the generalizability to new tasks that were seen during pre-training.
- Lack of theoretical understanding and ablation studies on the impact of multi-dataset training.
- The model’s novelty is questioned due to insufficient methodological explanations and the orthogonality of the model's structure to the proposed tokens and training methods.
- Issues with writing clarity, especially in explaining the roles of the tokens and experimental setups.
- Missing comparisons with similar models using unified training or prompt-based approaches.

Some concerns have been addressed by the authors during the rebuttal period. 

Reviewers generally lean positive for this paper. One of the reviewers had concerns about some of the overclaim, downgrading the rating from 6 to 5, but the concern seemed later addressed by the authors, and the reviewer adjusted the rating back to 6. Overall, I agree with the reviewers that this is a good contribution to the community and therefore recommend acceptance.
Decision: Accept (poster)