UniTS: A Unified Multi-Task Time Series Model

Authors: shanghua  gao
teddy  koker
owen  queen
thomas  hartvigsen
theodoros  tsiligkaridis
marinka  zitnik
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-30 00:25:11

Abstract: Although pre-trained transformers and reprogrammed text-based LLMs have shown strong performance on time series tasks, the best-performing architectures vary widely across tasks, with most models narrowly focused on specific areas, such as time series forecasting. Unifying predictive and generative time series tasks within a single model remains challenging. We introduce UniTS, a unified multi-task time series model that utilizes task tokenization to integrate predictive and generative tasks into a single framework. UniTS employs a modified transformer block to capture universal time series representations, enabling transferability from a heterogeneous, multi-domain pre-training dataset—characterized by diverse dynamic patterns, sampling rates, and temporal scales—to a wide range of downstream datasets with varied task specifications and data domains. Tested on 38 datasets across human activity sensors, healthcare, engineering, and finance, UniTS achieves superior performance compared to 12 forecasting models, 20 classification models, 18 anomaly detection models, and 16 imputation models, including adapted text-based LLMs. UniTS also demonstrates strong few-shot and prompt capabilities when applied to new domains and tasks. In single-task settings, UniTS outperforms competitive task-specialized time series models. Code and datasets are available at https://github.com/mims-harvard/UniTS.

Main paper: 
 introduction: 
 Foundation models, particularly large language models (LLMs), have transformed deep learning by enabling a single pre-trained model to support multiple tasks, eliminating the need for taskspecific models. Language and vision models [9, 101, 92, 50, 32] can be adapted to new tasks with minimal additional training through approaches such as multi-task learning [125] , few-shot learning [108, 86] , and prompting [66] . Beyond language and vision, there is a growing need for similarly versatile models in time series that can accommodate data from diverse domains-including medicine [34] , engineering [102] , and science [48] -and support a wide range of tasks, such as forecasting, classification, imputation, and anomaly detection.Developing multi-task time series models that unify predictive and generative tasks under a single framework remains an open challenge. Time series datasets span multiple domains and exhibit varied temporal scales, sampling rates, and dynamic patterns, making them complex to manage [124, 78] .Existing models often fall short in adaptability, as they either struggle to handle samples with varying numbers of variables [112, 67, 14] or treat each variable as independent, overlooking important interdependencies [82] . Time series tasks are also highly diverse, encompassing distinct objectives and specifications across generative and predictive tasks. For example, generative forecasting tasks aim to produce future values within a time series, while predictive tasks may involve making discrete predictions for entire samples. Additionally, task requirements can vary significantly even within the same task type; for instance, generative tasks may involve different forecast lengths, and predictive tasks may feature multiple classification categories. As a result, time series models have mainly remained task-specific, with unique architectures typically designed and trained from scratch for forecasting [67, 82, 119] , classification [30, 113] , or other specialized tasks [116, 112] . Recent efforts to pre-train unified models [36, 22] or adapt LLMs for time series [118, 12, 129, 47, 97, 100] still heavily depend on extensive fine-tuning or the addition of task-and dataset-specific modules. Some models have explored generative pre-training transformers specifically for time series forecasting [10, 118, 47, 28] , reporting strong results but focusing exclusively on forecasting without addressing other types of time series tasks. Consequently, these approaches require users to design and train new modules for each task or limit their application to a single type of tasks. To achieve a versatile, unified time series model-akin to foundational models in vision and language that operate across unified task spaces-a model must accommodate both generative and predictive tasks. Such a unified model would leverage a single set of weights for multiple tasks, removing the need to develop task-specific models from scratch. This approach would support a broad range of tasks and facilitate rapid adaptation to new datasets. Present work. To address these challenges, we introduce UNITS, a unified multi-task time series model capable of handling a broad spectrum of time series tasks. We rigorously compare UNITS against 12 forecasting methods, 20 classification methods, 18 anomaly detection methods, and 16 imputation methods, including transformer-based, LLM-based, RNNbased, and traditional approaches, to highlight UNITS's generalizability to new tasks. This capability is achieved through the following model design: 1) Task tokenization: UNITS encodes task specifications into a unified token representation, enabling universal task specification without post-hoc architectural modifications. 2) Unified time series architecture: UNITS processes heterogeneous time series data with varying numbers of variables and sequence lengths without altering its network structure. To accomplish this, UNITS employs self-attention across time and variable dimensions to adapt to diverse temporal dynamics. We introduce a dynamic linear operator to model complex relationships between data points along the time dimension and a module to reduce interference in the feature space of heterogeneous data. 3) Support for generative and predictive tasks: The combination of universal task specification and a unified time series architecture allows UNITS to share weights across tasks by co-training on multiple datasets. We use a masked reconstruction pre-training approach, enabling UNITS to be jointly optimized for generative and predictive tasks.In the single-task setting, where models are trained individually for each dataset, UNITS outperforms task-specialized time series models and repurposed LLMs across forecasting, classification, anomaly detection, and imputation. In a challenging multi-domain, multi-task setting, we find that a single shared-weight UNITS model successfully handles 38 tasks, demonstrating its versatility as a multi-task time series model. UNITS surpasses top baselines that rely on data-and task-specific modules, achieving the highest average performance across tasks and excelling in 27 out of 38 tasks. Additionally, UNITS supports prompt-based learning and direct multi-step forecasting with flexible sequence lengths, capabilities not offered by models using task-and data-specific heads. In direct multi-step forecasting, UNITS outperforms the strongest baseline (which uses a sliding-window approach) by 10.5%. UNITS can also adapt to new tasks through parameter-efficient prompting, achieving results comparable to its fully fine-tuned counterpart. For example, across 20 forecasting datasets, prompted UNITS slightly outperforms the fully fine-tuned model, reducing MAE from 0.381 to 0.376. Furthermore, UNITS demonstrates effective few-shot transfer, successfully addressing tasks like imputation, anomaly detection, and out-of-domain forecasting and classification without requiring specialized modules. For instance, UNITS improves on the strongest baseline by 12 .4% in MSE on imputation and 2.3% in F1-score on anomaly detection. UNITS paves the way toward unified time series models, offering strong performance and adaptability across tasks and domains. 
related work: 
 Traditional time series modeling. Time series analysis has been extensively explored in both the statistics and machine learning communities for many years [45, 103, 123, 18, 80] . Numerous neural architectures have been developed for specific time series tasks such as forecasting [114, 65, 68, 67, 107] , classification [115, 71, 70] , anomaly detection [25, 56, 16] , and imputation [17, 49, 3] . Task-specific models are typically trained via supervised learning on individual datasets, necessitating specialized modules. For example, a classification model requires a classification head with a specific number of classes, while data processing modules must handle a predetermined number of variables.In contrast, UNITS aims to unify various tasks into a universal task specification, enabling the handling of diverse data with a single, unified network architecture. This approach facilitates training a multi-task model capable of addressing multiple time series tasks.General time series modeling. Foundation models, including language models [9, 101] and vision models [62, 50] , are trained on broad data at scale to address diverse tasks with no or minimal additional training [8] . Recent studies in time series analysis have sought to develop models with similar capabilities. This includes developing novel architectures to capture diverse time series signals. For instance, TimesNet [112] uses multiple frequency-based features obtained through Fourier transform to capture complex time series signals. There have been several efforts to reprogram LLMs for time series tasks [81, 12, 129, 47, 10] . Models such as GPT4TS [129] and Time-LLM [47] adapt LLMs by fine-tuning their embedding layers or aligning time series samples with LLM-based text prototypes (e.g., GPT-2 [89] ). Unlike these models, UNITS is trained exclusively on time series data rather than relying on LLM architectures. Another approach, Lag-Llama [90] , pre-trains a model on time series data from multiple domains specifically for forecasting tasks. Similarly, the Moment model [36] is pre-trained on a diverse range of time series data. However, these approaches still require task-specific modules and tuning for each task. In contrast, our UNITS model supports generative and predictive tasks without requiring extensive task-specific model adjustments.Prompt learning. Prompt learning has emerged as an efficient method for task adaptation in large models [55, 88, 121, 13, 42] . Some approaches construct prompts directly in the model's input domain, such as text prompts for LLMs [2] . Other methods involve tuning soft token inputs to frozen language models [58] . In time series, PromptCast [118] and LLMTime [81] convert time series data into prompts for LLMs to facilitate forecasting. TEMPO [10] is another prompt-based approach that uses a learned set of prompts for LLM-based forecasting applications, while GPT4MTS [46] integrates both textual and numerical data to fine-tune LLMs for forecasting. In contrast, UNITS is trained exclusively on time series data, eliminating the need for computationally expensive pre-trained LLMs. Moreover, the universal task tokenization enables a frozen UNITS to adapt to new tasks beyond forecasting, such as classification and imputation. Further discussion of related work can be found in Appendix A.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: In Section M, we discuss the limitations and future work regarding the size of dataset collections and the exploration of more advanced network architectures.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem formulation: 
 Notation. We are given a set of multi-domain datasets D = {D i |i = 1, . . . , n}, where each dataset D i can have a varying number of time series samples; samples can be of varying time lengths and have varying numbers of sensors/variables. Each dataset is described asD i = (X i , Y i ),where X i denotes time series samples and Y i specifies a task defined on X i . Let X and Y be collections, defined as X = {X i |i = 1, . . . , n} and Y = {Y i |i = 1, . . . , n}, respectively. A time series sample in datasets is denoted as x ∈ R t×v , where t and v are the length of the time series sample and the number of variables, respectively. We use time dimension and variable dimension to indicate the row and column dimensions in x. Y i contains four common time series tasks: forecasting, classification, anomaly detection, and imputation. Further, each task type can be instantiated in numerous ways, e.g., forecasting over different time lengths and classification with varying numbers of classes. We use F (X , θ) to denote a multi-task model trained on X . See Table 12 for notation details. Desiderata for a unified multi-task time series model. Unlike specialized time series models designed and separately trained for each specific dataset D i , a unified time series model F (X , θ) is a single model with weights θ that are shared across all types of tasks and satisfies the following three desiderata: 1) Heterogeneous time series: To process time series from all sources, the model F must be agnostic with any input samples in X , given the heterogeneity in time series lengths t and variable counts v in time series samples x from various sources. 2) Universal task specification: For easy multi-task support and swift adaption to new tasks, the model F should adopt a universal task specification F (X , θ) → Y applicable across all type of tasks Y. 3) One shared model: Sharing weights θ across tasks enables the unified model F to handle multiple tasks simultaneously. It contrasts with existing methods that typically train separate models on task-specific datasets, often involving elaborately tuned training parameters.To realize the above desiderata, UNITS supports multi-task, prompt-based, and few-shot learning.Multi-task learning: UNITS specifies a single model F (X , θ) → Y for tasks Y defined on datasets X . Multi-task learning showcases the flexibility of the model to learn across time series domains and tasks. Prompt learning: By leveraging prompt tokens, UNITS supports prompt learning, Prompting{F (X , θ), token} → Y, across tasks while keeping the model frozen. Additionally, UNITS can be trained in a single-task manner, following the same setup as used by many existing models. Other settings are described in Appendix C.1.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the broader impacts in Section N. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
units model: 
 UNITS is a multi-task model with a unified network architecture. It uses a token-based format to describe tasks and time series from different domains. We introduce a novel approach with three distinct token types: sample, prompt, and task tokens, each serving a unique purpose in time series analysis. The input time series sample is tokenized into sample tokens. Prompt tokens provide essential context for the task, guiding the model to accomplish the user-specified task. Task tokens (GEN and CLS) are combined with other tokens and used for generative and predictive tasks. UNITS then converts task tokens into task predictions to produce the final model output. Unlike transformers such as PatchTST [82] , UNITS introduces new token types: sample tokens allow for modeling of multivariate time series, prompt tokens enable efficient multi-task and prompt learning [101] , and task tokens unify predictive and generative tasks into one format.We introduce how to use unified tokens to unify different task types and data for inference. Tokens on different network layers have the same shape, so we omit the layer index for simplicity. Sample tokens. We divide time series input sample x ∈ R t×v into patches along the time dimension using a non-overlapping patch size of k. A linear layer projects each patch into an embedding vector of length d, obtaining sample tokens z x ∈ R s×v×d , where s = t/k. Since v and s vary across time series data domains, we keep the variable and time dimension in tokens. z x are then added with learnable positional embeddings. Prompt tokens. Prompt tokens z p ∈ R p×v×d are defined as learnable embeddings, where p is the number of tokens. In a multi-task setting, each dataset has its own set of prompt tokens. These tokens incorporate the specific context related to the data and the task the model needs to complete. For each sample in the dataset, these prompt tokens are appended to the sample tokens and sent to the network to provide context information about the current sample. For prompt learning, with the pre-trained model weights being frozen, UNITS adapts to new tasks by utilizing prompt tokens learned with the prompt tuning. Prompt learning is more efficient than tuning new data/task-specific heads and achieves comparable performance to full model fine-tuning, as shown by few-shot learning experiments on new tasks (Tables 4 and 5 ) and new datasets (Table 3 ). Task tokens. In Figure 2ab , we categorize task tokens into two types: 1) GEN (Generation) tokens used in forecasting, imputation, and anomaly detection, and 2) CLS (Classification) tokens, which are used for classification tasks (in a given task, the number of CLS tokens corresponds to the number of classes in the task). Task tokens define a general format for representing tasks and support flexible adaptation to new tasks. For tasks involving forecasting, in Figure 2a , the GEN token z m ∈ R 1×v×d , is replicated f -times based on desired forecasting length to get ẑm ∈ R f ×v×d . These tokens ẑm are then concatenated with the sample and prompt tokens and fed into the UNITS network:EQUATION)where CA is the concatenation operation along the time dimension. At the output of the model, embedding vectors with length d in ẑm are unpatchified to patches with size e to obtain the forecasting sample x, i.e. x = Proj(ẑ m ) ∈ R (f ×e)×v . This approach allows the UNITS model to perform direct multi-step forecasting [99, 76, 119] over arbitrary time lengths, as illustrated in Figure 3 . For classification, in Figure 2b , CLS token z c ∈ R 1×v×d is concatenated along the time dimension with the prompt and sample tokens, resulting in:EQUATIONwhich is then fed into the model. We define class embeddings z e ∈ R e×v×d for each of e classes in the task. These class embeddings are either trained or generated by averaging CLS tokens of training samples in each class. Finally, the class for sample x is predicted by finding the class embedding vector in z e that is the closest to the CLS token z c from the model output:EQUATIONFor imputation, missing values are imputed using the GEN tokens. For anomaly detection, the model takes a time series sample containing any number of potentially anomalous values, generates the output sample by reading out the sample tokens, and then determines anomalous values based on the reconstruction error between the input sample and the generated sample. Details on using tokens for imputation and anomaly detection are in Appendix C.2. All tokens and embeddings are trained to achieve their functions.Time series samples can have varying numbers of variables, temporal dynamics, and time lengths across different domains and types of tasks. UNITS uses a modified transformer architecture [104] to handle heterogeneous multi-domain data with varying dynamics and the number of variables (Figure 2c ). In the following, we describe key modules of UNITS architecture. Note that UNITS can also be used with other backbones, such as Mamba [38] . Time and variable self-attention. We use a two-way self-attention to both variable and time dimensions. This approach contrasts with previous methods that apply self-attention to either time [82] or variable dimension [67] , but not to both dimensions. Time and variable self-attention effectively handle time series samples with various numbers of variables v and different time lengths t. DyLinear. We modify the transformer block by adding a dynamic operator (DyLinear) into the feed-forward network layer (FFN). This modification enables the FFN to capture dependencies between tokens. In contrast to the standard FFN, which processes embedding vectors on a point-wise basis, DyLinear uses weight interpolation to accommodate varying time lengths. Given a sequence of sample tokens z t ∈ R lin×d , DyLinear interpolates weights w ∈ R wout×win to accommodate varying time lengths as follows:EQUATIONwhere Interp is a bi-linear interpolation to resize w from shape w out × w in to l out × l in to match the input and output length. DyLinear captures dependency patterns across time series samples, which leads to improved performance on generative tasks (Table 23 ).Gating module. We add a gating module after each layer to mitigate interference in the latent representation space caused by multi-domain and multi-task datasets (Figure 2 ). This module dynamically re-scales features in layer-wise latent spaces and promotes the stability of latent representations.Generative and predictive towers. We design a shared GEN tower (H GEN ) and CLS tower (H CLS ) for transferring GEN/CLS tokens to generate time series samples and classification classes, as introduced in Section 4.1. Unlike existing works that use standalone, task-specific heads for individual datasets, our approach leverages GEN tower and CLS tower for all generative and predictive tasks, respectively, ensuring a more unified and efficient model architecture.The UNITS architecture includes the backbone network composed of N modified transformer blocks described above, a CLS tower, and a GEN tower. Implementation details are in Appendix C.3. Ablations in Appendix F verify the effectiveness of this architecture.Unified masked reconstruction pre-training. To enhance UNITS's abilities to 1) learn general features applicable to both generative and predictive tasks and 2) efficiently adapt to downstream tasks via prompt learning, we introduce a unified mask reconstruction pre-training scheme. It leverages the semantics of both prompt and CLS tokens (Section 4.1) for masked reconstruction pre-training, therefore learning representations for both generative and predictive capabilities. This is distinct from pre-training strategies that use either generative [82, 120, 26, 54] or predictive [72, 109, 117, 29, 124, 87] approach. Unlike these approaches that pre-train only the model backbone, our strategy pre-trains all components of UNITS, including the backbone and GEN/CLS towers (Section 4.2), enabling prompt and zero-shot learning over a frozen pre-trained model. For each time-series sample x, a handful of sample tokens get masked and replaced with GEN tokens. These masked sample tokens is then concatenated with prompt tokens and CLS tokens, sent to the UNITS backbone network. In the unified pre-training loss, tokens from the backbone network output are sent to the CLS/GEN towers to reconstruct the input sample x, formulating as follows:EQUATION)L MSE is the MSE loss to predict the full sample x. For the left side of the loss, prompt token z p is sent along with sample token z x to GEN tower H GEN to help with the reconstruction. For the right side of the loss, to leverage the semantics of the CLS token and train the CLS tower H CLS for predictive tasks, z Pred (Eq. 2) from the model output is processed by the CLS tower H CLS to get classification-related embedding vectors ẑPred = H CLS (z Pred ), and another GEN tower ĤGEN takes in ẑPred and z x to predict the full sample. ĤGEN is only used for pre-training and will be removed for downstream tasks. This unified pre-training strategy involves pre-training both tokens, the backbone network, and the GEN/CLS towers for both generative and predictive abilities.Training UNITS models. We implement and evaluate two UNITS models, each trained in a different regime. We start with a pre-trained UNITS that is optimized using self-supervised L pretrain in Eq. 5 and trained across a collection of multi-domain datasets. Given a self-supervised pre-trained UNITS whose weights are frozen, we consider a fine-tuned model where only tokens for predictive or generative tasks are fine-tuned (denoted as UNITS-PMT in Experiments). We also consider a standard multi-task supervised learning regime, where a single UNITS model is trained from scratch to simultaneously perform many tasks (denoted as UNITS-SUP in Experiments). These two regimes use a multi-task setup, where a single model is trained and tested on multiple tasks and datasets.During multi-task training, we sample batches of time series samples and aggregate dataset-centric loss values:L total = I i=1 λ i L i (D i ), where L i is the loss of batch i, λ i is the weight for each loss, and I denotes the number of batches. We follow [112] and use the MSE loss for forecasting and cross-entropy loss for classification. For fair comparison with models trained in a single-task manner, we follow the experimental setup of [112, 67] and benchmark UNITS in a single-task setting (denoted as UNITS-ST in Experiments), where the model is trained separately on each dataset/task. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0Table 1 : Single-task comparison with existing methods on forecasting, classification, anomaly detection, and imputation tasks where each model is separately trained on each dataset. Full results are shown in Table 30 , Table 31 , Table 32 , and Table 33 . 
experiments: 
 Datasets. For multi-task learning on forecasting and classification, we compiled 38 datasets from several sources [79, 33, 82] . These datasets span domains including human activity, healthcare, mechanical sensors, and finance domains and include 20 forecasting tasks of varying forecast lengths ranging from 60 to 720, as well as 18 classification tasks featuring from 2 to 52 categories. Time series samples have varying numbers of readouts (from 24 to 1,152) and sensors (from 1 to 963). Details are in Table 7 . When evaluating multi-task few-shot learning on new datasets, a novel dataset collection comprising 6 classification tasks and 9 forecasting tasks (Table 8 ) is utilized. For multi-task few-shot learning on new tasks, we use the 6 datasets (Table 10 ) for imputation tasks and 5 datasets (Table 11 ) for anomaly detection tasks. On the single-task setting, we following existing works [112, 67] to use 36 datasets for forecasting (Table 30 ), 10 datasets for classification (Table 31 ), 4 datasets for imputation (Table 10 ), and 5 datasets for anomaly detection (Table 11 ).Baselines. We conduct an extensive comparison between UNITS and 12 time series forecasting methods, 20 classification methods, 18 anomaly detection methods, and 16 imputation methods, as listed in Table 13 . For comparison on the challenging multi-task setting, we excluded methods that overly rely on task-specific modules and lack a shared backbone, and we select 6 strong time series methods: iTransformer [67] , TimesNet [82] , PatchTST [82] , Pyraformer [65] , Autoformer [114] , and the LLM-reprogrammed method GPT4TS [129] . Many of these methods are designed and evaluated only for one type of tasks, e.g., GPT4TS and iTransformer are forecasting models. To include these methods in our benchmarking, when necessary, we add task-specific input/output modules to support multiple tasks. Training and evaluation details are shown in Appendix D.2.Setup. For fair comparisons with baseline methods, we benchmark single-task UNITS-ST on forecasting, classification, anomaly detection, and imputation. Models are separately trained from scratch with configuration tailored to datasets. Details are in Appendix K.ITRANS. SHARED ✓ ✓ × × × × × ×Table 2 : Multi-task benchmarking across 20 forecasting tasks and 18 classification tasks. Both UNITS-SUP and UNITS-PMT process all 38 tasks using a single model. GPT4TS reprograms a pre-trained LLM (GPT-2) to time series and has dataset/task-specific modules, thus, it is excluded from best count evaluations to ensure fair comparisons." P " is forecasting length. "Class./Num." denotes the "number of classes in each task"/"number of datasets".Results. Table 1 shows the single-task performance for four types of tasks. On forecasting tasks with forecasting lengths of 92, 196, 336 , and 720, compared with 11 forecasting methods, UNITS-ST achieves the best results on 32 out of 36 datasets for MSE and 27 out of 36 for MAE, surpassing the previous best method, iTransformer, by a clear margin. In Table 34 , we demonstrate that UNITS-ST outperforms the concurrent MOMENT [36] model, which was trained on a large and diverse collection of time series data. Additionally, UNITS-ST achieves stronger performance than LLMreprogrammed methods that are pre-trained with extensive natural language data, e.g. GPT4TS [129] , TEST [97] , LLM4TS [12] , and TEMPO [10] . On 10 classification datasets, UNITS-ST outperforms 19 classification methods on the average accuracy, such as the transformer/MLP/frequency-based methods. It has a gain of 1.4% compared to the previous best TimesNet model. On 5 anomaly detection datasets, UNITS-ST has a clear gain of 3.95% in F1 score compared to the TimesNet and also beat other 15 anomaly detection methods, such as Anomaly Transformer [116] . On 16 imputation datasets with a mask ratio of 12.5%, 25%, 37.5%, UNITS-ST has the best results on all datasets in terms of MSE and MAE, outperforming 14 baseline methods. UNITS-ST has the SoTA performance on these single-task benchmarks, showing its effectiveness.Setup. In a multi-task setting, we benchmark a single UNITS model co-trained and evaluated on 38 datasets, comprising 20 forecasting tasks and 18 classification tasks, with variations in the number of variables/sensors, classification classes, and forecasting lengths. We consider two variants of UNITS; the fully supervised UNITS-SUP and the more challenging UNITS-PMT with prompting, as introduced in Section 4.3. Baselines use the same fully supervised multi-task training as our approach but cannot handle differences across data types and task specifications with a single model. To benchmark them, a shared backbone is used for all tasks, augmented by data-specific input modules and task-specific output modules.Results: Model benchmarking. Table 2 shows multi-task learning performance. UNITS consistently outperforms baseline methods, achieving the best results in 17 out of 20 forecasting tasks (MSE) and 10 out of 18 classification tasks (accuracy). Performance gains are especially remarkable because UNITS has one fully shared model, whereas all existing methods require task or dataset-specific modules. We find that baseline methods encounter difficulties performing well across different types of tasks. For example, TimesNet, which excels in classification tasks, underperforms in forecasting tasks. Conversely, iTransformer, the top-performing forecaster, struggles with classification tasks. In contrast, the UNITS model exhibits robust performance across classification and forecasting. On forecasting, UNITS-SUP surpasses the leading baseline, iTransformer, by 5.8% (0.439 vs. 0.466) in MSE and 3.3% (0.381 vs. 0.394) in MAE. On classification, UNITS-SUP has an average gain of 0.7% accuracy (81.6% vs. 80.9%) over the strongest baseline (TimesNet). UNITS shows promising potential to unify data and task diversity across time series domains.Recent research has adapted pre-trained LLMs to time series [47, 12, 129, 37] . Most approaches [47, 12, 129] , such as GPT4TS, incorporate additional task-specific modules to align the modalities of time series and natural language. We compare UNITS with GPT4TS that reprograms pre-trained GPT-2 model [89] . Despite the substantial data amount and model scale gap, e.g., GPT4TS is 48× larger than UNITS-SUP (164.5M vs. 3.4M), UNITS-SUP still compares favorably to GPT4TS. On forecasting tasks, UNITS-SUP even outperforms GPT4TS by 2.2% (0.439 vs. 0.449; MSE). Results: Prompting is competitive with supervised training. Using tokens to prompt a frozen UNITS, the SSL-pre-trained UNITS achieves performance comparable to its fully supervised counterpart (Table 2 ). UNITS-PMT even outperforms the supervised model in forecasting, with a lower MAE score (0.379 vs. 0.381), highlighting the effectiveness of prompt learning in UNITS. Furthermore, prompt learning with UNITS surpasses the performance of supervised baseline methods with separate modules. This indicates that the SSL-pre-trained model captures valuable time series representations and that prompt learning allows the model to efficiently adapt to target tasks.Setup. Direct multi-step forecasting predicts across varying time horizons by adjusting from the original trained length, with offsets ranging from 0 to 384. We use 14 out of 20 forecasting datasets with varying lengths. UNITS achieves this flexibility by repeating the GEN token, as described in Section 4.1, a capability not supported by existing methods. For comparison with baseline models, we implement a sliding-window approach for forecasting. In this method, predictions are made over a fixed window size, which then shifts forward incrementally to cover progressively extended time horizons. This sliding mechanism allows us to adapt the model to forecast over new, unseen time periods while maintaining consistency with the evaluation setup used by baseline methods.Results: Direct multi-step inference outperforms sliding window approach. In Figure 3 , UNITS demonstrates improved performance over baseline models across various forecasting lengths when using the sliding-window approach. For example, in the longest forecasting extension of +384, UNITS outperforms the iTransformer by 8.7% in MSE, achieving a score of 0.451 compared to 0.494. When using direct multi-step inference, UNITS gains an even larger advantage over the iTransformer, reducing MSE by 10.5% (0.442 vs. 0.494). This approach also reduces the average number of inference steps from 3.66 to 1, resulting in a 3× speedup.For transfer learning on new tasks and datasets, we load the model weights pre-trained on 38 datasets and apply them in a multi-task setting. We evaluate two approaches: the fully fine-tuned UNITS-FT model and the prompted UNITS-PMT model, in which task-specific tokens are trained. Setup: Few-shot classification and forecasting. Pre-trained models, undergo fine-tuning using 5%, 15%, and 20% of the 11 training set shown in Table 8 . Average performance is reported.Results. UNITS achieves superior performance compared to iTransformer across all training data ratios (Table 3 ). At the 20% data ratio, UNITS-FT achieves a gain of 8.8% in classification accuracy and a reduction of 5.7% in forecasting MSE. UNITS-PMT surpasses the fully supervised iTrans- 0 
conclusion: 
 We have developed UNITS, a unified model for time series that uses a universal specification of time series tasks. UNITS handles multi-domain time series data with heterogeneous representations, outperforming task-specific models and reprogrammed LLMs on 38 multi-domain and multi-task datasets. UNITS also shows strong few-shot and prompt-based performance and can generalize to new domains and tasks. The unified token scheme in UNITS allows it to represent data and tasks in a general manner. UNITS uses a transformer architecture, and we plan to explore other types of backbones, such MLP-based blocks [107, 14] and Mamba [38]  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We use open source datasets and codes based on their licenses. Most baselines are released under mit lisxxx. We use opsource datasets preprocessed xxx we did not crxx build new dataset in this paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Comparison of the abilities required by a unified time series model. We evaluate whether existing works in time series possess the necessary capabilities for constructing a unified time series model, as outlined in Table 6 . Most methods fail to support these requirements. For instance, PatchTST [82] processes each variable independently, enabling it to handle multi-domain time series datasets without the need for data-specific heads. However, it still requires task-specific heads for tasks like making forecasts over a fixed length or performing classifications within a predetermined number of classes. × × × PatchTST [82] ✓ × × iTransformer [67] × × × Dlinear [119] × × × FEDFormer [128] × × × MICN [106] × × × Pyraformer [65] × × × Autoformer [114] × × × UNITS ✓ ✓ ✓Dataset details. We introduce the details of the multi-task dataset collection used by our work in Table 7 . The dataset collection used for few-shot learning on classification and forecasting are listed in Table 8 , the collection used for zero-shot forecasting are listed in Table 9 , the collection used for imputation is listed in Table 10 , and the collection used for anomaly detection is listed in Table 11 . Datasets were aggregated from the Monash Forecasting Repository [33] , Time Series Classification Website [79] , and Time Series Library [112] . The combined training set consists of over 35 million timesteps and over 6,000 variables. For subsets of a dataset such as ETTh1, we start by splitting the data into training and testing sets based on distinct time intervals of a long time series sequence, following splits in [112] . Within these training and testing intervals, we generate samples using various sliding windows, ensuring that there is no data leakage between the training and testing sets.Dataset for direct multi-step forecasting on new forecasting lengths. For evaluating zero-shot learning capabilities over new forecasting lengths, we initially consider 20 forecasting datasets utilized in the multi-task setting, as detailed in Table 7 . However, to adapt to 384 additional forecasting lengths that the model was not trained on, we exclude specific datasets that are incompatible with this requirement. These datasets include NN5 P 112 , ECL P 720 , ETTh1 P 720 , ILI P 60 , Traffic P 720 , and Weather P 720 . Consequently, our analysis is conducted using 14 remaining forecasting datasets.C.1 All learning settings supported by UNITS UNITS incorporates multi-task, prompt, few-shot, and zero-shot learning, as well as the single-task learning same to existing methods. We introduce the multi-task and prompt learning in the manuscript, here we introduce the other settings supported by UNITS.Notations for zero-shot/few-shot learning. X is an out-of-domain dataset collection not included in X , and Ŷ is used to denote a new type of tasks not contained in Y.Zero-shot learning. UNITS has zero-shot learning ability where model F (X , θ) trained on all datasets in D is tested on multiple types of new tasks that are not trained for, i.e. F (X , θ) → X , X / ∈ X . New zero-shot learning tasks include direct multi-step forecasting with a new length and forecasting on out-of-domain datasets with a new number of variables. Zero-shot learning shows the adaptability of UNITS to different time series tasks. Few-shot learning. UNITS model F (X , θ) pre-trained on X , can be fine-tuned on a few samples on new data X and new tasks Ŷ, i.e., Few-Shot{F (X , θ), X } = F ( X , θ) → Ŷ. We verify the few-shot learning ability of UNITS on forecasting and classification tasks on new, out-of-domain datasets and on new types of tasks, including imputation and anomaly detection.Single-task learning. UNITS model can also conduct the single-task learning same as the existing works, where each model is separately trained on each dataset D i = (X i , Y i ), i.e., F (X i , θ i ) → Y i .We introduce how to use tokens for forecasting and classification tasks in the manuscript. Here we present the implementation of using tokens for imputation and anomaly detection tasks. Imputation task. In tasks that require imputation, GEN token z m is inserted in the positions where sample tokens z x are missing. This process creates an augmented sequence of tokens represented by ẑx . These augmented tokens are then concatenated along the time dimension with prompt tokens, forming the input tokens for the network: where CA denotes the concatenation operation along the time dimension. Similar to the approach in forecasting tasks, the output for augmented sample tokens ẑx are unpatchified to obtain the imputed sample x, i.e. x = Proj(ẑ x ). Anomaly detection task. For the anomaly detection task, we follow TimesNet [112] to form it as a generative task, where the model is trained to reconstruct the time series sample using reconstruction error as the anomaly criterion. The prompt tokens and the sample tokens are concatenated along the time dimension to form the input tokens for the network:EQUATIONEQUATIONThe output for sample tokens z x is unpatchified to obtain the predicted sample x. During inference, following the approach in [112] , we determine a threshold of reconstruction error from the training and testing data, which is then used to detect anomalous time series points. Specifically, we sort the reconstruction errors between the input and output samples from our model across all training and testing sets. A predefined anomaly ratio is then applied to determine the threshold that distinguishes normal from anomalous data points.The UNITS network architecture is composed of N UNITS blocks, one CLS tower, and one GEN tower. We introduce more implementation details of UNITS network architecture, including the Time MHSA, Variable MHSA, Dynamic FFN, and Gate Module in the UNITS block, as well as the GEN/CLS towers shared for generative and predictive tasks. UNITS block: time and variable MHSA. For attention across the time dimension, the standard MHSA is applied as done by [82] . For variable MHSA, to capture relations among variables across all time points while minimizing the computational overhead associated with long time lengths, we average the Q and K over the time dimension to get shared Q and K as follows:EQUATIONwhere mean t is the mean along the time dimension. Then,Output = Attn v V = Softmax Q KT √ dV is obtained where Attn v ∈ R v×v is the attention map among variables, which is shared for all time points. The notations for multi-head attention are omitted for simplicity. We show the effectiveness of both time and variable MHSA in Table 22 .UNITS block: Dynamic FFN. By argument the FFN layer in transformers with the proposed DyLinear operator, we present the Dynamic FFN module, as shown in Figure 5 . In the Dynamic FFN, we replace the first linear layer in the standard FFN layer with a 3-kernel convolution across the time dimension to capture the local details. The second linear layer is kept the same as the standard FFN layer, and the DyLinear is inserted in between the input convolution and the output linear layer. Specifically, after processed by the convolution layer, the embeddings with d dimension are split into two groups, resulting in (z 1 mid , z 2 mid ) ∈ R s×v×d/2 . z 1 mid and z 2 mid are processed as follows:EQUATIONwhere DyLinear M processes the sample and prompt tokens in z 1 mid with two DyLinear operators, while CLS token is skipped to ensure consistency for all tasks. z 2 mid is kept unprocessed. This separation of routes for z 1 mid and z 2 mid leads to a scale combination effect, enhancing multi-scale processing ability [31] . UNITS block: gate module. The gate module is placed as the output of each component in the UNITS block, including time MHSA, variable MHSA, and Dynamic FFN. Specifically, given an input z in ∈ R s×v×d , a linear layer maps it to a scaling factor x g ∈ R s×v×1 along the embedding dimension. This is followed by a Sigmoid function to ensure the scaling factor lies between 0 and 1. The final gating operation involves elementwise multiplication of the input by the Sigmoid-activated scaling factor, i.e.,EQUATIONGEN tower. The GEN tower H GEN is designed to transform tokens into time points prediction results. One GEN tower is shared by all generative tasks, including forecasting, imputation, and anomaly detection. As shown in Figure 4 , take the forecasting task as an example, the z Fore ∈ R (p+s+f )×v×d from Eq. 1 is processed by the GEN tower to get the full time-series sample as follows:x = Proj(MLP((z Fore + DyLinear(z Fore ))),where the MLP is composed of two linear layers with an activation layer in between, and Proj is the unpatchify operation that transfers the embedding back to the time series patch as introduced in Section 4.1. For imputation and anomaly detection tasks, only the tokens are modified while the GEN tower remains unchanged. CLS tower. The CLS tower H CLS transforms CLS tokens into classification classes. The CLS tower is shared across all classification tasks from different datasets. As illustrated in Figure 4 , the CLS tower processes z Pred ∈ R (p+s+1)×v×d from Eq. 2, which includes the CLS token z ′ c , to produce the final CLS token z c as follows:EQUATIONwhere the CLS token z ′ c serves as a query to perform cross-attention with all tokens in z Pred . Subsequently, the processed CLS token z c is matched with class embeddings to determine the predicted class as described in Eq. 3.By default, in a multi-task setting, the UNITS network comprises three UNITS blocks, one GEN tower, and one CLS tower. For each data source, the prompt tokens and task tokens are defined. Forecasting tasks on the same data source but with different forecast lengths share the same prompt and GEN token. For zero-shot learning on new datasets, we use a shared prompt and GEN token across all data sources to facilitate zero-shot learning. Tokens are trained to achieve their functions. The number of embedding dimensions, d, is set to 64 for UNITS-SUP and 128 for UNITS-PMT. All blocks in UNITS maintain the same feature shape, following the Transformer architecture. DLinear [119] LightTS [122] Transformer-based iTransformer [67] PatchTST [82] Transformer [104] Reformer [51] Informer [127] Pyraformer [65] Autoformer [114] Stationformer [69] FEDformer [128] ETSformer [111] Flowformer [ DLinear [119] LightTS [122] Transformer-based iTransformer [67] PatchTST [82] Reformer [51] Informer [127] Pyraformer [65] Autoformer [114] Stationformer [69] FEDformer [128] ETSformer [111] LogTransfomer [57] TCN-based TCN [30] RNN-based LSTM [41] LSSL [39] Anomaly detectionDLinear [119] LightTS [122] Transformer-based iTransformer [67] PatchTST [82] Transformer [104] Reformer [51] Anomaly Transformer [116] Informer [127] Pyraformer [65] Autoformer [114] Stationformer [69] FEDformer [128] ETSformer [111] For multi-task settings, all models are jointly trained on multiple tasks following the same training protocol. To match the size of the largest dataset, samples from each dataset are repeated in every training epoch. In each inference step, datasets are randomly sampled with equal probability, utilizing a batch size of 32. Supervised training involves 5 epochs using gradient accumulation for an effective batch size of 1024, starting with a learning rate of 3.2e-2 and adjusted with a multi-step decayed schedule. The λ i in L total are all set to 1 in this work. For self-supervised pre-training, the models are trained over 10 epochs with an effective batch size of 4096 and an initial learning rate of 6.4e-3, using a cosine decay schedule. All experiments are conducted using A100-40G GPUs. Each experiment is conducted with one or two GPUs, and the maximum running time is under 48 hours.Since all models are jointly trained across multiple tasks, we report the average performance for each task type. For tasks involving forecasting and imputation, model performance is assessed using Mean Squared Error (MSE) and Mean Absolute Error (MAE). In classification tasks, accuracy is used as the primary evaluation metric. For anomaly detection tasks, performance is measured using precision, recall, and the F1-score.No task-specific hyper-parameter tuning. UNITS is designed for multi-task settings where tasks share the same model weights. In UNITS, we do not need to perform any task-specific hyperparameter tuning. The baseline methods follow the same training setting as our method to ensure a fair comparisons.During the unified pre-training, we introduce two distinct masking schemes: the random masking scheme and the right masking scheme. The time series sample is initially truncated to a length randomly selected within the range of 50% to 100% of its original length. Subsequently, in the random masking scheme, a certain proportion p rand of tokens are masked at random positions within the time dimension. For the right masking scheme, designed to enhance the model's forecasting ability, a random proportion p right of tokens on the right side of the sample is masked. Both p rand and p right are set to 70%-80%. Each training step randomly utilizes one of these two schemes with equal probability.The baseline methods used in this paper are summarized in Table 13 . Unlike UniTS, which can handle diverse data and tasks within a single model, baseline methods cannot be directly used for unified training because: 1) To accommodate data with varying numbers of variables, baseline methods typically use a data-specific input head to project features from the variable count to a fixed number of embedding dimensions. 2) Similarly, to manage different tasks, such as classification with various classes and forecasting with different lengths, baseline methods employ task-specific output heads to transform the features into the appropriate task outputs. Since baseline methods are designed for single-task training, in their original setting, data/task-specific heads are used for each data and task.In the multi-task learning setting, to make baseline methods support unified training, we add separate input heads to project data into a shared embedding space and separate output heads to convert the shared model output into task-specific outputs. However, using separate input and output heads makes it hard to generalize to new datasets and tasks. We employ the same fully supervised multi-task training approach as UniTS. In this setting, model networks are stacked with 3 basic building blocks, except for GPT4TS, which utilizes the prescribed setting of 6 GPT blocks. For both the proposed method and patch-based baseline approaches, the patch size and stride are fixed at 16. The input and output heads of baseline methods are duplicated for each task to create data/task-specific heads tailored for each data source and task. For single-task learning settings, we follow the original settings of baseline methods and compare results reported in their papers.We do more analysis on the prompting and pre-training of UNITS. The average performance under 38 datasets with the multi-task setting is reported.Prompt learning with model scaling. In Table 14 , we further explore the capabilities of prompt learning in the SSL pre-trained UNITS model across different model sizes. As UNITS model size grows, we observe consistent improvements in performance for both classification and forecasting, suggesting that larger SSL models contain more robust representations for prompt learning. Effect of prompt tokens. Prompt tokens learn the contextual information related to the given data source and task types. By default, we use 10 prompt tokens for each task. We present an ablation study on the use of different numbers of prompt tokens in Table 15 . Utilizing prompt tokens leads to notable improvements in both forecasting and classification tasks. The average classification accuracy improves from 81.0% to 81.6%, and the average MSE and MAE improve from 0.460 to 0.439 and 0.391 to 0.381, respectively. Employing 10 instead of 5 prompt tokens results in greater gains in forecasting tasks and a marginal improvement of 0.1% in classification accuracy, indicating that forecasting tasks benefit more from the contextual information provided by the prompt tokens. We also evaluate the case where all prompt tokens are shared among tasks in Table 16 . Using shared prompt tokens across different tasks results in a performance decline, yet this approach still surpasses the performance of models that do not utilize prompt tokens. Cross-task pre-training. We evaluate the effect of cross-task pre-training by pre-training a model using our pre-training strategy on either generative tasks (forecasting) or predictive tasks (classification). Table 20 shows that UniTS, pre-trained solely on forecasting datasets, achieves similar performance to the model pre-trained on both forecasting and classification data. Despite not encountering any Effect of gate module. In Table 24 , we present a comparison of the UNITS model with and without the inclusion of the gate module. Incorporating the gate module yields consistent enhancements relative to the baseline model that lacks it. Specifically, the addition of the gate module results in an increase in classification accuracy, moving from 81.1% to 81.6%. For the forecasting task, the MSE sees an improvement from 0.459 to 0.439, and the MAE decreases from 0.387 to 0.381. These results show the effectiveness of the gate module in mitigating task interference by adjusting the scaling of embedding vectors.Table 25 : Zero-shot multi-task learning on forecasting tasks on 5 out-of-domain data with new forecasting length and new number of variables. We set shared prompt tokens and GEN tokens for UNITS. One sample from each dataset is used following [81] . Comparison with Transformer. To verify the effectiveness of UNITS structure, we compare the original Transformer with UNITS. The unified tokenization and co-training strategy are applied to both models. The results shown in Table 26 indicate that UNITS clearly outperforms the Transformer in both classification and forecasting tasks, suggesting that merely using a transformer structure is insufficient for achieving robust multi-task performance on time series datasets.Setup. When UNITS is trained with shared prompt and GEN tokens across all forecasting tasks, it acquires the ability to perform zero-shot forecasting on datasets with new lengths and variable numbers that were not part of its training domain. We evaluate UNITS in a zero-shot setting on five new forecasting tasks as referenced in Table 9 . These tasks have varying forecasting lengths and numbers of variables compared to those seen by UNITS during pre-training. We benchmark against LLMTime [81] , a model designed for zero-shot forecasting using LLMs. Following LLMTime, we utilize one sample from each dataset to manage the extensive inference costs. We exclude a related method, Time-LLM [47] , from experiments. Time-LLM supports zero-shot learning but requires that the forecasting length and the number of variables/sensors for zero-shot prediction are the same as those used for training. We calculate the similarity between prompt tokens across datasets, as illustrated in Figure 7 . Datasets within the same class, for instance, FaceDetection and SelfRegulationSCP2, which both consist of EEG data, demonstrate a higher similarity. While some out-of-domain datasets still exhibit strong similarities, indicating that they share certain similar requirements.To compare the difference among tokens before and after training, beyond similarity comparison, we show UMAP plots generated with the prompt tokens before and after training, in Figure 8 and Figure 9 . Before training, the prompt tokens from all datasets are dispersed. In contrast, the UMAP of prompt tokens after training reveals that tokens from the same datasets are clustered. However, some tokens from different datasets remain closely positioned, indicating that data from different domains share similar information.We present the performance of multi-task classification on each dataset in Table 27 . Average inference steps comparison. In Table 6 , we present a comparison of the average number of inference steps required by our direct multi-step inference method and the multi-step sliding window-based inference approach. Contrary to the direct multi-step inference, which is completed in a single step, the sliding window-based method necessitates multiple inference steps. Specifically, for the maximum extra inference length of 384, the sliding window-based approach demands, on average, 3.66 times more inference steps.K Additional Results: Benchmarking in the Single-Task RegimeSetup. As we are the first work that focuses on time series multi-task learning with one model, to make fair comparisons with existing time series methods, we compare them with the single-task setting.In this setting, for each dataset, one model is independently trained with tuned hyperparameters. Following existing works [112, 67, 14] , we tune the following hyperparameters, including number of channels, patch size, number of layers, learning rate, and dropout ratio. The baseline methods for time series forecasting, classification, anomaly detection, and imputation, are listed in Table 13 . We following existing works [112, 67] to use 36 commonly used datasets for forecasting (Table 30 ), 10 For fair comparisons, we follow the settings of [112] to only use reconstruction error for Anomaly Transformer. TimesNet are reproduced from the https://github.com/thuml/Time-Series-Library to ensure fair comparisons. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0To verify the gap between multi-task and single-task learning under fair comparisons, we conduct a experiment to train the single-task models using the same hyper-parameters as the multi-task cotraining. As shown in The datasets collected by this work do not yet cover all available time series datasets, such as some of the univariate datasets in UCR dataset collections [23] and the more physiologic time series signals from PhysioNet [34] . We will explore using larger dataset collections to further improve UNITS.UNITS primarily aims to unify predictive and generative tasks within a single multi-task model. We demonstrate this by showcasing its adaptability to new data and tasks through prompt learning and few-shot learning. While adapting to new time series data differs fundamentally from generalizing to entirely new data, we will further explore UNITS's generalization ability for zero-shot learning. This paper focuses on analyzing time series sequences from various domains and introduces a versatile machine-learning approach designed for this purpose. While our research has numerous potential societal impacts, we believe none require specific emphasis in this context. Justification: The abstract and introduction cover the contributions and scope of the paper regarding building a unified time-series model.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper proposes a unified pre-trained model for time series data called UNITS, designed to handle multiple predictive and generative tasks across different datasets. The model employs a transformer architecture enhanced with prompt and task tokens, enabling it to perform prompt tuning and few-shot learning. UNITS is pre-trained on multiple datasets from various domains and is evaluated on its transferability across different downstream tasks, demonstrating comparable or superior performance to state-of-the-art models.

**Strengths:**
- Strong experimental results, showing effective cross-dataset training on multiple tasks.
- The issue of handling diverse domains and multiple tasks within a single model is of great importance.
- UNITS achieves state-of-the-art results in several tasks and can adapt to new tasks via prompt learning.
- The paper is generally well-written and easy to follow.

**Weaknesses:**
- Overclaimed contributions, particularly regarding the generalizability to new tasks that were seen during pre-training.
- Lack of theoretical understanding and ablation studies on the impact of multi-dataset training.
- The model’s novelty is questioned due to insufficient methodological explanations and the orthogonality of the model's structure to the proposed tokens and training methods.
- Issues with writing clarity, especially in explaining the roles of the tokens and experimental setups.
- Missing comparisons with similar models using unified training or prompt-based approaches.

Some concerns have been addressed by the authors during the rebuttal period. 

Reviewers generally lean positive for this paper. One of the reviewers had concerns about some of the overclaim, downgrading the rating from 6 to 5, but the concern seemed later addressed by the authors, and the reviewer adjusted the rating back to 6. Overall, I agree with the reviewers that this is a good contribution to the community and therefore recommend acceptance.

Reviews: review: paper_summary: paper_summary: This work aims to present a pre-trained foundation model for time series. They proposed a model called UNITS, that performs multi-task learning (both generative and discriminative) on time series datasets. Specifically, they embed a transformer with prompt tokens and task tokens to perform prompt tuning or few-shot learning. The proposed method is pre-trained on multi-domain datasets, and showed transferability across multiple downstream datasets with diverse task specifications and domains. The authors performed experimental results that show UNITS performs favorably, or comparably against many state-of-the-art models, and can be used in few-shot and prompt tuning ways.
strength_weakness: strength_weakness: Major concern 1: Overall, the contributions from the paper are overclaimed. 
- The authors claim "generalizability to new tasks", yet (1) the proposed model is pre-trained on all datasets that are later experimented on; (2) for all available tasks, the authors perform prompt tuning on an aggregate dataset-centric loss value that includes all tasks that are later considered. If the model does not see a new dataset or new tasks during test time, why claim generalizability to new tasks?

Major concern 2: Lack of analytical or theoretical understanding of how multi-dataset training benefits performance.
- Lack of ablation experiments. For example: How much does the amount of pre-training datasets help (in terms of training steps or amount of datasets)? More systematic evaluation of the interrelationship between pre-training tasks (beyond Table 17)? How does the performance vary when the number of prompt tuning parameters change? 
- Lack of better understanding of how to handle varying channel-wise relationships. One of the biggest challenges in building foundational model for time series is that while temporal dynamics may be generalizable, the channel-wise relationships differ significantly for each dataset. How does the proposed method address this issue through pre-training, or does it just see all possible channel-wise relationships, that are going to appear at test time, during the pre-training process?
- Lack of understanding of the learned tokens. How do the sample tokens capture the dynamics of the data, can they be visualized or categorized? How do the task tokens or prompt tokens differ from other task tokens or prompt tokens after training beyond similarity comparison? Does the similarity comparison show the intrinsic dynamical properties of the datasets?

Minor concern 1: In experimental results, the baseline numbers seem to be slightly lower than that are reported in other papers.

Minor concern 2: Lack of methodological novelty. Due to the lack of ablations, it is challenging for me to dissect which parts of the proposed model are critical to the results. The multi-task parts seem to be the work that tries to show that the more datasets one uses to train their model, the better the performance will be, as long as one uses the right combination of existing methods.

Minor concern 3: Can the authors compare the number of prompt tuning parameters against other baseline models?
questions: questions: See above.
limitations: limitations: Can be more comprehensive.
paper_summary: paper_summary: This paper proposes UNITS, a multi-task time series model that handles multiple predictive and generative tasks within a single model. UNITS uses the mask modeling pre-training framework. To handle multiple downstream tasks, two new sets of tokens are concatenated with data tokens: 1)prompt tokens indicating the task corresponding to each dataset and are used for future prompt learning; 2)task tokens to generate generative/discriminative outputs. A modified transformer block is also proposed. Experiments of single-task and multi-task learning on forecasting, anomaly detection, imputation, and classification demonstrate the effectiveness of the proposed model.
strength_weakness: strength_weakness: 1. The writing needs improvement. 1) Section 4.3 requires a more detailed description with formal equations. Actually, it is hard to get the role of three kinds of tokens in pre-training; 2) Experiment setting in Section 5 also requires re-organization to clarify what experiments are conducted and which datasets are used for which tasks. Moreover, several datasets are generated from the same dataset with different sliding windows and thus should be addressed in the paper.

2. The modified model structure is orthogonal to the tokens and training methods proposed in the paper, and they are not well-coupled. For instance, training can still be conducted using the original Transformer.

3. Compared with traditional single-task single-dataset supervised training models (Table 25), the proposed multi-datasets multi-task pre-training model (Table 2) does not show a better performance. As training a simple model like DLinear for each dataset and task is efficient, this calls into question whether joint training across multiple domains of data can provide real benefits.

4. Some details of some experimental setups are not rigorous: for example in Table 4 (few-shot learning for imputation), datasets ECL, ETTh1 and Weather are generated from the same datasets used for pre-training, which the model can get access to, so these settings can not be viewed as few-shot learning.
questions: questions: 1. In Equation 5, what is the role of task tokens $z_m$? Compared with the left side, the right side encourages $H_{cls}(z_{pred}) = z_p$. How can it be discriminative enough for classification?

2. How is $\lambda_i$ in line 270 be determined?

3. Please compare UNITS trained by multi-task learning with that trained by single-task learning to confirm the necessity of unified multi-task training.
limitations: limitations: No potential negative societal impact.
paper_summary: paper_summary: The authors propose a unified model trained over multiple datasets to solve multiple tasks such as forecasting, imputation, anomaly detection, and classification. In the paper, the authors demonstrate their model abilities through extensive empirical results comparing with a large variety of baseline methods and suggesting new benchmark to compare multi-task results.
strength_weakness: strength_weakness: - The details for comparing with previous baseline methods are missing, and thus it is hard to assess how well the model performs in the unified set-up in contrast to previous methods. (see questions)


- Although most parts of the paper are clearly written and described, it is hard to follow the prompt learning part of the methods’ methodology. What exactly are the prompt tokens? The authors perform an extensive ablation study and include the number of prompts as part of it but do not describe or give an example of such prompts. (Is it just weights?)


- The empirical evidence lacks comparison to similar methods that employ unified training or prompt-based approaches, as in [1,2,3]. Although the author’s unified multi-task model is indeed interesting, it still remains unclear whether the given approach is at least comparable to other unified forecasting models that use similar benchmarks such as unified training, few-shot, zero-shot, etc.


[1]  TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting

[2]  UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecasting

[3]  Time-LLM: Time Series Forecasting by Reprogramming Large Language Models
questions: questions: - Eq. 2 is slightly confusing, $\textbf{z}_c$ is defined as an input to the model but at the same time is used for the final class choice. Isn’t it should be the output of the model compared to each $\textbf{z}_e(i)$?


- In Eq. 4, $W_{\text{Interp}}$ should be defined as $l_{\text{out}} \times l_{\text{in}}$ instead of $l_{\text{in}} \times l_{\text{out}}$, otherwise the DyLinear operations does not compile in terms of dimensions.

- The implementation details of the baseline methods should be extended. Specifically, it is hard to understand what exactly was done to train the models for the new benchmark compared to UniTS-PM and UniTS-SUP. As it does not support unified training, in the appendix part, the authors mention: “For multi-task learning settings, models are configured with 3 blocks…” “The input and output heads of baseline methods are duplicated for each task to create data/task-specific heads tailored for each data source and task.” How is it different from the original set-up of those baseline methods? what do the authors mean by models configured with 3 blocks?
- How do you perform zero-shot? Now that you do not have the corresponding prompt tokens, what do you feed to the network instead of those tokens?
- The bold and underline in Table 2 for the ILI row seem to be incorrect as GPT4TS outperforms the given results.
limitations: limitations: See the Weaknesses and Questions sections.
paper_summary: paper_summary: This paper presents UNITS, a multi-task time series model that uses task tokenization to express predictive and generative tasks within a single model. UNITS can process heterogeneous time series with diverse variables and lengths without modifying the network architecture. Experiments show that UNITS demonstrates effective few-shot and prompt learning capabilities when evaluated on new data domains and tasks.
strength_weakness: strength_weakness: Although the idea of a multi-task model is somewhat novel, it seems that the UNITS on Single-Task Learning outperforms Multi-Task Learning.
questions: questions: 1. Is the experiment setting in Section 5.2 consistent with that in Section 5.1? Does GPT4TS also use the same fully supervised multi-task training as UNITS?
limitations: limitations: yes

Comments: title: Thank you!
Thanks
Reminder of our new response to Reviewer h6vS's concerns.
New response to Reviewer h6vS Part II
New response to Reviewer h6vS
Excessive parameters trained during tuning & channel-wise relationship
Follow up on rebuttal
Follow-up on reponses
Response to Reviewer h6vS Part IV (Part IV of IV)
Response to Reviewer h6vS Part III (Part III of IV)
Response to Reviewer h6vS Part II (Part II of IV)
Response to Reviewer YrSc Part III (Part III of III)
Response to Reviewer YrSc Part II (Part II of III)
Response to Reviewer EsEH Part II (Part II of II)
Response to Reviewer bSMF Part II (Part II of II)
comment: Dear Reviewers,

We sincerely appreciate your thoughtful feedback and the time you invested in reviewing our paper! We will carefully revise the manuscript based on your valuable comments.

Thank you again for your time and dedication!

Best regards, 

The Authors
Thanks for your reply. It is a detailed plan for tuning down the claims. Please make sure they are included in the revision.
Thank you for your valuable feedback! 
We've provided a detailed response to your new comments. In case the OpenReview system didn’t notify you, we wanted to bring it to your attention here as well.

If you find our response unsatisfactory, please don't hesitate to share your concerns with us. We are more than happy to continue the discussion. 

Thank you for your time and consideration.

Best regards,

The Authors
# New response to Reviewer h6vS Part II (Part II of II)
### Claim about zero-shot ability  
We fully understand your concerns about adapting to new channel-wise relationships rather than generalizing, i.e. few-shot versus zero-shot learning. We agree that zero-shot learning is significantly more challenging than few-shot learning. **Our work primarily focuses on few-shot learning, with some initial exploration of zero-shot learning. We will emphasize this distinction and moderate our claims regarding zero-shot capabilities to avoid any overclaim.** Below, we provide a detailed explanation of our initial attempts in zero-shot learning, our work in few-shot learning, and the difference between UniTS and PatchTST.

* **For initial attempts in zero-shot learning**, the table below shows UniTS considerably surpasses LLMTime, a model designed for zero-shot forecasting using LLM, across most of the tested datasets, demonstrating superior performance in handling unseen channel-wise relationship, different forecasting lengths, and different variable numbers. Note that the testing datasets are never seen during pre-training.  For example, UniTS achieves a considerable improvement in MSE over LLMTime (0.030 vs.0.265) on Solar. Remarkably, UniTS exhibits an inference speed approximately 10e6 times faster than LLMTime. We also show in Figure 3 of Figure PDF (Figure 3 of the manuscript, page 8\) that UniTS can generalize to new forecasting lengths not seen during training. We agree that these attempts are only initial explorations of using the pre-trained model for zero-shot learning, and we will stress this in our manuscript.
| Solar dataset | MSE | Infer. Time (seconds) |  | 5 datasets (Full results in Table 21\) | Best count | Var. number | Pred length |
| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |
| LLM-Time | 0.265 | 2.0e3 | \- | LLM-Time | 1/5 | 1 to 767 | 16 to 128 |
| UniTS | **0.030** | **6.8e−3** | \- | UniTS | **4/5** | 1 to 767 | 16 to 128 |

* **For few-shot learning that UniTS mainly focuses on**, the table below shows UniTS clearly outperforms the best-performing single-task model iTransformer, on forecasting, classification, imputation, and anomaly detection tasks. (Full comparisons with other methods are shown in Tables 3,4,5,23,24.)
| Method/Best count | Forecast  (9 datasets) | Classification  (6 datasets) | Imputation  (6 datasets) | Anomaly detection  (5 datasets) |
| :---- | :---- | :---- | :---- | :---- |
| iTrans | 0/9 | 1/6 | 0/6 | 0/5 |
| UniTS | **9/9** | **5/6** | **6/6** | **4/5** |

* The difference between our work and PatchTST in handling varying amounts of channels is that PatchTST doesn’t consider the inter-relationship among channels, treating all channels independently, while we design a new architecture and pretraining scheme to enable the model to learn these inter-relationships, which can generalize to new data, as we responded to you in Major Concern 2.4.

### Overclaiming on the UniTS components  
Thank you for highlighting this important point. We would like to clarify that we do not claim UniTS to be a foundational model for time series that learns tasks. The word "foundation" or "foundational" appears 10 times in the main paper and supplementary materials. Five instances are in the Reference list, and the remaining five instances occur in the background sections of the abstract, introduction, and related work. We are careful not to claim that UniTS is a foundational model, and we have tried to carefully pace our claims and support them by experimental results (e.g., UniTS is a multi-task model unifying predictive and generative tasks, as reflected in our evaluations).

---

That said, understanding your excellent comment, we will additionally adjust our claims regarding the zero-shot capability of UniTS. We will also incorporate the points you raised into the limitations section to guide future work that would bring us closer towards a foundation model for time series.

If you find our response unsatisfactory, please don't hesitate to share your concerns with us. We are more than happy to continue the discussion. Given this new understanding and clarification on misunderstanding wrt point 1, we kindly ask that you reconsider the adjustment of your initial score.
# New response to Reviewer h6vS Part I (Part I of II)
Thank you for pointing out these two important questions\! We totally understand your concerns and we are clarifying your concerns about the parameters in UniTS and we will tune down our claim about the zero-shot ability of UniTS.  We will further add these points you mentioned in the limitation section to prompt future work to explore the unified time series model. However, we would also like to clarify a misunderstanding on the first point that we respond to below. 

If you think our response is still unsatisfactory, please share your concerns with us, we are more than happy to have further discussion with you.

---

### Parameters of UniTS

* Thank you for your comment, but we would like to kindly point out a misunderstanding. **The total number of parameters in UniTS is 3.4M**, including 1.0M for the backbone network and 2.4M for the tokens, **which is smaller than the 3.9M parameters in TimesNet.** **Again, 3.4M is the TOTAL number of parameters in UNiTS, not only the number of trainable parameters in the prompt tuning stage** ***(PLEASE SEE THE 5TH COLUMN IN THE TABLE BELOW)*.**   
* **The number of tokens can be adjusted to balance performance and parameter count.** We use 10 prompt tokens for each dataset, but as shown in table below (Table 14, page 24), reducing the number of prompt tokens or even eliminating them can significantly decrease the parameter count while still outperforming TimesNet. For example, reducing the number of prompt tokens from 10 to 5 decreases the token parameters to 1.5M with only a minor performance drop, such as a 0.1% drop in accuracy. **Even without prompt tokens, UniTS, with 0.7M token parameters and 1.7M total parameters, still outperforms TimesNet, which has 3.9M parameters (with only 43% of TimesNet's parameters), by a wide margin in forecasting tasks**, e.g., 0.460 vs. 0.525 in MSE.
| Method | Prompt token Num. | All Token parameters | Model network parameters | Total number of parameters (tokens+network) | Acc ↑ | MSE ↓ | MAE↓ |
| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |
| TimesNet | \- | \- | 3.9M | 3.9M | 80.9 | 0.525  | 0.412  |
| **UniTS** | **0** | **0.7M** | **1.0M** | **1.7M** | **81.0** | **0.460** | **0.391** |
| UniTS | 5 | 1.5M | 1.0M | 2.5M | 81.5 | 0.455 | 0.387  |
| UniTS | 10 | 2.4M | 1.0M | 3.4M | 81.6 | 0.439 | 0.381 |

* **We want to clarify that UniTS is not based on LLMs; instead, it is trained exclusively on time series data.** As shown in Table 29 (page 32), UniTS demonstrates superior performance across most datasets, even without utilizing the GPT-2 model (1,500M parameters), which is trained with additional textual data and used by other LLM-based methods. Furthermore, unlike these methods that focus solely on forecasting, UniTS supports both generative tasks (e.g., forecasting, imputation, anomaly detection) and predictive tasks (e.g., classification).
|  | UniTS | TEMPO | UniTime | TIME-LLM |
| :---- | :---- | :---- | :---- | :---- |
| Additional data necessary beyond the training set | **No** | Yes | Yes | Yes |
| Multi-task support | **Yes** | No | No | No |
| ETTm1 | 0.337/0.376 | 0.501/0.458 | 0.385/0.399 | 0.329/0.372 |
| ETTm2 | 0.254/0.315 | 0.281/0.328 | 0.293/0.334 | 0.251/0.314 |
| ETTh1 | 0.405/0.426 | 0.428/0.427 | 0.442/0.448 | 0.408/0.424 |
| ETTh2 | 0.331/0.387 | 0.361/0.398 | 0.378/0.403 | 0.334/0.383 |
| ECL | 0.156/0.253 | 0.216/0.308 | 0.216/0.305 | 0.159/0.253 |
| Traffic | 0.409/0.278 | 0.503/0.358 | \- | 0.388/0.264 |
| Weather | 0.216/0.259 | 0.282/0.316 | 0.253/0.276 | 0.226/0.258 |

**Please see more content in PART II**
Thank you for the careful response. While the response did clarify some of my questions, I think it also further highlighted my doubts about the work. I want to highlight two points below:

1. UNITS trains **2.4M** parameters during the prompt tuning stage, while the near-SOTA task-specific model TimesNet only has **3.9M** parameters in total. The new parameters, by themselves, are sufficient to train a brand-new model to achieve decent performance. It would be unfair to compare against huge models that attempt to engineer LLMs into TS models because their objective is to adapt a very large model to be useful in this case.
2. Being adaptive to new channel-wise relationships is fundamentally different from being generalizable to new channel-wise relationships. I understand that the model could be used on data with varying amounts of channels, and many works have different ways to achieve that (e.g. PatchTST). However, as the model claims itself to be a foundation model for time series that learns tasks in a zero-shot way, being merely adaptive is not sufficient. The authors have to provide the fundamental logic of using the pre-trained model on a completely unseen channel-wise relationship. Otherwise, it is an overclaim.

Overall, I like the engineering perspective of the work, yet I found the overclaiming components very concerning, and the limitations of the proposed approach are not very well addressed/discussed in the paper. I'd give a much higher score if the paper does not overclaim or lead the readers to assume beyond what is told by experimental results. Given my current understanding I'd adjust my score from 6 to 5.
Thank you for the responses, clarifications, and additional results. Considering other reviewers' comments and the author's rebuttal, I have updated my score.
Thanks for your response. The rebuttal addressed my concern about Multi-task vs. Single-task learning. I have raised my score to 5. 

However, I do suggest authors improve the representation in the final version, including: 1) giving detailed descriptions of the pre-training process, better to provide a visualization to show the role of two towers in pre-training; 2) reorganizing the experiment section to clarify what experiments are conducted in a more structured way.
Thank you for the responses. Considering other reviewers' comments and Author Rebuttal as well, I have updated my score.
Dear Reviewer h6vS,

We sincerely appreciate your insightful comments and the time you have taken to review our work. In response, we have carefully addressed each of your concerns and made the necessary revisions. We kindly ask if these revisions meet your expectations and if there are any further concerns or feedback you would like to discuss. We are more than happy to continue the conversation and address any additional points you may have.

Thank you once again for your time and thoughtful consideration.

Best regards,
The Authors
Dear Reviewer YrSc,

We sincerely appreciate your valuable comments! We have provided detailed responses to each of your concerns and made the necessary revisions. We kindly ask if you are satisfied with our revisions and if you have any further concerns or feedback. We would be more than happy to continue the discussion and address any additional points you may have.

Thank you for your time and consideration.

Best regards,

The Authors
# Response to Reviewer h6vS Part IV (Part IV of IV)

### Minor concern 1: Slightly lower baseline numbers.

We have two settings: a single-task setting and a multi-task setting. For the single-task setting, we report results from previous works to ensure fair comparisons. For the multi-task setting, we reimplement the baseline to support multi-task capabilities. The multi-task setting is intrinsically more challenging than the single-task setting, so the results for baseline methods are lower than those for the single-task setting.

### Minor concern 2: Methodological novelty and ablations. How does multi-task work?

We show in Table 2 that UniTS outperforms existing methods in the multi-task setting, and it’s not a combination of existing methods. Instead, we propose new designs to handle multi-task learning as follows:

* The unified tokenization strategy unifies the predictive and generative tasks, which is the first time that two types of tasks have been unified in one model, as far as we know.  
* The Variable/Time MHSA separately handles the features with varying numbers of variables and time slots, enabling one model to handle different datasets, while existing works have to define data-specific input heads to support each dataset.  
* The Dynamic FFN adapts the weight shape to capture the shared temporal representations of different data.  
* The Gate module modulates the feature scales to mitigate interference in the latent representation space caused by multi-domain and multi-task datasets.

Due to the limited space in the manuscript, we show the ablations in Tables 15, 16, 17, 18, 19, and 20 of the appendix. From these tables, we summarize the ablation results of key modules in UniTS in the following table. For classification, both Time MSHA, Variable MSHA, and Dynamic FFN play similar roles in performance, indicating that time/variable/channel level features are all important to the time series classification. For forecasting, the Dynamic FFN and Gate module plays more important roles than the Time/Variable MHSA. Still, Time/Variable MHSA is important to forecasting performance.

|  | Acc↑ (Classification) | MSE↓ (Forecasting) |
| :---- | :---- | :---- |
| UniTS | 81.6 | 0.439 |
| UniTS w/o Time MHSA | 80.7  | 0.449 |
| UniTS w/o Variable MHSA | 80.8 | 0.444 |
| UniTS w/o Dynamic FFN  | 80.8 | 0.465 |
| UniTS w/o Gate module  | 81.1  | 0.459 |

### Minor concern 3: Compare the number of prompt tuning parameters against other baseline models.  
The table below compares the prompt tuning parameters of UniTS against strong baseline methods. The tokens used for prompt tuning in UniTS are only about 1.4% of parameters used in the LLM-based GPT4TS. Additionally, compared to pure time series models like TimesNet and PatchTST, the parameters for prompt tuning in UniTS are still smaller.

| Model | Prompt-tuning parameters in UniTS | GPT4TS | TimesNet | PatchTST |
| :---- | :---- | :---- | :---- | :---- |
| Trainable parameters | 2.4M | 164.5M | 5.0M | 3.9M |
# Response to Reviewer h6vS Part III (Part III of IV)

### Major concern 2.3 How does the performance vary when the number of prompt tuning parameters change?

We have performed an ablation study on the number of prompt tokens in table below (Table 15 of the manuscript, page 24). We find that larger numbers of prompt tokens lead to better performance, and this performance gains levels off with the increased number of prompt tokens. However, the performance remains consistently high across different token counts. For instance, using 10 versus 5 tokens results in only a \+0.1 increase in accuracy and a \-0.016 decrease in MSE, indicating that the results are robust to variations in this hyperparameter.
| Prompt token (Count) | Acc↑ (Classification) | MSE↓ (Forecasting) |
| :---- | :---- | :---- |
| 0 | 81.0  | 0.460 |
| 5 | 81.5 | 0.455 |
| 10 | **81.6** | **0.439** |

### Major concern 2.4: How to handle varying channel-wise relationships?

Handling the varying channel-wise (variable-wise) relationships is a key issue we addressed in designing UniTS. We tackle this challenge by designing UniTS such that it can handle arbitrary numbers of variables. UniTS specifies a unified masked reconstruction pre-training task to ensure the model can capture channel/variable-wise relationships without assuming a fixed number of variables/channels across all pre-training/fine-tuning datasets.

At the architecture level:

* The input to the network is tokenized to features that have independent channel and variable dimensions.    
* The UniTS network is designed to handle a varying number of variables. For example, we use the Variable MHSA to get the relation among variables without the need to use linear layers with a fixed number of channels to learn relation among variables.  
* Prompt tokens in UniTS are variable-dependent, enabling the network to adapt to data with varying numbers of variables seamlessly.

At the model pre-training level: 

* Based on the unified network that handles a varying number of variables, our pre-training approach is designed to be independent of the number of variables. It employs a unified masked reconstruction pre-training task to randomly mask input tokens. The model is forced to learn general and shared representations of the relationships among variables to reconstruct missing tokens. This ensures that the model can capture the essential relational dynamics without being tied to a specific number of variables.   
* The prompt tokens for datasets are also pretrained, making the prompt tokens learn the conditions that can prompt the network to capture data-specific variable-wise relationships during inference.

### Major concern 2.5:  How do the sample tokens capture the dynamics of the data, can they be visualized or categorized?

In UniTS, the sample tokens are extracted from the input sample to capture data dynamics using time/variable MHSA, DynamicFFN, and Gate modules. These tokens are then transformed into sample sequences via the GEN tower, allowing visualization as time series sequences. Additionally, the CLS token enables categorization by transferring sample tokens to the classification classes in the CLS tower.

### Major concern 2.6: How do tokens differ from other tokens after training beyond similarity comparison?

To compare the difference among tokens before and after training, beyond similarity comparison, we add new UMAP plots generated with the prompt tokens before and after training, in Figure 1 and Figure 2 of Figure PDF file. Before training, the prompt tokens from all datasets are dispersed. In contrast, the UMAP of prompt tokens after training reveals that tokens from the same datasets are clustered. However, some tokens from different datasets remain closely positioned, indicating that data from different domains share similar information.

### Major concern 2.7: Does similarity comparison show the intrinsic dynamical properties of the datasets?

We categorize the datasets and visualize the similarity among learned prompt tokens for datasets in Figure 6 of the manuscript (page 27). Datasets within the same class, for instance, FaceDetection and SelfRegulationSCP2, which both consist of EEG data, demonstrate a higher similarity. While some out-of-domain datasets still exhibit strong similarities, indicating that they share certain similar requirements.
# Response to Reviewer h6vS Part II (Part II of IV)

### Major concern 2.1: Lack of ablation experiments of pre-training.

Thank you for your great suggestion\! Following your suggestion, we conduct new experiments of pre-training UniTS by varying the size of the pre-training dataset and the amount of training epochs. 

- **Pre-training with different numbers of epochs**: As demonstrated in the table below, increasing the number of pre-training epochs improves performance on both forecasting and classification tasks.

  *New experiments of UniTS under different pre-training epochs, average performance on 20 forecasting and 18 classification are reported.*
   | Pre-training steps | 1 epoch | 3 epochs | 5 epochs | 8 epochs | 10 epochs |
   | :---- | :---- | :---- | :---- | :---- | :---- |
   | Acc↑  (Cls.) | 75.1 | 76.8 | 78.2 | 77.0 | 79.0 |
   | MSE↓ (Fore.) | 0.493 | 0.479 | 0.484 | 0.473 | 0.460 |
   | MAE↓ (Fore.) | 0.410 | 0.391 | 0.389 | 0.386 | 0.383 |

- **Pre-training with different data sizes**: Similarly, increasing the size of pre-training dataset improves performance on both forecasting and classification tasks, as shown in the table below.

   *New experiments of UniTS under different pre-training epochs, average performance on 20 forecasting and 18 classification are reported.*
  | Pre-training data size | 10% of the total training set | 30% | 50% | 80% | 100% |
  | :---- | :---- | :---- | :---- | :---- | :---- |
  | Acc↑  (Cls.) | 74.2 | 76.3 | 77.6 | 78.8 | 79.0 |
  | MSE↓ (Fore.) | 0.502 | 0.462 | 0.483 | 0.465 | 0.460 |
  | MAE↓ (Fore.) | 0.417 | 0.385 | 0.391 | 0.384 | 0.383 |

### Major concern 2.2: More systematic evaluation of the interrelationship between pre-training tasks. 

We have added three new experiments to systematically evaluate interrelationship and interdependencies between pre-training tasks:

* **Results on scaling pre-training data sizes and training steps**: We examine the impact of considering different sizes of pre-training datasets and the amount of model training in our response to the major concern 2.1.  
* **Results on cross-task pre-training**: We evaluate the effect of cross-task pre-training by pre-training a model using our pre-training strategy on either generative tasks (forecasting) or predictive tasks (classification). The table below shows that UniTS, pre-trained solely on forecasting datasets, achieves similar performance to the model pre-trained on both forecasting and classification data. Despite not encountering any classification datasets during pre-training, it still performs well on classification tasks. When the model is pre-trained exclusively on classification datasets, performance on both classification and forecasting tasks drops significantly compared to the model pre-trained on both types of data. Given that the data amount of forecasting datasets is larger than classification datasets (22920 vs. 5022 iterations per epoch), this suggests that the larger amount of data plays a more crucial role in pre-training effectiveness than the data type.

  *New experiment on cross-task pre-training evaluation on UniTS, average performance on 20 forecasting and 18 classification tasks are reported.*
  | Pre-training data type | Acc↑ (Cls.) | MSE↓ (Fore.) | MAE↓ (Fore.) |
  | :---- | :---- | :---- | :---- |
  | 20 forecasting datasets | 78.5 | 0.454 | 0.379 |
  | 18 classification datasets | 74.1 | 0.583 | 0.807 |
  | Full 38 datasets | 79.0 | 0.460 | 0.383 |

* **Results on cross-domain pre-training:** We evaluate the effect of cross-domain data pre-training, where the model is pre-trained on either Weather-domain datasets or Traffic-domain datasets. In the table below, compared to joint pre-training on both domains, the performance decreases with single-domain pre-training, where pre-training is conducted solely on the downstream dataset's domain, showing the advantage of joint pretraining. For instance, the MSE on Weather datasets goes from 0.253 to 0.259. Compared to single-domain pre-training, cross-domain pre-training leads to larger performance drops, e.g., pre-training on Traffic datasets and then evaluating on Weather datasets results in an MSE increase from 0.259 to 0.289. Interestingly, pre-training on Weather datasets achieves better performance across both domains, suggesting that data from certain domains might be more beneficial for pre-training.

  *New experiment on cross-domain pre-training evaluation on UniTS, average performance on 4 Weather or Traffic dataset domains are reported.*
  | Evaluation data➡️ | Weather datasets (4 sets) | Traffic datasets (4 sets) |
  | :---- | :---- | :---- |
  | Pre-training data⬇️ | MSE↓/MAE↓ (Fore.) | MSE↓/MAE↓ (Fore.) |
  | Weather domain (4 datasets) | 0.259/0.287 | 1.338/0.768 |
  | Traffic domain (4 datasets) | 0.289/0.314 | 0.680/0.438 |
  | Weather \+ Traffic domains (8 sets) | 0.253/0.282 | 0.511/0.320 |
# Response to Reviewer YrSc Part III (Part III of III)

###  W4: Some datasets in few-shot learning for imputation are generated from the same datasets used for pre-training, thus these settings can not be viewed as few-shot learning.

Thank you for pointing this out. We have revised the term "few-shot learning for imputation" to "downstream imputation task with few fine-tuning examples." The reason for using “few-shot learning” in the initial version is that 1\)  The imputation task is not seen during pretraining, making it a new task that requires a few fine-tuning examples. 2\) Following established practice in recent studies, e.g., \[ICLR2023times\], we use four datasets for imputation, in which the ETTm1 dataset is not used during pre-training.

###  Q1: Clarification of Equation 5. 

We have clarified Equation 5 in the revision for better understanding.

* Role of task token $z_m$ (GEN token): Tokens of a full input time series sample $x$ are randomly masked, i.e., replaced with GEN tokens $z_m$, resulting in the sample token $z_x$ as shown in Equation 5\.  
* For the right part of the loss, instead of using information from the prompt token to assist with mask reconstruction, we use the feature $H_{CLS}(z_{Pred})$ to enforce the mask reconstruction. This approach trains the CLS token and $H_{CLS}$ head to learn semantic information related to classification. As demonstrated in the table below (with full results in Table 17), removing the right part of the loss results in a significant drop in classification performance (from 78.0% to 33.1%), indicating that this part of the loss is crucial for maintaining discriminative power in classification.

| UniTS | Unified pretraining | Remove the right part of the loss | Remove the left part of the loss |
| :---- | :---- | :---- | :---- |
| Acc ↑ | 78.0 |  33.1 | 76.8 |

###  Q2: How to set $𝜆_𝑖$?

In this work, we set $𝜆_𝑖$ to 1 for simplicity. In future research, we will explore more effective methods for determining $𝜆_𝑖$, as the contribution of different datasets may vary.

###  Q3: Compare UniTS trained by multi-task learning with that trained by single-task learning.

We conduct new experiments to show that multi-task learning with UniTS outperforms single-task learning under the same hyperparameters. Following established protocols \[ICLR2024itrans, ICLR2023patch, ICLR2023times\], results for single-task models shown in Table 25 are obtained by training on each dataset with separately tuned hyperparameters, meaning that hyperparameters are optimally tuned for each dataset. To ensure fair comparisons, we train the single-task models using the same hyperparameters as the multi-task co-training. As shown in the table below, multi-task learning achieves stronger performance on both forecasting and classification tasks. Interestingly, under the same hyperparameters, some classification models fail to converge in the single-task setting, whereas the multi-task model does not have this issue, demonstrating the robustness of multi-task training.

| UniTS | Acc↑ (Classification) | MSE↓ (Forecasting) |
| :---- | :---- | :---- |
| **Multi-task** (same hyperparameters) | 81.6% | 0.439 |
| **Single-task** (same hyperparameters) | 65.3% | 0.464 |
# Response to Reviewer YrSc Part II (Part II of III)

###  W3: The performance gain of the multi-datasets multi-task pre-training model over single-task single-dataset supervised training models. A DLinear for each dataset and task is efficient. Whether joint training across multiple domains of data can provide real benefits?

Thank you for your great question\! The advantages of multi-task learning in UniTS over single-task learning include: 1\) enhanced performance without the need for task-specific hyperparameter tuning, and 2\) improved generalization capabilities, such as few-shot and zero-shot learning. The detailed evidence is presented in the following three parts:
* **For UniTS, multi-task learning outperforms single-task learning under the same hyperparameters:** Following common practice\[ICLR2024itrans, ICLR2023patch, ICLR2023times\], the single-task results shown in Table 25 are obtained by training on each dataset with separately tuned hyperparameters. To ensure fair comparisons, we conduct a new experiment to train the single-task models using the same hyperparameters as the multi-task co-training. As shown in the table below, multi-task learning achieves stronger performance on both forecasting and classification tasks. Interestingly, under the same hyperparameters, some classification models fail to converge in the single-task setting, whereas the multi-task model does not have this issue, demonstrating the robustness of multi-task training.
| UniTS | Acc↑ (Classification) | MSE↓ (Forecasting) |
| :---- | :---- | :---- |
| **Multi-task** (same hyperparameters) | 81.6% | 0.439 |
| **Single-task** (same hyperparameters) | 65.3% | 0.464 |
* **Multi-task UniTS outperforms hyperparameters-tuned single-task DLinear:** We compare multi-task performance of UniTS with single-task performance of DLinear, as summarized in the table below (full results in Tables 1 and 2). Despite DLinear being able to optimize and select dataset-specific hyperparameters for each dataset, UniTS, co-trained in a multi-task setting, consistently outperforms DLinear. Specifically, across 32 experiments (rows in the table below) across 16 datasets, we find that UniTS outperforms DLinear in 27 out of 32 experiments, showing that joint training across data domains is an effective approach.
| Dataset | UniTS (Multi-task; MSE/MAE) | DLinear (Single-task; MSE/MAE) |
| :---- | :---- | :---- |
| ECL_P96 | **0.157/0.258** | 0.197/0.282 |
| ECL_P192 | **0.173/0.272** | 0.196/0.285 |
| ECL_P336 | **0.185/0.284** | 0.209/0.301 |
| ECL_P720 | **0.219/0.314** | 0.245/0.333  |
| Traffic_P96 | **0.465/0.298** | 0.650/0.396  |
| Traffic_P192 | **0.484/0.306** | 0.598/0.370  |
| Traffic_P336 | **0.494/0.312** | 0.605/0.373 |
| Traffic_P720 | **0.534/0.335** | 0.645/0.394 |
| ETTH1_P96 | 0.390/0.411 | **0.386/0.400** |
| ETTH1_P192 | **0.432**/0.438 | 0.437/**0.432** |
| ETTH1_P336 | **0.480**0.460 | 0.481/**0.459** |
| ETTH1_P720 | 0.542/**0.508** | **0.519**/0.516  |
| Weather_P96 | **0.157/0.206** | 0.196/0.255  |
| Weather_P192 | **0.208/0.251** | 0.237/0.296 |
| Weather_P336 | **0.264/0.291** | 0.283/0.335 |
| Weather_P720 | **0.344/0.344** | 0.345/0.381  |
| Multi-task training regime | **Yes** | No |
| Best Count | **27/32** | 5/32 |
* **Multi-task UniTS introduces new abilities beyond the single-task model**: Using a single-task model is insufficient to handle tasks that require strong generalization ability, such as few-shot learning and zero-shot learning.   
  - For few-shot learning, the table below shows UniTS clearly outperforms the best-performing single-task model iTransformer, on forecasting, classification, imputation, and anomaly detection tasks. (Full comparisons with other methods are shown in Tables 3,4,5,23,24.)
| Method/Best count | Forecast  (9 sets) | Classification  (6 sets) | Imputation  (6 sets) | Anomaly detection  (5 sets) |
| :---- | :---- | :---- | :---- | :---- |
| iTrans | 0/9 | 1/6 | 0/6 | 0/5 |
| UniTS | **9/9** | **5/6** | **6/6** | **4/5** |
   - For zero-shot learning, the table below shows UniTS considerably surpasses LLMTime, a model designed for zero-shot forecasting using LLM, across most of the tested datasets, demonstrating superior performance in handling different forecasting lengths and variable numbers.  For example, UniTS achieves a considerable improvement in MSE over LLMTime (0.030 vs.0.265) on Solar. Remarkably, UniTS exhibits an inference speed approximately 10e6 times faster than LLMTime. We also show in Figure 3 of Figure PDF (Figure 3 of the manuscript, page 8\) that UniTS can generalize to new forecasting lengths not seen during training. 
| Solar dataset | MSE | Infer. Time (seconds) |  | 5 datasets (Full results in Table 21) | Best count | Var. number | Pred length |
| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |
| LLM-Time | 0.265 | 2.0e3 | - | LLM-Time | 1/5 | 1 to 767 | 16 to 128 |
| UniTS | **0.030** | **6.8e−3** | - | UniTS | **4/5** | 1 to 767 | 16 to 128 |
# Response to Reviewer EsEH Part II (Part II of II)

###  Q5: Incorrect bold and underline in Table 2.

Thank you for pointing this out. For fair comparisons, we mark GPT4TS with gray and exclude it from model ranking for the following reasons: 1\) GPT4TS is based on the GPT-2, which is pre-trained with large-scale extra-textual data  (such as the internet-scale Pile dataset). In contrast, UniTS is not trained on additional datasets beyond those described in our papers. 2\) GPT4TS is 48 times larger than UniTS (164.5M vs. 3.4M parameters) in terms of model scale. 

Despite the significant differences in data volume and model scale, UniTS still performs favorably compared to GPT4TS. On forecasting tasks, UniTS-SUP even outperforms GPT4TS by 2.2% (0.439 vs. 0.449 MSE).
# Response to Reviewer bSMF Part II (Part II of II)

### Q1.1: Are experiment settings of Sec.5.1 and Sec.5.2 the same?

The settings in Sec. 5.1 represents a single-task setting, following existing works (e.g., \[CLR2024itrans, ICLR2023patch, ICLR2023times\]), where the model is trained separately on each dataset with unique training settings for each dataset. The settings in Sec. 5.2 represents a multi-task learning setting, where one model is co-trained on all datasets.

### Q1.2: Does GPT4TS also use the same fully supervised multi-task training as UniTS?

For the single-task setting, the results of GPT4TS are obtained using the training setup outlined in the GPT4TS paper. In the multi-task learning setting, since GPT4TS doesn’t inherently support multi-task learning, we added data and task-specific heads to GPT4TS and employed the same fully supervised multi-task training approach as UniTS. We have updated the implementation details in the appendix to include this. Compared to GPT4TS, UniTS offers the following advantages: 1\) Unlike GPT4TS, which is built on the GPT-2 model trained with large-scale data (such as the internet-scale Pile dataset), UniTS is not trained on additional datasets beyond those described in our papers. 2\) GPT4TS is 48 times larger than UniTS (164.5M vs. 3.4M parameters) in terms of model scale. Despite the significant differences in data volume and model scale, UniTS still performs favorably compared to GPT4TS. On forecasting tasks, UniTS-SUP even outperforms GPT4TS by 2.2% (0.439 vs. 0.449 MSE).