NuwaTS: a Foundation Model Mending Every Incomplete Time Series

## Abstract

Abstract Time series imputation is critical for many real-world applications and has been widely studied. However, existing models often require specialized designs tailored to specific missing patterns, variables, or domains which limits their generalizability. In addition, current evaluation frameworks primarily focus on domain-specific tasks and often rely on time-wise train/validation/test data splits, which fail to rigorously assess a model’s ability to generalize across unseen variables or domains. In this paper, we present NuwaTS , a novel framework that repurposes Pre-trained Language Models (PLMs) for general time series imputation. Once trained, NuwaTS can be applied to impute missing data across any domain. We introduce specialized embeddings for each sub-series patch, capturing information about the patch, its missing data patterns, and its statistical characteristics. By combining contrastive learning with the imputation task, we train PLMs to create a versatile, one-for-all imputation model. Additionally, we employ a plug-and-play fine-tuning approach, enabling efficient adaptation to domain-specific tasks with minimal adjustments. To evaluate cross-variable and cross-domain generalization, we propose a new benchmarking protocol that partitions the datasets along the variable dimension. Experimental results on over seventeen million time series samples from diverse domains demonstrate that NuwaTS outperforms state-of-the-art domain-specific models across various datasets under the proposed benchmarking protocol. Furthermore, we show that NuwaTS generalizes to other time series tasks, such as forecasting. Our codes are available at https://github.com/Chengyui/NuwaTS .

## 1 Introduction

Time series data are pervasive across numerous fields, including transportation (Li et al., ), healthcare (Tonekaboni et al., ), education (Mao et al., ), and meteorology (Bi et al., ). However, real-world time series data often suffer from missing values. Incomplete data complicate various time series applications such as forecasting and classification, ultimately degrading the quality of data analysis (Ma et al., ). This makes time series imputation especially critical (Wang et al., ). Traditionally, time series imputation methods have leaned heavily on statistical techniques like mean imputation and interpolation (Van Buuren & Groothuis-Oudshoorn, ). Yet, with the rise of deep learning, there has been a notable shift towards deep learning models for imputation tasks (Cao et al., ; Du et al., ).

In traditional deep learning-based imputation models, the typical protocol involves training a model on relatively complete or incomplete time series data within a closed domain from historical records. During the testing phase, the model is validated or tested on newly observed incomplete data, which are future observations of the variables from the training set (Cao et al., ; Wu et al., ) (Figure 1(b)). However, this approach presents a key limitation: the lack of cross-variable generalization capability. Models trained in this manner may struggle to extend to variables not observed in the training set. For instance, in one particular factory, a newly launched production line generates several new time series data, and it’s uncertain whether the model trained on existing production lines can generalize effectively to those new ones. A more challenging scenario arises when we lack training data from a specific domain and must rely on a model trained from other domains for imputation. This introduces a more complex cross-domain generalization problem (Jin et al., ; Wilson et al., ). For example, generalizing from traffic time series to factory time series presents significant challenges in transferring learned knowledge across fundamentally different domains (Figure 1(a)).

Figure: Figure 1: (a) Cross-variable and cross-domain generalization: time series data across different variables and domains may exhibit both shared and distinct patterns. (b) The conventional train/validation/test division protocol of partitioning datasets along the time dimension. (c) Variable-wise division: the proposed approach trains, validates, and tests models on distinct sets of variables, ensuring the model’s ability to generalize across unseen variables during deployment.
Refer to caption: x1.png

In this paper, we propose a novel framework and benchmarking methodology specifically designed to address and assess time series imputation methods that are capable of both cross-variable and cross-domain generalization. We draw inspiration from the success of foundation models trained on diverse datasets spanning multiple domains (Kirillov et al., ; Brown et al., ; Peng et al., ). These models have demonstrated remarkable ability to generalize across tasks and domains using techniques like fine-tuning (He et al., ; Zhuang et al., ) or prompting (Jia et al., ; Kirillov et al., ). In a similar vein, we propose a method that pushes time series imputation beyond the traditional single-domain paradigm, enabling generalization both across variables and across domains.

Specifically, we introduce NuwaTS, a foundation model designed for incomplete time series imputation. Our model segments the time series into patches and employs novel token designs to capture both statistical information and missing data patterns. Additionally, we leverage contrastive learning combined with missing data reconstruction to train an one-for-all imputation model on time series data from diverse domains. Finally, for cases requiring domain-specific adaptation, we have developed a domain-specific prefix embedding and a plug-and-play fine-tuning mechanism. This mechanism introduces continuous prompts at each layer of the frozen pre-trained foundation model without modifying its weights, enabling efficient domain specialization.

Furthermore, we introduce a novel benchmarking paradigm for time series imputation models. Rather than the conventional time-wise train/validation/test partitioning, we partition multivariate time series data along the variable dimension, allocating different variables to the training, validation, and test sets (Figure 1(c)). This approach closely simulates real-world deployment scenarios, where models trained on one domain are required to impute missing data for entirely new variables and domains, effectively assessing the model’s cross-variable and cross-domain generalization capabilities.

Our contributions are summarized as follows:

- 1.
We propose a novel and more practically relevant benchmark which divides the multivariate time series data along the variable dimension for time series imputation, which better assesses a model’s ability to generalize to new data.
- 2.
We introduce NuwaTS, designed to handle missing data imputation tasks for any incomplete time series. NuwaTS is trained on data from diverse domains and incorporates a light-weight “plug-and-play” fine-tuning technique that requires minimal data and computational resources, making it capable of mending every incomplete time series.
- 3.
Under the proposed benchmarking protocol, the one-for-all NuwaTS consistently outperforms domain-specific state-of-the-art methods in imputation tasks across nearly all missing rates. Moreover, fine-tuned NuwaTS can be extended to time series forecasting, where its forecasting results are comparable to or even better than existing domain-specific time series forecasting models.

## 2 Related Works

### 2.1 Incomplete Time Series Imputation

Many time series imputation models are tailored to specific missing data patterns and domains, such as randomly missing traffic time series. For instance, matrix factorization models are designed for multivariate time series imputation (Yu et al., ). For a dataset from a specific domain with a particular missing rate, it often requires optimizing a specialized model using a low-rank prior as the optimization target for imputation. Recent advancements in deep learning techniques have shown promising results in addressing missing data imputation. Generative models such as Generative Adversarial Networks (GANs) (Luo et al., ; Tashiro et al., ) and diffusion models are used to learn the underlying distribution of the incomplete time series. Several architectures, such as Recurrent Neural Networks (RNNs) (Cao et al., ; Ma et al., ; Liu et al., ; Tang et al., ) and attention mechanisms (Du et al., ), are proposed to capture temporal dependencies within incomplete time series. While achieving good performance on narrow domains and missing data patterns, these models lack versatility and generalizability to diverse domains and missing data patterns.

### 2.2 Foundation Model for Time Series

In recent years, foundational models for time series analysis have made notable strides, primarily leveraging pre-trained backbones from NLP and aligning modalities to extend their reasoning capabilities to time series tasks. For example, Gruver et al. (Gruver et al., ) discovered that by encoding time series as strings of numerical digits, LLMs can perform time series forecasting with zero-shot capabilities.
GPT4TS (Zhou et al., ) trains time series models using pre-trained GPT weights. UniTime (Liu et al., ) integrates domain-specific instructions into LLMs, enhancing their generalization across domains. Time-LLM (Jin et al., ) employs LLMs’ reasoning by framing time series data and statistical information in textual prompts, aligning time patches with word embeddings via cross-attention mechanisms for improved zero-shot learning. Autotimes (Liu et al., ) utilizes precomputed text embedding as positional embeddings and an autoregressive approach for long-term forecasting. TEST (Sun et al., ) uses text-prototype-aligned embeddings to enhance LLMs’ reasoning in time series data. $S^{2}$IP-LLM (Pan et al., ) applies seasonal-trend decomposition to time series and uses semantic space-informed prompting to retrieve appropriate prompts from word token embeddings as prefixes. aLLM4TS (Bian et al., ) adapts LLMs for time series representation learning by directly converting individual patches into time series sequences. Chronos (Ansari et al., ) trains language models from scratch on a large collection of time series data, using scaling and quantization for tokenization. This model demonstrates strong capabilities in zero-shot probabilistic forecasting. These models primarily focus on time series forecasting and do not specifically address missing data issues or the design of embeddings for missing data patterns.

## 3 Methodology

### 3.1 Preliminaries and Problem Statement

###### Definition 3.1 .

Incomplete Time Series: We define an incomplete time series $\mathbf{x}=\{x_{1},x_{2},\ldots,x_{T}\}\in\mathbb{R}^{T}$ from domain $\mathcal{S}$ as a sequence of $T$ observations. Each observation $x_{t}$ is associated with a timestamp $s_{t}$. In practice, an observation $x_{t}$ may not be observable due to various reasons. To represent the missing values in $\mathbf{x}$, we introduce a masking vector $\mathbf{m}\in\mathbb{R}^{T}$ defined by:

$$ $m_{t}=\begin{cases}1,&\text{if }x_{t}\text{ is observed}\\ 0,&\text{otherwise}\end{cases}$ (1) $$

Suppose we have acquired $N$ time series datasets $\mathcal{D}=\{\mathbf{x}^{1},\mathbf{x}^{2},\ldots,\mathbf{x}^{N}\}$ from a diverse set of domains $\{\mathcal{S}^{1},\mathcal{S}^{2},\ldots,\mathcal{S}^{K}\}$, where $\mathbf{x}^{n}\in\mathbb{R}^{T_{n}}$ represents the $n^{th}$ time series with length $T_{n}$. Our objective is to utilize $\mathcal{D}$ to train an imputation model, denoted as $f_{\mathbf{\Phi}}$, characterized by parameters $\mathbf{\Phi}$. For any incomplete time series $\hat{\mathbf{x}}\in\mathbb{R}^{\hat{T}}$ accompanied by any missing pattern $\hat{\mathbf{m}}\in\mathbb{R}^{\hat{T}}$ from any domain $\hat{\mathcal{S}}$, the model $f_{\mathbf{\Phi}}$ can impute the missing values in $\hat{\mathbf{x}}$ as accurately as possible.

###### Definition 3.1 .

### 3.2 Model Architecture

In this work, we initialize model parameters $\mathbf{\Phi}$ with weights from PLMs. Given the constraints of computational resources, we limit our use to the parameters of the first six layers, thereby making NuwaTS more accessible to applications with limited computational resources.

Figure: Figure 2: Overview of NuwaTS. To fully leverage the semantic information of time series and their missing patterns, NuwaTS introduces the tokenization of time series in patches. It utilizes the missing data patterns, statistical information for each pattern and the entire series, and a domain-specific embedding, trained through imputation and contrastive learning tasks.
Refer to caption: x2.png

To train PLMs to adapt to various domains and missing data patterns, we have implemented several key design features for training. These include Instance Normalization & Patching, Statistical Embedding, Missing Embedding, a Domain-Specific Embedding for fine-tuning and Contrastive Learning with Variable Missing Patterns. We visualize the model architecture in Figure 2.

Instance Normalization & Patching
Time series from different domains can vary in magnitude and distribution. To address these differences, we apply reversible instance normalization (Kim et al., ) to each variable before feeding it into the model, with missing values set to zero. To enhance the model’s ability to recognize domain information, we use a patching technique, dividing each time series segment into non-overlapping patches. These patches are then embedded into a hidden space using a shared, learnable linear projection denoted by $\mathbf{Z}_{i,(p)}\in\mathbb{R}^{D\times N}$, enabling more effective modeling of the time series data across domains.

Statistical Embedding.
Previous work primarily used hard textual prompts to translate dataset descriptions and statistical information (Liu et al., ; ; Jin et al., ), which proved challenging due to the mismatch between time series patches and textual prompts. In this work, we move away from this approach and instead generate statistical information, such as minimum, median, maximum values, and trends, for both the entire variable (denoted by $\mathbf{z}_{i,(v_{g})}\in\mathbb{R}^{D}$) and individual patches (denoted by $\mathbf{Z}_{i,(v_{p})}\in\mathbb{R}^{D\times N}$). This information is embedded into a hidden space using a shared, learnable linear projection, allowing for better alignment with the time series data.

Missing Embedding.
Adapting to the missing data pattern is crucial. We design a Missing Embedding, a learnable parameter that captures the missing rate of each patch. This embedding is multiplied with the corresponding patch’s mask ratio and added to the corresponding embed patches: $\mathbf{E}_{i,(p)}=\mathbf{Z}_{i,(p)}+\mathbf{Z}_{i,(v_{p})}+\mathbf{z}_{i,(m)
}\times\mathbf{r}_{i}$, allowing the model to better account for missing data across the target time series.

Domain-Specific Embedding.
In cases where the model needs to function within a specific domain while preserving cross-domain generalization, we introduce a domain-specific embedding, $\mathbf{k}\in\mathbb{R}^{D}$. This embedding learns to capture domain-specific knowledge during training and is inserted before the patch embeddings. This embedding is beneficial for the proposed fine-tuning process, as discussed in Section 3.3.

The final input embeddings are expressed as $\mathbf{E}_{i}=[\mathbf{k},\mathbf{z}_{i,(v_{g})},\mathbf{E}_{i,(p)}]$, integrating both domain-specific and global statistical information. This design could help model generalize well on new in-domain data or even out-of-domain data.

Contrastive learning.
To improve the model’s adaptability to different missing patterns, we incorporate a Contrastive Learning module into the training process. For each input $\mathbf{x}_{i}$, we generate two random masks with different mask ratios and input the masked time series into the PLM. The module ensures the model learns similar representations for the same patch under different masks, treating them as positive samples, while treating representations from other patches and series as negative samples. We use the InfoNCE (Oord et al., ) loss combined with MSE loss to optimize the model (details in Appendix B.3).

Output layer.
After passing through the PLM backbone, we discard the domain-specific and variable-wise statistical embeddings, retaining only the $N$ patch representations as inputs for the final layer. While the prefixes contribute to causal attention calculations, they are excluded from the final output. The remaining representations are flattened and linearly mapped back to their original dimensions, producing the model outputs $\mathbf{o}_{i}\in\mathbb{R}^{L}$.

### 3.3 Domain-Specific Fine-Tuning

To achieve a domain-specific model, we borrow the idea of P-tuningv2 strategy (Liu et al., ). Unlike pre-training phase, where the domain-specific embedding $\mathbf{k}$ is only added to the input embedding.

We incorporate this domain-specific information into every layer of the PLM. Initially, we employ a domain-transfer layer—a two-layer MLP network—to map $\mathbf{k}\in\mathbb{R}^{D}$ to $\bm{\mathcal{\hat{K}}}\in\mathbb{R}^{2\times\text{Layer}\times D}$.

Figure: Figure 3: Illustration of domain-specific fine-tuning.
Refer to caption: x3.png

Following the P-tuningv2 approach, we also randomly initialize a continuous prompt $\bm{\mathcal{P}}\in\mathbb{R}^{2\times\text{Layer}\times D}$ for each layer.
We then combine the randomly initialized $\bm{\mathcal{P}}$ with the domain-specific information $\bm{\mathcal{\hat{K}}}$ to serve as the domain-specific prefix Key and Value in the PLM’s hidden layers. Thus, we have $[\mathbf{Key}_{p},\mathbf{Value}_{p}]=\bm{\mathcal{P}}+\beta\bm{\mathcal{\hat{
K}}}$, where $\beta=0.01$. Here, $\mathbf{Key}_{p}$ and $\mathbf{Value}_{p}\in\mathbb{R}^{\text{Layer}\times D}$ contain the domain-specific prefix key and value for every layer of the pre-trained model, thereby enhancing its representational capacity.

During fine-tuning on time series data from the target domain, we freeze all parameters except for the randomly initialized $\bm{\mathcal{P}}$ and the domain-transfer layer. The fine-tuning process is illustrated in Figure 3.
If the domain-specific prefix is removed, the domain-specific model reverts to the original one-for-all foundation model. Therefore, the fine-tuning strategy we propose is essentially “plug-and-play”.
Additionally, the domain-specific prefix is lightweight. For example, when using GPT-2 as the PLM backbone, the prefix requires less than 100KB of storage, compared to the 331.77MB required for the entire model. This makes our approach highly practical for real-world deployment, particularly in edge computing environments, as a single NuwaTS model can be trained on large-scale time series data. When deploying domain-specific models locally, only the corresponding prefix needs to be stored, significantly reducing storage requirements and enabling efficient deployment with minimal computational resources.

Additionally, we can incorporate inter-series correlations into the prefix. For the subsequent forecasting tasks in section 4.7, we design an inter-variable fine-tuning network to generate the layer prefix, which contains inter-series correlation information. Specific implementation details can be found in Algorithm 3 and Section B.4.

## 4 Experiment

We conducted comprehensive experiments on NuwaTS across ten commonly used datasets from various domains. To facilitate comparison, we provided four versions of the model based on the GPT2 architecture in Figure 4: (a) a specific model trained on a single domain, (b) an one-for-all model trained on a general fused dataset (17.6M collected samples from various domains), (c) a fine-tuned model for a specific domain (fine-tuned based on the one-for-all model) and (d) a cross-domain model pretrained only on LargeST (Liu et al., ) (comprising 100.1 million collected samples). Cross-domain model will be evaluated on time series from other domains to verify the domain-transfer zero-shot capability in Section 4.3. Moreover, we evaluated BERT (Devlin et al., ) and LLaMA2 (Touvron et al., ) as backbones for NuwaTS (details in Appendix D.5).

Figure: Figure 4: The four different versions of NuwaTS trained in this study.
Refer to caption: x4.png

Baselines.
Following the configuration outlined in (Du et al., ), we evaluated two naive imputation methods: Median, where missing values are filled with the median value, and Last, where missing values are filled with the last previous observations. To ensure a fair comparison, all methods followed a unified pipeline(^1^11[https://github.com/thuml/Time-Series-Library](https://github.com/thuml/Time-Series-Library)). We assessed two classic deep learning-based imputation models, BRITS(Cao et al., ) and SAITS(Du et al., ).
We also compared our model with other foundation models for time series such as DLinear(Zeng et al., ), PatchTST(Nie et al., ), iTransformer(Liu et al., ), TimesNet(Wu et al., ), Autoformer(Wu et al., ), Fedformer(Zhou et al., ), and GPT4TS(Zhou et al., ). We trained GPT4TS separately on single datasets with the number of layers set to 6 to align with NuwaTS. We trained all these models using a domain-specific approach. Since PatchTST is a transformer-based model with bi-directional attention, we also trained and tested it on the general fused dataset.

Setups. We partitioned the dataset along the sensor (variable) dimension into training, validation, and test sets in a 1:1:1 ratio. This division simulates the process of collecting relatively complete time series data from a few sensors or variables with lower missing rates, training a model with this data, and then using it to impute data from other sensors or variables that have higher missing rates. The input time series length $L$ is 96, and we trained all baselines under random missing rates (sampled from 0.1 to 0.9) and tested them separately under 9 missing rates: 0.1, 0.2,…, 0.9.
We used a total of ten datasets
for domain-specific training and mixed them into a general fused dataset for one-for-all training. Appendix A.1 shows the details of datasets. For the cross-domain model, we chose LargeST(^2^22[https://github.com/liuxu77/LargeST](https://github.com/liuxu77/LargeST)), a large-scale traffic dataset including a total of 8,600 time series and over 100 million samples, to train NuwaTS and conduct zero-shot experiments on other domains such as ETTs, weather and electricity.

**Table 1: Imputation results are presented for ETTs, ECL, and weather datasets with nine test missing rates: 0.1, 0.2, …, 0.9. All results are averaged across the nine different missing rates. A lower MSE or MAE signifies better imputation performance. Green highlights the best results, while Yellow indicates the second-best.**
| Model | ETTh1 | ETTh2 | ETTm1 | ETTm2 | ECL | Weather |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |  |
| Median | 0.723 | 0.611 | 0.728 | 0.472 | 0.699 | 0.584 | 0.744 | 0.464 | 1.010 | 0.834 | 0.998 | 0.497 |
| Last | 0.432 | 0.476 | 0.089 | 0.149 | 0.336 | 0.413 | 0.059 | 0.122 | 0.965 | 0.826 | 0.731 | 0.349 |
| Autoformer(2021) | 0.552 | 0.547 | 0.472 | 0.420 | 0.346 | 0.409 | 0.211 | 0.312 | 0.137 | 0.262 | 0.610 | 0.450 |
| Fedformer(2022) | 0.354 | 0.436 | 0.538 | 0.429 | 0.076 | 0.185 | 0.055 | 0.156 | 0.129 | 0.258 | 0.237 | 0.211 |
| Dlinear(2023) | 0.356 | 0.414 | 0.245 | 0.301 | 0.274 | 0.357 | 0.266 | 0.351 | 0.317 | 0.419 | 0.355 | 0.309 |
| iTransformer(2024) | 0.639 | 0.572 | 0.369 | 0.376 | 0.352 | 0.413 | 0.356 | 0.416 | 0.092 | 0.200 | 0.515 | 0.412 |
| BRITS(2018) | 0.213 | 0.313 | 0.117 | 0.172 | 0.096 | 0.183 | 0.071 | 0.123 | 0.317 | 0.427 | 0.793 | 0.474 |
| TimesNet(2022) | 0.166 | 0.280 | 0.021 | 0.091 | 0.066 | 0.155 | 0.011 | 0.063 | 0.362 | 0.450 | 0.681 | 0.280 |
| PatchTST(2023) | 0.185 | 0.298 | 0.022 | 0.093 | 0.080 | 0.183 | 0.011 | 0.066 | 0.136 | 0.266 | 0.271 | 0.122 |
| SAITS(2023)^1 | 0.196 | 0.289 | 0.215 | 0.190 | 0.087 | 0.163 | 0.135 | 0.139 | 0.445 | 0.512 | 0.932 | 0.484 |
| GPT4TS(2024) | 0.196 | 0.290 | 0.025 | 0.092 | 0.078 | 0.161 | 0.012 | 0.063 | 0.296 | 0.401 | 0.939 | 0.321 |
| NuwaTS(specific) | 0.182 | 0.293 | 0.020 | 0.091 | 0.070 | 0.164 | 0.011 | 0.064 | 0.086 | 0.191 | 0.307 | 0.127 |
| PatchTST(one-for-all) | 0.178 | 0.288 | 0.019 | 0.088 | 0.075 | 0.172 | 0.011 | 0.065 | 0.121 | 0.243 | 0.230 | 0.116 |
| NuwaTS(one-for-all) | 0.164 | 0.263 | 0.018 | 0.084 | 0.064 | 0.147 | 0.010 | 0.060 | 0.085 | 0.186 | 0.206 | 0.088 |
| NuwaTS(fine-tuned) | 0.156 | 0.255 | 0.017 | 0.082 | 0.060 | 0.142 | 0.010 | 0.058 | 0.081 | 0.183 | 0.207 | 0.086 |

Figure: Table 2: Imputation results on four PEMS datasets with the same setting as Table 1.
Refer to caption: extracted/5895493/fig/main_results.jpg

### 4.1 Main Results

We present the average MAE and MSE across different missing rates for specific, one-for-all, and fine-tuned NuwaTS models, along with other baselines, across 10 different datasets in Table 1 and Table 2, detailed results are shown in Appendix E. Furthermore, we visualize the average MSE across all datasets at various missing rates for the NuwaTS models and several baselines, as shown in Figure 5. On nearly all datasets, the NuwaTS model outperforms other domain-specific models. Moreover, we observed that the generalization ability of both NuwaTS and PatchTST (one-for-all) was further enhanced after training on a fused dataset spanning multiple domains, supporting the presence of a scaling law (Kaplan et al., ) in time series imputation tasks. We visualized the imputation results in Appendix C.1. We also conducted experiments to verify NuwaTS on real-world dataset in Appendix D.1.

### 4.2 Imputation Results under Continuous Missing

In Section 4.1, we conduct a comprehensive evaluation of the performance of different approaches under purely random missing data scenarios. In real-world scenarios, missing data often occurs in a continuous manner (Du et al., ).
Therefore, we also evaluated NuwaTS’s performance under conditions of continuous missing data. The experimental setup is set as the following: Assuming a missing rate of $r$, we randomly selected a continuous segment of the time series with a length of $L\times r$ for masking.
We evaluate the models trained on randomly missing data by directly testing them on datasets with continuous missing patterns.

**Table 3: Imputation results on continuous missing data. The final results are averaged across 9 missing rates: 0.1, 0.2… 0.9. A lower MSE or MAE indicates better imputation performance. Green : the best.**
| Model | ETTh1 | ETTh2 |  |  |
| --- | --- | --- | --- | --- |
| MSE | MAE | MSE | MAE |  |
| BRITS | 0.717 | 0.619 | 0.580 | 0.519 |
| SAITS | 0.519 | 0.469 | 0.318 | 0.250 |
| TimesNet | 0.497 | 0.494 | 0.110 | 0.154 |
| GPT4TS | 0.483 | 0.480 | 0.104 | 0.154 |
| NuwaTS | 0.465 | 0.455 | 0.095 | 0.141 |

As it is shown in Table 3, our findings reveal that NuwaTS exhibits strong domain adaptation capabilities and is highly robust in handling diverse missing patterns, further highlighting its generalizability across varying data missing patterns.

### 4.3 Domain-Transfer Zero-shot Capability

We validated the zero-shot capability of our model by directly evaluating a trained model on target data without further training. TimesNet and GPT4TS, which use a channel-dependent approach, can only perform zero-shot across datasets with the same number of variables, specifically the four ETT datasets in our study. In contrast, NuwaTS and PatchTST, as channel-independent approaches, support inference on datasets with different variable dimensions. We trained them on LargeST and then evaluated them on ECL and Weather datasets. The results in Table 4 shows NuwaTS exhibits superior zero-shot capability, achieving better performance on all cases. The reason may lie in the use of specialized missing data embedding and statistical embedding. Additionally, the language knowledge embedded in the PLM has strong generalization capabilities, whereas the PatchTST model trained on LargeST does not possess such strong capabilities.

**Table 4: Zero shot evaluation results. All methods were trained on one dataset and zero shot to the other. A lower MSE or MAE indicates better imputation performance. Green : the best. /: model failed to work.**
| Metrics | NuwaTS | PatchTST | GPT4TS | TimesNet |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |  |
| $ETTh1$ $\Rightarrow$ $ETTh2$ | 0.023 | 0.091 | 0.024 | 0.092 | 0.029 | 0.095 | 0.026 | 0.095 |
| $ETTh1$ $\Rightarrow$ $ETTm2$ | 0.013 | 0.072 | 0.013 | 0.073 | 0.015 | 0.073 | 0.014 | 0.072 |
| $ETTm1$ $\Rightarrow$ $ETTh2$ | 0.021 | 0.094 | 0.022 | 0.096 | 0.027 | 0.093 | 0.024 | 0.096 |
| $ETTm1$ $\Rightarrow$ $ETTm2$ | 0.011 | 0.063 | 0.011 | 0.067 | 0.013 | 0.064 | 0.013 | 0.066 |
| $LargeST$ $\Rightarrow$ $ECL$ | 0.338 | 0.366 | 0.385 | 0.433 | / | / | / | / |
| $LargeST$ $\Rightarrow$ $Weather$ | 0.217 | 0.087 | 0.236 | 0.121 | / | / | / | / |

### 4.4 Few-shot Domain-Specific Fine-Tuning

The ability to generalize to the target domain with a small amount of data is an important criterion for evaluating the generality of a model. We use $10\%$ and $1\%$ of the data for fine-tuning the cross-domain NuwaTS model which is trained on LargeST. The results shown in Table 5 indicate that domain-specific fine-tuning is particularly effective for fields with limited data. On the ETT datasets, we achieved the same results using only 10% of the data as we did using 100% of the data.

**Table 5: Fine-tuned results from cross-domain NuwaTS model pre-trained on LargeST (Liu et al., ). A lower MSE or MAE indicates better imputation performance. Green best, Yellow second best, and ${\downarrow}$ indicates fine-tuning effectiveness.**
| Methods | ECL | ETTh1 | ETTh2 | ETTm1 | ETTm2 |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |  |
| Zero-Shot | 0.338 | 0.366 | 0.212 | 0.306 | 0.021 | 0.089 | 0.066 | 0.145 | 0.011 | 0.059 |
| Fine-tuning with $1\%$ data | 0.284${\downarrow}$ | 0.338${\downarrow}$ | 0.205${\downarrow}$ | 0.305${\downarrow}$ | 0.021${\downarrow}$ | 0.090 | 0.063${\downarrow}$ | 0.147 | 0.010${\downarrow}$ | 0.059 |
| Fine-tuning with $10\%$ data | 0.190 ${\downarrow}$ | 0.292 ${\downarrow}$ | 0.180 ${\downarrow}$ | 0.282 ${\downarrow}$ | 0.019 ${\downarrow}$ | 0.086 ${\downarrow}$ | 0.060 ${\downarrow}$ | 0.142 ${\downarrow}$ | 0.010 ${\downarrow}$ | 0.058 ${\downarrow}$ |
| Fine-tuning with $100\%$ data | 0.143 ${\downarrow}$ | 0.253 ${\downarrow}$ | 0.180 ${\downarrow}$ | 0.280 ${\downarrow}$ | 0.019 ${\downarrow}$ | 0.085 ${\downarrow}$ | 0.061 ${\downarrow}$ | 0.143 ${\downarrow}$ | 0.010 ${\downarrow}$ | 0.058 ${\downarrow}$ |

### 4.5 Ablation Study

Table: Table 6: Ablation results of the model training on ETTh1 and LargeST and zero-shot on other data domain. A lower MSE or MAE indicates better imputation performance. Green : the best.

We conducted ablation experiments via zero-shot evaluation.
We trained the default model, several ablated models on ETTh1 with 34650 samples and LargeST with more than 100.1 million samples.
We then performed zero-shot evaluation on other datasets for out-of-domain evaluation.
As shown in Table 6, after ablation of the Statistic Embedding, Missing Embedding, and contrastive learning module, the model’s zero-shot ability decreases significantly, proving the necessity of these key components. Freezing the PLM backbone leads to poor performance. Finally, we discovered that when training the model from scratch without using weights pre-trained on NLP tasks, its zero-shot generalization ability on other datasets deteriorates significantly. This indicates that training the model on NLP tasks benefits its performance on time series tasks, demonstrating that cross-modality training is meaningful (Zhang et al., ).

When comparing the ablation results on ETTh1 and LargeST, where the former is a very small dataset and the latter is significantly larger. Our findings show that the specially designed tokens and contrastive learning modules provide more noticeable improvements on the smaller ETTh1 dataset. This suggests that for foundation models, data quantity might be more critical than the inductive biases introduced by specialized modules.

### 4.6 Enhancing Forecasting on Incomplete Time Series

**Table 7: We trained and evaluated TimesNet (Wu et al., ) on incomplete time series with original missing rates of 0.2 and 0.5, comparing its forecasting performance on data imputed by PatchTST and NuwaTS. We set the length of input sequence and forecasting both to 96. A lower MSE or MAE indicates better forecasting performance.**
| Methods | ETTh1 | ETTh2 | ETTm1 | ETTm2 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |  |
| Complete Data | 0.384 | 0.402 | 0.340 | 0.374 | 0.338 | 0.375 | 0.187 | 0.267 |
| Missing Rate: 0.2 |  |  |  |  |  |  |  |  |
| Default | 0.457 | 0.450 | 0.353 | 0.389 | 0.389 | 0.404 | 0.200 | 0.284 |
| +PatchTST | 0.439 | 0.445 | 0.346 | 0.382 | 0.353 | 0.388 | 0.189 | 0.270 |
| +NuwaTS | 0.428 | 0.437 | 0.341 | 0.378 | 0.344 | 0.381 | 0.190 | 0.269 |
| Missing Rate: 0.5 |  |  |  |  |  |  |  |  |
| Default | 0.585 | 0.515 | 0.370 | 0.403 | 0.572 | 0.496 | 0.238 | 0.318 |
| +PatchTST | 0.450 | 0.452 | 0.354 | 0.389 | 0.357 | 0.391 | 0.189 | 0.270 |
| +NuwaTS | 0.441 | 0.446 | 0.344 | 0.382 | 0.345 | 0.383 | 0.190 | 0.270 |

In practical applications, forecasting models often have to train on incomplete data, which can significantly impair their performance. To address this issue, we applied pre-trained imputation models to impute the incomplete data before training the forecasting model. We chose TimesNet (Wu et al., ) as the forecasting model and used four ETT datasets for training. We randomly discarded 20% and 50% of the data in the training sets of four ETT datasets.

To prevent data leakage during imputation, we used cross-domain NuwaTS and PatchTST pre-trained on LargeST to impute the training data. Finally, we trained TimesNet (Wu et al., ) on the imputed time series and tested it on the complete time series.

As shown in Table 7, the data imputed by NuwaTS improved the forecasting model’s performance across most metrics. Additionally, the imputed data from NuwaTS was of higher quality for downstream tasks compared to PatchTST, demonstrating that NuwaTS can effectively address the issue of incomplete time series data in practical applications.

### 4.7 Fine-tuning Imputation Model to Forecasting Model

**Table 8: Forecasting results. We set the length of input sequence and forecasting both to 96. Forecasting results from other baselines come from (Wu et al., ; Liu et al., ; Cai et al., ). We fine-tuned the cross-domain model in order to avoid data leakage. A lower MSE or MAE indicates a better performance. Green : the best, Yellow : the second best.**
| Model | ETTh1 | ETTh2 | ETTm1 | ETTm2 | ECL | Weather |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |  |
| Autoformer(2021) | 0.449 | 0.459 | 0.346 | 0.388 | 0.505 | 0.475 | 0.255 | 0.339 | 0.201 | 0.317 | 0.266 | 0.336 |
| Fedformer(2022) | 0.395 | 0.424 | 0.358 | 0.397 | 0.379 | 0.419 | 0.203 | 0.287 | 0.193 | 0.308 | 0.217 | 0.296 |
| TimesNet(2022) | 0.384 | 0.402 | 0.340 | 0.374 | 0.338 | 0.375 | 0.187 | 0.267 | 0.168 | 0.272 | 0.172 | 0.220 |
| Dlinear(2023) | 0.397 | 0.412 | 0.333 | 0.387 | 0.345 | 0.372 | 0.193 | 0.292 | 0.197 | 0.282 | 0.196 | 0.255 |
| PatchTST(2023) | 0.460 | 0.447 | 0.308 | 0.355 | 0.352 | 0.374 | 0.183 | 0.270 | 0.190 | 0.296 | 0.186 | 0.227 |
| MSGNet(2024) | 0.390 | 0.411 | 0.328 | 0.371 | 0.319 | 0.366 | 0.177 | 0.262 | 0.165 | 0.274 | 0.163 | 0.212 |
| iTransformer(2024) | 0.386 | 0.405 | 0.297 | 0.349 | 0.334 | 0.368 | 0.180 | 0.264 | 0.148 | 0.240 | 0.174 | 0.214 |
| NuwaTS(without inter-variable correlation) | 0.375 | 0.404 | 0.292 | 0.348 | 0.319 | 0.360 | 0.185 | 0.268 | 0.183 | 0.267 | 0.172 | 0.215 |
| NuwaTS(with inter-variable correlation) | 0.374 | 0.403 | 0.308 | 0.357 | 0.314 | 0.357 | 0.184 | 0.267 | 0.151 | 0.242 | 0.179 | 0.221 |

By directly inserting the masked padding tokens $\mathbf{p}$ where $\mathbf{p}\in\mathbb{R}^{M\times D}$ after the original input embeddings $\mathbf{E}_{i}\in\mathbb{R}^{(N+2)\times D}$, we get $\mathbf{E}_{i}^{f}=[\mathbf{k},\mathbf{z}_{i,(v_{g})},\mathbf{E}_{i,(p)},
\mathbf{p}]$ where $\mathbf{E}_{i}^{f}\in\mathbb{R}^{(N+2+M)\times D}$. The model automatically imputes the future $M$ patches, effectively transforming into a forecasting model.
We discard the prefixal part and obtain the final forecasting represenation $\mathbf{h}^{(\text{Layer})}_{i}\in\mathbb{R}^{M\times D}$. Then $\mathbf{h}^{(\text{Layer})}_{i}$ pass through the output layer and we get the final forecasting results.
In order to further help NuwaTS adapt to the forecasting task, we conducted two types of fine-tuning module based on the cross-domain model, the first one using two-layer MLP as the domain-transfer layer which accounts for only 9.35% of the model’s total parameters (details in Appendix A.3), and the other one computing the inter-variable correlation. (details in Appendix B.4)
We visualized the forecasting results in Appendix C.2.

## 5 Conclusion

In this paper, we address the challenging problem of cross-variable and cross-domain generalization in time series imputation. We introduce NuwaTS, a time series imputation model built on a PLM backbone that explicitly captures patch-wise statistical information and missing patterns. Additionally, we propose an efficient domain-specific fine-tuning technique that allows the model to seamlessly adapt from an all-in-one imputation model to a domain-specific one with minimal cost, and even transform into a forecasting model when needed. To benchmark the model’s performance, we design a novel variable-wise train/validation/test partitioning strategy, providing a rigorous evaluation framework. Experimental results show that our approach significantly outperforms state-of-the-art methods, demonstrating strong adaptability across diverse domains and missing rates, even in zero-shot scenarios. To the best of our knowledge, this is the first foundational time series imputation model capable of generalizing across domains. We believe our method sets a new benchmark in the field and provides valuable insights for future research.

## Appendix A Experimental Details

### A.1 Dataset Details

Our research encompasses experimental evaluations utilizing a selection of ten popular multivariate datasets, including (1) ETT(^3^33[https://github.com/zhouhaoyi/ETDataset](https://github.com/zhouhaoyi/ETDataset)) reports seven factors of electricity transformers, encompassing four subsets (ETTm1, ETTm2, ETTh1, ETTh2). These are divided into two categories based on temporal granularity, with data recorded at intervals of 15 minutes (m) and 1 hour (h), respectively. (2) ECL(^4^44[https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014](https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014)) records the hourly electricity consumption of 321 customers. (3) Weather(^5^55[https://www.bgc-jena.mpg.de/wetter/](https://www.bgc-jena.mpg.de/wetter/)) comprises 21 meteorological factors, recorded at ten-minute intervals throughout the year 2020, including variables such as temperature and humidity. (4) PEMS encompasses four datasets (03, 04, 07, and 08), each of which collects data on California’s public traffic network at a frequency of every five minutes.
For the LargeST (Liu et al., ), which consists of 8,600 sensors and spans from 2017 to 2021, we only selected the 2019 data for pre-training NuwaTS in this paper.

For the eleven datasets mentioned above, we split the data from the variable dimension rather than the time dimension, using a 1:1:1 ratio for training, validation, and test datasets. This approach is necessary because some baselines use a channel-dependent method, which requires fixed input dimensions for time series.

For the general fused datasets, we processed each time variable separately. We sliced time variables from all datasets (ETT, Weather, ECL, PEMS) of varying lengths with a stride of 1, creating a combined dataset with 17.6 million time segments of fixed length. In this study, we set it to 96. These segments were then randomly masked and used for model training.

Using the same approach, we sliced LargeST into 100.1 million segments to train NuwaTS and PatchTST. This was done for zero-shot evaluation and fine-tuning for forecasting tasks on ETT, weather, and ECL, ensuring no data leakage.

### A.2 Implementation Details

Regarding the training details of our model compared to others, we mainly followed the imputation task configuration, including optimizer, learning rate, and early stop strategy in the [https://github.com/thuml/Time-Series-Library](https://github.com/thuml/Time-Series-Library) for fair comparisons. We trained three different PLM backbones: GPT2 (Radford et al., ), BERT (Devlin et al., ), and LLaMA (Touvron et al., ). All the comparison models, as well as GPT2 and BERT, were trained on NVIDIA RTX 3090-24G GPUs. LLaMA-version (the first six layers) was trained on an NVIDIA A6000-48G GPU.

### A.3 Different parameter-efficient training strategy

We trained NuwaTS using three different PLM backbones. Table 9 presents the detailed training specifics for each experiment and backbone, including fine-tuned network layers, memory usage, and the number of parameters.

Table: Table 9: Different parameter-efficient training strategies on GPT2, BERT, and LLaMA2.

## Appendix B Training Algorithm and Mathematical formula

### B.1 Algorithm

Figure: Algorithm 1 One-for-all Model Training

Figure: Algorithm 2 Domain Specific Fine-tuning

Figure: Algorithm 3 Fine-tuning with inter-variable correlation

### B.2 Instance Norm

We applied reversible instance normalization (RevIN) (Kim et al., ) to individual series before splitting them into patches. Each series was normalized to have zero mean and unit standard deviation, with the original mean and standard deviation stored for later reversal, thereby reducing the domain distribution shift caused by non-stationary information. The details are as follows:

$$ $\begin{split}&\mathbb{E}_{t}\left[\mathbf{x}_{i}\right]=\frac{1}{L}\sum_{j=1}^ {L}\mathbf{x}_{i,j},\\ &\operatorname{Var}\left[\mathbf{x}_{i}\right]=\frac{1}{L}\sum_{j=1}^{L}\left( \mathbf{x}_{i,j}-\mathbb{E}_{t}\left[\mathbf{x}_{i}\right]\right)^{2},\\ &\hat{\mathbf{x}}_{i}=\left(\frac{\mathbf{x}_{i}-\mathbb{E}_{t}\left[\mathbf{x }_{i}\right]}{\sqrt{\operatorname{Var}\left[\mathbf{x}_{i}\right]+\epsilon}} \right).\\ \end{split}$ (2) $$

### B.3 Contrastive Learning Loss

Figure: Figure 6: Illustration of contrastive learning: Representations of each patch under different masks are treated as positive samples for each other, while representations from other time series and different patches from the same sequence under varying masks are considered negative samples.
Refer to caption: x5.png

Given a time patch represenation as a query $q$, and a set of keys $\mathbb{K}=\left\{k_{0},k_{1},\ldots\right\}$, including a positive target $k_{+}$. We employ the InfoNCE (Oord et al., ) as an auxiliary loss function to optimize the model’s performance:

$$ $\mathcal{L}_{cl}=-\log\frac{\exp\left(q^{T}Wk_{+}\right)}{\exp\left(q^{T}Wk_{+ }\right)+\sum_{i=0}^{K-1}\exp\left(q^{T}Wk_{i}\right)},$ (3) $$

where we employ bi-linear inner product and $W$ is a learnable matrix.

### B.4 Fine-tuning with inter-variable correlation

Unlike the two-layer MLP used in the channel-independent paradigm in Section 3.3, inspired by TimeXer (Wang et al., ) and iTransformer (Liu et al., ), we map each time series variable into a token. We then apply a stack of lightweight transformer layers, with the same number of layers as the PLM, to obtain the hidden states from each layer. These hidden states are passed through a linear layer to be directly mapped to $\bm{\mathcal{\hat{K}}}\in\mathbb{R}^{N\times 2\times\text{Layer}\times D}$, thus generating the prefix. This prefix now contains inter-variable correlation information, addressing the limitation of NuwaTS in modeling relationships between variables.

## Appendix C Visualization

### C.1 Imputation Visualization

We visualized the imputation results of NuwaTS in Figure 7, Figure 8, Figure 9 and Figure 10.

Figure: Figure 7: Case visualization when missing rate set to 0.1, 0.5, 0.9. We also visualized the case where the two patches are missing.
Refer to caption: x6.png

Figure: Figure 8: Case visualization when missing rate set to 0.1, 0.5, 0.9. We also visualized the case where the two patches are missing.
Refer to caption: x7.png

Figure: Figure 9: Case visualization when missing rate set to 0.1, 0.5, 0.9. We also visualized the case where the two patches are missing.
Refer to caption: x8.png

Figure: Figure 10: Case visualization when missing rate set to 0.1, 0.5, 0.9. We also visualized the case where the two patches are missing.
Refer to caption: x9.png

### C.2 Forecasting Visualization

We visualized the forecasting results of fine-tuned NuwaTS whose backbone had not undergone any forecasting training in Figure LABEL:forecasting_vis.

### C.3 Domain Recognition

We also visualized the domain recognition ability of NuwaTS in Figure 20. By directly appending domain-specific embeddings $\mathbf{k}$ again to the end of the input embedding $\mathbf{E}_{i}\in\mathbb{R}^{(N+2)\times D}$, we get $\hat{\mathbf{E}_{i}}=[\mathbf{k},\mathbf{z}_{i,(v_{g})},\mathbf{E}_{i,(p)},
\mathbf{k}]$, where $\hat{\mathbf{E}_{i}}\in\mathbb{R}^{D\times(N+3)}$. We feed the $\hat{\mathbf{E}_{i}}$ into the NuwaTS (one-for-all) model, and get the final representation $\hat{\mathbf{h}}^{(\text{Layer})}_{i}\in\mathbb{R}^{(N+3)\times D}$.
Then We extract the last embeddings $\mathbf{k^{\prime}}\in\mathbb{R}^{D}$ from $\hat{\mathbf{h}}^{(\text{Layer})}_{i}$ which is mixed with the patch embeddings in front of the input embeddings, thus exhibiting distinct domain characteristics. We collected $N$ time series from different domains and extracted $\mathbf{k^{\prime}}$. After applying the t-SNE (Van der Maaten & Hinton, ) method to reduce the dimension to 2, we obtained scattered points $\mathbf{k^{\prime}}^{p}\in\mathbb{R}^{N\times 2}$ and visualized them. The scattered points corresponding to time series from the same domain tend to cluster together, which indicates that NuwaTS can recognize domain information.

Figure: Figure 20: Illustration of domain recognition.
Refer to caption: extracted/5895493/fig/gpt2_vis.jpg

## Appendix D Extra Experiments

### D.1 real-world dataset evaluation

**Table 10: Imputation results on the Air-Quality dataset with missing values. 10% of the observed values in the validation set and test set are held out as ground truth for evaluation. A lower RMSE or MAE indicates better imputation performance. Green : the best.**
| Model | Air-Quality |  |
| --- | --- | --- |
| RMSE | MAE |  |
| GP-VAE | 0.614 | 0.268 |
| BRITS | 0.525 | 0.153 |
| SAITS | 0.518 | 0.137 |
| NuwaTS (zero-shot) | 0.370 | 0.151 |
| NuwaTS (domain fine-tuned) | 0.363 | 0.144 |

We conducted experiments on Beijing Multi-Site Air-Quality (Zhang et al., ) in Table 10 following pipeline in SAITS (Du et al., ) where we observed that, despite not using inter-series correlations and the model not being directly trained on the data, the performance of the pre-trained one-for-all NuwaTS was close to that of SAITS, surpassing both BRITS and GP-VAE (Fortuin et al., ).

### D.2 Ablation Study of Domain-Specific Fine-tuning

We also verified the design of domain-specific fine-tuning strategy of NuwaTS in Table 11. When the randomly initialized prefix key&value and the knowledge-transfer layer were ablated, the model’s fine-tuning performance showed a significant decline which indicates that domain-transfer layer retains the learned knowledge while newly added prefix key&value provide flexibility in transferring to specific domain. Additionally, we find that merely appending the prefix to the input layer and fine-tuning it significantly reduces the effectiveness of fine-tuning. Inserting the prefix into each layer of the PLM enhances the prefix’s representation capacity during domain transfer (Liu et al., ).

**Table 11: Domain-specific fine-tuning ablation study on ETTs based on the model pre-trained on general fused dataset. A lower MSE or MAE indicates better imputation performance. Green : the best.**
| Model | ETTh1 | ETTh2 | ETTm1 | ETTm2 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |  |
| Domain-Specific Fine-Tuning | 0.156 | 0.255 | 0.017 | 0.082 | 0.060 | 0.142 | 0.010 | 0.058 |
| w/o random initialized prefix key&value | 0.160 | 0.260 | 0.019 | 0.085 | 0.063 | 0.149 | 0.010 | 0.060 |
| w/o domain-transfer layer | 0.158 | 0.256 | 0.018 | 0.083 | 0.060 | 0.142 | 0.010 | 0.058 |
| only apply to input embedding | 0.163 | 0.263 | 0.018 | 0.084 | 0.062 | 0.145 | 0.010 | 0.060 |
| no fine-tuning | 0.164 | 0.263 | 0.018 | 0.084 | 0.064 | 0.147 | 0.010 | 0.060 |

### D.3 Learning From Incomplete Training data

**Table 12: When model training on incomplete data with 0.2 and 0.5 original missing rate. The comparison between NuwaTS and PatchTST. A lower MSE or MAE indicates a better imputation performance.**
| Methods | ETTh1 | ETTh2 | ETTm1 | ETTm2 | ECL | Weather |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |  |
| Default | 0.164 | 0.263 | 0.018 | 0.084 | 0.064 | 0.147 | 0.010 | 0.060 | 0.085 | 0.186 | 0.206 | 0.088 |
| NuwaTS-Origin Missing Rate 0.2 | 0.169 | 0.276 | 0.018 | 0.085 | 0.070 | 0.160 | 0.010 | 0.061 | 0.130 | 0.233 | 0.214 | 0.099 |
| NuwaTS-Origin Missing Rate 0.5 | 0.170 | 0.274 | 0.020 | 0.086 | 0.077 | 0.167 | 0.011 | 0.062 | 0.160 | 0.258 | 0.228 | 0.114 |
| PatchTST | 0.178 | 0.288 | 0.019 | 0.088 | 0.075 | 0.172 | 0.011 | 0.065 | 0.121 | 0.243 | 0.230 | 0.116 |
| PatchTST-Origin Missing Rate 0.2 | 0.185 | 0.295 | 0.019 | 0.089 | 0.079 | 0.179 | 0.011 | 0.066 | 0.150 | 0.264 | 0.239 | 0.118 |
| PatchTST-Origin Missing Rate 0.5 | 0.202 | 0.316 | 0.021 | 0.093 | 0.079 | 0.181 | 0.011 | 0.067 | 0.293 | 0.387 | 0.244 | 0.124 |

We simulated the training of the model on incomplete time series by randomly omitting 20% and 50% of the original training data and then tested its ability of resisting disturbance. The experimental results indicate that NuwaTS has strong resilience and can achieve good generalization performance when trained on data with a high missing rate (details shown in Table 12).

### D.4 The effect of the number of the GPT2 layer.

Through experiments, we tested the impact of the number of GPT-2 layers and patch size on the final results in Figure 21. The experimental results showed that using 6 GPT-2 layers yielded the best performance.

Figure: Figure 21: The effect of the number of GPT2 layers on Imputation Performance.
Refer to caption: extracted/5895493/fig/gpt_layer.jpg

### D.5 Effect of the Type of PLM Backbone

**Table 13: Effect of different PLM backbone. A lower MSE or MAE indicates a better imputation performance. Green : the best.**
| Model | ETTh1 | ETTh2 | ETTm1 | ETTm2 | ECL | Weather |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |  |
| GPT2 | 0.164 | 0.263 | 0.018 | 0.084 | 0.064 | 0.147 | 0.010 | 0.060 | 0.085 | 0.186 | 0.206 | 0.088 |
| BERT | 0.169 | 0.267 | 0.018 | 0.084 | 0.065 | 0.148 | 0.010 | 0.060 | 0.091 | 0.193 | 0.217 | 0.091 |
| LLaMA2 | 0.170 | 0.272 | 0.018 | 0.084 | 0.067 | 0.152 | 0.010 | 0.060 | 0.096 | 0.200 | 0.210 | 0.090 |

We experimented with three different backbones in Table 13: GPT2 (Radford et al., ), BERT (Devlin et al., ), and LLaMA (Touvron et al., ), uniformly using the first six layers of each. Among them, BERT is a transformer with bi-directional attention, while the other two, GPT2 and LLaMA, employ causal attention.

Using GPT-2 as the backbone outperformed the much larger LLaMA2, likely due to the relatively smaller amount of training data compared to LLaMA2’s massive parameter count.

Additionally, LLaMA2’s large embedding dimensions produced sparse features and the slow speed of LLaMA2 during training and evaluation makes deployment on edge devices challenging.

BERT, with the similar size as GPT-2, mainly differs in its bidirectional attention. It slightly lags behind GPT-2 due to the causal nature of time series data. However, BERT’s bidirectional attention allows the domain-specific embedding to capture domain information from the following embed patches.

Based on the BERT model pre-trained on general fused dataset, We have $\mathbf{E}_{i}=[\mathbf{k},\mathbf{z}_{i,(v_{g})},\mathbf{E}_{i,(p)}]$ before feeding into the backbone, where $\mathbf{E}_{i}\in\mathbb{R}^{D\times(N+2)}$. We feed the $\mathbf{E}_{i}$ into the backbone, and get the final representation $\mathbf{h}^{(\text{Layer})}_{i}\in\mathbb{R}^{(N+2)\times D}$.
Then We extract the first embedding $\mathbf{k^{\prime}}\in\mathbb{R}^{D}$ from $\mathbf{h}^{(\text{Layer})}_{i}$ and take it as [CLS] token (Devlin et al., ) which is mixed with the patch embeddings through bidirectional attention mechanism. With same approach as Appendix C.3, we obtain scattered points $\mathbf{k^{\prime}}^{p}\in\mathbb{R}^{N\times 2}$ and visualize them in Figure 22.

Figure: Figure 22: Domain information visualization. We take the model’s first output embedding as the [CLS] token, which carries domain information from following sequences. We then use the t-SNE (Van der Maaten & Hinton, ) method to visualize the domain information from different datasets.
Refer to caption: extracted/5895493/fig/bert_vis.jpg

### D.6 Embed with simple linear layer VS. Embed with text-alignment.

For tokenization, we employ a simple linear layer to embed patches. We acknowledge that recent research (Jin et al., ) has proposed utilizing Patch Reprogramming mechanism to align time patches with PLM’s pre-trained word embeddings, thereby activating the model’s time series understanding and reasoning capabilities. However, we contend that, due to the high proportion of missing values in the input masked patches and the varying locations of these missing values, modality alignment is not effective in representing such complex incomplete time series. Table 14 shows that simple linear embedding is better than the text-alignment strategy.

**Table 14: Embedding methods comparison.**
| Model | ETTh1 | ETTh2 | ETTm1 | ETTm2 | ECL | Weather |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |  |
| Simple linear layer | 0.164 | 0.263 | 0.018 | 0.084 | 0.064 | 0.147 | 0.010 | 0.060 | 0.085 | 0.186 | 0.206 | 0.088 |
| Embed with text-alignment (Jin et al., ) | 0.250 | 0.357 | 0.024 | 0.100 | 0.128 | 0.242 | 0.014 | 0.076 | 0.205 | 0.311 | 0.323 | 0.153 |

## Appendix E Main Results in Detailed

We reported detailed results for various baselines and different variants of our model across ten datasets, with missing rates ranging from 0.1 in Table 15 to 0.9 in Table 23.

Table: Table 15: Detailed Imputation results with missing rate set to 0.1. We set the input length to 96. A lower MSE or MAE indicates a better imputation performance.

Table: Table 16: Detailed Imputation results with missing rate set to 0.2. We set the input length to 96. A lower MSE or MAE indicates a better imputation performance.

Table: Table 17: Detailed Imputation results with missing rate set to 0.3. We set the input length to 96. A lower MSE or MAE indicates a better imputation performance.

Table: Table 18: Detailed Imputation results with missing rate set to 0.4. We set the input length to 96. A lower MSE or MAE indicates a better imputation performance.

Table: Table 19: Detailed Imputation results with missing rate set to 0.5. We set the input length to 96. A lower MSE or MAE indicates a better imputation performance.

Table: Table 20: Detailed Imputation results with missing rate set to 0.6. We set the input length to 96. A lower MSE or MAE indicates a better imputation performance.

Table: Table 21: Detailed Imputation results with missing rate set to 0.7. We set the input length to 96. A lower MSE or MAE indicates a better imputation performance.

Table: Table 22: Detailed Imputation results with missing rate set to 0.8. We set the input length to 96. A lower MSE or MAE indicates a better imputation performance.

Table: Table 23: Detailed Imputation results with missing rate set to 0.9. We set the input length to 96. A lower MSE or MAE indicates a better imputation performance.

## Appendix F Limitation

The model was trained on time series segments fixed at a length of 96, imputing the masked series based on the unmasked portions. Although our model can adapt to different domains and missing patterns, showing strong adaptability after domain-transfer fine-tuning, in practical applications, the model may require further fine-tuning when dealing with segments longer than 96, as well as the segments that are entirely missed.

Future work will aim to improve NuwaTS’s capability in handling longer missing segments.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Are there any specific advantages of using pre-trained LMs over standard transformer based time-series models such as PatchTST, iTransformer, Autoformer, Ti-MAE, etc.? How is pre-training on natural languages (using self-supervised learning) helpful for time-series imputation? Would randomly initialized LMs be able to achieve the same level of accuracy (i.e., training from scratch)?  
2. Holistically comparing univariate time series imputation (using the variable-wise division paradigm) to the standard train/val/test splits in time (which allows for multivariate imputation) is important to demonstrate that univariate time-series imputation is sufficient. Imputation methods like SAITS, BRITS, etc. might be optimized for multivariate time-series imputation, and hence are not as good when compared using the proposed benchmarking paradigm.
3. The motivation behind using statistical embeddings is not clear. Will the performance of the NuwaTS significantly drop without this? Empirically demonstrating it using an ablation study could help highlight the importance. 
4. Line 198: $Z_{i, (p)}$ notation needs clarification. $I$, $p$, $D$, and $N$ are not defined before.
5. Line 213: $z_{i, (p)}$ not defined. $r_i$ is not defined (I am guessing it is the mask ratio). Why is it multiplied with the mask ratio?
6. While the experiments on synthetic masking such as random masking and continuous masking are interesting and illustrative to an extent, adding more experiments on real-world datasets with missing values could strengthen the paper.
soundness: 3
strengths: 1. Imputation is an important step in almost every real-world time-series analysis, and designing a foundational model for general purpose time-series imputation is both interesting and useful for the community.
2. The proposed domain-specific embedding is novel, and allows for the lightweight (and plug-and-play) domain-specific fine-tuning of the NuwaTS model.
summary: The paper introduces a framework called NuwaTS that utilizes pre-trained LMs as a backbone for performing time-series imputation. The paper employs normalization and embedding schemes from prior works, and introduces a novel way to represent missing values and domain-specific information as embeddings. These embedding schemes along with the pre-trained LMs are trained in an end-to-end manner to perform imputation. The paper empirically demonstrates the efficacy of the proposed NuwaTS framework on several benchmark time-series datasets, and domain-transfer capabilities that are crucial for a foundational model.
weaknesses: 1. While the paper details a few time-series modeling approaches using pre-trained LMs in related works, the motivation for using them for imputation is missing. See additional details in questions.
2. The paper defines time-series imputation as a univariate problem. It leverages this formulation to introduce a novel benchmarking paradigm (called variable-wise division). However, multivariate time-series imputation has unique advantages over univariate formulations, such as the ability to leverage cross-variate correlations. See additional details in questions.
3. The proposed NuwaTS model has marginal improvements over the PatchTST (one-for-all) variant.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. [Main] I am curious on the setting itself. I agree that sometimes missingnesses on variable-dimension is important. However, Is it indeed possible to impute new dimension which had never been exposed to the model? I assume this is only possible if the trend of the target variable is similar with the observed variables. In other words, the generalization performance indeed extremely rely on the variable splitting, which is randomly performed in this paper. Moreover, regarding marginal improvements on few-shot fine tuning results (table5), it seems that ETT dataset itself has high variable correlations. I think this paper must contain experiments or discussions with (synthetic) data which is trained with one pattern of variables then target variables have completely different missing patterns. I think this is the way to evaluate genuine generalization across variables and complete the claim of this paper. I would happy to dramatically raise my score if authors could show more careful analysis on across-variable generalization (and why this model is capable of those generalizability).
2. [Minor] Please provide variances of the results, and evaluation details such as number of iterations. 
3. [Minor] Please include more recent probabilistic imputation models (VAE, GAN, DIffusion, and others..). If there is reason why those models are not applicable, please provide those reasons.
soundness: 3
strengths: 1. This paper includes three different settings to use PLMs for time series imputation. Different settings have their own challenges. 
2. The paper includes comprehensive experimental results on diverse settings. Ablation studies are also comprehensive.
summary: This paper proposes to use pretrained language models (PLMs) for general times series imputation. This paper especially focused on the generalization on the variable dimension, within the domain and cross-domain. They proposed three different ways to pretrain / finetune PLMs with (optionally) few more parameters. They tested the performance on general imputation (missing at variable dimension), zero-shot domain transfer, few-shot finetuning, and downstream forecasting with TimesNet.
weaknesses: 1. I couldn't find variances of the results. In my experience, the time series imputation performance often highly depends on the seed settings.
2. Related to 1: Table 4 zero-shot performance gain seems marginal compared to PatchTST. Please provide variances.
3. Table 8 results makes me doubt the model. For forecasting (which is an extreme case of imputation), the model seems worse (or just comparable) to baselines yet this uses much more parameters. I know the authors only trained small MLPs but it indeed uses large language model, which seems inefficient to me.
4. The baselines only includes predictive models.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: Please refer to weaknesses
soundness: 3
strengths: - This paper is well written and organized. I enjoy reading this paper.
- I agree with the idea and appreciate the effort to deal with cross-domain generalization which is important in time-series modeling since the different patterns among domains are more significant than other data modalities. 
- The authors provide well organized code base for the method which helps reproducibility and usability.
- Imputation-specific zero-shot model is new, and some designs(e.g. using missing information) can potentially beneficial for other tasks using time-series with missing data.
summary: This paper introduces NuwaTS, which is a general time series imputation framework leveraging pre-trained language models to address limitations in existing models' generalizability. Through tailored embeddings and contrastive learning, NuwaTS can impute missing data across any domain, with relatively small adjustments. The authors provide multiple experiments to show its performance over other models across diverse datasets.
weaknesses: - Although this paper introduce several interesting concepts, I think the authors should justify their design choices since the authors integrates several existing components and conducts heavy engineering to boost the zero-shot capability, but there are sufficient techniques for existing works in time-series forecasting literature which can be extended to imputation task. I think the performance compared to other LLM based time series method is not that impressive 
- I think proposing new benchmarking strategy(e.g. 1:1:1 split of train/valid/test in feature-wise manner) is insufficient to claim it as novel contribution. Since I think cross-variable generalization can be considered as part of domain generalization, how cross-variable generalization is specifically important? Just conducting zero-shot imputation experiments is not enough for measuring the generalization capabilities?
- This paper mainly lists the results of experiments. I think more insights and analysis beyond the abalation studies would be beneficial

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: None
soundness: 2
strengths: Experiments are conducted thoroughly.

Novel designs such as statistics patch embedding, missingnes embedding, time series path contrastive learning and p-tuningv2 style domain specific fine-tuning.
summary: This paper proposes a single pre-trained model to imput on time series from different domains. Several designs are proposed such as statistics patch embedding, missingness embedding, time series path contrastive learning, and p-tuningv2 domain-specific fine-tuning. Experiments show that this one-for-all model achieves SOTA performance compared to strong baselines. Also, this model has zero-shot, few-shot and forecasting capability. Ablation studies show that the novel designs proposed are indeed helpful.
weaknesses: Though several designs are proposed, they are not as effective as the paper claims. In table 6, I don't think the improvements of these designs are significant enough.

Modeling each individual dimension separately is a limitation of the proposed method. Althought the authors proposes to include inter-series correlation information during fine-tuning for forecasting, they don't show that inter-series correlation information can be used for imputation. I think this method is not applicable to datasets with cross-dimensional interactions. Thus this is not a one-for-all model.