Title: Addressing Spatial-Temporal Heterogeneity: General Mixed Time Series Analysis via Latent Continuity Recovery and Alignment
Venue: NeurIPS 2024
Accepted: yes
Keywords: Mixed Time Series

General Time Series Modeling

Spatial-Temporal Heterogeneity

Latent Continuity Recovery

Adversarial Learning
Abstract: Mixed time series (MiTS) comprising both continuous variables (CVs) and discrete variables (DVs) are frequently encountered yet under-explored in time series analysis. Essentially, CVs and DVs exhibit different temporal patterns and distribution types. Overlooking these heterogeneities would lead to insufficient and imbalanced representation learning, bringing biased results. This paper addresses the problem with two insights: 1) DVs may originate from intrinsic latent continuous variables (LCVs), which lose fine-grained information due to extrinsic discretization; 2) LCVs and CVs share similar temporal patterns and interact spatially. Considering these similarities and interactions, we propose a general MiTS analysis framework MiTSformer, which recovers LCVs behind DVs for sufficient and balanced spatial-temporal modeling by designing two essential inductive biases: 1) hierarchically aggregating multi-scale temporal context information to enrich the information granularity of DVs; 2) adaptively learning the aggregation processes via the adversarial guidance from CVs. Subsequently, MiTSformer captures complete spatial-temporal dependencies within and across LCVs and CVs via cascaded self- and cross-attention blocks. Empirically, MiTSformer achieves consistent SOTA on five mixed time series analysis tasks, including classification, extrinsic regression, anomaly detection, imputation, and long-term forecasting. The code is available at https://github.com/chunhuiz/MiTSformer.

Addressing Spatial-Temporal Heterogeneity: General Mixed Time Series Analysis via Latent Continuity Recovery and Alignment

Abstract

Mixed time series (MiTS) comprising both continuous variables (CVs) and discrete variables (DVs) are frequently encountered yet under-explored in time series analysis. Essentially, CVs and DVs exhibit different temporal patterns and distribution types. Overlooking these heterogeneities would lead to insufficient and imbalanced representation learning, bringing biased results. This paper addresses the problem with two insights: 1) DVs may originate from intrinsic latent continuous variables (LCVs), which lose fine-grained information due to extrinsic discretization; 2) LCVs and CVs share similar temporal patterns and interact spatially. Considering these similarities and interactions, we propose a general MiTS analysis framework MiTSformer , which recovers LCVs behind DVs for sufficient and balanced spatial-temporal modeling by designing two essential inductive biases: 1) hierarchically aggregating multi-scale temporal context information to enrich the information granularity of DVs; 2) adaptively learning the aggregation processes via the adversarial guidance from CVs. Subsequently, MiTSformer captures complete spatial-temporal dependencies within and across LCVs and CVs via cascaded self- and cross-attention blocks. Empirically, MiTSformer achieves consistent SOTA on five mixed time series analysis tasks, including classification, extrinsic regression, anomaly detection, imputation, and long-term forecasting. The code is available at https://github.com/chunhuiz/MiTSformer .

1 Introduction

Multivariate time series analysis is energized in various real-world applications, such as weather forecasting [7], activity recognition [20], and industrial maintenance [47]. Empowered by deep learning, plentiful time series models have been proposed based on foundation models such as RNNs [23, 25, 22], CNNs [6, 36, 40], Transformers [30, 41, 10] and modernized MLPs [50, 45, 39]. These sophisticated models have achieved increasingly remarkable performance in various time series analysis tasks, e.g., classification [17], forecasting [7, 42], imputation [40] and anomaly detection [9].

The primary challenge in time series analysis is effectively modeling spatial-temporal patterns, including intra-variable temporal variations and inter-variable spatial correlations [46, 48, 16, 11, 43]. So far, most current approaches naturally assume that time series data are composed solely of continuous variables, and then uniformly model spatial-temporal patterns in continuous spaces. Yet, in broad practical scenarios, the acquired data are often mixed time series (MiTS) that encompass both continuous variables (CVs) and discrete variables (DVs). Take meteorological data as an example (Fig. 1 (Left)), some sensor-derived variables are commonly recorded as CVs (e.g., temperature,

Days

Temperature

Humidity

Cloudage

Rainfall

1

0

1

0

high

low

high

low

Auto-Correlation

Periodicity

Fluctuation

p

Sudden Change Limited Information

p

0

1

Temporal Heterogeneity

>

0

0

humidity, and wind speed), while certain variables like cloudage and rainfall patterns are typically tracked as DVs due to measurement restrictions or distinctive nature. Up to now, mixed time series analysis is still a formidable yet under-explored problem. Essentially, MiTS presents spatial-temporal heterogeneity problems as depicted in Fig. 1 (Right). On one hand, CVs commonly encapsulate rich temporal variation information, exhibited in autocorrelations, periodical patterns, local fluctuations , etc, while DVs often undergo sudden changes or steady states due to limited value ranges, resulting in the temporal variation discrepancy between CVs and DVs that complicates temporal modeling. On the other hand, CVs generally adhere to Gaussian distributions, while DVs follow Bernoulli distributions, resulting in the distribution type discrepancy that hinders spatial correlation analysis between CVs and DVs. Neglecting these heterogeneities and equally treating mixed variables would yield insufficient and imbalanced spatial-temporal modeling problems, i.e., the model may struggle to characterize distinct temporal patterns of DVs and CVs and fail to reliably capture spatial correlations within and across DVs and CVs, posing a bottleneck for MiTS analysis.

The key to addressing the spatial-temporal heterogeneity lies in bridging the information gap between DVs and CVs. Essentially, in real-world MiTS, the observed DVs may originate from intrinsic continuous-valued factors, which are unobservable owing to extrinsic factors such as measurement limitations, storage requirements, and transmission interference. Continuing with the aforementioned meteorological example, cloud cover percentage is an intrinsically continuous variable, whose finegrained values are hard to measure directly. In practice, discretizing it with coarse-grained discrete variable-cloudage (reflect 'cloudy' or not) is sufficient for most applications and is memory-efficient. In this paper, we introduce latent continuous variables (LCVs) to describe the intrinsic continuous factors behind DVs. Given its numerically continuous nature, the LCV of cloudage, i.e., cloud cover percentage, may be spatially correlated with other observed CVs (e.g., humidity and wind speed) and exhibit similar temporal variation patterns (e.g., autocorrelation and seasonal fluctuation) with them, as both of them originate from the same meteorological system. Thereby, we can progressively recover the LCVs behind DVs by leveraging these temporal similarities and spatial interactions among CVs and LCVs. In this way, spatial-temporal dependencies of mixed variables can be completely and reliably modeled in a unified continuous numerical space, and the spatial-temporal heterogeneity problem is mitigated. Also, by bridging the mutual spatial-temporal interactions, LCVs and CVs can supply complementary information for various downstream analysis tasks.

Enlightened by the above insights, we reconcile the intrinsic tension between the two highly dependent problems - Latent Continuity Recovery and Spatial-Temporal Modeling - in one coherent and synergistic framework, MiTSformer , for general mixed time series analysis. By leveraging the temporal similarities and spatial interactions between LCVs and CVs, MiTSformer can gradually decipher the LCVs behind DVs and capture the complete spatial-temporal dependencies within and across LCVs and CVs. Specifically, we design a recovery network to portray LCVs behind DVs by adaptatively and hierarchically aggregating temporal contextual information. Followingly, an adversarial variable modality discrimination objective and smoothing constraints are devised to guide the learning of the recovery network, ensuring the recovered LCVs share similar temporal properties and distributions with CVs. Additionally, MiTSformer employs self-attention to learn spatial-temporal dependencies within LCVs or CVs and cross-attention to exploit those across LCVs and CVs, facilitating various downstream analysis tasks. Our contributions lie in three aspects:

Novel Problem : To the best of our knowledge, our paper pioneers the exploration of the general mixed time series analysis, which is practical and challenging. We reveal the crucial spatial-temporal heterogeneity problem, which is caused by the discrepancies in temporal variation properties and distribution types between DVs and CVs.

Customized Framework : To solve the spatial-temporal heterogeneity problem, we propose a task-general framework MiTSformer customized for MiTS, which adaptively recovers LCVs behind DVs by leveraging the adversarial guidance of CVs and task supervisions. Moreover, MiTSformer can capture spatial-temporal dependencies within and across CVs and LCVs via self- and cross-attention blocks, thus learning sufficient and balanced spatialtemporal representations and being amenable to various mixed time series analysis tasks.

Versatile Effectiveness : Empirically, our proposed MiTSformer establishes the state-ofthe-art performance on five mainstream mixed time series analysis tasks with 34 datasets covering wide-ranging real-world application domains. We believe our work makes a predominant attempt at general mixed time series analysis in practical applications.

2 Related Work

General Time Series Analysis General time series analysis aim at learning universal temporal representations by developing task-general backbones and task-specific heads for diverse tasks. To date, these methods have been designed predominantly for time series comprising continuous variables solely. As a pioneering work, TimesNet [40] leverages fast Fourier transform and parameterefficient inception block to capture intra-period and inter-period variations for time series modeling. Followingly, ModernTCN [16] modernizes and modifies the traditional TCN by introducing larger effective receptive fields and cross-variable dependency modeling, bringing great performance and efficiency. Meanwhile, GPT2TS [52] leverages pre-trained language models, e.g., GPT2 [32], for various time series analysis tasks with task-specific fine-tuning.

Yet, most of the current works naturally assume uniformity in variable types of time series and are inapplicable for MiTS, as they lack the differentiation of distinct variable modalities. In this paper, we develop a systematic framework with delicate differentiation and deft alignment of mixed variables to support sufficient and balanced spatial-temporal modeling for general mixed time series analysis.

Mixed Data Analysis Previous studies have revealed the significance of mixed data analysis and made several attempts to shed light on this challenging problem. A naive solution for mixed data modeling is roughly pre-processing DVs and CVs into the same variable modality, e.g., discarding DVs or discretizing CVs by certain policies [35, 13]. However, these methods may lose vital finescaled information and bring errors inevitably. Towards concurrently modeling of DVs and CVs, Mixed Data RBF-ELM method [24] adopts a distance-based learning scheme for efficient and direct mixed data classification, while Mixed-variate Restricted Boltzmann Machine (Mv.RBM) [15, 14] construct ensembles of mixed-data Deep Belief Nets with varying depths for anomaly detection of mixed data. Another line of work treats DVs as semantic attributes of CVs and establishes the relationships between DVs and CVs by developing specific inference rules [18, 44]. More recently, researchers have utilized mixed naive Bayes models [37, 38] or variational inference [12] for mixed data modeling in industrial processes with different distribution priors of CVs and DVs.

However, current studies mostly focus on specific analysis tasks (e.g., designed only for classification) and may be restricted by linear, non-temporal data, or other rigorous assumptions, which are not capable of handling real-world MiTS that exhibit intricate spatial-temporal patterns. Instead, our proposed MiTSformer can model complete spatial-temporal dependencies within and across DVs and CVs and can produce task-general representations for various MiTS analysis tasks.

3 MiTSformer

3.1 Problem Formulation and Motivation Analysis

Mixed Time Series . Given a collection of multivariate time series X = { x 1 , x 2 , ..., x p } comprising p variables with length T . Among them, there are p -n continuous variables X C = { x 1 , x 2 , ..., x p -n } ∈ R ( p -n ) × T with continuous numerical values, and n discrete variables X D = { x p -n +1 , x p -n +2 , ..., x n } ∈ A n × T with discrete states. Without loss of generality, we consider the binary-valued discrete variables, whose value can be 0 or 1 , i.e., A = { 0 , 1 } . Mixed time series are used as model input to support various analysis tasks, such as regression and classification.

Hampered by the spatial-temporal heterogeneity problem, directly modeling spatial-temporal dependencies of CVs and DVs without considering their discrepancies may inevitably yield non-negligible

biases. Such research bottlenecks prompt us to exploit the underlying generation and interaction mechanism of DVs and CVs. As aforementioned, the observed DVs are potentially derived from LCVs, which undergo discretization processes due to external interferences as depicted in Fig. 2. For each DV x D , we adopt a corresponding LCV x LC to portray its latent continuity. To reliably and completely model inherent spatial-temporal patterns within MiTS, deciphering and recovering LCVs behind DVs is indispensable. Yet, it is challenging since a single discrete state can not be directly transformed into a continuous value without proper supervision. In this study, we address this challenge by revealing and leveraging the temporal similarity and spatial interaction between LCVs and CVs:

Temporal Similarity The unobserved LCVs share similar temporal variation patterns (e.g., autocorrelation, periodicity, trend, etc.) with the observed CVs.

Spatial Interaction LCVs and CVs exhibit information interactions and inter-variable spatial correlations. The synergistic effect of LCVs and CVs provides complementary information for various downstream tasks.

Built upon these two insights, we design

External interference factors

Observed Discrete Variables

Observed Continuous Variables



Temporal Similarity



Spatial Interaction

high

low

high

low

1

0

T

MiTSformer, which will be elaborately introduced in the following parts. 方法概览

Observed DVs

Observed CVs

Embedding

Embedding

Self-Attention

Sub-Block

Self-Attention

Sub-Block

Cross-Attention

Sub-Block

Cross-Attention

Sub-Block

Reconstruction

Decoder

Reconstruction

Decoder

Downstream

Tasks:

Spatial-Temporal Attention Blocks

Classification

Regression

Forecasting

Imputation

口

XN

3.2 Framework Overview

As aforementioned, there are two key steps for general MiTS analysis: 1) unifying the temporal characteristics and distribution types of DVs and CVs; 2) modeling sufficient and balanced spatialtemporal dependencies for effective representation learning. MiTSformer facilitates these two steps in a highly versatile manner with a coherent framework as depicted in Fig. 3. The overall pipeline contains two key parts: 1) Latent Continuity Recovery that adaptively aggregate contextual information of DVs to recover LCVs with the adversarial alignment guidance of CVs and temporal smoothness constraints; 2) Spatial-Temporal Attention Blocks that model intra- and inter-variable modality spatial-temporal dependencies with cascaded self-attention and cross-attention sub-blocks.

3.3 Latent Continuity Recovery

Since single time points of DVs contain limited information, it is difficult to directly transform discrete states to latent continuous values. Fortunately, time series commonly present auto-correlation natures, i.e., the value of a certain time step is correlated with its adjacent ones. Specifically, for a discrete state of '1' surrounded by states '1', its latent continuous value would be relatively large, e.g., 0.9. In contrast, for a discrete state of '1' surrounded by states '0', its latent continuous value would be relatively small, e.g., 0.6. Thus, an intuitive solution for enriching the information density of a single time point of DVs is to properly leverage its adjacent context information.

Contextual Aggregation Recovery Network We convert the above insights into the model inductive bias and realize the latent continuity recovery by adaptively and hierarchically aggregating multi-scale adjacent context information of DVs. Specifically, convolutional neural networks (CNNs) own an inductive bias that can aggregate receptive local information by convolutional kernels. Technically, we devise the recovery networks that receive DVs as input to generate LCVs as x LC = Rec -Net ( x D ) . As depicted in Fig. 4, the recovery network is composed of several residual dilated convolutional blocks, which employ dilated convolutional kernels along the temporal dimension to aggregate adjacent context information, and utilize residual connections to adaptively accumulate multi-scale temporal information to characterize intricate temporal patterns of LCVs. The residual dilated convolutional network is implemented by the iterative process as h i = Conv d i ( h i -1 )+ h i -1 , where h i represents the output of the i -th residual block, i = 1 , 2 , . . . , n , Conv d i denotes the convolution operation along the temporal axis to aggregate contextual information with dilation

Recovery Network

Recovered LCVs

Embedding

Observed CVs

Embedding

Residual Dilated CNN Blocks

Conv Block

Conv Block

Conv Block

Norm

Embeddings of LCVs

Embeddings of CVs

Variable-modality

Discriminator

GRL

GRL

Variable-modality Labels

Dilated Conv

Dilated Conv

1

0

Observed DVs

?

?

rate d i . The final output, x LCV = h n , is the result after n residual blocks.

Temporal Adjacent Smoothness Constraint : To facilitate spatial-temporal modeling across LCVs and CVs, the recovered LCVs should be equipped with interpretable autocorrelation or trend properties, instead of 'sudden changes' as DVs. To this end, we encourage the smoothness of LCVs across time with a regularization term as

where S is the smoothness matrix and ⊗ denotes the Hadamard product operation. Abs ( S x D ) = Abs ( x D [2: T ] -x D [1: T -1] ) denotes the absolute value of the first-order difference of DVs and can reflect the ' sudden change ' points (with state '1'). Overall, minimizing L smooth is equivalent to minimizing ∑ T -1 t =1 Abs ( x D t+1 -x D t ) ( x LC t+1 -x LC t ) 2 by introducing the multiplication of the constant matrix S .

Alternatively, we can adopt the K-Lipschitz continuity as smoothness constraints, where the CVs can act as guidance to determine the smoothness degree of the recovered LCVs. Empirically, we find out such design would bring similar performance.

After latent continuity recovery, raw series of x LC and x C are independently embedded as tokens z LC and z C through variate-wise linear projection to describe the properties of each variable.

Adversarial Variable-Modality Discrimination : Inspired by the temporal similarity property, the recovered LCVs should exhibit similar temporal patterns and distributions to CVs. Accordingly, we devise a variable modality discrimination objective, which is optimized in an adversarial manner as

where Dis denotes the variable modality discriminator. Dis is trained to distinguish LCVs and CVs as accurately as possible, while the recovery networks are facilitated to generate LCVs with characteristics similar to CV as much as possible to confuse the discriminator. We adopt gradient reversal layer [19](GRL) to achieve the adversarial learning objective. In this way, we bridge the interactions between CVs and DVs and enable CVs to supply supervision signals for LCV recovery.

3.4 Intra- and Inter-Variable-Modality Spatial-Temporal Dependency Learning

Spatial-temporal dependencies are of vital importance for time series representation learning. Specifically for MiTS, the complete spatial-temporal correlations include the ones within LCVs or CVs

Multivariate

Self-Attention

Q

K

V

LayerNorm

Feed-forward

LayerNorm

Input

Output

FCN

Act&Norm

FCN

Query

Key

Value

MatMul

Multivariate

Correlation

MatMul

Multivariate

Cross-Attention

Q

K

V

LayerNorm

Feed-forward

LayerNorm

Output

Multivariate

Cross-Attention

LayerNorm

Feed-forward

LayerNorm

Output

V

K

Q

Input

Input

LCVs

CVs

CVs

LCVs

within

CVs

across

CVs and LCVs

across

LCVs and CVs

within

LCVs

Variable Adjacency Matrix

Self-Attention Sub-block

Cross-Attention Sub-block

CiC

1

TLC

3

Features

Variate

SelfAttn(z)

CrossAttn(，c)

CrossAttn(C)

SelfAttn(c)

↑

and the ones across LCVs and CVs, which are explicitly characterized by spatial-temporal attention blocks in MiTSformer (as shown in Fig. 5).

Intra-Variable-Modality Self-Attention Within each variable modality, we adopt self-attention to model their spatial-temporal dependencies as

where Self -Attn denotes the Multi-head Self-Attention that captures intra-variate spatial correlations by computing Softmax scores with query and key embeddings and weighted aggregating the value embeddings. FFN denotes the feed-forward network that processes each variable token to learn intra-variate global temporal representations. LN denotes Layer Normalization, which is applied to individual variate tokens and has been proven effective in tackling non-stationary problems [28].

Inter-Variable-Modality Cross-Attention We adopt symmetric cross-attention sub-blocks to model the spatial-temporal interactions across LCVs and CVs as

Subsequently, the acquired token embeddings z LC L and z C L are utilized both for 1) original MiTS reconstruction with L Rec as Eq. 5, and 2) downstream tasks with task supervision loss L Task , e.g., Cross Entropy loss for the classification task. See Appendix A for pipelines of each task.

Self-Reconstruction We devise variate-wise decoders based on MLPs to reconstruct the original DVs and CVs, which can not only provide self-supervision signals for spatial-temporal dependency learning but also guarantee the recovered LCVs retain the information of the observed DVs.

3.5 Synergy of Latent Continuity Recovery and Downstream Tasks

Considering both latent continuity recovery and downstream analysis task, the overall optimization objective of MiTSformer is expressed as

While seemingly separated from each other, the loss components of MiTSformer work in a collaborative fashion on two aspects: 1). The first three self-supervision losses facilitate latent continuity recovery synergetically to tackle the spatial-temporal heterogeneity problem. Specifically, the smoothness loss L Smooth favorably manifests the autocorrelation of LCVs and alleviates the sudden

changes. Also, the discrimination loss L Dis guarantees LCVs to be equipped with similar temporal dynamics with CVs adversarially, which guarantees a crucial condition for spatial-temporal modeling. Meanwhile, the reconstruction loss provides auxiliary self-supervision signals and constrains the LCV recovery processes to be reversible. Our experiments, particularly the ablation study displayed in Section 4.2, further justify the mutual dependency of the synergy between the three components; 2) The latent continuity recovery losses and task loss also work collaboratively , as reliable latent continuity recovery can bring excellent downstream task performance, while task supervision loss may, in turn, also reciprocate the recovery processes. Take MiTS classification as an example, the ideally recovered LCVs can prompt learning discriminative representations for classification, while the class label loss provides additional supervision for learning appropriate recovery functions.

4 Experiments

To verify the effectiveness and versatility of MiTSformer, we extensively experiment on five mainstream mixed time series analysis tasks, including mixed time series classification, extrinsic regression, long-term forecasting, imputation, and anomaly detection .

Implementations Table 1 summarizes the experiment benchmarks. For each dataset, we randomly select half the variables and discretize them as DVs to generate MiTS data. More information about datasets and experimental platforms, hyperparameters and experimental configurations, and algorithm implementations can be found in Appendix A.1, A.2,A.3, respectively. The pipelines of different mixed time series analysis tasks can be found in Appendix A.5 ∼ A.8.

Baselines We extensively compare MiTSformer with the latest and advanced models in the time series community, including CNN-based models: ModernTCN (2024), TimesNet (2023) and MICN (2023); Transformer-based models: iTransformer (2024), PatchTST (2023), Crossformer (2023), FEDformer (2022) and Pyraformer (2022); MLP-based models: LightTS (2023), DLinear (2023) and FiLM (2022). To guarantee fairness, we keep the original backbone for each method as the feature extractor, and we adopt universal task-specific heads and loss functions consistently for all methods.

4.1 Main Results on Different Tasks

(1) Mixed Time Series Classification We select 10 multivariate datasets from the UEA Time Series Classification Archive [5] and pre-press them following [40]. As shown in Fig. 6, MiTSformer achieves the best performance with an average accuracy of 71.9%, surpassing all powerful baselines. Besides, it can be observed that frequency-based methods FiLM and FEDformer show inferior performance, as the introduction of DVs may make it difficult to estimate the frequency reliably and yield non-negligible errors. In comparison, MiTSformer adaptively recovers and aligns the LCVs behind DVs with the guidance of both CVs and class-label supervision,

Pyraformer

FEDformer

FiLM

Dlinear

LightTS

MICN

Crossformer

PatchTST

TimesNet

ModernTCN.

iTransformer

MiTSformer

60

62

64

66

68

70

72

Accuracy(%)

68.9

64.8 64.9

68.1 68.2

66.4

69.0

67.6

69.6

70.1

68.9

71.9

Averaged baseline

Best baseline

Figure 6: Classification Results (Acc ↑ ) thereby facilitating high-level representation learning and benefiting the classification tasks.

(2) Mixed Time Series Extrinsic Regression

We select 10 multivariate datasets from TSER repository [34] and pre-process them as [34]. As summarized in Fig. 7 (b), MiTSformer outperforms all the rivals with an average MAE of 0.524, verifying its capacity to model complex extrinsic regression patterns. Besides, some transformer-based models, e.g., PatchTST and Crossformer, present competitive performance by learning global dependencies with selfattention. Notably, Dlinear shows inferior results. This is probably because Dlinear adopts single-layer MLP to model temporal dependencies, which might be suitable for some autoregressive endogenous patterns, but are not appli-

Pyraformer

FEDformer

FiLM

Dlinear

LightTS

MICN

Crossformer

PatchTST

TimesNet

ModernTCN.

iTransformer

MiTSformer

0.48

0.50

0.52

0.54

0.56

0.58

0.60

0.62

MAE

0.54

0.576

0.598

0.607

0.554

0.536 0.535 0.534

0.553

0.535

0.55

0.524

Averaged baseline

Best baseline

cable for nonlinear exogenous regression relationships and degenerate performance.

Mixed Time Series Anomaly Detection We conduct experiments on five widely-used anomaly detection benchmarks: SMD [33], MSL [21], SMAP [21], SWaT [29] and PSM [4].

For fair comparisons, we adopt the reconstruction loss for both CVs and DVs to train base models and use the reconstruction error as the shared anomaly criterion for all experiments. Specifically for MiTSformer, no additional taskorientation loss is added since there are already reconstruction losses. Fig. 8 presents the performance comparison evaluated by the F1-score ( ↑ ), demonstrating that MiTSformer consistently achieves state-of-the-art performance on five benchmarks. Besides, iTrasformer also achieves great performance by adaptively modeling multivariate correlations with self-attentions. In comparison, MiTSformer not only recovers the LCVs behind DVs but also comprehensively and

























































explicitly characterizes the spatial-temporal dependencies within and across DVs and CVs, thereby equipping meticulous anomaly detection ability.

Mixed Time Series Imputation Following [40], we select datasets of ETT [49], Weather (Wetterstation) and Electricity (UCI) benchmarks. In practice, imputating the values of CVs is more important and meaningful than those of DVs. Therefore, we focus on imputing the CVs in each experiment, while DVs are used as inputs to provide auxiliary information. To compare the model capacity under different proportions of missing data, we randomly mask the time points in the ratio of { 12 . 5% , 25% , 37 . 5% , 50% } in length-96 mixed time series and report the averaged imputation accuracy of CVs from 4 different mask ratios.

Due to the missing values, the imputation task requires the model to deeply exploit the underlying temporal dependencies and spatial correlations from partial observations. According to Table 2, MiTSformer achieves the best performance on most tasks. Through the recovery of LCVs and the intra- and inter-variable modality attention mechanisms, MiTSformer can effectively bridge spatial-temporal information interactions between CVs and DVs. In this way, not only can CVs exploit useful information for self-imputation by mining temporal and spatial correlation themselves, but also DVs can provide reliable auxiliary information for more accurate imputation of CVs.

(5) Mixed Time Series Long-term Forecasting We follow the settings of prediction lengths and benchmarks as [40], including ETT [49], Electricity (UCI), Weather (Wetterstation), Exchange [23] and ILI (CDC), corresponding to different applications. We focus on forecasting both DVs and CVs.

Since CVs contain more fine-grained information granularity and can more adequately evaluate the model forecasting performance, we mainly focus on the prediction accuracy of CVs, which are summarized in Table. 3. As reported, MiTSformer presents the best performance on most tasks (76 % according to Table 16), surpassing extensive advanced MLP-based, Transformer-based and CNN-based models. In addition, recent baselines- modernTCN and iTransformer present great performance due to their delicate design of global temporal receptive fields.

0

48

0

1

Observed DV

0

48

96

Actual LCV

Recovered LCV

(a) ETTh1 case#1

0

48

0

1

Observed DV

0

48

96

Actual LCV

Recovered LCV

(b) ETTh1 case#2

0

48

0

1

Observed DV

0

48

96

Actual LCV

Recovered LCV

(c) ETTh1 case#3

0

48

0

1

Observed DV

0

48

96

Actual LCV

Recovered LCV

(d) ETTh1 case#4

0

48

0

1

Observed DV

0

48

96

Actual LCV

Recovered LCV

(e) ETTh1 case#5

0

48

0

1

Observed DV

0

48

96

Actual LCV

Recovered LCV

(f) ETTh2 case#1

0

48

0

1

Observed DV

0

48

96

Actual LCV

Recovered LCV

(g) ETTh2 case#2

0

48

0

1

Observed DV

0

48

96

Actual LCV

Recovered LCV

(h) ETTh2 case#3

0

48

0

1

Observed DV

0

48

96

Actual LCV

Recovered LCV

(i) ETTh2 case#4

4.2 Ablation Studies

To further verify the effectiveness of each key design, we conduct ablation experiments on mixed time series classification, long-term forecasting, and anomaly detection tasks. The results are summarized in Table 4, where ' w/o LCVs' denotes removing L Dis , L Smooth , and L Rec together, and ' w/o Cross-Att' denotes removing the cross-attention sub-block and adopt self-attention sub-block solely.

Latent Continuity Recovery The three loss terms, L Dis , L Smooth , and L Rec support the latent continuity recovery from different perspectives and work collaboratively. As shown in Table 4, ablating each loss term would lead to performance degradation for different tasks. Remarkably, the employment of all recovery loss functions in conjunction yields the optimal result.

Attention Backbones The cross-attention block is devised to bridge the information prorogation among LCVs and CVs by modeling spatial-temporal dependencies across LCVs and CVs, as LCVs and CVs provide complementary information for analysis tasks. The results in Table 4 also emphasize the importance of cross-variable-modality dependency modeling for MiTS analysis.

4.3 Visualization and Model Investigations

Visualization of the Recovered LCVs To verify the interpretability of LCV recovery, we provide visualization plots in Fig. 9. We find that DVs commonly show patterns of ' sudden changes ' or ' steady states ', which are less informative for analyzing spatial-temporal correlations. Fortunately, MiTSformer not only recovers their latent fine-grained and informative temporal variation patterns but also further leverages them for spatial-temporal representation learning, thereby mitigating the spatial-temporal heterogeneity challenge and achieving superior performance.

Additional Analysis We analyze the model efficiency in the Appendix. B, showing MiTSformer maintains great performance and efficiency compared with most baselines. In addition, we investigate hyperparameter sensitivity in the Appendix C. The results demonstrate that (1) MiTSformer is relatively stable in the selection of odel capacity-related hyper-parameters d model and number of layers L ; (2) MiTSformer is quite robust to the weights of loss items (i.e., λ 1 , λ 1 , and, λ 3 ), and moderate weights bring optimal performance.

5 Conclusion and Future Work

This paper focuses on a challenging yet seldomly explored problem in the time series community and provides a systematic and universal solution for mixed time series analysis. To address the spatial-temporal heterogeneity problem, we first reveal the LCVs behind DVs and try to recover them for sufficient and balanced spatial-temporal modeling. Accordingly, MiTSformer is developed as a task-general mixed time series analysis framework by leveraging the temporal similarities and spatial interactions between LCVs and CVs. MiTSformer can perform adaptive LCV recovery for DVs via the adversarial guidance of CVs as well as smooth constraints, and model complete spatial-temporal dependencies via self- and cross-attention blocks. With extensive empirical evaluations, MiTSformer shows great practicality, superiority, and versatility in five mainstream mixed time series analysis tasks. In the future, it is of interest to empower MiTSformer with advanced pre-training techniques and powerful large language models for broader applications of mixed time series.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No. 62450020 and No. 62125306), and Zhejiang Key Research and Development Project (No. 2024C01163).

References

Cdc. illness. https://gis.cdc.gov/grasp/fluview/fluportaldashboard. html.

Uci. electricity. https://archive.ics.uci.edu/ml/datasets/ electricityloaddiagrams20112014/.

Wetterstation. weather. https://www.bgc-jena.mpg.de/wetter/.

Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. Practical approach to asynchronous multivariate time series anomaly detection and localization. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining , pages 2485-2494, 2021.

Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075 , 2018.

Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 , 2018.

Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium-range global weather forecasting with 3d neural networks. Nature , 619(7970):533-538, 2023.

David Campos, Miao Zhang, Bin Yang, Tung Kieu, Chenjuan Guo, and Christian S Jensen. Lightts: Lightweight time series classification with adaptive ensemble distillation. Proceedings of the ACM on Management of Data , 1(2):1-27, 2023.

Alberto Castellini, Francesco Masillo, Davide Azzalini, Francesco Amigoni, and Alessandro Farinelli. Adversarial data augmentation for hmm-based anomaly detection. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Jiawei Chen, Pengyu Song, and Chunhui Zhao. Multi-scale self-supervised representation learning with temporal alignment for multi-rate time series modeling. Pattern Recognition , 145:109943, 2024.

Jiawei Chen, Pengyu Song, Chunhui Zhao, and Jinliang Ding. Spatiotemporal multiscale correlation embedding with process variable reorder for industrial soft sensing. IEEE Transactions on Instrumentation and Measurement , 2023.

Junhao Chen, Chunhui Zhao, and Jinliang Ding. A flexible probabilistic framework with concurrent analysis of continuous and categorical data for industrial fault detection and diagnosis. IEEE Transactions on Industrial Informatics , 2023.

Kochenderfer M J. Chen Y C, Wheeler T A. Learning discrete bayesian networks from continuous data. In Journal of Artificial Intelligence Research , pages 59: 103-132, 2017.

Kien Do, Truyen Tran, and Svetha Venkatesh. Multilevel anomaly detection for mixed data. arXiv preprint arXiv:1610.06249 , 2016.

Kien Do, Truyen Tran, and Svetha Venkatesh. Energy-based anomaly detection for mixed data. Knowledge and Information Systems , 57(2):413-435, 2018.

Luo donghao and wang xue. ModernTCN: A modern pure convolution structure for general time series analysis. In The Twelfth International Conference on Learning Representations , 2024.

Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee-Keong Kwoh, Xiaoli Li, and Cuntai Guan. Self-supervised contrastive representation learning for semi-supervised time-series classification. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Liangjun Feng and Chunhui Zhao. Fault description based attribute transfer for zero-sample industrial fault diagnosis. IEEE Transactions on Industrial Informatics , 17(3):1852-1862, 2021.

Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning , pages 1180-1189. PMLR, 2015.

Rong Hu, Ling Chen, Shenghuan Miao, and Xing Tang. Swl-adapt: An unsupervised domain adaptation model with sample weight learning for cross-user wearable human activity recognition. Proceedings of the AAAI Conference on Artificial Intelligence , 2023.

Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Soderstrom. Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 387-395, 2018.

Yuxin Jia, Youfang Lin, Xinyan Hao, Yan Lin, Shengnan Guo, and Huaiyu Wan. Witran: Water-wave information transmission and recurrent acceleration network for long-range time series forecasting. Advances in Neural Information Processing Systems , 36, 2024.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval , pages 95-104, 2018.

Qiude Li, Qingyu Xiong, Shengfen Ji, Yang Yu, Chao Wu, and Hualing Yi. A method for mixed data classification base on rbf-elm network. Neurocomputing , 431:7-22, 2021.

Shengsheng Lin, Weiwei Lin, Wentai Wu, Feiyu Zhao, Ruichao Mo, and Haotong Zhang. Segrnn: Segment recurrent neural network for long-term time series forecasting. arXiv preprint arXiv:2308.11200 , 2023.

Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference on learning representations , 2022.

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations , 2024.

Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems , 35:9881-9893, 2022.

Aditya P Mathur and Nils Ole Tippenhauer. Swat: A water treatment testbed for research and training on ics security. In 2016 international workshop on cyber-physical systems for smart water networks (CySWater) , pages 31-36. IEEE, 2016.

Zelin Ni, Hang Yu, Shizhan Liu, Jianguo Li, and Weiyao Lin. Basisformer: Attention-based time series forecasting with learnable and interpretable basis. Advances in Neural Information Processing Systems , 36, 2024.

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations , 2023.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog , 2019.

Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 2828-2837, 2019.

Chang Wei Tan, Christoph Bergmeir, François Petitjean, and Geoffrey I Webb. Time series extrinsic regression: Predicting numeric values from time series data. Data Mining and Knowledge Discovery , 35(3):1032-1060, 2021.

Giulio Ventura and Elena Benvenuti. Advances in Discretization Methods: Discontinuities, Virtual Elements, Fictitious Domain Methods , volume 12. Springer, 2016.

Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. Micn: Multi-scale local and global context modeling for long-term series forecasting. In The Eleventh International Conference on Learning Representations , 2023.

Min Wang, Donghua Zhou, and Maoyin Chen. Anomaly monitoring of nonstationary processes with continuous and two-valued variables. IEEE Transactions on Systems, Man, and Cybernetics: Systems , 53(1):49-58, 2022.

Min Wang, Donghua Zhou, and Maoyin Chen. Hybrid variable monitoring: An unsupervised process monitoring framework with binary and continuous variables. Automatica , 147:110670, 2023.

Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y Zhang, and JUN ZHOU. Timemixer: Decomposable multiscale mixing for time series forecasting. In The Twelfth International Conference on Learning Representations , 2024.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations , 2023.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems , 34:22419-22430, 2021.

Haixu Wu, Hang Zhou, Mingsheng Long, and Jianmin Wang. Interpretable weather forecasting for worldwide stations with a unified deep model. Nature Machine Intelligence , 5(6):602-611, 2023.

Wanke Yu, Chunhui Zhao, and Biao Huang. Moninet with concurrent analytics of temporal and spatial information for fault detection in industrial processes. IEEE Transactions on Cybernetics , 52(8):8340-8351, 2022.

Jiaqi Yue, Jiancheng Zhao, and Chunhui Zhao. Similarity makes difference: Sshtn for generalized zero-shot industrial fault diagnosis by leveraging auxiliary set. IEEE Transactions on Industrial Informatics , 2024.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 11121-11128, 2023.

Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations , 2023.

Chunhui Zhao. Perspectives on nonstationary process monitoring in the era of industrial artificial intelligence. Journal of Process Control , 116:255-272, 2022.

Chunhui Zhao, Junhao Chen, and Hua Jing. Condition-driven data analytics and monitoring for wide-range nonstationary and transient continuous processes. IEEE Transactions on Automation Science and Engineering , 18(4):1563-1574, 2021.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 11106-11115, 2021.

Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, et al. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances in Neural Information Processing Systems , 35:12677-12690, 2022.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning , pages 27268-27286. PMLR, 2022.

Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained lm. Advances in neural information processing systems , 36, 2024.

A Implementation Details

A.1 Datasets and Experimental Platforms

Despite being a fundamental issue, modeling mixed time series remains underexplored in academia, with a lack of specialized benchmark datasets for mixed time series. To meet the tasks for mixed time series, we employed benchmark time-series datasets and implemented discretization to convert some CVs into DVs. Our conversion process simulates the generation process of DVs and discretizes variables while preserving their inherent coupling relationships and properties. We summarized the dataset descriptions in Table 5. All experiments are repeated three times, implemented in PyTorch 2 and conducted on Linux servers with Intel(R) Xeon(R) Gold 6246 CPUs and NVIDIA 3090 24GB GPUs. The versions of Python and Pytorch are 3.9.7, and 1.10.0 respectively.

A.2 Hyperparameters and Experimental Configurations

All the baselines that we reproduced are implemented based on their official codes or Time Series Library (TSlib) 3 . Since some baseline models are designed for specific analysis tasks, we modify them to serve different analysis tasks by replacing task heads and loss functions. Specifically, we keep the original backbone architecture for each method as the feature extractor, and we adopt universal task-specific heads and loss functions consistently for all methods. The architectures of task heads and related loss functions will be introduced in the following section.

The hyper-parameter configurations of MiTSformer are summarized in Table 6 and the hyperparameter configurations of baseline models are summarized in Table 7. Some hyperparameters, including batch size, training epochs, dropout rate, and the number of attention heads (transformerbased models only) are fixed and kept the same for all baseline models and MiTSformer. Moreover, to compare the upper bound of different models specifically for mixed time series analysis tasks, we conduct grid searches of model capacity-related hyper-parameters, including number of layers and d model / hidden size and optimization-related hyper-parameter-learning rate. Compared with baseline models, MiTSformer additionally fine-tuned the weight of smoothness loss weight λ 1 to adapt to different temporal variations of different datasets. The variable modality discrimination loss weight is fixed at 1 . 0 and the reconstruction loss weight is fixed at 1 . 0 for all experiments.

A.3 Algorithm Implementations of MiTSformer

Mixed time series modeling differs significantly from typical time series tasks due to variable heterogeneity. Thereby, each component in MiTSformer is essential to address it via latent continuity recovery and alignment. We summarized the model feed-forward, loss-calculation, and parameterupdate procedures of MiTSformer with pseudo-codes, which are presented in Algorithm 1. We also report the standard deviation of MiTSformer performance with different random seeds in Table 8, which exhibits that the performance of MiTSformer is stable. Here we introduce the detailed architectures of each module in MiTSformer.

Recovery Network The recovery network, receiving the input of DVs X D with shape (Batchsize × n × T ) and outputting LCVs X LCV with shape (Batchsize × n × T ) , is composed of 3 dilated convolution blocks with residual connections. Specifically, the l th dilated convolution block is composed of 2 dilated 1D convolutional layers with kernel _ size = 2 , hidden _ channel = 8 , dilation = 2 l and GELU activations. Additionally, padding operation Padding = ((kernel _ size -1) × dilation + 1) // 2 is adopted for the convolutional layer to ensure that the input and output have the same length T , where (kernel _ size -1) × dilation + 1 denotes the receptive field). Finally, we perform z-score normalization for each x LC ∈ R 1 × T to ensure training stability as

where Mean and Std denote the mean and the standard deviation along the time axis, respectively.

Self-attention and Cross-attention Sub-blocks The self-attention sub-block is composed of vanilla FullAttention with 8 attention heads and 0.1 dropout rate. The same for the cross-attention sub-block.

† We set the batch size to 16 for the PEMS-SF dataset due to its high dimensionality. We set the batch size to 64 for other classification datasets. The same for Table .7

Variable Modality Discriminator The variable modality discriminator, receiving the input variate embeddings with shape (Batchsize × p × d model ) and outputting the variable modality class with shape (Batchsize × p × 2) , is composed of 3-layer MLP with structure of { d _ model , 4 ∗ d _ model , 4 ∗ d _ model , 2 } . Relu activation and batch normalization are also adopted.

Reconstruction Decoders The discrete variable reconstruction decoder, receiving the input of LCV embeddings (Batchsize × n × d model ) and outputting the reconstructed DVs (Batchsize × n × T × 2) , is composed of a linear layer with input _ size = d _ model , output _ size = 2 ∗ T (the discrete variable reconstruction can be treated as a binary classification task). The continuous variable reconstruction decoder, receiving the input of CV embeddings (Batchsize × ( p -n ) × d model ) and outputting the reconstructed CVs (Batchsize × ( p -n ) × T ) , is composed of a linear layer with input _ size = d _ model , output _ size = T .

In the following, we will introduce the architectures of task heads and loss functions for different downstream tasks.

A.4 Pipeline for Classification

Classification is a long-standing task in the time series community and is widely used to evaluate the high-level representation capacity of models. The overall pipeline of MiTSformer-based classification is depicted in Fig. 10. For the classification task, both token embeddings of LCVs and CVs provide complementary information. Thereby, we concatenate the embeddings of LCVs and CVs. The fused embeddings are flattened and fed into a classifier network to predict the class labels. For MiTSformer and baseline models, the classifier is composed of a single-layer MLP with GELU activation and

Algorithm 1 The training process of MiTSformer

Input: Input mixed time series with continuous variables X C = { x 1 , x 2 , ..., x p -n } ∈ R ( p -n ) × T , and discrete variables X D = { x 1 , x 2 , ..., x n } ∈ n × T , number of attention blocks L , loss weights λ 1 , λ 2 and λ 3 .

Output: Optimized model parameters of MiTSformer.

1: Initialize the parameters of MiTSformer;

2: while not converge do

3: (1) Feedforward Computation:

4: ▷ Latent Continuity Recovery for DVs

5: x LC ← Rec -Net( x D ) ;

6: ▷ Variate-wise Token Embedding of LCVs and CVs

8: z C 0 ← Embed CV ( x C )

9: ▷ Interand Intra-Variable-Modality Spatial-Temporal Modeling

10: for l = [0 , 1 , ..., L -1 ] do

11: ▷ Intra-Variable-Modality Self-Attention Sub-Block

16: ▷ Inter-Variable-Modality Cross-Attention Sub-Block

18:

z

C

(

C

(

C

))

LN

z

+FFN

z

21: end for

22: ▷ Task Prediction (if needed)

23: ˆ y ← Task -Heads( z LC L , z C L )

24: (2) Loss Calculation:

25: ▷ Task Supervision Loss (if needed)

26: L Task ← Task -Criterion(ˆ y, y ) .

27: ▷ Smoothness Constraint Loss

28: L smooth = ∥ ∥ Abs ( S x D ) ⊗ ( S x LC )∥ ∥ 2 2 .

29: ▷ Reconstruction Loss

30: L Rec = ∑ p -n i =1 MSE(Rec -Decoder ( z C L,i ) , x C i ) + ∑ n i =1 CE(Rec -Decoder ( z LC L,i ) , x D i ) .

31: ▷ Variable Modality Discrimination Loss (via Gradient Reverse Layer)

32: L Dis = E [ log ( GRL(Dis ( z C )) ) ] + E [ log ( 1 -GRL(Dis ( z LC )) ) ]

33: ▷ Overall Loss

34: L All = L Task + λ 1 L Smooth + λ 2 L Rec + λ 3 L Dis

35: (3) Parameter Update:

36: Update the parameters of models using Adam optimizer to minimize L all ;

37: end while

38: return Optimized model parameters.

a dropout layer with a dropout rate of 0.1. The task supervision loss is Cross-Entropy Loss. The classification accuracy is used as the performance evaluation metric.

Concat

Classifier

Fused Embeddings

Output

Reconstruct

Decoder

Reconstruction

Decoder

GeLU Act.

Drop Out

Linear

Observed DVs

Observed CVs

MiTSformer

Backbone

A.5 Pipeline for Extrinsic Regression

Closely related to classification, time series extrinsic regression aims to learn the relationship between a time series and a continuous scalar variable. The overall pipeline of MiTSformer-based extrinsic regression is depicted in Fig. 11. Similar to classification, both token embeddings of LCVs and CVs provide complementary information for regression tasks. Thereby, we concatenate the embeddings of LCVs and CVs. The fused embeddings are flattened and fed into a regressor network to predict the numerical values. For MiTSformer and baseline models, the regressor is composed of a single-layer MLP with GELU activation and a dropout layer with a dropout rate of 0.1. The task supervision loss is MSE Loss. We adopt the mean absolute error (RMAE) and the root mean square error (RMSE) as the performance evaluation metrics as

where N y denotes the number of test samples, ˆ y i denotes the predicted labels and y i denotes the ground-truth label.

Concat

Reconstruct

Decoder

Reconstruction

Decoder

Observed DVs

Observed CVs

Regressor

GeLU Act.

Drop Out

Linear

Output

Fused Embeddings

MiTSformer

Backbone

A.6 Pipeline for Imputation

The imputation techniques are developed to impute the missing values based on the partially observed time points. For the imputation task, we mainly focus on imputating the missing value of CVs, while DVs are adopted as input to provide auxiliary information. The overall pipeline of MiTSformer-based imputation is depicted in Fig. 11. For MiTSformer, the task head is an imputation decoder composed of a linear layer, which performs variate-wise imputation for each CV. For baseline methods that adopt the channel-independent strategy, we adopt the same task head as MiTSformer. For baseline methods that adopt the channel-mixing strategy, we adopt a linear layer with input size of d model and output size of continuous variable dimension p -n to reconstruct the CVs. We use a mask matrix M ∈ R T × ( p -n ) to represent the missing values in input CVs. The state of M denotes whether the corresponding element value of CVs is missing (denoted by 0) or not (denoted by 1). The task supervision loss is the MSE loss calculated on the masked observations. Specifically for MiTSformer, the reconstruction loss of CVs is calculated on unmasked observations. We adopt the mean absolute error (MAE) and the root mean square error (RMSE) as the performance evaluation metrics that are computed on masked elements.

Imputation

Decoder

Linear

Reconstruct

Decoder

Reconstruction

Decoder

Observed DVs

Observed CVs

Imputed CVs

MiTSformer

Backbone

Features

Variate

Features

Variate

A.7 Pipeline for Long-term Forecasting

Forecasting is a fundamental problem in the time series community, and long-term forecasting is a more practical and challenging task. The long-term forecasting task for MiTS includes the prediction of both DVs and CVs. The overall pipeline of MiTSformer-based long-term forecasting is depicted in Fig. 13. For MiTSformer, the task head is composed of linear layer-based DVforecastor and CVforecaster, which perform variate-wise forecasting for each DV and each CV, respectively. For baseline models, we also adopt the linear layer-based forecaster to forecast the DVs with the extracted temporal features. We adopt the mean absolute error (MAE) and the root mean square error (RMSE) as the forecasting performance evaluation metrics.

DVForecaster

Future DVs

Future CVs

CVForecaster

Linear

Linear

Reconstruct

Decoder

Reconstruction

Decoder

Observed DVs

Observed CVs

MiTSformer

Backbone

TL

A.8 Pipeline for Anomaly Detection

The anomaly detection task is achieved by self-supervised autoencoding with the reconstruction of both CVs and DVs. The reconstruction errors of DVs and DVs are utilized as anomaly criteria. In our experiment, we estimate the anomaly thresholds respectively for DVs and CVs. For a test sample, the reconstruction error of DVs exceeding the threshold or the reconstruction error of DVs exceeding the threshold is considered an anomaly. The overall pipeline of MiTSformer-based anomaly detection is depicted in Fig. 13. Since there already exist reconstruction decoders and corresponding losses, no task head and additional losses are devised. For baseline models, we also adopt the linear layer-based reconstruct decoders to reconstruct the DVs and linear layer-based reconstruct decoders to reconstruct the CVs with the extracted temporal features. MSE loss is used for reconstruction of CVs and CrossEntropy loss is used for reconstruction of DVs. We adopt the Precision, Recall and F1-Socre as matrices.

Reconstruct

Decoder

Reconstruction

Decoder

Observed DVs

Observed CVs

MiTSformer

Backbone

口

B Model Efficiency Analysis

Complexity of MiTSformer Considering the self-attention and cross-attention sub-blocks, the complexity of MiTSformer can be derived as

where n denotes the number of DVs and p -n denotes the number of CVs, p is the number of total variables. In our experimental setting, we keep n = ⌊ 0 . 5 p ⌋ . Thereby, the complexity of MiTSformer is quadratic to the number of DVs as O ( n 2 ) .







































































































(a) ETTh1 (3DVs, 4CVs)































































































(b) Electricity (160DVs, 161CVs)

Model Efficiency Comparison We comprehensively compare the model performance, training speed, and memory footprint of MiTSformer and baseline models. The results are with the optimal model configurations as reported in Table 6 and 7. Two representative long-term forecasting datasets ETTh1 (3DVs, 4CVs) and Electricity (160DVs, 161CVs) with input-96 and output-720 settings are adopted for efficiency comparison. The results are depicted in Fig. 15.

In general, MiTSformer maintains great performance and efficiency compared with most baselines in datasets with a relatively small number of variables (ETTh1). When encountering datasets with a relatively large number of variables (Electricity), MiTSformer occupies a relatively large memory footprint, specifically compared with some advanced efficient models such as modern TCN and DLinear. However, the training time of MiTSformer is still efficient.

C Hyper-parameter Sensitivity Analysis

We evaluate the hyper-parameter sensitivity of MiTSformer with respect to the following two aspects:

Sensitivity of Model Capacity The hidden dimension d model and number of layers L influence the model capacity of MiTSformer. We evaluate the sensitivity of these two hyper-parameters on typical long-term forecasting datasets, including ETTh1, ETTh2, and Weather with four different prediction horizon settings. The results are presented in Fig. 16. We can find that MiTSformer is relatively stable in the selection of d model and L , particularly for ETTh2 and Weather datasets. Specifically for the ETTh1 dataset, the model capacities are not essentially favored to be as large as possible.

Sensitivity of Loss Functions We further investigate the effects of loss items, including smoothness loss weight λ 1 , reconstruction loss weight λ 2 , and variable modality discrimination loss weight λ 3 on classification datasets (JapaneseVowels, SpokenArabicDigits, and SelfRegulationSCP1). The results are presented in Fig. 17. In general, we can observe that MiTSformer is quite robust to the weights of loss items, and moderate weights bring optimal performance. For example, λ 1 controls the magnitude of the smoothing constraints for latent continuity recovery. Too small λ 1 would make the smoothness invalid while too large λ 1 may lead to over-smoothing problems. Therefore, a moderate setting of λ 1 is favored for MiTSforemer.

D Limitations and Future Work

Handling Discrete Variables with Natural Sudden Changes The smoothness loss in our method is suitable for DVs with sudden changes that are caused by inherent smooth variations. However, it may not adequately account for DVs with inherent sudden changes that are essential characteristics of the dataset. For such cases, we can adjust the coefficient λ 1 with a relatively small value, or we can

d_model

32

64

128

256

Layers

1

2

3

4

MAE

0.350

0.355

0.360

0.365

0.370

0.375

0.380

0.385

0.390

0.384

0.386

0.388

0.390

(a) ETTh1

96

-

96

d_model

32

64

128

256

Layers

1

2

3

4

MAE

0.380

0.385

0.390

0.395

0.400

0.405

0.410

0.415

0.420

0.410

0.412

0.414

0.416

(b) ETTh1

96

-

192

d_model

32

64

128

256

Layers

1

2

3

4

MAE

0.400

0.405

0.410

0.415

0.420

0.425

0.430

0.435

0.440

0.426

0.428

0.430

0.432

0.434

0.436

0.438

(c) ETTh1

96

-

336

d_model

32

64

128

256

Layers

1

2

3

4

MAE

0.40

0.41

0.42

0.43

0.44

0.45

0.46

0.47

0.48

0.4600

0.4625

0.4650

0.4675

0.4700

0.4725

0.4750

0.4775

(d) ETTh1

96

-

720

d_model

32

64

128

256

Layers

1

2

3

4

MAE

0.30

0.32

0.34

0.36

0.38

0.3800

0.3825

0.3850

0.3875

0.3900

0.3925

0.3950

0.3975

(e) ETTh2

96

-

96

d_model

32

64

128

256

Layers

1

2

3

4

MAE

0.38

0.39

0.40

0.41

0.42

0.43

0.44

0.430

0.432

0.434

0.436

(f) ETTh2

96

-

192

d_model

32

64

128

256

Layers

1

2

3

4

MAE

0.40

0.41

0.42

0.43

0.44

0.45

0.46

0.47

0.458

0.460

0.462

0.464

0.466

(g) ETTh2

96

-

336

d_model

32

64

128

256

Layers

1

2

3

4

MAE

0.40

0.41

0.42

0.43

0.44

0.45

0.46

0.47

0.48

0.4600

0.4625

0.4650

0.4675

0.4700

0.4725

0.4750

0.4775

(h) ETTh2

96

-

720

d_model

32

64

128

256

Layers

1

2

3

4

MAE

0.200

0.205

0.210

0.215

0.220

0.225

0.230

0.235

0.231

0.232

0.233

0.234

(i) Weather

96

-

96

d_model

32

64

128

256

Layers

1

2

3

4

MAE

0.25

0.26

0.27

0.28

0.29

0.30

0.2925

0.2930

0.2935

0.2940

0.2945

0.2950

0.2955

(j) Weather

96

-

192

d_model

32

64

128

256

Layers

1

2

3

4

MAE

0.30

0.31

0.32

0.33

0.34

0.35

0.35300

0.35325

0.35350

0.35375

0.35400

0.35425

0.35450

0.35475

(k) Weather

96

-

336

d_model

32

64

128

256

Layers

1

2

3

4

MAE

0.30

0.32

0.34

0.36

0.38

0.40

0.42

0.4260

0.4265

0.4270

0.4275

0.4280

(l) Weather

96

-

720

0.1

0.2

0.3

0.5

0.8

Weight of Smoothness Loss

1

90.0

92.5

95.0

97.5

100.0

Accuracy (%)

JapaneseVowels

SelfRegulationSCP1

SpokenArabicDigits

(a) Sensitivity of

λ

1

0.1

0.5

1.0

5.0

10

Weight of Reconstruction Loss

2

90.0

92.5

95.0

97.5

100.0

Accuracy (%)

JapaneseVowels

SelfRegulationSCP1

SpokenArabicDigits

(b) Sensitivity of

λ

2

0.1

0.5

1.0

5.0

10

Weight of Discrimination Loss

3

90.0

92.5

95.0

97.5

100.0

Accuracy (%)

JapaneseVowels

SelfRegulationSCP1

SpokenArabicDigits

(c) Sensitivity of

λ

3

further leverage less restrictive constraints such as K-Lipschitz continuity to determine the proper smoothness level.

Modeling Generalized Mixed Time Series MiTSformer is based on the assumption that the observed discrete variables are derived from latent continuous variables, which can be guaranteed for integer discrete variables whose discrete states represent the magnitude of the numerical value. However, a limitation of MiTSformer is that it can not be directly applied to categorical discrete variables whose discrete states represent different and mutually exclusive class labels (e.g., color and gender). It is noted that integer discrete variables are more frequently encountered in real-world time series data, specifically in the fields of medical analysis, finance, and industrial processes. In contrast, categorical discrete variables are often recorded as independent samples that are not presented in time series forms. In the future, it is of interest to exploit the analysis of generalized variables (including both integer discrete variables and categorical discrete variables) by leveraging advanced cross-modality modeling and fusion technologies.

Incorperating Powerful LLMs and Pre-training Techniques Recently, large language models, such as the Generative Pre-trained Transformer (GPT), have shown impressive performance on various applications and attracted numerous research interests. It is intriguing to explore the potential of GPT-type architectures for mixed time series analysis tasks with proper inductive biases and elaborate prompt strategies. Additionally, by utilizing efficient pre-training techniques, we can enhance the scalability of MiTSformer in large-scale or real-time real-world applications. Meanwhile, the mixed time series analysis on few-shot and zero-shot settings can also be a promising research direction.

E Additional Experiments and Discussions

E.1 Comparison with Mixed Naive Bayes and Variational Inference-based methods

To deal with mixed variables, naive Bayes (NB)-based models [38] and variational inference (VI)based models [12] are proposed to match the distributions of continuous variables (CVs) and discrete variables (DVs) with different distribution priors for industrial process data analysis. Essentially, mixed NB and VI methods are typically designed for tabular data. They struggle with time series and often rely on certain assumptions like conditional independence, limiting their ability to model the correlations between DVs and CVs. In contrast, MiTSformer leverages temporal adjacencies to achieve latent continuity recovery, making it capable of handling time series data and effectively capturing inherent nonlinear correlations within MiTS. Empirically, we compare MiTSformer against typical mixed NB-based models HVM [1] and VI-based methods VAMDA [2] on mixed time series classification datasets and summarized the results in Table 9, showing MiTSformer consistently outperforms HVM and VAMDA.

E.2 Investigation of Non-binary Discrete Variable

In our main experimental setting, we chose the binary DVs as they are the most challenging and commonly encountered in the real world. Actually, DVs can take on multiple states ( ⩾ 2) that reflect the magnitude and can be directly input into the recovery network to obtain LCVs outputs. It is noted

that more states in a DV imply richer information granularity. Also, we conducted experiments on mixed time series classification and anomaly detection datasets with various numbers of discrete states in Tabel 10 and Table 11, respectively. In general, the results shows that as the number of states increases, performance improves due to the richer information.

Kindly note: For dataset abbreviations,'Appli.' denotes 'AppliancesEnergy'; 'House.1' denotes 'HouseholdPowerConsumption1'; 'House.1' denotes 'HouseholdPowerConsumption2'; 'Benze.' denotes 'BenzeneConcentration'; 'BJ.PM25 ' denotes 'BeijingPM25Quality'; 'BJ.PM10 ' denotes 'BeijingPM10Quality'; 'Live.' denotes 'LiveFuelMoistureContent'; 'Austra.' denotes 'AustraliaRainfall'; 'PPGD.' denotes 'PPGDalia'; and 'IEEE.' denotes 'IEEEPPG'.

F Full Experimental Results

The full results of all experiments are presented in the following: classification in Table 12, extrinsic regression in Table 13, imputation in Table 14, anomaly detection in Table 15. Besides, long-term forecasting results of CVs are presented in Table 16 and those of DVs are presented in 17.

Models

MiTSformer

(Ours)

iTransformer

(2024)

M-TCN

(2024)

TimesNet

(2023)

PatchTST

(2023)

Crossformer

(2023)

MICN

(2023)

LightTS

(2023)

Dlinear

(2023)

FiLM

(2022)

FEDformer

(2022)

Pyraformer

(2022)

M-Ratio

MAE MSE

MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE

ETTm1

12.5%

0.143

0.042

0.149

0.046

0.123 0.031

0.119 0.029

0.151 0.048 0.150 0.045

0.137 0.038 0.156 0.050 0.159 0.053 0.161 0.054 0.144 0.046 0.178 0.063

25%

0.152

0.047

0.162

0.053

0.130 0.034

0.131 0.035 0.159 0.053 0.155 0.048

0.156 0.049 0.172 0.058 0.182 0.069 0.183 0.070 0.161 0.055 0.192 0.080

37.5%

0.160

0.051

0.175

0.060

0.138 0.038

0.140 0.040 0.167 0.058 0.162 0.051

0.169 0.057 0.185 0.068 0.203 0.087 0.204 0.087 0.172 0.059 0.204 0.085

50%

0.170

0.057

0.190

0.070

0.149 0.043

0.164 0.052 0.180 0.067 0.172 0.058

0.178 0.065 0.200 0.079 0.226 0.110 0.227 0.110 0.202 0.079 0.222 0.096

0.156

0.049

0.169

0.057

0.135 0.037

0.139 0.039 0.164 0.057 0.160 0.051

0.160 0.052 0.178 0.064 0.193 0.080 0.194 0.080 0.170 0.060 0.199 0.081

ETTm2

12.5%

0.104

0.030

0.163

0.062

0.169 0.064 0.134 0.040 0.132 0.046 0.194 0.080

0.174 0.066 0.197 0.087 0.196 0.086 0.211 0.100 0.178 0.065 0.234 0.105

25%

0.112

0.034

0.182

0.076

0.178 0.068 0.159 0.055 0.138 0.050 0.211 0.091

0.224 0.104 0.205 0.093 0.245 0.135 0.246 0.136 0.211 0.092 0.246 0.121

37.5%

0.120

0.038

0.195

0.087

0.193 0.079 0.180 0.072 0.147 0.056 0.225 0.113

0.268 0.151 0.213 0.098 0.272 0.165 0.274 0.169 0.258 0.140 0.279 0.159

50%

0.128

0.043

0.205

0.096

0.212 0.095 0.205 0.092 0.162 0.066 0.232 0.128

0.313 0.203 0.219 0.102 0.300 0.200 0.302 0.203 0.303 0.193 0.302 0.179

0.116

0.036

0.186

0.080

0.188 0.077 0.170 0.065 0.145 0.055 0.216 0.103

0.245 0.131 0.209 0.095 0.253 0.147 0.258 0.152 0.238 0.123 0.265 0.141

ETTh1

12.5%

0.203

0.080

0.213

0.091

0.190 0.070

0.201 0.078 0.221 0.100 0.225 0.104 0.196

.077

0.239 0.114 0.223 0.103 0.231 0.109

.201

0.075 0.214 0.085

25%

0.217

0.091

0.232

0.108

0.204 0.077

0.222 0.097 0.241 0.118 0.240 0.116

0.221 0.095 0.254 0.131 0.250 0.130 0.256 0.135 0.226 0.095 0.227 0.095

37.5%

0.230

0.100

0.250

0.124

0.223 0.093

0.247 0.119 0.259 0.136 0.251 0.126

0.247 0.115 0.274 0.155 0.275 0.156 0.279 0.165 0.257 0.122 0.254 0.118

50%

0.243

0.111

0.269

0.142

0.244 0.126 0.279 0.152 0.281 0.163 0.266 0.139

0.272 0.139 0.296 0.181 0.300 0.192 0.304 0.198 0.292 0.157 0.291 0.152

0.223

0.096

0.241

0.116

0.215 0.092

0.237 0.112 0.251 0.129 0.246 0.121

0.234 0.107 0.266 0.145 0.262 0.145 0.268 0.152 0.244 0.112 0.247 0.113

ETTh2

12.5%

0.175

0.073

0.224

0.109

0.259 0.148 0.234 0.120 0.237 0.122 0.277 0.163

0.246 0.129 0.290 0.180 0.255 0.140 0.271 0.163 0.279 0.158 0.328 0.195

25%

0.182

0.079

0.242

0.125

0.293 0.186 0.300 0.201 0.253 0.140 0.303 0.195

0.286 0.170 0.308 0.202 0.284 0.168 0.317 0.220 0.325 0.209 0.342 0.226

37.5%

0.189

0.085

0.258

0.141

0.325 0.230 0.332 0.243 0.268 0.158 0.311 0.206

0.338 0.239 0.322 0.220 0.340 0.246 0.365 0.291 0.370 0.276 0.383 0.269

50%

0.199

0.093

0.276

0.160

0.360 0.287 0.408 0.390 0.282 0.173 0.325 0.226

0.403 0.343 0.335 0.235 0.368 0.289 0.413 0.372 0.424 0.390 0.415 0.320

0.186

0.083

0.250

0.134

0.309 0.213 0.319 0.239 0.260 0.148 0.304 0.198

0.318 0.220 0.314 0.209 0.312 0.211 0.342 0.262 0.350 0.258 0.367 0.253

Electricity

12.5%

0.154

0.055

0.184

0.071

0.178 0.065 0.189 0.075 0.180 0.067 0.181 0.066

0.198 0.077 0.191 0.075 0.206 0.086 0.206 0.085 0.221 0.093 0.259 0.130

25%

0.177

0.069

0.204

0.086

0.200 0.082 0.204 0.087 0.194 0.079 0.193 0.075

0.220 0.093 0.214 0.091 0.243 0.115 0.243 0.115 0.250 0.117 0.271 0.143

37.5%

0.195

0.082

0.221

0.099

0.214 0.093 0.216 0.099 0.207 0.091 0.205 0.117

0.240 0.111 0.233 0.106 0.273 0.144 0.274 0.134 0.276 0.142 0.279 0.157

50%

0.216

0.097

0.239

0.114

0.236 0.113 0.233 0.115 0.231 0.222 0.257 0.131

0.259 0.131 0.252 0.122 0.304 0.175 0.304 0.176 0.292 0.169 0.288 0.169

0.186

0.076

0.212

0.093

0.207 0.088 0.211 0.094 0.203 0.115 0.209 0.097

0.229 0.103 0.223 0.099 0.257 0.130 0.257 0.128 0.260 0.130 0.274 0.150

Weather

12.5%

0.055

0.029

0.080

0.033

0.074 0.030 0.093 0.035 0.083 0.034 0.116 0.044

0.094 0.035 0.095 0.036 0.097 0.037 0.100 0.038 0.103 0.039 0.082 0.038

25%

0.060

0.030

0.088

0.036

0.081 0.032 0.136 0.051 0.085 0.035 0.111 0.041

0.122 0.047 0.107 0.040 0.116 0.045 0.118 0.046 0.119 0.045 0.093 0.041

37.5%

0.065

0.032

0.095

0.039

0.083 0.033 0.173 0.074 0.088 0.037 0.114 0.043

0.148 0.060 0.106 0.041 0.129 0.051 0.131 0.052 0.144 0.058 0.105 0.042

50%

0.069

0.034

0.102

0.042

0.087 0.035 0.195 0.098 0.097 0.041 0.119 0.045

0.176 0.076 0.118 0.046 0.143 0.059 0.145 0.600 0.188 0.086 0.118 0.047

0.062

0.031

0.091

0.038

0.081 0.033 0.149 0.065 0.088 0.037 0.115 0.043

0.135 0.055 0.107 0.041 0.121 0.048 0.124 0.184 0.139 0.057 0.100 0.042

0.155

0.062

0.192

0.086

0.189 0.090 0.204 0.102 0.185 0.090 0.208 0.102

0.220 0.111 0.216 0.109 0.233 0.127 0.240 0.160 0.233 0.123 0.242 0.130

Top1 Counts.

34

0

12

2

0

0

0

0

0

1

0

0

Models

MiTSformer

(Ours)

iTransformer

(2024)

M-TCN

(2024)

TimesNet

(2023)

PatchTST

(2023)

Crossformer

(2023)

MICN

(2023)

LightTS

(2023)

Dlinear

(2023)

FiLM

(2022)

FEDformer

(2022)

Pyraformer

(2022)

Length

MAE MSE

MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE

ETTm1

96

0.342

0.271

0.348

0.279

0.346 0.272 0.363 0.288 0.346 0.276 0.368 0.296

0.352 0.275 0.364 0.296 0.344 0.275 0.362 0.297 0.398 0.319 0.443 0.408

192

0.363

0.310

0.371

0.321

0.367 0.314 0.400 0.351 0.367 0.313 0.401 0.349

0.369 0.321 0.382 0.331 0.368 0.312 0.392 0.348 0.415 0.356 0.467 0.441

336

0.382

0.338

0.391

0.347

0.387 0.344 0.413 0.371 0.387 0.342 0.416 0.368 0.389

0.328

0.401 0.358 0.388 0.336 0.404 0.367 0.436 0.393 0.492 0.479

720

0.415

0.391

0.430

0.414

0.421 0.404 0.437 0.416 0.416

0.389

0.442 0.417

0.429 0.399 0.430 0.407 0.423 0.399 0.428 0.414 0.451 0.414 0.532 0.552

0.376

0.328

0.385

0.340

0.380 0.334 0.403 0.357 0.379 0.330 0.407 0.358

0.385 0.331 0.394 0.348 0.381 0.331 0.397 0.357 0.425 0.371 0.484 0.470

ETTm2

96

0.293

0.231

0.298

0.241

0.297 0.239 0.300 0.250 0.299 0.232 0.434 0.397

0.352 0.277 0.388 0.312 0.376 0.304 0.305 0.256 0.320 0.255 0.459 0.381

192

0.344

0.324

0.351

0.331

0.339

0.319 0.345 0.334 0.340

0.316

0.566 0.649

0.480 0.484 0.481 0.473 0.443 0.421 0.357 0.345 0.365 0.349 0.581 0.582

336

0.383

0.395

0.388

0.402

0.385 0.400 0.400 0.437 0.390 0.399 0.735 0.970

0.575 0.670 0.555 0.614 0.539 0.604 0.394 0.418 0.403 0.417 0.856 1.195

720

0.439

0.503

0.445

0.518

0.444 0.525 0.459 0.558 0.443 0.509 1.776 2.209

0.703 0.963 0.690 0.913 0.688 0.943 0.449 0.538 0.457 0.519 1.688 4.769

0.365

0.363

0.371

0.373

0.366 0.371 0.376 0.395 0.368 0.364 0.878 1.056

0.528 0.599 0.529 0.578 0.512 0.568 0.376 0.389 0.386 0.385 0.896 1.732

ETTh1

96

0.381

0.323

0.393

0.338

0.382 0.335 0.411 0.354 0.382 0.330 0.392 0.327

0.390 0.328 0.451 0.426 0.385 0.331 0.397 0.365 0.401 0.335 0.483 0.483

192

0.408

0.369

0.419

0.383

0.408 0.381 0.434 0.390

0.404

0.371 0.418 0.371

0.426 0.374 0.475 0.462 0.413 0.375 0.419 0.399 0.430 0.373 0.518 0.533

336

0.424

0.402

0.436

0.419

0.426 0.417 0.464 0.446

0.417

0.408 0.446 0.418

0.476 0.442 0.488 0.491 0.427 0.404 0.443 0.441 0.449 0.416 0.535 0.563

720

0.442

0.398

0.460

0.430

0.445 0.418 0.473 0.459 0.459 0.416 0.443

0.389

0.498 0.472 0.496 0.479 0.444 0.395 0.514 0.516 0.469 0.434 0.567 0.591

0.414

0.373

0.427

0.393

0.415 0.388 0.446 0.412 0.416 0.381 0.425 0.376

0.448 0.404 0.478 0.465 0.417 0.376 0.443 0.430 0.437 0.390 0.526 0.543

ETTh2

96

0.380

0.369

0.386

0.380

0.395 0.402 0.428 0.429 0.391 0.388 0.737 1.016

0.504 0.546 0.555 0.622 0.394 0.382 0.404 0.411 0.426 0.429 1.014 1.778

192

0.430

0.458

0.438

0.482

0.442 0.490 0.442 0.494 0.437 0.460 0.899 1.447

0.612 0.775 0.636 0.795 0.621 0.777 0.444 0.496 0.471 0.515 1.326 2.645

336

0.455

0.494

0.464

0.508

0.454

0.506 0.468 0.529 0.457 0.495 0.944 1.430

0.723 1.028 0.765 1.113 0.736 1.067 0.463 0.518 0.499 0.546 1.414 2.819

720

0.455

0.473

0.478

0.519

0.478 0.523 0.472 0.509 0.476 0.511 1.088 1.795

0.930 1.651 0.960 1.720 0.948 1.701 0.487 0.536 0.507 0.542 1.462 2.951

0.430

0.449

0.442

0.472

0.442 0.480 0.453 0.490 0.440 0.464 0.917 1.422

0.692 1.000 0.729 1.063 0.675 0.982 0.450 0.490 0.476 0.508 1.304 2.548

Weather

96

0.231

0.147

0.243

0.160

0.232

0.140

0.242 0.152 0.247 0.163 0.244 0.141

0.283 0.180 0.269 0.171 0.267 0.173 0.266 0.183 0.309 0.205 0.259 0.152

192

0.294

0.216

0.302

0.228

0.292

0.218 0.307 0.230 0.297 0.220 0.301

0.201

0.327 0.233 0.319 0.229 0.318 0.228 0.317 0.245 0.365 0.279 0.334 0.228

336

0.353

0.298

0.361

0.310

0.355

0.299 0.374 0.322 0.356 0.303 0.357

0.272

0.379 0.304 0.381 0.304 0.366 0.309 0.366 0.318 0.405 0.347 0.390 0.310

720

0.426

0.412

0.429

0.417

0.431 0.420 0.467 0.479 0.428 0.418 0.448 0.415

0.435 0.394 0.447 0.402 0.432

0.384

0.461 0.418 0.465 0.457 0.446 0.405

0.326

0.268

0.334

0.279

0.328 0.269 0.348 0.296 0.332 0.276 0.338

0.257

0.356 0.278 0.354 0.277 0.346 0.274 0.353 0.291 0.386 0.322 0.357 0.274

Exchange

96

0.221

0.094

0.227

0.098

0.225 0.097 0.266 0.131 0.224 0.095 0.345 0.225

0.236 0.101 0.264 0.120 0.238 0.103 0.251 0.112 0.324 0.187 0.481 0.390

192

0.327

0.194

0.335

0.211

0.333 0.205 0.369 0.246 0.341 0.219 0.503 0.443

0.339 0.198 0.384 0.248 0.336 0.198 0.340 0.213 0.430 0.325 0.577 0.528

336

0.445

0.339

0.455

0.366

0.460 0.369 0.501 0.429 0.452 0.360 0.690 0.762

0.432 0.311

0.473 0.370 0.433 0.321 0.447 0.358 0.552 0.519 0.721 0.815

720

0.788

0.963

0.790

0.974

0.774 0.942 0.876 1.186 0.795 0.995 0.844 1.099

0.640 0.683 0.714 0.869

0.629 0.651

0.757 0.910 0.948 1.365 0.819 0.982

0.445

0.398

0.452

0.412

0.448 0.403 0.503 0.498 0.453 0.417 0.596 0.632

0.412 0.323 0.459 0.402

0.409 0.318

0.449 0.398 0.564 0.599 0.650 0.679

ILI

24

0.771

1.542

0.950

2.112

0.938 2.060 0.882 1.918 0.950 2.061 1.079 2.744

2.211 2.587 1.688 5.046 1.456 3.552 1.123 2.642 1.377 3.322 1.101 2.533

36

0.761

1.387

0.961

2.066

0.933 1.966 0.886 1.994 0.939 2.046 1.100 2.852

1.047 2.155 1.696 5.217 1.351 3.231 1.208 2.621 1.275 3.056 1.063 2.745

48

0.777

1.482

0.962

2.040

0.907 1.904 0.898 2.078 0.985 2.183 1.158 3.014

1.055 2.203 1.749 5.574 1.273 2.969 1.090 2.664 1.207 2.826 1.104 2.832

60

0.805

1.516

1.105

2.309

0.911 1.897 0.897 2.071 1.017 2.269 1.222 3.238

1.117 2.496 1.804 5.890 1.278 3.034 1.330 2.881 1.208 2.806 1.115 2.877

0.779

1.482

0.995

2.132

0.922 1.957 0.891 2.015 0.973 2.140 1.140 2.962

1.358 2.360 1.734 5.432 1.340 3.197 1.188 2.702 1.267 3.003 1.096 2.747

Electricity

96

0.235

0.143

0.268

0.180

0.243 0.145 0.263 0.161 0.275 0.187 0.268 0.171

0.278 0.167 0.318 0.218 0.297 0.207 0.286 0.207 0.330 0.220 0.390 0.314

192

0.250

0.159

0.278

0.189

0.259 0.165 0.279 0.179 0.280 0.190 0.315 0.226

0.286 0.176 0.326 0.225 0.300 0.206 0.305 0.216 0.336 0.227 0.401 0.319

336

0.265

0.171

0.296

0.207

0.272 0.174 0.290 0.189 0.296 0.204 0.349 0.257

0.298 0.186 0.342 0.240 0.314 0.217 0.313 0.227 0.352 0.247 0.407 0.326

720

0.291

0.198

0.331

0.252

0.306 0.210 0.318 0.217 0.327 0.245 0.371 0.292

0.319 0.209 0.368 0.276 0.343 0.251 0.361 0.284 0.385 0.296 0.401 0.315

0.260

0.168

0.293

0.207

0.270 0.174 0.288 0.187 0.295 0.207 0.326 0.237

0.295 0.185 0.339 0.240 0.314 0.220 0.316 0.234 0.351 0.248 0.400 0.319

Traffic

96

0.292

0.459

0.357

0.560

0.358 0.597 0.333 0.762 0.350 0.575

-

-

0.333 0.652 0.464 0.831 0.432 0.774

-

-

0.409 0.756 0.423 0.858

192

0.310

0.494

0.361

0.573

0.360 0.627 0.353 0.790 0.349 0.584

-

-

0.350 0.669 0.456 0.792 0.410 0.715

-

-

0.396 0.746 0.444 0.908

336

0.316

0.508

0.371

0.595

0.364 0.640 0.359 0.801 0.363 0.611

-

-

0.361 0.696 0.460 0.809 0.412 0.719

-

-

0.409 0.765 0.452 0.957

720

0.330

0.534

0.397

0.645

0.382 0.676 0.370 0.860 0.379 0.642

-

-

0.375 0.750 0.479 0.863 0.429 0.758

-

-

0.451 0.830 0.504 1.055

0.312

0.499

0.372

0.593

0.366 0.635 0.354 0.803 0.360 0.603

-

-

0.355 0.692 0.465 0.824 0.421 0.742

-

-

0.416 0.774 0.456 0.945

Top1 Counts.

55

0

5

0

4

4

2

0

5

0

0

0

G Broader Impacts

This paper copes with the general analysis of real-world mixed time series, which are under-explored in literature but frequently encountered in various practical applications, such as industrial maintenance, healthcare monitoring, and financial analysis. Since previous studies struggle to address the spatial-temporal heterogeneity problem for real-world MiTS, we present MiTSformer that fundamentally reveals and recovers the latent continuous variables for discrete variables to facilitate exploiting the intricate spatial-temporal patterns within MiTS, thereby being amenable to various analysis task. Our model achieves the state-of-the-art performance on five mainstream tasks that cover 30+ real-world datasets from diverse application domains. Therefore, the proposed model makes it promising to tackle real-world MiTS analysis tasks, which helps our society make better decisions. Our paper mainly focuses on scientific research and has no obvious negative social impact.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have clearly claimed our contributions in the end of the introduction from the perspective of (1) Problem Formulation, (2) Technical Design, and (3) Empirical Evaluation.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have analyzed the limitations of our proposal in Appendex D. Also, we investigated the computational efficiency of our approach in Appendix B.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided the details regarding reproducibility, including computational platforms, dataset descriptions, model configurations, task-specific pipelines, hyperparameters settings, and pseudo-code of our approach in Appendix A. Also, We have released the source code on Anonymous Github as stated in the Abstract. The download links of the datasets and pre-processing functions are also included in the project homepage.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The datasets used in our paper are benchmarks and are publicly available. We have provided the relevant references and hyperlinks of datasets. Also, We have released the source code on Anonymous Github as stated in the Abstract. The download links of the datasets and pre-processing functions are also included in the project homepage.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided all the experimental implementation details in Appendix A, including (1) dataset descriptions and splits of training/validation/test sets (Table 5); (2) Experiment configurations and hyperparameter settings of our approach (Table 6) and baselines (Table 7).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All experiments are repeated three times and we have reported the average results. Also, we have investigated the sensitivity of our approach in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have clearly stated the experimental platforms (e.g., GPUs and CPUs) and software (the version of Python and Pytorch) used in our paper in Appendix A, and we have analyzed the computational efficiency of our approach in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics and we have ensured to preserve anonymity.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed both potential positive societal impacts and negative societal impacts of our work in Appendix G.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the original paper (or URL) of the code package or dataset that is used in our paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We have released the source code on Anonymous Github as stated in the Abstract. The instructions for running our codes are provided on the project homepage at Anonymous Github. If our paper is accepted, we commit to releasing the relevant code as open-source and will provide detailed documentation to support its use and replication of our results.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer:[NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose to model mixed time series with both continuous variables and discrete variables by constructing latent continuous variables (LCVs) from discrete variables (DVs). Several super-supervised learning constraints are proposed to help improve the effectiveness of LCVs as well as the co-learning of LCVs and CVs with both cross-attention and self-attention modules.\n\nSTRENGTHS:\n1. Clarity in paper-writing. Both figures and languages are easy to understand and well-polished. \n2. Originality. The idea is novel, easy to follow, and most importantly, it makes sense. \n3. Quality. The paper is conducted on various datasets for various tasks to support its claim as a general model. \n4. Code is released and sufficient training details are provided.\n\nWEAKNESSES:\n1. The paper lacks evidence to support the effectiveness of the proposed self-supervised objective functions. There is no theoretical or strong intuition behind them. \n\n1.1 For example, why does Temporal Adjacent Smoothness Constraint matter? I know from experiments it seems necessary, but intuitively and by commonsense, some discrete variables could indeed have sudden changes. In your example of meteorology, rainfall could indeed suddenly stop at a time point because the cloud moves from where the measurement device is. How do you defend your model in case where a DV is indeed subject to sudden changes by its nature? Moreover, for this constraint, do we have to use this specific formulation, because minimizing every two adjacent points could be too strong? How about K-lipschitz continuity as constraint, where k can be determined by the CVs? I would love to see some discussion on why you select this formulation specifically.\n\n1.2 In addition, the adversarial framework that discriminates between CV and LCV seems weak. There is no guarantee that it will work in theory, as GAN-like structures are unstable. Even if the optimization is stable, in what condition would a model think \"I cannot discriminate between CV and LCV\"? What specific properties are in CVs that LCV could learn to trick a discriminator? For facial recognition, we can tell if the object in an image is like a real human face or not, based on whether some details are distorted. For time series, how to interpret that? If at some time points, the time series are experiencing abnormal variations, that does not mean it is “fake”. What specific properties are you trying to look into when you design this loss function? If you just want the LCVs to reflect some of the variations in the CVs, you could design some simpler constraints just like your Temporal Adjacent Smoothness Constraint, correct? I would love to see some discussions.\n\n2. Some issues with presentation. Variable z is first used in Eq.2 and then in Eq.4, but it was never explained in text. Usually, the audience is much more interested in the error bars instead of showing both MAE and MSE, as both metrics are extremely similar. Please adjust accordingly. \n\n3. Some issues with the experimental setting. The reconstruction loss is essential, but the key issue is, is the generated LCVs actually correct? Wouldn't it be wonderful if you could test on some datasets where some DVs are discretized from CVs, and you can try to recover the CVs and plot the results? If the accuracy of actually recovering CVs is not guaranteed, we do not need to have to “recover” a continuous time series, but we can obtain some latent embedding from the DV, which should also work well by intuition. Therefore, this experiment should be added as a demonstration that recovering LCV is possible. It would be a prerequisite to prove it is helpful.\n\nQUESTIONS:\nThe paper lacks evidence to support the effectiveness of the proposed self-supervised objective functions. There is no theoretical or strong intuition behind them. For example, why does Temporal Adjacent Smoothness Constraint matter? I know from experiments it seems necessary, but intuitively and by commonsense, some discrete variables could indeed have sudden changes. In your example of meteorology, rainfall could indeed suddenly stop at a time point because the cloud moves from where the measurement device is. How do you defend your model in case where a DV is indeed subject to sudden changes by its nature? Moreover, for this constraint, do we have to use this specific formulation, because minimizing every two adjacent points could be too strong? How about K-lipschitz continuity as constraint, where k can be determined by the CVs? I would love to see some discussion on why you select this formulation specifically.\n\nIn addition, the adversarial framework that discriminates between CV and LCV seems weak. There is no guarantee that it will work in theory, as GAN-like structures are unstable. Even if the optimization is stable, in what condition would a model think \"I cannot discriminate between CV and LCV\"? What specific properties are in CVs that LCV could learn to trick a discriminator? For facial recognition, we can tell if the object in an image is like a real human face or not, based on whether some details are distorted. For time series, how to interpret that? If at some time points, the time series are experiencing abnormal variations, that does not mean it is “fake”. What specific properties are you trying to look into when you design this loss function? If you just want the LCVs to reflect some of the variations in the CVs, you could design some simpler constraints just like your Temporal Adjacent Smoothness Constraint, correct? I would love to see some discussions.\n\nLIMITATIONS:\nI do not see limitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n•\tThis paper introduces a type of spatial-temporal heterogeneity caused by the gap between continuous variables (CVs) and discrete variables (DVs). \n•\tThe author introduces latent continuous variables to create a unified continuous numerical space for both CVs and DVs, with the aim to address the heterogeneity caused by the gap between these types of variables.\n•\tThe Latent Continuity Recovery architecture is innovative.\n\nSTRENGTHS:\nThe introduction of latent continuous variables (LCVs) to bridge the gap between DVs and CVs is an innovative approach. The proposed method for latent continuity recovery through adaptive and hierarchical aggregation of multi-scale adjacent context information is a creative combination of existing ideas.\n\nThe code is open-sourced, clean, and well-organized, demonstrating a high standard of technical implementation.\n\nThe proposed solution addresses a fundamental issue in spatial-temporal modeling, making it highly relevant for various applications in fields such as precipitation, temperature, humidity, etc.\n\nWEAKNESSES:\n1. The presentation for the transformation process for LCVs can be improved for better clarity. A more explicit and detailed connection between the mathematical formulas and the LCV transformation process would strengthen the clarity and comprehensiveness of the paper.  \n2. The formulation and application of the regularization term to encourage smoothness across time need to be better explained.\n3. The experiment is mostly comprehensive, but the analytical explanation of latent variables can be strengthened. This analysis could be interesting to broad audience.\n\nQUESTIONS:\n1.How does the proposed method scale with large datasets or in real-time applications? what is the computational complexity of the proposed method? \n2. Could you explain more explicit and detailed connection between the mathematical formulas and the LCV transformation process?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"Addressing Spatial-Temporal Heterogeneity: General Mixed Time Series Analysis via Latent Continuity Recovery and Alignment\" introduces MiTSformer, a framework designed to address the challenges of mixed time series (MiTS) data, which include both continuous variables (CVs) and discrete variables (DVs). The framework recovers latent continuous variables (LCVs) behind DVs to ensure sufficient and balanced spatial-temporal modeling. MiTSformer employs hierarchical aggregation of temporal context and adversarial learning to align DVs with CVs. The framework is validated on five MiTS analysis tasks, showing state-of-the-art performance across multiple datasets.\n\nSTRENGTHS:\nNovel Approach: The introduction of latent continuous variables (LCVs) to handle the spatial-temporal heterogeneity in mixed time series is innovative and addresses a significant gap in current methodologies.\nComprehensive Framework: MiTSformer is versatile, capable of handling various tasks such as classification, regression, anomaly detection, imputation, and long-term forecasting.\nRobust Performance: The framework demonstrates superior performance across a wide range of datasets and tasks, indicating its robustness and effectiveness.\nDetailed Analysis: The paper provides thorough empirical evaluations and ablation studies, which validate the effectiveness of the proposed method.\n\nWEAKNESSES:\nComplexity: The framework's complexity may pose implementation challenges for practitioners, potentially limiting its accessibility and usability.\nData Dependency: The performance heavily relies on the quality and diversity of the training data, which may limit its applicability in scenarios with limited data availability.\nScalability: While the method shows promising results, its scalability to very large datasets or real-time applications is not fully demonstrated.\nLimited Explanation of Hyperparameters: The paper could benefit from a more detailed explanation of the choice and tuning of hyperparameters, which is crucial for replication and practical application.\n\nQUESTIONS:\nHow does MiTSformer handle highly non-linear temporal patterns in both CVs and DVs?\nCan the framework be extended to handle non-binary discrete variables, and if so, how?\nWhat are the computational requirements for training MiTSformer on large-scale datasets?\nHow does the framework perform in real-time applications where data arrives sequentially?\n\nLIMITATIONS:\nThe paper presents a compelling and innovative approach to handling mixed time series data. However, the complexity of the method, data dependency, and scalability issues suggest that further development and validation are needed to ensure its broad applicability and practical utility.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe MiTSformer framework includes Latent Continuity Recovery, which recovers latent continuous variables (LCVs) from discrete variables (DVs) using multi-scale temporal context and adversarial guidance, and Spatial-Temporal Attention Blocks, which capture dependencies within and across LCVs and continuous variables (CVs) through self- and cross-attention mechanisms. The paper explores general mixed time series analysis, addressing spatial-temporal heterogeneity. MiTSformer adapts to recover LCVs and capture spatial-temporal dependencies, demonstrating state-of-the-art performance across five tasks on 34 datasets.\n\nSTRENGTHS:\nThe paper introduces a fresh problem definition by addressing the heterogeneity between continuous variables (CVs) and discrete variables (DVs) in mixed time series analysis.  The MiTSformer framework effectively recovers latent continuous variables (LCVs) from DVs and captures spatial-temporal dependencies, offering a balanced and comprehensive modeling approach. The framework is highly effective in handling various types of variables, providing clear insights into their relationships, which is invaluable for designing robust foundation models for time series.\n\nWEAKNESSES:\nFirstly, the experimental setup relies on converting continuous variables (CVs) to discrete variables (DVs) for experimentation, rather than using datasets with naturally mixed CVs and DVs. This may introduce unnecessary complexity and potential information loss, weakening the paper's motivation. Secondly, the complexity of the framework, which includes the recovery network and adversarial learning components, may pose challenges for practical implementation and scalability. The claim that the framework effectively recovers the inherent continuous nature of DVs and maintains temporal similarity through adversarial learning lacks robust verification and evidence. The improvements attributed to L_Dis are minimal, as shown in Table 4's ablation study, and further analytical reasons should be provided to support these findings.\n\nMiTSformer\n\nQUESTIONS:\nThe paper claims that using MiTSformer allows for freedom from the strict limitations of mixed naive Bayes models and variational inference previously used to match the distributions of continuous variables (CVs) and discrete variables (DVs). However, the reasons for this claim are not clearly explained. A detailed explanation is needed to justify why MiTSformer is less restrictive. Additionally, a performance comparison between MiTSformer and these traditional methods would be beneficial to substantiate its effectiveness.\n\nSecondly, did you compare the latent continuous variables (LCVs) obtained from randomly sampled DVs in the forecasting tasks with the actual CV information? Such an analysis would be interesting and could provide deeper insights into the model's performance.\n\nLIMITATIONS:\nOne limitation mentioned is that the framework cannot be directly applied to categorical discrete variables. In practice, many time series datasets combine categorical data with time series data, such as event data and product sales demand. This limitation is significant since these types of datasets are common in real-world applications.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-24"}