Addressing Spatial-Temporal Heterogeneity: General Mixed Time Series Analysis via Latent Continuity Recovery and Alignment
Authors: Jiawei Chen
Chunhui Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: Mixed Time Series
General Time Series Modeling
Spatial-Temporal Heterogeneity
Latent Continuity Recovery
Adversarial Learning
Primary area: deep_learning_architectures
TL;DR: A mixed time series analysis framework, which recovers and aligns latent continuity of mixed variables for complete and reliable spatial-temporal modeling, being amenable to various analysis tasks.

Abstract: Mixed time series (MiTS) comprising both continuous variables (CVs) and discrete variables (DVs) are frequently encountered yet under-explored in time series analysis. Essentially, CVs and DVs exhibit different temporal patterns and distribution types. Overlooking these heterogeneities would lead to insufficient and imbalanced representation learning, bringing biased results. This paper addresses the problem with two insights: 1) DVs may originate from intrinsic latent continuous variables (LCVs), which lose fine-grained information due to extrinsic discretization; 2) LCVs and CVs share similar temporal patterns and interact spatially. Considering these similarities and interactions, we propose a general MiTS analysis framework MiTSformer, which recovers LCVs behind DVs for sufficient and balanced spatial-temporal modeling by designing two essential inductive biases: 1) hierarchically aggregating multi-scale temporal context information to enrich the information granularity of DVs; 2) adaptively learning the aggregation processes via the adversarial guidance from CVs. Subsequently, MiTSformer captures complete spatial-temporal dependencies within and across LCVs and CVs via cascaded self- and cross-attention blocks. Empirically, MiTSformer achieves consistent SOTA on five mixed time series analysis tasks, including classification, extrinsic regression, anomaly detection, imputation, and long-term forecasting. The code is available at https://github.com/chunhuiz/MiTSformer.

Addressing Spatial-Temporal Heterogeneity: General Mixed Time Series Analysis via Latent Continuity Recovery and Alignment
Jiawei Chen, Chunhui Zhao
State Key Laboratory Industrial Control Technology, College of Control Science and Engineering, Zhejiang University, China
Abstract
Mixed time series (MiTS) comprising both continuous variables (CVs) and discrete variables (DVs) are frequently encountered yet under-explored in time series analy- sis. Essentially, CVs and DVs exhibit different temporal patterns and distribution types. Overlooking these heterogeneities would lead to insufficient and imbalanced representation learning, bringing biased results. This paper addresses the problem with two insights: 1) DVs may originate from intrinsic latent continuous vari- ables (LCVs), which lose fine ine-grained information due to extrinsic discretization; 2) LCVs and CVs share similar temporal patterns and interact spatially. Con- sidering these similarities and interactions, we propose general MiTS analysis framework MiTSformer, which recovers LCVs behind DVs for sufficient and balanced spatial-temporal modeling by designing two essential inductive biases: 1) hierarchically aggregating multi-scale temporal context information to enrich the information granularity of DVs; 2) adaptively learning the aggregation processes via the adversarial guidance from CVs. Subsequently, MiTSformer captures com- plete spatial-temporal dependencies within and across LCV' and CVs via cascaded self- and cross-attention blocks. Empirically, MiTSformer achieves consistent SOTA on five mixed time series analysis tasks, including classification, extrinsic regression, anomaly detection, imputation, and long- term forecasting. The code is available at ltttt::// :/gg:t///itte hitggitt.bb.... com/
Introduction
Multivariate time series analysis is energized in various real-world applications, such as weather forecasting 77]. activity recognition [20], and industrial maintenance [47]. Empowered by deep learning, plentiful time series models have been proposed based on foundation models such as RNNs
23 25
CNNs 6 36 40), Transformers 30
41
10 and modernized MLPs
(50) 45 39 These
sophisticated models have achieved increasingly remarkable performance in various time series
analysis tasks, e.g., classification [17], forecasting 7 42.. imputation [40] and anomaly detection (9]
The primary challenge in time series analysis is effectively modeling spatial-temporal patterns, including intra-variable temporal variations and inter-variable spatial correlations [46 48) 16 II 43
So far, most current approaches naturally assume that time series data are composed solely of continuous variables, and then uniformly model ceatial-temporal patterns in continuous spaces. Yet, in broad practical scenarios, the acquired data are often mixed time series (MiTS) that encompass both continuous variables (CVs) and discrete variables (DVs). Take meteorological data example (Fig. T (Lett), some sensor-derived variables are commonly recorded as CVs (e.g., temperature,
*Corresponding author: Chunhui Zhao (chhzhao@zju, edu.cn)
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Auto
Periedicly
Distribution
Temperoture
Humidity
Temporal Hetarogenelty
Spatial Heterogeneity
Cloudage
Dayss
Suddan Change Limited Information Stoady Shotes
Bemoulli Distribution
Figure 1: Left: Illustration of mixed time series. Right: Spatial-temporal heterogeneity problem.
humidity, and wind speed), while certain variables like cloudage and rainfall patterns are typically tracked as DVs due to measurement restrictions or distinctive nature. Up to now, mixed time series analysis is still formidable yet under-explored problem. Essentially, MiTS presents spatial-tempon heterogeneity problems as depicted Fig. [ (Rghht) On one hand, CVs commonly encapsulate rich temporal variation information, exhibited in autocorrelations, periodical patterns, local fluciuations, etc, while DVs often undergo sudden changes or steady states due to limited value ranges, resulting in the temporal variation discrepancy between CVs and DVs that complicates temporal modeling. On the other hand, CVs generally adhere to Gaussian distributions, while DVs follow Bernoulli distributions, resulting in the distribution type discrepancy that hinders spatial correlation analysis between CVs and DVs. Neglecting these heterogencities and equally treating mixed variables would yield insufficient and imbalanced -peppolltemporal modeling problems, i.e., the model may struggle to characterize distinct temporal patterns of DVs and CVs and fail to reliably capture spatial correlations within and across DVs and CVs, posing bottleneck for MiTS analysis.
The key to addressing the spatial-temporal heterogeneity lies in bridging the information gap between DVs and CVs. Essentially, in real-world MiTS, the observed DVs may originate from intrinsic continuous-vail factors, which are unobservable owing to extrinsic factors such as measurement limitations, storage requirements, and transmission interference. Continuing with the aforementioned meteorological example, cloud cover percentage is an intrinsically continuous variable, whose fine- grained values are hard to measure directly. In practice, discretizing it with coarse- coarse-grained discrete variable-cloudage (reflect "cloudy" or not) is sufficient for most applications and is memory-efficien In this paper, we introduce latent continuous variables (LCVs) to describe the intrinsic continuous factors behind DVs. Given its numerically continuous nature, the LCV of cloudage, i.e., cloud cover percentage, may be spatially correlated with other observed CVs (e.g., humidity and wind speed) and exhibit similar temporal variation patterns (e.g., autocorrelation and seasonal fluctuation) with them, both of them originate from the same meteorological system. Thereby, we can progressively recover the LCVs behind DVs by leveraging these temporal similarities and spatial interactions among CVs and LCVs. In this way, spatial-temporal dependencies of mixed variables can be completely and reliably modeled in a unified continuous numerical space, and the teatial-emmpral heterogeneity problem is mitigated. Also, by bridging the mutual spatial-temporal interactions, LCVs and CVs can supply complementary information for various downstream analysis tasks.
Enlightened by the above insights, we reconcile the intrinsic tension between the two highly dependent problems Latent Continuity Recovery and synergistic framework, MiTSformer, for general mixed time series analysis. By leveraging the temporal similarities and spatial interactions between I. I.C's and CVs, MiTSformer can gradually decipher the LCVs behind DVs and capture the complete spatial-temporal dependencies within and across LCVs and CVs. Specifically, we design a recovery network to portray LCVs behind DVs by adaptatively and hierarchically aggregating temporal contextual information. Followingly, an adversarial variable modality discrimination objective and smoothing constraints are devised to guide the learning of the recovery network, ensuring the recovered LCVs share similar temporal properties and distributions with CVs. Additionally, MiTSformer employs self-attention to learn dependencies within LCVs or CVs and cross-attention attention to exploit those across LCVs and CVs, facilitating various downstream analysis tasks. Our contributions lie in three aspects:
Modeling in one coherent and
(1) Novel Problem: To the best of our knowledge, our paper pioneers the exploration of the general mixed time series analysis, which is practical and challenging. We reveal the crucial spatial-temporal heterogeneity problem, which is caused by the discrepancies in temporal variation properties and distribution types between DVs and CVs.

---

(2) Customized Framework: To solve the ppatial-tempont heterogeneity problem, we propose a task-general framework MiTSformer customized for MiTS, which adaptively recovers LCVs behind DVs by leveraging the adversarial guidance of CVs and task supervisions. Moreover, MiTSformer can capture spatial-temporal dependencies within and across CVs and LCVss via self- and cross-attention blocks, thus learning sufficient and balanced spatial- temporal representations and being amenable to various mixed time series analysis tasks.
(3) Versatile Effectiveness: Empirically, proposed MiTSformer establishes the state-of- the-art performance on five mainstream mixed time series analysis tasks with 34 datasets covering vide-ranging real-world application domains. We believe our work makes predominant attempt at general mixed time series analysis in practical applications.
2 Related Work
General Time Series Analysis General time series analysis aim at learning universal temporal representations by developing task-general backbones and task-specific heads for diverse tasks. To date, these methods have been designed predominantly for time series comprising continuous variables solely. As pioncering work, TimesNet [40] leverages fast Fourier transform and parameter- efficient inception block to capture intra-period and inter-period variations for time series modeling. Followingly, ModemTTN [66] modernizes and modifies the traditional TCN by introducing larger effective receptive fields and rross-variable dependency modeling, bringing great performance and efficiency. Meanwhile, GPT2TS [52] leverages pre- pre-trained language models, e.g.. GPT2 [32), for various time series analysis tasks with task-specific fine-tuning.
Yet, most of the current works naturally assume uniformity in variable types of time series and inapplicable for MiTS, as they lack the differentiation of distinct variable modalities. In this paper, we develop a systematic framework with delicate differentiation and deft alignment of mixed variables to support sufficient and balanced spatial-temporal modeling for general mixed time series analysis.
Mixed Data Analysis Previous studies have revealed the significance of mixed data analysis and made several attempts to shed light on this challenging problem. A naive solution for mixed data modeling is roughly pre-processing DVs and CVs into the same variable modality, e.g., discarding DVs or discretizing CVs by certain policies [35 [3]. However, these methods may lose vital fine- scaled information and bring errors inevitably. Towards concurrently modeling of DVs and CVs, Mixed Data RBF-ELM method (24) adopts a distance-based learning scheme for efficient and direct mixed data classification, while Mixed-varial Restricted Boltzmann Machine (Mv.RBM) [5 [4] construct ensembles of mixed-data Deep Belief Nets with varying depths for anomaly detection of mixed data. Another line of work treats DVs semantic attributes of CVs and establishes the relationships between DVs and CVs by developing specific inference rules [88 44). More recently,
researchers have utilized mixed naive Bayes models [37 38 variational inference 12 for mixed
data modeling in industrial processes with different distribution priors of CVs and DVs.
However, current studies mostly focus on specific analysis tasks (e.g., designed only for classification) and may be restricted by linear, non-temporal data, or other rigorous assumptions, which are not capable of handling real-world MiTS that exhibit intricate spatial-temporal patterns. Instead, our proposed MiTSformer can model complete spatial-temporal dependencies within and across DVs and CVs and can produce task-general representations for various MiTS analysis tasks.
3 MiTSformer
3.1 Problem Formulation and Motivation Analysis
Mixed Time Series. Given a collection of multivariate time series X = {I1, #2, T2,,..,,r,) comprising p variables with length T. Among them, there are p - n continuous variables Xc {21, 82, -2. f2, Tp-n] R(p-n)xT with continuous numerical values, and discrete vari- ables x {Ip-n+1, fp-n+t2 ..., Fnh AxXT with discrete states. Without loss of generality, consider the binary-valued discrete variables, whose value can be 0 or 1, i.e., A = {0. 1}. Mixed time series used as model input support various analysis tasks, such as regression and classification.
Hampered by the spatial-tempone heterogeneity problem, directly modeling spatial-temporal depen- dencies of CVs and DVs without considering their discrepancies may inevitably yield non-negligible
3

---

biases. Such research bottlenecks prompt us to exploit the underlying generation and interaction mechanism of DVs and CVs. As aforementioned. the observed DVs are potentially derived from I.Css, which undergo discretization processes due to external interferences as depicted in Fig. 2 For each DV xP, we adopt a corresponding I.V xLCC' to portray its latent continuity. To reliably and completely model inherent spatial-temporal patterns within MiTS, deciphering and recovering LCVs behind DVs is indispensable. Yet, it is challenging since a single discrete state can not be directly trans- formed into continuous value without proper supervision. In this study, we address this challenge by revealing and leveraging the temporal similarity and spatial interaction between LCVs and CVs:
Temporal Similarity The unobserved LCVs share similar temporal variation patterns (e.g., autocorrelation, periodicity, trend, etc.) with the observed CVs.
Latent Continuous
high?
fr
External interference factors
Spatial Interaction LCVs and CVs exhibit information interactions and inter-variable spatial correlations. The synergistic effect of LCVs and CVs provides complementary information for various downstream tasks.
"
Temporal Similarity Spotial Interaction
Hish
Observed Continuous Variables
Observed Discrate Variables
Built upon these two insights, we design MiTTSformer, which will be elaborately introduced in the following parts.
Figure Connections among DVs, CVs, and LCVss.
Decoder
Downstream Tasks:
munun
Self Attention Bub-Blouk
Cross-Attntto But-&iook
mumU
Crowidng
Classifictiio
Obsonved
Voriate Takinn LOWs
LfVL
Regression
mmmma
Fartancs
Forecasting
Eresccinng
Self Attordonn
Gross Altenntonn
Imputation
Sue Block
Anomaly Det.
Observed
Varann Tolumss of CVds
Spaiss Tertss--Trmpprall Aftertion Blocks
Veriate of
Deconier
Figure 3: Overall pipeline of MiTSformer. First, MiTS undergo latent continuity recovery (DVs only) and are embedded variate tokens, which are then refined through spatial-temporal attention blocks. The acquired variate tokens are utilized both for reconstructing the original MiTS and serving various downstream tasks.
3.2 Framework Overview
As aforemmentioeed, there are two key steps for general MiTS analysis: 1) unifying the temporal characteristics and distribution types of DVs and CVs; 2) modeling sufficient and balanced spatial- temporal dependencies for effective representation learning. MiTSformer facilitates these two steps in a highly versatile manner with a coherent framework as depicted in Fig. B The overall pipeline contains key parts: 1) Latent Continuity Recovery that adaptively aggregate contextual information of DVs to recover LCV's with the adversarial alignment guidance of CVs and temporal smoothness constraints; 2) Spatial-Tempont Atlention Blocks that model intra- and inter-variable modality apatial-tempore dependencies with cascaded self-attention and cross-attention sub-blocks.
3.3 Latent Continuity Recovery
Since single time points of DVs contain limited information, it is difficult to directly transform discrete states to latent continuous values. Fortunately, time series commonly present auto-correlation natures, i.e., the value of certain time step correlated with its adjacent ones. Specifically, for a discrete state of "I" surrounded by states "I", its latent continuous value would relatively large, e.g., 0.9, In contrast, for a discrete state of "I" surrounded by states "0", its latent continuous value would be relatively small, e.g., 0.6. Thus, an intuitive solution for enriching the information density of single time point of DVs is to properly leverage its adjacent context information.

---

Contextual Aggregation Recovery Network We con- vert the above insights into the model inductive bias and realize the latent continuity recovery by adaptively and hierarchically aggregating multi-scale adjacent con- text information of DVs. Specifically, convolutional neural networks (CNNs) own an inductive bias that can aggregate receptive local information by convolu- tional kernels. Technically, we devise the recovery net- works that receive DVs as input to generate LCVs x LG = Rec-Net (z²)) ) As depicted in Fig. 4 the recovery network is composed of several residual di- lated convolutional blocks, which employ dilated con- volutional kernels along the temporal dimension ag- gregate adjacent context information, and utilize resid- ual connections to adaptively accumulate multi-scale temporal information to characterize intricate tempo- ral patterns of LCVs. The residual dilated convolu- tional network is implemented by the iterative process as h4 Conv (h4-1)+ + h,,1, where h; represents the out- put of the i-th residual block, i = 1.2, n, Conv", denotes the convolution operation along the temporal axis to aggregate contextual information with dilation rate d; The final output, L.CVV = Jn,, is the result after n residual blocks.
Obsanved DVs
Recovery Network
Nor
Resicul Diated CNN Blooks
m Obsarved CVd
Recovere LVv
Enbedding
Fimbelding
Enbeddiings
Embediinge CVvs
Variable-modally Discrmbhator
Variable modallty Label
Figure 4: LCV recovery with adversarial variable modality discrimination.
Temporal Adjacent Smoothness Constraint: To facilitate spatial-temporal modeling across LCVs and CVs, the recovered LCVss should be equipped with interpretable autocorrelation or trend proper-
ties, instead of "sudden changes" as DVs. To this end, encourage the smoothness of LCVs across time with a regularization term as
Lsmonth = =|| ||Abs (SxP) s (Sx (1
E ITT-T)XT
(1)
where S is the smoothness matrix and @ denotes the Hadamard product operation. Abs (Sx?) = denotes the absolute value of the first-order difference of DVs and can reflect the "sudden change" points (with state "1"). Overall, minimizing Lsmooth is equivalent to minimizing IT-1 1= Abs (zf.1 xP) (#ff - 2100)² by introducing the multiplication of the constant matrix S.
Abs (wf-TT - -(iTT-1)
Alternatively, we can adopt the K-Lipschitz continuity smoothness constraints, where the CVs can act as guidance to determine the smoothness degree of the recovered LCVs. Empirically, we find out such design would bring similar performance.
After latent continuity recovery, raw series of LC and a:C are independently embedded as tokens LC and c through variate-wise linear projection to describe the properties of each variable.
Adversarial Variable-Modality Discrimination: Inspired by the temporal similarity property, the recovered LCVs should exhibit similar temporal patterns and distributions to CVs. Accordingly, we devise a variable modality discrimination objective, which is optimized in an adversarial manner as
argmin Prs
max
(CDis = E [log (Dis (= + E [log (1- Dis (z1))))
(2)
(Ree.
where Dis denotes the variable modality discriminator. Dis is trained to distinguish LCVs and CVs as accurately as possible, while the recovery networks are facilitated to generate LCVs with characteristics similar CV as much as possible to confuse the discriminator. We adopt gradient reversal layer (GRL) to achieve the adversarial learning objective. In this way, we bridge the interactions between CVs and DVs and enable CVs to supply supervision signals for LCV recovery.
3.4 Intra- and Inter Varr--aaabbbaodollle
Spatial-Tempont Dependency Learning
Spatial-temporal dependencies are of vital importance for time series representation learning. Specif- ically for MiTS, the complete spatial-temporal correlations include the ones within LCVs CVs
5

---

Variable Adjacancy Maini
Self-Attention Sub-block
Crass-Attention Sub-blood block
Ceoat Aee
Toteer &c
20
Rie
sic
Leve
Lapeloom
Laperkare
SuifAtta (=")
Crankttine wi
within CVs
across
Feed foraadd
Feee forwad
CVa and LCVa
PCN
NCN
tierrs
sic
LCVe
%
SelfAtta(s**))
Mat wittle
Natouriils
across
within
420
LOVS
Vial
text
Figure 5: Spatial-temponts attention blocks. Left: MITS variable adjacency matrix, including the variable relationships i) within CVs LCVs and ii) across CVs and I.Vs;; Middle: -arialeariable- modality self-attention for modeling spatial-temporal dependencies within CVs or LCVs, and Right: pier-variable-moddae cross-attention for modeling those across CVs and LCVs.
and the ones across LCVs and CVs, which are explicitly characterized by spatial-temporal attention blocks in MiTSformer (as shown in Fig. 5
Intra- VariableeModdllii Self-Attention Within each variable modality, we adopt self-attention to model their spatial- temporal dependencies as
2f = LN (zf + Self-Attn (lof Kf, (f..f))) of = LN &2 + FFN (FF) 21.0 = LN 2 Sclf-Attn ([o!O K.C. v.o.)))))) 2/.0 LN 22 FEN (2(0)
(3)
where Self-Atta denotes the Multi-head Self-Attention that captures intra-variate spatial correlations by computing Softmax scores with query and key embeddings and weighted aggregating the value embeddings. FFN denotes the intra-variate global temporal representations. LN denotes Layer Normalization, which is applied to individual variate tokens and has been proven effective in tackling -onsttaiionayy problems [28]
network that processes each variable token learn
Inter- Variabbe-Moaaality Cross-Attention We adopt symmetric cross-attention sub-blocks to model the spatial-temporal interactions across L.V's and CVs
= = LN (2 + Cross-Attn (lof. 10f,Kto KfC,
(t/C.vlo)) v)()). (10 Kf, v Z141
E LN (2f41 + FFN (=f+1)
(4)
LC *11
LN LC 2i+1 + Cross-Attn
LC LN (#/4 + FFN (=(4)
Subsequently, the acquired token embeddings 2/C and zf are utilized both for 1) original MiTS reconstruction with Leo as Eq. [5] and 2) downstream tasks with task supervision loss Crasks e.g., Cross Entropy loss for the classification task. See Appendix for pipelines of each task.
Self-Reconstruction We devise pariate-wise decoders based on ML.ss to reconstruct the original DVs and CVs, which not only provide self-supervision signals for spatial-temporal dependency learning but also guarantee the recovered LCVs retain the information of the observed DVs.
p-
LReG (SeeeeeDDeooeerr (zf.,) x + CE(Rec-Decoder..c (= .
(5)
3.5 Synergy of Latent Continuity Recovery and Downstream Tasks
Considering both latent continuity recovery and downstream analysis task, the overall optimization objective of MiTSformer expressed as
L L + X + >a
(6)
Self Supervision
Task- Supervision
While seemingly separated from each other, the loss components of MiTSformer work in collabora- tive fashion on two aspects: 1). The first three self-supervision losses facilitate latent continuity recovery synergetically to tackle the spatial-temporal heterogeneity problem. Specifically, the smoothness loss LSmooth favorably manifests the autocorrelation of LCVs and alleviates the sudden

---

Table 1: Summary of experiment benchmarks. For each dataset, we randomly select n [0.5p variables as DVs, whose values are first MinMax normalized and then discretized into the value of 0 or with the threshold 0.5 int(MinMax(2) 0.5). See Table for more details.
Tasks
Benchmarks
Metrics
Seeries length
"Variables (p)
Classification
UEA (10 subsots)
Accuracy
29-v1751
3--963
Extrinsic Regression UCR (10 subsets)
MAF, MAF,RMSE RMSE
24-11440
4--24
Imputation
ETT (4 subiets), Electricity, Weather
MSE, MAE
7ru321
Anomaly Detection
SMD. MSL. SMAP, SWaT, PSM
Precision, Recall. F1-Soce
100
25,w55
96-720
Long-term Forccasting ETT (4 subsets), Electricity, Traffic, Weather, Exchange, ILI
MSE, MAE
(ILI: 24--0)
7 862
changes. Also, the discrimination loss LDis guarantees LCVs to be equipped with similar temporal dynamics with CVs adversarially. which guarantees a crucial condition for spatial-temporal modeling. Meanwhile, the reconstruction loss provides auxiliary self-supervision signals and constrains the LCV recovery processes to be reversible. Our experiments, particularly the ablation study displayed in Section 4.2, further justify the mutual dependency of the synergy between the three components; 2) The latent continuity recovery losses and task loss also work collaboratively. as reliable latent continuity recovery can bring excellent downstream task performance, while task supervision loss may, in turn, also reciprocate the recovery processes. Take MITS classification as an example, the ideally recovered LCVs can prompt learning discriminative representations for classification, while the class label loss provides additional supervision for learning appropriate recovery functions.
4 Experiments
To verify the effectiieness and versatility of MiTSformer, we extensively experiment on five main- stream mixed time series analysis tasks, including mixed time series classification, extrinsic regression, long-term forecasting. imputation, and anomaly detection.
Implementations Table |summarizes the experiment benchmarks. For each dataset, we randomly select half the variables and discretize them DVs generate MiTS data. More information about datasets and experimental platforms, hyperparameters and experimental configurations, and algorithm implementations can be found Appendix A. A.2 A.3 respectively. The pipelines of different mixed time series analysis tasks can be found in Appendix A.5 A.8
Baselines We extensively compare MiTSformer with the latest and advanced models in the time series community, including CNN-based models: Modern (2024), TimesNet (2023) and MICN
(2023); Transformer-basse models: iTransformer (2024) PatchTST (2023) Crossformer (2023)
FEDformer (2022) and Pyraformer (2022) MLP-based models: LightTS (2023) DLinear (2023) and
FiLM (2022) To guarantee fairness, keep the original backbone for each method the feature extractor, and we adopt universal task-specific heads and loss functions consistently for all methods.
4.1 Main Results on Different Tasks
(1) Mixed Time Series Classification We se- lect multivariate datasets from the UEA Time Series Classification Archive (5) and pre-press them following [40] As shown Fig. MiTS- former achieves the best performance with
Best buseline
70.1
70
$9.0
(ccuracy(%)
Averaged basel
basel168.1 168.1 68.2
average accuracy of 7 71.9%, surpassing all pow-
66.4
66
erful baselines. Besides, it can be observed that trequency-basee methods FiLM and FEDformer show inferior performance, the introduction of DVs may make it difficult to estimate the fre- quency reliably and yield non-negligible errors. In comparison, MiTSformer adaptively recovers and aligns the LCVs behind DVs with the guid- ance of both CVs and class-label supervision, thereby facilitating high-level representation learning and benefiting the classification tasks.
62
Pynofommer
PEDfomeer
RLA
Dinear
Lisars
Meni
Cressfrmeer
PaiGhYST
timesttet
MostinTcH
Wrnnoomer
AaTtformer
Figure 6: Classification Results (Ace 1)
7

---

Table 2: Imputation Task. The best results are bolded and the second-bes results are underlined. The
same goes for Table 3 See Table 14 for full results.
Medill
MiTSformedt
Thans
M-ICN.
Teme
ParcTTST
Cross (2025)
MICN
Ligh:TS
Dlincur
FILM (2022]
FED
Parn (0022)
(Ours)
2024)
(2024)
(2023)
(2023)
(2025)
(2023]
(2023)
22022
Metric
MAE MSE
MAE MSE
MAE MSE
MAE MSE
MAE MSE
MAE MSE
MAE MSE
MAE MSE
MAE MSE
MAE MSE
MAE MSE
MAE MSE
ETTml
(0115
0.049
0.169
0.05 0.037
8:057
0.39 0. 061000000000000000000000000000 00160 h1t.
0.080
ETTm2
0.116
0.036
0.186
0.080 0.077
ETTai
0.223
0.09
0.241
FTTh2
(0.166
0.083
[0255 8:1 33..3...000000..... 309
Electric
(0.186
0.076
(.2212
0.207 0.085)o.211 0.09-|0.200
Weather
0.031
|.091 0.038
0.081 0.03:Jo.149 0.065
12.
(2) Mixed Time Series Extrinsic Regression We select 10 multivariate datasets from TSER
0.62
0.607
0.60
repository (34) and pre-process them 33 As summarized in Fig. [ MiTSformer out-
0.58
0.576
performs all the rivals with an average MAE of 0.524, verifying its capacity to model com- plex extrinsic regression patterns. Besides, some transformer-based models, e.g., PatchTST and Crossformer, present competitive perfor- mance by learning global dependencies with self- attention. Notably, Dlinear shows inferior re- sults. This is probably because Dlinear adopts single-layer MLP to model temporal dependen- cies, which might be suitable for some autore- gressive endogenous patterns, but are not appli- cable for nonlinear exogenous regression relationships and degenerate performance.
0.56
0.554
0.553
0.13
0.54
0.66 rage- baseline
0.536u 5350.534
0.535
0:524
Bust bacaline
0.2-
050
048
Pyatormer
FEDromer
PIL
Dainear
Ligwr7
MICH
Crowsromer
PatchsT
Timestee
MostenTCH
fransforme
MIrsformer
Figure 7: Regression Results (MAE +)
(3) Mixed Time Series Anomaly Detection We conduct experiments on five widely-used anomaly detection benchmarks: SMD 33 MSL. 21], SMAP (21) SWaT (29 and PSM [4]
For fair comparisons, we adopt the reconstruc- tion loss for both CVs and DVs to train base models and use the reconstruction error as the shared anomaly criterion for all experiments. Specifically for MiTSformer, no additional task- orientation loss is added since there are already reconstruction losses. Fig. 8 presents the per- formance comparison evaluated by the Fl-score (t), demonstrating that MiTSformer consistently achieves state-of of-the-art performance on five benchmarks. Besides, iTrasformer also achieves great performance by adaptively modeling multi- variate correlations with self-attentions. In comparison, MiTSformer not only recovers the LCVs behind DVs but also comprehensively and
SMD
88,02
MITSformer
iTransformer
McdernTCN
MSL
PSM
TimesNei
05,00
95,90
PatchTST
Croesformer
MICN
LightTS
Diinear
FLM
FEDlomer
82,09
96.23
Pyraformer
SMAP
SWaT
Figure 8: Anomaly detection results (F1-score). explicitly characterizes the ppatial-tempont dependencies within and across DV' and CVs, thereby equipping meticulous anomaly detection ability.
(4) Mixed Time Series Imputation Following
Weather In practice, imputating the values of CVs is more important and meaningful than those of DVs. Therefore, we focus on imputing the CVs in each experiment, while DVs are used as inputs to provide auxiliary information. To compare the model capacity under different proportions of missing data, we randomly mask the time points in the ratio of (1..5% 25%, 37. 75,% 50%} in length-96 mixed time series and report the averaged imputation accuracy of CVs from 4 different mask ratios.
40 we select datasets of ETT 49
( (Wetterstation) and Electricity
8

---

Table Long Term Forecasting of CVs. "_" denotes out of memory. See Table 16 for full results.
Modals
MiTSGermer
(Oars)
Trnn. (2124)
M-TCN
(2024))
(2023)
(3025)
(3023)
(2023)
MRCN
LishTT
(2023)
Diinzer
(2025)
20223
(2022)
FED
(2022)
Metric
MAE MSE
MAE MSE
MAE MSG
MAB
MSC
MAE
MSE
MAC
MSC
MAE
MSC
MAC MSE
MAC
MSE
MAE
MSE
MAE
MSE
MAE
MSC
ETTml
0.376
0.326
0 5.540
0.350
0.334
0,440
0.357
0,579
0.330
0..477
0.358
0.385
0.394
0.345
0.381
0:331
0.397
0:357
1,425 0.37
0.484 0.470
ETTm2
0.365
0,363
0.371 0.373
0,306 0.371
10,776
0.995
0,508
0.377
1.456
0.528
0,500
0.529.0,5770...1
0,512
0,568
0.376
3899
0.34
1.732
ETTh
0.414
0.373
0.427
0.393
0.415
0.388
(1,446
0.412
D.416
0.576
0.445
0.404
(0.478
0.417
0.376
0.437
9.000
ETT22
0.445
0.449
0.472 [0,442
00,442
0.480
[1453
(1440))
(1.40
0146
0:917
1.422
0.692
1,000
j.729
1,063
6.675
0,982
(:450
0.490
4900
0.476
0.508
2,548
Weather
0.336
0.260
0.534
0.219
0.322
0.209
[1,348
0.296
0..2%
0.257
(0.338
0.257
0.27%
0.337
0.346
0:274
0,353
0.291
Exchango
0,398
(1A52 0.412
[0,448
0.403
[1,503
(1408)
(1453
0.417
(16022
0,412
0.324
0.445
0.409
0.318
(.449
0.398
0.399
1.504
01.99
0.400
11.1
0.779
0.995
2:1132
0.920
1,957)
2.01
0,973
2.140
2,960
1,358
2,360
1.734
5.430
340
3.197
188
2.700
3:00
2.747
Flectric
|0260
0.168
(.200
0:207
0.270
0.174
0.288
0.187
0.205
0.207
(0.3660.233
0.295 D. 0.185 (0.399
314 0,220
233160223
h 351 0.248
0.24000000.00000000000000000 0.400.0199
Traffic
|0.312
0.499
0:372
0.503
0.366
0.635
0.:554
0.503
0.360
|0,355 0.690 0.0.00.............
0.742
|44167744455095
Due to the missing values, the imputation task requires the model to deeply exploit the underlying temporal dependencies and spatial correlations from partial observations. According to Table 2 MiTSformer achieves the best performance on most tasks. Through the recovery of LCVs and the intra- and inter-variable modality attention mechanisms, MiT'Sformer can effectively bridge spatial-temporal information interactions between CVs and DVs. In this way, not only can CVs exploit useful information for self-imputation by mining temporal and spatial correlation themselves, but also DVs can provide reliable auxiliary information for more accurate imputation of CVs.
(5) Mixed Time Series Long-term Forecasting We follow the settings of prediction lengths and benchmarks as [40], including ETT [49], Electricity (UCI) Weather (Watterr(WWtttstatiion)) Exchange 23 and ILI (CDC) corresponding to different applications. We focus on forecasting both DVs and CVs.
Since CVs contain more fine-graing information granularity and can more adequately evaluate the model forecasting performance, mainly focus the prediction accuracy of CVs, which are summarized in Table. 3 As reported, MiT'Sformer presents the best performance on most tasks (76 % according to Table (6), surpassing extensive advanced MLP-based, Transformer-basel and CNN-based models. In addition, recent baselines- modern'TCN and iTransformer present great performance due to their delicate design of global temporal receptive fields.
(a) ETThl case#1
(b) ETThl case#2
(c) ETThl case#3
DERING
KOUFTEN
(d) ETThl case#4
(c) ETThl case#5
(f) ETTh2 case#1
(g) ETTh2 case#2
(h) ETTh2 case#3
(i) ET case#4
Figure Visualization of LCV recovery. For each subfigure, the Left plots the observed DVs, and the Right plots the actual LCVss (red line) and recovered LCV's (black line). The grey rectangular patches denotes the area where the observed DV is "1"

---

Table 4: Ablation analysis. For anomaly detection tasks, we do not ablate Rec. as it is needed to support anomaly criterion calculation. The corresponding results are omitted with /".
Models
Components
Cls. (Acc.)
Long-term-foree form--ooeeettinng 96-720 (MAE)
Anomaly detection (FI)
Dix Smmothh
Aross-Ar
UCR Avg.
ETTml |
ETThi
[TT
Avg.
[MD]
MSL.
SMAP
Avg.
MrTSformer
71.9
0.415
0.439
0.422
0.455
0.438
87.84
85.30
96.83
89.99
w/o Dis
x
70.1
0.416
0.440
0.466
0.459
0.440
86.56
83.24
95.70
88.50
Smooth
x
0.437
0.445
87.58
w/o Ruc
x
69.7
0.463
0.478
0.624
0.509
0.519
a/a LCNs
x
69.3
0.501
0.479
0.626
0.509
0.529
x/o Aross-Ai
X
70.2
0.428
0.442
0.451
0.461
0.446
84,92
84.10
95.79
88.27
4.2 Ablation Studies
To further verify the effectiveness of each key design, we conduct ablation experiments mixed time series classification, long-term forecasting, and anomaly detection tasks. The results are summarized in Table 4 where "w/o I.CVs" denotes removing LDis. Ssmooth, and LRe together, and "w/o Cross-Atl denotes removing the cross-attention sub-block and adopt self-attention sub-block solely,
Latent Continuity Recovery The three loss terms, LDis. LSmooth. and LRee support the latent continuity recovery from different perspectives and work collaboratively. As shown in Table 4 ablating each loss term would lead to performance degradation for different tasks. Remarkably, the employment of all recovery loss functions in conjunction yields the optimal result.
Attention Backbones The cross-attention block is devised to bridge the information prorogation among LCVs and CVs by modeling spatial-temporal dependencies across LCVs and CVs, as LCVs and CVs provide complementary information for analysis tasks. The results in Table emphasize the importance of cross-variable-mont modality dependency modeling for MiTS analysis.
4.3 Visualization and Model Investigations
Visualization of the Recovered LCVs To verify the interpretability of LCV recovery, we provide visualization plots Fig. 9 We find that DVs commonly show patterns of "sudden changes" or "steady states", which are less informative for analyzing opatial-temporal correlations. Fortunately, MiTSformer not only recovers their latent fine- grained and informative temporal variation patterns but also further leverages them for spatial-temporty representation lcarning, thereby mitigating the spatial-tempol heterogeneity challenge and achieving superior performance.
B showing MiTSformer maintains great performance and efficiency compared with most baselines. addition, we investigate hyprraaraeeter sensitivity in the Appendix C The results demonstrate that (1) MiTSformer is relatively stable the selection of odel capacity-related hyper-parameters dmmotet and number of layers L; (2) MiTSformer is quite robust to the weights of loss items (i.e., Åi, d1. and, Ag), and moderate weights bring optimal performance.
Additional Analysis We analyze the model efficiency in the Appendix.
5 Conclusion and Future Work
This paper focuses on a challenging yet seldomly explored problem the time series community and provides systematic and universal solution for mixed time series analysis. To address the spatial-temporal heterogeneity problem, we first reveal the LCVs behind DVs and try to recover them for sufficient and balanced apatial-temporal modeling. Accordingly, MiTSformer developed as task-general mixed time series analysis framework by leveraging the temporal similarities and spatial interactions between LCVs and CVs. MiTSformer can perform adaptive LCV recovery for DVs via the adversarial guidance of CVs as well as smooth constraints, and model complete spatial-temporal dependencies via self- and cross-attention blocks. With extensive empirical evaluations. MiTSformer shows great practicality, superiority, and versatility in five mainstream mixed time series analysis tasks. In the future, is of interest to empower MiT'formme with advanced pre-training techniques and powerful large language models for broader applications of mixed time series.
10

---

Acknowledgments
This work was supported by the National Natural Science Foundation China (No. 62450020 and
No. 62125306), and Zhejiang Key Research and Development Project (No. 2024C01163).
References
[1] Cdc. illness.
html.
[2] Uci. electricity. http://aacchiieessss....cdduuuuu oltp:///sssscccccoooooooooaaaane 12014/.
[3] Wetterstation. weather. https//wwww..ggeejee..gg.veeeeee
[4] Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. Practical approach to asynchronous multivariate time series anomaly detection and localization. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 2494, 2021.
[5] Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flyun, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv: 18/ 1.00005, 2018.
[6] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint 1803.01000.01277 2018.
[7] Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium-range global weather forecasting with 3d neural networks. Nature, 61977970):533-538, 2023.
[8] David Campos, Miao Zhang, Bin Yang, Tung Kieu, Chenjuan Guo, and Christian Jensen. Lightts: Lightweight time series classification with adaptive ensemble distillation. Proceedings of the ACM Management of Data, 12)):1-27, 2023.
[9] Alberto Castellini, Francesco Masillo, Davide Azzalini, Francesco Amigoni, and Alessandro Farinelli. Adversarial data augmentation for amm-based anomaly detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[10] Jiawei Chen, Pengyu Song, and Chunhui Zhao. Multi-scale self-supervised representation learning with temporal alignment for multi-rate time series modeling. Pattern Recognition, 145:109943, 2024.
[11] Jiawei Chen, Pengyu Song, Chunhui Zhao, and Jinliang Ding. Spatiotemporal multiscale corre- lation embedding with process variable reorder for industrial soft sensing. IEEE Transactions on Instrumentation and Measurement, 2023.
[12] Junhao Chen, Chunhui Zhao, and Jinliang Ding. A flexible probabilistic framework with concurrent analysis of continuous and categorical data for industrial fault detection and diagnosis. IEEE Transactions on Industrial Informatics, 2023.
[13] Kochenderfer M continuous data. In Journal of Artificial Intelligence Research, pages 59; 103-132, 2017.
Chen C. Wheeler A. Learning discrete bayesian networks from
[14] Kien Do, Truyen Tran, and Svetha Venkatesh. Multilevel anomaly detection for mixed data. arXiv preprint arXiv:1610.061: 2016.
[15] Kien Do, Truyen Tran, and Svetha Venkatesh. Energy-based anomaly detection for mixed data. Knowledge and Information Systems, 57(2):413 435, 2018.
[16] Luo donghao and wang xue. ModemnTNN A modern pure convolution structure for general time series analysis. In The Twelfth International Conference on Learning Representations, 2024.

---

[17] Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee-Keong Kwoh, Xiaoli Li, and Cuntai Guan. Self-supervised contrastive representation learning for semi-supervised time-series classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[18) Liangjun Feng and Chunhui Zhao. Fault description based attribute transfer for zero-sample industrial fault diagnosis. IEEE Transactions on Industrial Informatics,
2021.
[19] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1 1180-1189. PMLR, 2015.
[20] Rong Hu, Ling Chen, Shenghuan Miao, and Xing Tang. Swl-adapt: An unsupervised do- main adaptation model with sample weight learning for cross-user wearable human activity recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
[21] Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Soderstrom. Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 387-395, 2018.
[22] Yuxin Jia, Youfang Lin, Xinyan Hao, Yan Lin, Shengnan Guo, and Huaiyu Wan. Witran: Waver-wave information transmission and recurrent acceleration network for long-range time series forecasting. Advances in Neural Information Processing Systems, 36, 2024.
[23] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research development in information retrieval, pages 1041 2018
[24] Qiude Li, Qingyu Xiong, Shengfen Ji, Yang Yu, Chao Wu, and Hualing Yi. A method for mixed data classification base on rbf-elm network. Neurocomputiing 431:7-22, 2021.
[25] Shengsheng Lin, Weiwei Lin, Wentai Wu, Feiyu Zhao, Ruichao Mo, and Haotong Zhang. Segrnn: Segment recurrent neural network for long-term time series forecasting. arXiv preprint arXiv:2308. 1 1200, 2023.
[26] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting, In International conference learning representations, 2022.
[27] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024.
[28] Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems, 35:9881-9892 2022.
[29] Aditya P Mathur and Nils Ole Tippenhauer. Swat: water treatment testbed for research and training on ics security. In 2016 international workshop on cyber-physical systems for smart water networks (CySWater). pages 31-36. IEEE, 2016.
[30] Zelin Ni, Hang Yu, Shizhan Liu, Jianguo Li, and Weiyao Lin. Basisformer: Attention- based time series forecasting with learnable and interpretable basis. Advances in Neural Information Processing Systems, 36, 2024.
[31] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2023.
[32] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
12

---

[33] Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25tth ACM SIGKDD international conference on knowledge discovery & data mining, pages 2828 2837, 2019.
[34] Chang Wei Tan, Christoph Bergmeir, François Petitjean, and Geoffrey I Webb. Time series extrinsic regression: Predicting numeric values from time series data. Data Mining and Knowledge Discovery, 1533):1032-1061 2021.
[35] Giulio Ventura and Elena Benvenuti. Advances in Discretization Methods: Discontinuities Virtual Elements, Fictitious Domain Methods, volume 12. Springer, 2016.
[36] Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. Mien: Multi-scale local and global context modeling for long-term series forecasting. In The Eleventh International Conference on Learning Representations, 2023.
[37] Min Wang, Donghua Zhou, and Maoyin Chen. Anomaly monitoring of nonstationary processes with continuous and woluud variables. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 53(1):49-555 -5,, 2022.
[38] Min Wang, Donghua Zhou, and Maoyin Chen. Hybrid variable monitoring: An unsupervised process monitoring framework with binary and continuous variables. Automatica, 147:1 10670, 2023.
[39] Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y Zhang, and JUN ZHOU Timemixer: Decomposable multiscale mixing for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024.
[40] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations, 2023.
[41] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long, Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34:22419-22430, 2021.
[42] Haixu Wu, Hang Zhou, Mingsheng Long, and Jianmin Wang. Interpretable weather forecasting for worldwide stations with unified deep model. Nature Machine Intelligence, 5(6):602-611, 2023.
[43] Wanke Yu, Chunhui Zhao, and Biao Huang. Moninet with concurrent analytics of temporal and spatial information for fault detection in industrial processes. IEEE Transactions on Cybernetics, 52(8):8340-83511 2022.
[44] Jiaqi Yue, Jiancheng Zhao, and Chunhui Zhao. Similarity makes difference: Sshtn for gener- alized zero-shot industrial fault diagnosis by leveraging auxiliary set. IEEE Transactions Industrial Informatics, 2024.
[45] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121-11128, 2023.
[46] Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting, In The Eleventh International Conference on Learning Representations, 2023.
[47] Chunhui Zhao. Perspectives on nonstationary process monitoring in the era of industrial artificial intelligence. Journal of Process Control, 16:2555-22,, 2022.
[48] Chunhui Zhao, Junhao Chen, and Hua Jing, Condition-driven data analytics and monitoring for wide-range nonstationary and transient continuous processes. IEEE Transactions on Automation Science and Engineering, 18(4):1563-1574, 2021.
13

---

[49] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11115,
2021.
[50] Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun. Tao Yao, Wotao Yin, Rong Jin, et al. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances in Neural Information Processing Systems, 35:12677 2022.
[51] Tian Zhou, Ziqing Ma. Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. International Conference on Machine Learning, pages 27268-27286. PMLR, 2022.
[52] Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained Im. Advances in neural information processing systems, 36, 2024.
14

---

A Implementation Details
A.1 Datasets and Experimental Platforms
Despite being a fundamental issue, modeling mixed time series remains underexplored in academia, with lack of specialized benchmark datasets for mixed time series. To meet the tasks for mixed time series, we employed benchmark time-series datasets and implemented discretization to convert some CVs into DVs. Our conversion process simulates the generation process of DVs and discretizes variables while preserving their inherent coupling relationships and properties. We summarized the dataset descriptions in Table 5
All experiments are repeated three times, implemented in Py and conducted on Linux servers with Intel(R) Xeon(R) Gold 6246 CPUs and NVIDIA 3090 24GB GPUs. The versions of Python and Pytorch are 3.9.7, and 1.10.0 respectively.
A.2
Hyperparameters and Experimental Configurations
All the baselines that we reproduced are implemented based on their official codes or Time Series Library (TSlib |
Since some baseline models are designed for specific analysis tasks, we modify them to serve different analysis tasks by replacing task heads and loss functions. Specifically, we keep the original backbone architecture for each method as the feature extractor, and we adopt universal task-specific heads and loss functions consistently for all methods. The architectures of task heads and related loss functions will be introduced in the following section.
The apper-parameter configurations of MiTSformer are summarized in Table 6 and the hyper- parameter configurations of baseline models are summarized in Table 7 Some hyperparameters, including batch size, training epochs, dropout rate, and the number of attention heads (transformer- based models only) are fixed and kept the same for all baseline models and MiTSformer. Moreover, to compare the upper bound of different models specifically for mixed time series analysis tasks, we conduct grid searches of model capacity-relate dmodet/ hidden size and optimization-related models, MiTSformer additionally fine-tuned the weight of smoothness loss weight X to adapt to different temporal variations of different datasets. The variable modality discrimination loss weight is fixed at 1.0 and the reconstruction loss weight is fixed at 1.0 for all experiments.
prpee--prramters, including number of layers and rate. Compared with baseline
A.3
Algorithm Implementations of MiTSformer
Mixed time series modeling differs significantly from typical time series tasks due variable heterogeneity. Thereby, each component in MiTSformer is essential to address it via latent continuity recovery and alignment. We summarized the model feed-forward, loss-calculation, and parameter- update procedures of MiTSformer with pseudo-codes, which are presented in Algorithm [ We also report the standard deviation of MiTSformer performance with different random seeds in Table 8, which exhibits that the performance of MiTSformer is stable. Here we introduce the detailed architectures of each module MiTSformer.
Recovery Network The recovery network, receiving the input of DVs XD with shape (Batchsize T) and outputting LCVs XLCV with shape (Batchsize posed of 3 dilated convolution blocks with residual connections. Specifically, the Ith di- lated convolution block composed of 2 dilated ID convolutional layers with kernel size 2, hiddde_
T), is com-
8, dilation = 2' and GELU activations. Additionally, padding operation Padding ((kernelssize 1) dilation + 1/2 is adopted for the convolutional layer to ensure that the input and output have the same length T, where (kernel size 1) x dilation + 1 denotes the receptive field). Finally, we perform z-score normalization for each I.C. gIxT to ensure training stability as
11
x LC (C)
LC
x
Sid(zLC)
where Mean and Std denote the mean and the standard deviation along the time axis, respectively.
Self-attention and Cross-attention Sub-blocks The self-attention sub-block is composed of vanilla FullAttention with 8 attention heads and 0. dropout rate. The same for the cross- sub-block.
hntps://pytorch.orr/ https//gtthub..o//tuu///// Seriine---ee----eee Library
15

---

Table 5: Dataset deseriptions. The dataset size is organized in (Train, Validation, Test). "Dim. p" denotes the total variable dimension and "Dim. n² denotes the discrete variable dimension. Since current benchmark datasets are time series encompassing only continuous variables, we generate mixed time series from these datasets by discretizing partial variables. For each dataset, we randomly select half variables as DVs (n = [0.5p]). whose values are first MinMax normalized and then discretized into the value of 0 with the threshold 0.5 as int (MtMMMaaxx 0.5).
Task
Dataset
Dim p
Dim n
Series Length
Dataset Size
Semanties
ElhanolCConeenttttioon
3
1751
(261, 0. 263)
Alcobol Industry
PaceDetection
144
72
62
(5890, 0, 3524)
Face (250Hz)
Handwriting
3
152
(150, 0, 850)
Handwriting
Heartheat
61
30
405
(204, O, 205)
Heart Reat
Classification (UEA)
VopeneseVowels
12
(270, 0, 370)
Voice
PEMS-SF
963
481
144
(267, 0, 173)
Transportation (Daily)
SelfRegalatioSCCPI
6
896
(268, 0, 293)
Health (256H1z)
SelfRegulatinSCPP
7
1152
(200, 0, 180)
Health (256Hz)
SiokenArabicDigiss
13
93
(6599, 0, 2199)
Voice (1102HHz)
LWWaeGGsttneLibbary
3
315
(120, O, 320)
Gesture
AppliancesEnery
24
12
144
(96,0,42)
Energy Monitoring
HouseholdPower.
5
1440
(746,0,694)
Inergy Monitoring
HouscholdPower.2
5
2
1440
(746,0,694)
Energy Monitoring
Extrinsic Regression
8
240
(3433,0,5445)
Envirommnnt Monitoring
BeijingPM25Qualiny
9
24
(12432,0,5100)
Environment Monitoring
(UCR)
BeijingPMIQQulltty
9
24
(12432,0,5100)
Environment Monitoring
LiveRuclMoisure.
7
365
(3493,0,1510)
Enviromment Monitoring
AustrliiRRinfall
3
24
(112186,0,4888)
Enviroment Monitoring
PPGDalia
4
256
(43215,0,21482)
Health Monitoring
IEEPPGG
5
1000
(1768,0,
Monitoring
ETTmL
7
96
(34465, 11521, 11521)
Electricity (15 mins)
Imputation
ETTh1ETTh2
7
96
(8545, 288 I, 288
Electricity (15 mins)
Electricity
321
160
(18317, 2633, 5261)
Electricity (Hourly)
Weather
21
10
96
(36792, 5271, 10540)
Weatber (10 mins)
SMD
38
19
100
(566724, 141681, 708420)
Server Machine
MSL
55
100
(44653, 11664, 73729)
Spacecraft
Anomaly Detection
SMAP
25
12
100
(108146, 27037, 427617)
Spacecraft
SWaT
51
100
(396000, 99000, 449919)
Infrasructtre
PSM
25
12
100
(105984, 26497, 87841)
Machine
ETTml,ETT22
7
192.36.12
(34465, 11521, 11521)
Electricity (15 mins)
ETThJETTh2
7
196:96.19..336.720)
(8545, 288 1, 288
Electricity (15 mins)
Long-term
Electricity
321
160
[96;96,,922 192,96,1923336,720)
(18317, 2633, 5261)
Electricity (Hourly)
Forecasting
Traffic
862
431
(96;96,192,, 19,,36779))3
(12185, 1757, 3509)
Transportation (Hourly)
Weather
21
10
[96:96, 192,336,200)
(36792, 5271, 10540)
Weather (10 mins)
Exchange
8
(96.96.192.336,777
(5120, 665, 1422)
Exchange (Daily)
11.1
7
[24:24,36,48660)
(617, 74, 170)
Illness (Weekly)
16

---

Table 6: Experiment configuration of MiTSformer, All the experiments use the ADAM optimizer with the default hyperparameter configuration for (3,,82) as (0.9, 0.999) with proper early stopping, and adopt dropout rate of 0.1. A denotes the weight of smoothness loss, X2 denotes the weight of reconstruction loss, and X denotes the weight of variable modality discrimination loss. LR* denotes the initial learning rate. The number of attention heads set to 8 for all experiments.
Model Hyperparameters
Training Process
Task
Layers
dimoddtt
A1
Aa As
Batch size Epochs
Classification
[1,2)
(16,32,6,,228,556)
(0.1,0.3,0.5)
{10-3 9,5 10 10
64/16
100
Extrinsic Regression
[1,2)
[16.32.44.1282256]
[0.1,0.3,0.5]
(1079.5 5. 10
10
64
100
Imputation
(1,2)
(64,128,256)
(0.3,0.5)
1
{10-3,
(10",,10""
32
10
Anomaly Detection [1,2)
164,128,256
(0.3.0.5)
1
{10
10
32
10
Long-term Forecasting
{1,2)
124,128,256)
{0.3,0.5)
1
110-",10"*;
32
10
We set the batch size to 16 for the PEMS-SF dataset due to its high dimensionalityy We set the batch size to 64
for other classification datasets. The same for Table 7
Table 7: Experiment configuration of baseline models. All the experiments use the ADAM optimizer with the default hyperparameter configuration for (2) (0.9, 0.999) with proper early stopping, and adopt dropout rate of 0.1. LR* denotes the initial learning rate. For Transformer-based models, the number of attention heads is set to 8 for all experiments.
Model Hyperparameters
Training Process
Task
Layers
dmodee hidden size
LR"
Batch size
Epochs
Classification
(1,2)
(16,32,,4,11282566)
(10-3,5- 5 10 10-4)
64/16*
100
Extrinsic Regression
(1,2)
(16,32,,4,122,2566
(10-3,5 10 10
64
100
Imputation
(1,2)
(64,,28,256)
{10-3, 10-4)
32
10
Anomaly Detection
(1,2)
(64,12),256)
{10-9,
32
10
Long-term Forecasting
(1,2)
(6,,,2),256) {10-2, 10-4
32
10
Variable Modality Discriminator The variable modality discriminator, receiving the input variate embeddings with shape (Batchsize x p X dmodet) and outputting the variable modal- ity class with shape (Batchsize x p x 2). is composed of 3-layer MLP with structure of {d model, 4 d modell 4 modell 2}. Relu activation and batch normalization are also adopted.
Reconstruction Decoders The discrete variable reconstruction decoder, receiving the input of LCV embeddings (Batchsize x n x dmmode) and outputting the reconstructed DVs (Batchsize d d_model, output_sizz _size-iiee = 2 T (the discrete variable reconstruction can be treated as a binary classi- fication task). The continuous variable reconstruction decoder, receiving the input of CV embeddings (Batchsize x (p- n) dmodet) and outputting the reconstructed CVs (Batchsize x (p n) x T'), composed of linear layer with input_size sizee d_model, output_size - T.
T x 2). is composed of linear layer with input_sie _ize
In the following, we will introduce the architectures of task heads and loss functions for different downstream tasks.
A.4 Pipeline for Classification
Classification a long- -tanddignding task in the time series community and is widely used to evaluate the high-level representation capacity of models. The overall pipeline of SerSformer--ased classification depicted in Fig. 10 For the classification task, both token embeddings of LCVs and CVs provide complementary information. Thereby, we concatenate the embeddings of LCVs and CVs. The fused embeddings are flattened and fed into classifier network to predict the class labels. For MiTSformer and bascline models, the classifier is composed of a single-layer MLP with GELU activation and
17

---

Algorithm 1 The training process of MiTSformer
Input:
Input mixed time series with continuous variables X (2,, Ip-n ) R(P-a)xTT,
and discrete variables X { ... InI e"xT, number of attention blocks L, loss weights
X1 A2 and A3.
Output: Optimized model parameters of MiTSformer.
I: Initialize the parameters of MiTSformer;
2: while not converge do
3:
(I) Feedforward Computation:
4: > Latent Continuity Recovery for DVs
5:
x/C Rec-Net(x?);
6:
> Variate-wise Token Embedding of LCVs and CVs
7:
20 (mbed
8:
of + Embedcv(x
9:
> Inter- and attr--ariaable-odall Spatial-Temporal Modeling
10:
for = 00, L.., 1] do
11:
> Intra-Variable-
Self-Attention Sub-Block
& LN
of + Self-Attn
Qf KY,
& <- IN & FFN (E))
$/c + LN x + Selr-Altn
@otC KtC vtol))
2 + LN &t +FFN (t(
15:
16:
Inter- PnteerVarrian Variable-Modall by Cross Pross-Attention Sub-Block
17:
141 < LN & + Cross-Atth
C
Qi ,(_,v(() ))
vt=l))
18:
*41 - LN (2f+) + FFN (=t+1)
C
19:
241
LC LN
2(41 [CCC Cross-Attn (f0 KV,V(( vi]))
LC + LN (=LC FFFN (=(4))
21:
end for
22:
Task Prediction (if needed)
23:
ú +- (eaks xf)
24:
(2) Loss Calculation:
25:
Task Supervision Loss (if needed)
26:
T + Mask-Criterion(j.11 y).
>Smoothness Constraint Loss
28:
Lsmooth = Abs P) & (Sx40)
29:
<Reconstruction Loss
30:
LR Σ MSE(Rec-Decdder (zf,)) xx9) + Ei= CE(Rec-Decoder (z .::)) AVariable Modality Discriminatio Loss (via Gradient Reverse Layer)
31:
CDs [log (GRL(Dis (=))) E [log GRL((Dis (=40))
33:
+veralll Loss
All Lrask + A + do X + ds
35:
(3) Parameter Update:
36: Update the parameters of models using Adam optimizer to minimize Calt
37: end while
38: return Optimized model parameters.
18

---

Table 8: Robustness of MiTSffrmmer performance on forecasting datasets. Averaged MAE, MSE, and their standard deviations based on different random seeds are reported.
Datasel
ETThl
ETTml
ETTm2
Horizon
MAE
MSE
MAE
MSE
MAE
MSE
0.38140.001
0.32340.000
0.34240.001 0.271=0.000
0.29340.002 0.231=0.003
192
0,40840.000 0.369=0.000
0.363=0.0003 0.31
0.34420.001 0.324=0,002
336
0.42440.002 0.402+0.002
0.382+0.002
0.338=0.003
0.383+0.000 002 0.395=0.001
720
0.442+0.004 0.398+0.002 0.415=0.003
0.391=0.003 0.439 +0003 0.503=0.002
Dataset
Electricity
Weather
Traffic
Horizon
MAE
MSE
MAE
MSE
MAE
MSE
96
0.235+0.002 0. 0.14
0.231=0.000 0.147=0.001
0.292#0.002 0.459=0.0002
192
0.25040.000 0.15920.002
0.29420.000 0.21620.001
0.310 0.49420.002
336
0.265+0.002
0.171100.001
0.353=00.001 0.298=0.000 0010
0.31620 .003 0.508=0.003
720
0.291+0.003 0. 0.198*0.000
0.426#0.002 0.412##.001
0.330#0.005 0.534=0.003
a dropout layer with a dropout rate of 0.1. The task supervision loss is Cross-Entroe Loss. The classification accuracy is used as the performance evaluation metric.
Reconstruct
Decoder
TunTU ruwuu munu
Concal
Ourput
Observed
MiTSformer Backbone
Classifler
Fused Embeddings
Observed CVs
Variate Tokens CVs
Reconssturttin Decoder
Figure 10: Overall pipeline MiTSformer- based classification. The embeddings of LCVs and CVs concatenated, flattened, and fed into the classifier for classification.
A.5 Pipeline for Extrinsic Regression
Closely related classification, time series extrinsic regression aims learn the relationship between time series and continuous scalar variable. The overall pipeline of MiTSformer based extrinsic regression depicted in Fig. IT] Similar classification, both token embeddings of LCVs and CVs provide complementary information for regression tasks. Thereby, we concatenate the embeddings of LCVs and CVs. The fused embeddings are flattened and fed into a regressor network to predict the numerical values. For MiTSformer and baseline models, the regressor is composed of single-layer MIPP with GELU activation and dropout layer with dropout rate is MSE Loss. We adopt the absolute error (RMAE) and the root mean square error (RMSE) as the performance evaluation metrics as
The task supervision loss
RMSE
(0:
yi
(7)
N
1 N N
MAF
Σ 1
Vil
(8)
where Ny denotes the number of test samples, ÿ denotes the predicted labels and yi denotes the ground-truth label.
19

---

Recomstratt Decader
L m humU
Flatlen
Output
Observed DVss
MiTSformer Backbone
Regressor
mmmam Mamnn
Fused Embeddings
Observed CVs
Variate Takens of CVs
Recandiruction
Docoder
Figure 11: Overall pipeline of MiTSformer-basee extrinsic regression. The embeddings LCVs and CVs are concatenated, flattened, and fed into the regressor for regression.
A.6 Pipeline for Imputation
The imputation techniques are developed to impute the missing values based on the partially observed time points. For the imputation task, mainly focus imputating the missing value of CVs, while DVs are adopted as input to provide auxiliary information. The overall pipeline of MiT'Sforme--baee imputation is depicted in Fig. T1 For MiT'Sfommer, the task head is an imputation decoder composed of a linear layer, which performs variate-wise imputation for each CV. For baseline methods that adopt the channel-independee methods that adopt the channel-mixing strategy, we adopt a linear layer with input size of dmodet and output size of continuous variable dimension p M TT*(p=)) to represent the missing values in input CVs. The state of M denotes whether the corresponding element value of CVs is missing (denoted by 0) or not (denoted by 1). The task supervision loss the MSE loss calculated on the masked observations. Specifically for MiTSformer, the reconstruction loss CVs is calculated on unmasked observations. We adopt the mean absolute error (MAE) and the root mean square error (RMSE) the performance evaluation metrics that computed on masked elements.
strategy, adopt the same task head as MiTSformer. For baseline
reconstruct the CVs. We use mask matrix
Reconstruct Deoodar
WUnT munmn
Observed DVs
MITSformer Backbone
Variate Tokens LCVs
Imputation Decoder
Obeerxed
Imputed DVs
Takens
Reconstruction
Linear
Deccaer
Figure 12: Overall pipeline MiTSformer-basee imputation. The embeddings of CVs are individu- ally fed into the imputation decoder impute missing values of CVs.
A.7 Pipeline for Long-term Forecasting
Forecasting a fundamental problem in the time series community, and long-term forecasting is more practical and challenging task. The long-term forecasting task for MÎTS includes the prediction of both DV' and CVs. The overall pipeline of MiTSformer-ban based long-term forecasting depicted in Fig. 13 For MiTSformer, the task head is composed of linear layer-based DVforecastor and CVforecaster, which perform variate-wise forecasting for each DV and each CV, respectively. For baseline models, we also adopt the linear layer- based forecaster to forecast the DVs with the extracted temporal features. We adopt the mean absolute error (MAF) and the root mean square error (RMSE) the forecasting performance evaluation metrics.
20

---

Recorittust Decoder
DVForecaster
MUL
Observed
MiTSformer Backbone
Linear
Textanm
CVForecaster
Observed CVs
Linear
Future
Reconstraction
Decoder
Figure 13: Overall pipeline of MiTSformer-basee long-term forecasting, The embeddings LCVs individually fed into the DVForecaster predict the future value of corresponding DVs, and the embeddings of CVs are individually fed into the DVForecaster to predict the future value of corresponding CVs.
A.8 Pipeline for Anomaly Detection
The anomaly detection task is achieved by self- supervised autoencoding with the reconstruction of both CVs and DVs. The reconstruction errors of DVs and DVs are utilized as anomaly criteria. In our experiment, we estimate the anomaly thresholds respectively for DVs and CVs. For test sample, the reconstruction error of DVs exceeding the threshold or the reconstruction error DVs exceeding the threshold is considered an anomaly. The overall pipeline of MiTSformer- -oasoeee----eddd anomaly detection is depicted Fig. 13 Since there already exist reconstruction decoders and corresponding losses, no task head and additional losses are devised. For baseline models, we also adopt the linear layer-based reconstruct decoders to reconstruct the DVs and linear layer-based reconstruct decoders to reconstruct the CVs with the extracted temporal features. MSE loss is used for reconstruction of CVs and CrossEntropy loss used for reconstruction of DVs. We adopt the Precision, Recall and F1-Socre as matrices.
Reconstruet Decoderr
PLIMMSMM uuuUuu
Locaa
Observed DVs
MiTSformer Backbone
mmmma Wamme
Frehhrus
Observed CVA
Varieta Tokerns of CVs
Reconstraction
Decodder
Figure 14: Overall pipeline of MiTSformer-bart anomaly detection. The anomaly detection tasks only rely self-reconstruction and thus no task head is attached.
B Model Efficiency Analysis
Complexity of MiTStormer Considering the self-attention and cross-attention sub-blocks, the complexity of MiTSformer can be derived as
(n²) + (p-((1))) +1np--))++((pp1))>)
(9)
sell-attention
cross-attention
where n denotes the number of DVs and p n denotes the number of CVs, p is the number total variables. In our experimental setting, we keep [0.5p]. Thereby, the complexity of MiTSformer quadratic the number of DVs O (n²).
21

---

ETTh1 (96-720)
Eloctriclty96-7222
0.58
0,40
ICODUG
0.54
0,36
MARCOO
0.50
048
0.32
PAR TOT
Memory Foctpr int
0.50
300MS
10000NB
044
0.20
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.38
0.40
0.45
2
8
Trahiin Time (alter)
Trainin Time (sitar)
(a) ETThl (3DVs, 4CVs)
(b) Electricity (160DVs, 161CVs)
Figure 15: Model efficiency analysis. Experiments are carried out ETThl and Electricity datasets with "input-output" setting of "96-720". For each subfigure, dots with different colors represent different methods, and the size of the circle represents the magnitude of the memory footprint. The horizontal axis represents the training time (seconds) per iter, and the vertical axis represents the forecasting accuracy (MAE).
Model Efficiency Comparison We comprehensively compare the model performance, training speed, and memory footprint of MiTSformer and baseline models. The results are with the optimal model configurations reported in Table 6 and 7 Two representative long-term forecasting datasets ETTh1 (3DVs, 4CVs) and Electricity (160DVs, 161CVs) with input-96 and output-720 settings are adopted for efficiency comparison. The results are depicted in Fig. I5
In general, MiTSformer maintains great performance and efficiency compared with most baselines in datasets with a relatively small number of variables (ETTh1). When encountering datasets with relatively large number of variables (Flectricity), MiTSformer occupies a relatively large memory footprint, specifically compared with some advanced efficient models such modern TCN and DLinear. However, the training time of MiTSformer still efficient.
C Hyper-parameter Sensitivity Analysis
We evaluate the hyper-parameter sensitivity of MiTSformer with respect to the following two aspects:
Sensitivity of Model Capacity The hidden dimension dmodet and number of layers L influence the model capacity of MiTSformer. We evaluate the sensitivity of these two hyper- parameters on typical long-term forecasting datasets, including ETTh1, ETTh2, and Weather with four different prediction horizon settings. The results are presented in Fig. 16 We can find that MiTSformer is relatively stable in the selection of dymodel and I., particularly for ETTh2 and Weather datasets. Specifically for the ETTh1 dataset, the model capacities are not essentially favored to be as large possible.
Sensitivity of Loss Functions We further investigate the effects of loss items, including smoothness loss weight Å₁ reconstruction loss weight A2, and variable modality discrimination loss weight Xg on classification datasets (Japanese Vowels, Arabecrra
and SelfRegulltionSCCP)) The results are presented in Fig. 17 In general, we can observe that MiTSformer is quite robust to the weights of loss items, and moderate weights bring optimal performance. For example. A₁ controls the magnitude of the smoothing constraints for latent continuity recovery. Too small a would make the smoothness invalid while too large A may lead over-smoothing problems. Therefore, moderate setting of λ is favored for MiTSforemer.
Vowels,,
D Limitations and Future Work
Handling Discrete Variables with Natural Sudden Changes The smoothness loss in our method suitable for DVs with sudden changes that are caused by inherent smooth variations. However, it may not adequately account for DVs with inherent sudden changes that are essential characteristics of the dataset. For such cases, can adjust the coefficient X with relatively small value, can
22

---

0350
18,00
6440
6003
6425
1. 415
335
0A22
Layera
Layor
6420
Layera
monel
d. modee
monle
358
255
251
(a) ETTh1 96-96
(b) ETTh! 96-192 192
(c) ETThl 96-336
2.44
03375
042
0,330
0.32
0/0
828
Layers
Lapers
0,330
Layors
128
mose
d. Modul
123
294
(d) ETThl 96-720
(e) ETTH2 96-96
(i) ETTh2 96-192
0
454
2446
3.4
04725
642
041
205
646
0.40
34225
Layers
Leyon
Layers
2. modsi
325
2 mooul
(g) ETTh2 96-336
(h) ETTh2 96-720
(i) Weather 96-96
E.120
035
0330
3.40
6.475
03
0344
222
0,3500
331
0.2
625
628
133
Layers
Layers
03530
Layer
m.masess
d d_mosee
123
m_made
(i) Weather 96-192
(k) Weather 96-336
(I) Weather 96-720
Figure 16: Sensitivity analysis of model capacity- apper-parameters dmodel and number of layers L. Experiments are carried out on typical long-term forecasting datasets, including ETTh1, ETTh2, and Weather datasets. 000-9///19//3///77 denotes different forecasting length settings
23

---

Apaorctionn ScMlepulserrP
Sp gic
Jranneriiees Sc SchhqqualoneC71
Sellepuitoneee
100.0
100.0
100.
Accurbyy
95.0
Mccurnr
45.0
Mecuracy
95.0
92.5
62.5
925
90.0
90.0
90.0
0.5
0.8
0.5
5.0
10
0.1
5.0
Weight Smoothness
Weight of Reconstruction Loss
Weight Discrtmiaatoo Loss
(a) Sensitivity of X
(b) Sensitivity of X2
(c) Sensitivity of A
Figure 17: Sensitivity analysis of loss items, including smoothness loss weight A₁, reconstruction loss weight 22 and variable modality discrimination loss weight λ3. Experiments are carried out on classification datasets Vowels, Vowels, SpokenArabicDigits, and SeliRegulationSCPI.
further leverage less restrictive constraints such as K-Lipschitz continuity to determine the proper smoothness level.
Modeling Generalized Mixed Time Series MiTSformer is based on the assumption that the observed discrete variables are derived from latent continuous variables, which can be guaranteed for integer discrete variables whose discrete states represent the magnitude of the numerical value. However, limitation of MiTSformer is that it can not be directly applied to categorical discrete variables whose discrete states represent different and mutually exclusive class labels (e.g., color and gender). It is noted that integer discrete variables are more frequently encountered real-world time series data, specifically in the fields of medical analysis, finance, and industrial processes. In contrast, categorical discrete variables are often recorded as independent samples that are not presented in time series forms. In the future, it is of interest to exploit the analysis of generalized variables (including both integer discrete variables and categorical discrete variables) by leveraging advanced cross- modality modeling and fusion technologies.
Incorperating Powerful LLMs and Pre-training Techniques Recently, large language models, such as the Generative Pre-trained Transformer (GPT), have shown impressive performance on various applications and attracted numerous research interests. It is intriguing to explore the potential of GPT-type architectures for mixed time series analysis tasks with proper inductive biases and elaborate prompt strategies. Additionally, by utilizing efficient pre-training techniques, we can enhance the scalability of MiTSformer in large-scale or real-time real-world applications. Meanwhile, the mixed time series analysis on few-shot and zero-shot settings can also be a promising research direction.
E
Additional Experiments and Discussions
E.1
Comparison with Mixed Naive Bayes and Variational Inference-basee methods
To deal with mixed variables, naive Bayes (NB)-based models 38 and variational inference (VI)- based models (12] are proposed to match the distributions of continuous variables (CVs) and discrete variables (DVs) with different distribution priors for industrial process data analysis. Essentially, mixed NB and VI methods are typically designed for tabular data. They struggle with time series and often rely on certain assumptions like conditional independence, limiting their ability to model the correlations between DVs and CVs. In contrast, MiTSformer leverages temporal adjacencies to achieve latent continuity recovery, making it capable of handling time series data and effectively capturing inherent nonlinear correlations within MiTS. Empirically, typical mixed NB-based models HVM [1] and VI-based methods VAMDA [2] on mixed time series classification datasets and summarized the results in Table 9 showing MiTSformer consistently outperforms HVM and VAMDA.
compare MiTSformer against
F.2 Investigation of Non-binary Discrete Variable
In our main experimental setting, we chose the binary DVs as they are the most challenging and commonly encountered in the real world. Actually, DVs can take on multiple states (> 2) that reflect the magnitude and can be directly input into the recovery network to obtain LCVs outputs. It is noted
24

---

Table Compared to mixed NB- and VI-based methods. Accuracy(%) scores are reported. The best results are bolded.
Datasets
MiTSformer
HVM VAMDA
Handwriting
22.6
8.1
10.8
Heartheat
74.6
57.7
63.4
Japanese Vowels
94.6
86.4
78.4
SeIlRegulationSCPI
91.1
85,3
88.7
SelfRegulationSCI2
60.0
51.1
54.4
SpokenArabicDigits
98.5
90.2
90.8
Table 10: Performance of mixed time series classification under the different discrete states of DVs. iee, NDV. Accuracy (%) scores are reported. The best results are bolded.
Datasets
NDV. = NDv.
Concentionn
30.4
30.4
FaceDetection
67.9
68.3
Handwriting
22.6
23.1
Heartheat
74.6
74.1
Japanese Vowels
94.6
95.9
PEMS-SF SF
93.1
92.5
SelfRegulationSCPI
91.1
92.8
SellRegulatioSSPP2
60
61.3
Diaie
98.5
98.7
LWaaeGrstureeibbrary
86.3
85.9
Average
71.9
72.3
that more states in a DV imply richer information granularity. Also, we conducted experiments on with various numbers of discrete
mixed time series classification and anomaly detection datasets states in Tabel 10 and Table TT respectively. In general, the results shows that as the number of states increases, performance improves due the richer information.
Table 11: Performance of mixed time series anomaly detection under the different discrete states of DVs, i.e., Npvs. The best results are bolded.
Datasets
Metric
Novs = NDvs
Precision
88.92
88,37
SMD
Recall
86.78
87.92
F1-score
87.84
88.14
Precision
90.66
90.81
MSL.
Recall
80.54
81.22
F1-score
85.30
85.75
Precision
96.71
96.82
SMAP
Recall 72.21
75.26
Fl-score
82.69
84.69
Precision 96,31
94.33
SWaT
Recall 95.98
94.82
F1-score 96.15
94.57
Precision 97.88
98.41
PSM
Recall 94.85
95.62
F1-score
96.83
96.99
25

---

Table 12: Full classification results. We report the classification accuracy (%) as the result.
MrIS. (Ours)
ITans. (2024)
M-TCN (2024)
Times (2023)
Patch
Cross.
MICN
Light. (2023)
Dlincar (202))
FILM
FED.
Pyza
Datasett/Models
(2023)
(2023)
(2033)
(2022)
(2022)
(2022)
EthunoCCosccatration
30.4
26.6
28.5
30.0
25.9
31.9
28.9
30.4
29.3
27.8
27.0
24.8
FaceDeteccion
67.9
67.9
66.1
67.9
67.5
67.3
65.2
67.9
68.0
65.9
67.4
67.4
Handwriting
22.6
17.8
18.2
15.7
13.9
15.9
6.4
8.1
11.5
8.2
14.2
17.8
Heartbeat
74.6
66.8
76.6
73.7
72.2
72.2
72.7
72.2
74.1
72.2
73.7
74.1
JapaneseVowels
94.6
93.0
94.3
93.5
90.3
92.4
79.2
92.4
87.0
77.0
94.6
93.0
PEMS-SF
93.1
87.9
84.4
86.7
83.2
83.2
85.5
85.0
86.7
85.7
85.5
84.4
SelllegulationSCPP
91.1
87.4
92.1
89.1
86.0
89.4
89.1
90.4
89.8
84.9
73.7
89.8
SelfRegulationSCC2
60.0
58.3
57.8
56.1
56.1
57.2
56.7
56.1
57.2
55.6
55.66
54.0
SpolenArrabieiigiss
98.5
98.4
98.1
98.7
98.0
97.7
97.7
97.8
96.1
97.3
98.7
98.9
UWweeGesturelibrray
86.3
84.4
84.7
84.7
83.4
82.5
82.5
81.9
80.9
74.7
57.8
84.7
Average
71.9
68.9
70.1
69.6
67.6
69.0
66.4
68.2
68.1
64.9
64.8
68.9
Table 13: Full extrinsic regression Results. We report the MAE and RMSE the result.
Model
MiTNformmrr
(Oars)
TTansfomer (2024)
TCN
TuncaNies
20003)
PacTTST
Condeeter
(2008))
LishTT
(025)
FBD
(0023
Paraforuer
Metric
MAR RMSE
MAR RMSF
MAR RMSE
MAF RMSF
MAE RMSF
MAE RMST)
MAR RMSF
MAR RMSE MAF RMSE
RMSF
MAE MSR
RMSR
Appli
0.579
0,730
(0.622
0.864
60s 0.805 (6.664 0,800 Ju675 0.877
10.007 0.966
60.411 0.808
205 0.873
0..02 0.892
(0751 0,970
a 0:806
0.64 0,853
4.41
0.555
[0.412
0.54
40 045 (0:197 0.538
(53 0.462
0.477 0.308 0.516
40404 0.542
(.528 0.705
(u477 0.638
House.2
0.671
0,895
|0,630
0,907
(0,81 0.910 (6,570 0,920
(o,674 0,915
10,00
0,898
10,091
0:928
(0705 0,956
10,085 0.927
Jos8t
0,928
0,390
0.591
Benne.
603352
0,481
(0.371
0,518
(0,390
0.515 (0.365 0,475 (0.85 0.992
|.332
0.484
00.995 0,526
(0.303
0.486
(0359 0,477 (0.41
0.745
4.450
0.191
[0.39]
0.494
BJPM25
(0537
0,782
(0,569
0,790
(0.538 0.787 (0.514 0,720
(0,566
0,795
(0:57
0,788 10.525 0,796
(0.550 0.799 10.50 0,827 (0.49
0,803
0.539
0,782 0.773
0,886
(0,509
0,839
(0.54 0.899
(0.583 0,568
(0572 0.871 [0.509
0.867 10.55 0,962
(0.56s 0.867 (0.587 0,896 (05922 0,870
0.563
0.5030.8559..30
0.856
Live
60750
1,000 (0/752
1,000 (0.780 1.031 [0.009 1,017 j0225 1.016 10,947 1.000 0.775 1.001 (0,663 1.010 1.752 1007 007254 1.001
0.750
1,000
(0758 1.009
Austm.
10223
0,966
(0,263
0.965
(0,200
0.95 (0,062 0.988
(0255 0.954 10.267
0.96 00.364 0,900 (0.254 0.959 10255 0,968
10.76 0,964
0.255
0,966
0.53
PROD:
104003
0.613
10,436
0,692
(0.397
0.544 (0.997
0.586
402 0.594
(0.372
0.588 0.398 0,547 (0.43 0,579 1,.119 0.17 j.502 0.712
0.413
0.560 0.589
186E:
0.755
0.928
(0.779 0,973 ju755 0.939 (0.96 1.003 (.751 0.945 (0.655 0941 0.736 0,932 (u811 1.030 (..90 1,075
jos51 0.996
0.741 0.902 0,902 094s
Average
0.524
0,780
j0.550
(0.535 0.789 00.55 0.301 (a.534 0.791 (0.535 0.790 (0.366 0.788 (0.54 0,810 (0.507 0,368 0.599 0.822
0 0.36 0.78
Kindly note: For dataset abbreviations, "Appli." denotes "AppliancesEnergy", "House. 1" denotes "Houschold- PowerConsumpttinn "; "House. 1" denotes ArbuushhoddoweerCCossuniiiii
"Benze." denotes "BenzeneCon- centration"; "BJ.PM25 denotes "BeijingPM2SQuaitty" "BJ.PMI0" denotes "BejjiggMMIOQualii "Live." "Austra." denotes "AustraliaRainfal"" "PPGD." denotes "PPGDalia", and "TEEE." denotes "IEEEPPG".
denotes [Lvveuuc
F Full Experimental Results
The full results of all experiments are presented in the following: classification in Table 12 extrinsic
regression in Table [3] imputation in Table 14, anomaly detection in Table 15 Besides, long-term forecasting results of CVs are presented in [6 those of DVs are presented in 17
26

---

Table 14: Full imputation results. We randomly mask 12.5% 25%, 37.5% and 50% time points to compare the model performance under different missing degrees.
Model
MiTSforusrr
(Chnc)
Manddonuee
(204g)
M-TCN (2024)
TingsNer
(2023)
PacLTST
(2023)
Cressforseer
(2023)
(M03
MICN
Lichrrs 20033
Dlincer
(2033
HLM (2025)
|HHDiormed [2022]
Pyzaforner [2022]
M-Ratio
MAF
MSR
MAF MSF
MAF MSF
MAF MSF
MAF MSF
MAR MSF
MAF MSF
MAF MSF
MAF MSF
MAF MSF
MAR MSF
MAR MSF
12.5%
0.143
0.042
0.149
0,046
0.123 0.001
0.029
0.151
0.045
0.150
0.045
00115
0.038 0.156 0.050 0.53
0.53/0.011
0.101 0.08
0.144 0.0160.173 0,063
25%
0.152
E1047
0.162
0.053
0.130 0,004
0.131 0.035
0.159
0.053
0.155
0,043
00.55
0.049 0.172 ouss (.182 0.009
0.183
0.0700.111
0.055 0.192 0,060
37.5%
0.160
0.061
0.175
0,060
0,138
0.038
0.140
0.167
0.055
0,057
0.185 0.688 0.203 o
0.204
0.087
0.059 0,085
50%
0.170
0.057
0.190
0.070
0.149
0,043
0.164
0.052
0.18O
0.067
0.172
0.058
0.178
0.065 0.200 0.079 0.222 0.110 0.222 0.110
0.079 0,096
Avg.
0.156
0.049
0.169
0,057
0.135
0,037 0.139
0,1129
0.166 0.057
0.160
0.051
00166
0,152
(L178 00064 (0.195 0L080 0.0900 0.08000.170
0.060 0.081
12.5%
0.104
0.030
0.163
0.062
1,009
0,064
0.040
194
0.090
0.096 O. 0.087
0.086/..311
0.065/0.234
25%
0.112
IU034
0.182
0.177 0.178
0,368 0.159 0,055
0.138 Q.150
0.211 0,091 0.224 10104 0.205 OLIMB (1:44 0.135
0.246
0.136
0.042/0.246 0.121
51
0.120
0.195
0,067 0,198
0.079 0.180 0.012
0.147
0.056
0.268 0,151 0.213 OL09S (1272
0.274
0.169
0.34 0.159
50%
0.128
0.043
0.205
0.096
0,212 0.065 0.205 0.092
0.162
0.056
0.232
0.128 0.313 O. 0.219 0.102 (.2000 o 0.302
0,300
0.193
0.179
Avg.
0.116
0.836
0.185
0,080
0,188
0,007
OL17O 0L005
0.145 QJU55
0.216
0.103
0.245 0.131 0.209 OUIMS (1.255 0.147
0.258
0.152
0.228
0.123
0.205 0.341
12.5%
0.203
0.080
0.213
0.091
0.199
0.077 0.201 0:078
2.221
100
225
0.104
1.196 .07 2.239
22223 o 103
0.231
0.100
2011
0.075/0.214
0.085
67
0.117
0.091
0.252
0.108
0,204
0.222 0.097
0:240 0.116
0.095 0.254
(.220 0. 0.356
0.135
0.206
0.095
0.095
0.230
0.100
0.250
0.124
0.223
0.093 0.247 0. 0.259
56 2525 0.126
0.115 0.274 0.155 (1.275
0.169
0.279
0.165
0.322.0.254
0.118
50%
0.243
0.111
0.266
0,142
0.2444
12126 O. 277 0. 152 0.281
0.163 0.266 0.159 0.272 O. 150 O 296 1818 (0.900 c. 192 0.504
o 198 0.200 o 0.157 0.220 0.152
Avg.
0.223
1.096
0.241
0.136
0.215
0,092 0.257 0.112 0.251
0.1 0.12990.2000 0.240 0.121 0.254 O. 0.260 0.145 (12202 0.145 0.208 0.152 0.244 0.112 0.224 0.113
12.5%
0.175
0.073
0.224
100
0.259
0.148 0.2344 D. 0.237
122 227 0.163 0.246 O. 120 100.0000000005111011
0 0.165 o 155
0.195
0.182
0.242
0.125
0,293
156
0.253
140
195 0 0.286 0. 0.308 0020 0.284 011.00.0000000302202202000000030
0.209
0.206
37.5%
0.189
0.085
0.258
0.141
0,525
0,270 0.332 0. 0.243 0.268 155 0.311 0.206 0.338 0. 239 0.322 0.220 0.266
0,365
0.291 0.370
0,333 0.269
50%
0.199
0.093
0.276
0.160
0,990
0.287 0.408 O. 300
0.762 O. 1 325 0.226 0.4033 O. 343 0. 0.25 0.25
o. 0.500 0.515 0.300
Avg.
0.186
0.083
0.250
0. 1.15
0,309
0,213 0.319 230 0
O. 145 304 0. 0.196 (0.318 0. 414 0 200 2.022.00.01221111
0 0.2.2
o 0.258 0 0.253
12.5%
0.154
0.855
0.184
0,171
0.178
0.365 0.189 0.075
0.180 0.067
0..81 0.056
0.199 0,077 0.191 0.075 (0.206
0.086
0.085
0.083/0.259 0.130
Elaariity
0.177
0.066
0,200
0,062
0.067
0.079
0.075 0.220 0,093 0.214
(0.243
0.11
0.243
0.250
1.117 0,143
37.5%
0.195
0.082
0.221
0.099
0,214
0,083 0.216 0,099
0.207 0.091
0:205 0.117 0.240 0.111 0.233 0.105 (.227 0.144
0.274
0.136
0.142
0.157
50%
0.216
0,897
0.239
0.114
0.236
0.113 0.233 0.1
0.251
0.222
0.257
0.131
0.259 0.131 (.252 0.122 (1.204 0.175 00.204
0.176
0.169/0.258 0.169
Avg.
0.186
0.076
0.212
0.093
0.207
0,088 00.11 0
D.
203 0. 15 209
0.097 0.229 0, 100 0.223 0.099 o 130 0.257 o 228 0,060 0. 130 2.24 150
12.5%
8.055
B.829
0.080
0,033
0.074
0.303 0LUM3 0,035
0.0833
0,034
0.116
0.044
0.0944 0,035 0L095 0.036
0.036 (00997 0.037 0.100
0.039
0.039/0.052 0.00M
Weother
25%
0.060
0L088
0,1156
0,USE 0,082
0.051
OLONS 0,035
0.111
0.122 0.147 0.107 ouse0 01116 0.045
0.118
0.046.
0.045
0.199 0.041
37.5%
0.065
0.032
0.095
0.039
0,083
0.033 0.173
0.074
0.088
0.037
0.114 0.043
0.148
0,050 0.106 0.041 00.12
0.05]
0.131
0.0520.144
0.144
0.058
0. 0.04
50%
0J869
0.034
0.102
0,042
0,057
0,EE5 0.195 0,099
0.097
0,041
0.119
0,045
0.1176
0.076 0.118 0u986
(1.143 0.059
0..45
0.000 0.000 0.118 0.047
Arg.
0J062
1.031
0.091
0.158
0.USE
0.149 0.005
p.08
0.057
0.115
0,043
0.1155
0.055
0.107
0.143
(1.111 o.04s
01124 0. 0.144.0.123 0.139 0.057 0. 0.042
Overall Avg
8.155
0.062
0.086
081
0.000 O 100
0.120
0.216.
(00
127.0.2000000000 0.000...33 0. 0.123/0
Topl Counss.
34
12
2
0
0
0
0
0
0
Table 15: Full anomaly detection results. The "P", "R", and "F1" represent the precision, recall, and F1-score (%) respectively. Fl-score is the harmonic mean of precision and recall. A higher value of P. R. and Fl indicates better anomaly detection performance.
Datavet
SMD
MSL
SMAP
SWaT
PSM
Avg. F1 (%)
Metrics
R
F1
F1
MiTSformer
88.92
86.78
87.84
90.66
80.54
85,30
96.71
72.21
82.69
96.31
95.98
96.15
97.88
94,85
96.83
89.76
87.66
75.90
81,36
84.13
82.87
83,50
86.72
77.15
81.65
96.24
93.77
94.99
98.88
92.87
95.78
87.46
ModemICN (2024)
86,90
73.95
79.91
93:25
60.50
73.39
96.84
50.69
66.50
96.34
94.45
95.39
98.92
93,20
95.98
82.23
TimesNe (2023)
88.56
81.32
84.78
94.41
71.01
81.06
95.04
$4.50
69.27
95.69
95.74
95.71
98.56
92.49
95.43
85.25
PatchTTT 2023)
87.75
75.35
81.08
99488
67.27
78:71
94.51
54.85
69.53
96.28
95.16
95.72
99.15
92.70
95.82
84.17
Crossforme (2023)
(2023
83,98
75.76
79.60
995.73
69.85
80.77
95.22
60.07
73.67
94.47
94.77
96.11
98.39
92.89
95.57
85.14
MIC (2023)
89.65
82.47
85.91
155.35
71.42
81.67
95.001
56.62
67.73
95.25
95.72
95.48
98.92
88.22
93.27
84.81
LightTS (2023)
87.10
70.73
76.07
94.85
69.57
80.27
94.77
52.94
67.93
96.24
95.75
96.00
988.58
92.51
95.45
83.14
Dlinea (2023)
87.91
78.86
83.14
94.40
60.45
73.71
95.51
:2..33
67.36
97.22
94.88
96.04
98.99
92.80
95.79
83.21
FILM (2022)
87.07
75.46
80.85
774662
59.42
73:00
94.82
51.20
66.53
96.17
95.48 95.83
|8.
03
92.45
95.39
82.32
FEDformer (2022)
85.23
84.14
84.68
96.05
68.77
80.15
95.35
$4.82
69.61
96.99
94.31
95.63
99.11 1 92.08
95.47
85.11
Pyraforme (222)
83.27
71.55
76.97
96.07
71.99
82.31
95.53
62.08
75.26
99.05
92.81
95.83
98.85
91,46
95.01
85.08
27

---

Table 16: Full long-term forecasting results for CVs. We compare extensive competitive models under different prediction lengths. The input sequence length is set 36 for the ILI dataset and 96 for the others. Avg. is averaged from all four prediction lengths. The performance is evaluated by MAE and RMSE. "_" denotes "out of memory".
Models
MiTSformser
(Ours)
Manssonuee
(2024)
M-TCN
(2024)
TimesNet (2023)
PaccITT
(2023)
Cressformer (2025)
MICN
20233
LishrTs (2023
Dlincas
22123)
220223
HHDDoommed (2022)
Pynafomner
(30221
Langth
MAF
MSR
MAF MSF
MAR MSF
MAF MSF
MAF MSF
MAR MSF
MAF MSF
MAF MSF
MAF MSF
MAE
MSF
MAR MSF
MAR MSF
96
0.342 0.271
0.348
0.219
0,346
0.272
0.363 0.283
0.345
0,276
0.368
0.296
0.352 0.275 0.364 0.296
0.244
0.275
0.362
0.297 0,398
0.319
0,408
Em1ml
192
0.363
0,310
0.371
0.321
0.967 0.314 0.400 0.000.355
1.367
0.313
0-401
0.349
0.369 0. 921 0.982
0.366
0:112
0.392
0.348
0,34800445
0.415 0.35m 0.460 0:441
0.382
0.238
0.347
0,387
0,344
0.371
0.387
0.342
0.389 0.328 0:401 0.358
0..88
0.36
0,404
0.367
0,436
0.393.044992 0,479
720
0.415
0.291
0.430
0.414
0,421
0,404 0,437 0.416
0.416
0.389
0:442
0,417 0:429 0,399 0.450 0.407
(0:425 0.399
0.428
0.414
0.413 0.552
0.376
0.328
0.185
0,340
0,380 0,314 O 0.4033.00
0, 357
0.3790033300..0
407
0,358
( 0.385 O. 351 ( 348
0.331
0.307
0:357
0:357
0.435
0.371 00484 0,470
95
0.293
0.231
0.298
0.241
0.299
0,239 0.500 0 250
0.299
0.252
0.434
0.397 0.352 0.277 0.388 0.312
0.312
(0.276 0 300
0.305
2.5000.32
0.255/0.259
0.381
E13m2
0.344
0.324
0.351
0,331
0.339
0.319 0.345 0.334
0.340
0.316
0.566 0.649 0.480 0.454 01481 0.473
(1,443
0.421
0.357
0.345
00.55
0.349
0.551 0.582
0.383
0.395
0,402
0,385
0,400
0,437
0.390
0.399
0.735
0,970 0.575 0U670 0.555 0.614 0..53 o 604
0.294
0.418
0.417
720
0.439
0.503
0.445
0.518
0,444
0.525 0,445 0.558
0.443
0,509
1.776
2,209 0,703 O. 0.690 0.913 0.688
0.543
0.449
0,457
0.519 L.688 4,769
Avg.
0.365
0.36.3
0.371
0,373
0,350
0,371 0.376
0,395
0.368
0.364
0.878
1,056 0.528 0,599 0.529 0,575
(1.112 0.568
(0.576
0.385
0.385
0.899 1.732
0.381
0.323
0.393
0.338
0.383
0.335
00.41 0.354
182
0.330.
0.299
0.927
0.990 O. 928 4.451 0.426
0:131
00397
0:36510.40)
0 0.335/0.483 0.483
677h
0.408
11369
0.419
0,383
0,400
0.381 0.434
0.390
0.4044
0.371
0.418
0.371
0.426 0,374 0L475 0.462
0..15
(1.113
0.375
0.419
0.399
0.573 0.533
1:424
01402
0.436
0,419
0.425
0.466
0.446
0.417
0.426
0.446
0.476 0.442 0.488
01:427
0.4144
0,445
0..44
0.416 0.563
720
0.442
0.398
0.460
0,430
0.445
0.418
0.473
0.450
0,4
0.416
0.389
0.498 0.472 4.69
0.470
01444
3.39
0.514
0.5160044000
0.434/0,567 0.991
Avg.
0.414
0.373
0.427
0,393
00,455
0,388
0.466
0.412
0.416
0.383
0:425
0.376 0.448 O. 9044 0.478 0.465
(0.417
0.376
0.443
0.430
0.437
0.390
0.526 0,543
96
0.380
0.369
0.986
0.905
0,400
0:420
0.385
737
1.016 ( 504 546 0,555
0..944 0 383
0.404
0:41
00.476
1.429
1.014 1,778
61Th2
0.430
0.438
0,452
0,442
0:437
0:480
1.447
0.612 0,775 0.636 0.795
0,496
0.515
2.645
0.455
0.494
0.464
0.506
0.454
0.506
0,520
0.457
0,495
0.944
1,450
1,028 0.765 1.113 0.736 1.067
01,63
0.518
1.414 2.819
720
0.455
0.473
0.478
0,519
0,478
0.523
0.472 0 909
0.476
0.51
1,88
1,795
(.9300 1.651 0.960 0.720
1.701
0.487
0.536
0.542
| 1.462, 2.951
Avg.
0.430
0.449
0.442
0,472
0,442
0,450 0.453 0,490
0:440
464
0.917
1.422 (.692) 1,000 0.729 1.063 0.675 o 983
0.450
0.490
0. 5.00
| 2,548
96
0.231
0.147
0.243
0.233
04 1o 0.242.00 0.
0.161
2.44
0. 141 (0.283 0. 0.269 0.11 (0.767 0. 175
0.266
o 18310,300 0 0.000.259 0.152
Wenther
0.294
0.302
0.228
0,292
0.307
0.297
0.220
(0327 0,233 0.319 0.318 228
0.317
0.245
0.270
0.334 0.228
336
0.383
0.298
0.361 0,310
0.355
0.259 0.374 0.322
0.356 0.300
0.357 0.272 0.379 0. 0,381 0.304
(1.266 0 0.309
0.315
0.347
0. 0.310
720
0.426
0:412
0.429
0.417
0.431
0.470 0-467 0.470
0:428
0.418
0:44
0.415 (.:435 0. 394 0-447 0.402 0.384
0.461
0.445 0.457
0.446 0.406
Avg.
0.326
0.268
0.334
0,279
0,308
0.269 O. 348 0. 206
0.
0,276
0 338
0.257 0.356 0,278 0.354 0.277
0.346 o 274
0.353
0.291
0.336 0. 0.332
0.357 0.274
0.221
0.994
0.227
0.096
0.225
0,097 0.266 0.131
0.234
0.099
0.345
0.225 0.2336 0.101 0.264 0.120 o
0.251
0.112
0.334
0.187/0.481
0.187
0.441 0.390
Exchange
0.327
0.394
0.335
0.211
0,353
0.205
0.246
0.341
0.219
0.339 0, 0.384 0.245 01333
19%
0.340
0.213
0.430
0.325
0.577
336
0.445
0.239
0.455
0.366
0,460
0.369 0.501 0,429
0.452
0.360
690
0,762
0.432
0.311 0.370
0.321
0.447
0.358
0.519 0.815
720
0.788
0.963
0.790
0.974
0,7744
0.942 0.876 1,156
0.795
0.995
0.844
1.099
0.660
0,683 0.714 0.869
(1,629
0.651
0.757
0.910
1.365 0.819 0.962
Avg.
0.445
0.398
0.452
0,412
0,448 0,400 0 505 0,496
0.453
0:417
0 596 0.633
0.412
0.323
0.459
0.400
(0.409
0.318.04449
0.318
0.449
0.564
0 1.500 0.500/0.6500 0.679
0.771
1.542
0.950
2.112
0,938
2,060 0.882 19958
0.550
2.06:
1 1,079
2.744
2.211
2,567 1.088
5.046
1.456
3.552/1.323
1.323
2.642
1.377 3.322| 1.101 2.533
0.761
1,387
2,066
0,953
1.966 0.896 1.994
0.939
2,146
1.100
1.047
2,155 1.696
5:217
1.208
2.621
1,275 3.056
1.063 2.745
0.777
1.482
0.962
2,040
0.907
1.904 0 0.898
2.078
0.985
2,183
1158 3.014
1.055
2.201 1.749
5:574
1,273
2.969
1,000 2.664
1,207 2.826
1. 1.104 2,832
60
0,805
1.516
1.105
2,309
0.911
1,897 0.897 2,071
1.017
2,269
1 .,22 3.258
1.117
2,496 1.804
5,890
1.278
3.034
1.330
2,881
1,208
2,500
1.115 2,877
Avg.
0.779
1.482
0.995
2.152
0.922
1.957 0.991 2.035
0.573
2.140
1.1 1140 2,962
1.35% 2,360 1.734 5:452
1.340 3.197
1.188 2.702
1 1.267 3.003
1,096 2,747
0.235
0.143
0.268
0.150
0,243
0.545 0..63.00
0.275
87
(0.278 0, 0.318
0 2.770.0726
22007603000
0. 0.220
300300 0.314
Elecnielty
0.250
0.359
0.278
0.189
0,259
0.165 0.275 0. 179
0.28000.190
0.15 0.226
0.286 0. 0.326 0.225
(1.30) 0 2066
00.00
0.216
0.400 0.319
336
0.265
0.171
0.296
0.207
0.202
174 0.200 0. 180
0.296
0.204
0.349
0.257
0.298 0. 186 0.342 (.30022222223311-43333 0.217
0.313
0 0.347.0.40 0.306
720
0.291
0.198
0.331
0.252
0.306
0.210 0.318
0.217
0.322700
0.245
0:371
0.200
0.319 0.100.200 0.368 0.336 0.21
0.361
0.384..355
o 0.206 0.441 0,305
Arg.
0.260
1L168
0.293
0.207
0.2700
0.174 0.285 0.187
0.295
0.207
0.326 0.257
0.295 0.185 0L339 0.240
01311
0.351 0.248 0,400 0.519
0.240
0.292
0.459
0.357
0,560
0,358
0.333 0,760
0.555
0.333
0.464
0.7732.0.774
0,409 073660042
Inllix
192
0.310
0.494
0.361
0.573
0,360
0.627 00.833 0,
0.349
0.350 0.689 0.456.0.99
0.:11
0.396 0.74t 0.544 0.908
336
0.316
0.508
0.371
0.595
0.364
0.640 0.399 0. 801
0.363-0.611
0.361 0.696 0 0.809
0.412.0.19
0..40 0.765 0.765 0.957
720
0.330
0.534
0.397
0.645
0,382
0,676 0. 0.370 0. 860
0. 0.642
0.375 0.790 0.479 0.863
(0.429
0.445 0.83000.500 1.055
Avg.
0.312
0.499
0.372
0.593
0,396
0.835
0.354
0.800
0.600
0.355 0.692 0.824
(0.42 0.742
0.416 0.774 0.945
Topl Counts.
55
28

---

Table 17: Full long-term forecasting results for DVs. We compare extensive competitive models under different prediction lengths. The input sequence length is 36 for the ILI dataset and 96 for the others. Avg. is averaged from all four prediction lengths. The performance is evaluated by
forecasting accuracy (%).. "_" denotes "out of memory".
Models
Mixfo. (Ours)
Trans. (2024)
M-TCN (2024)
TimesN (2023)
Patch. (2023)
Cross. (3023)
MICN (2023)
Lieht. (2023)
Dlinctr (2023)
FILM (2022)
FEDior (2022)
Pyzafo (202))
96
73.73
71.84
73.49
71.26
72.24
68.50
73.38
71.11
70.72
71.50
72.12
68.28
192
70.72
68.80
71.14
68.33
69.74
68.00
70.58
68.58
68.71
70.16
67.50
65,95
65.87
68.37
720
67.81
65.99
66.94
65.68
66.58
63.36
67.47
64.35
64.91
67.57
63.30
62.95
Avg.
70.36
68.51
70.17
68.20
69.00
66.44
69.89
67.48
67.67
69.40
67.24
65.67
96
73.31
71.19
74.37
73,80
71.44 14.44
70.24
71.36
71.49
70.80
70.91
72.00
64,20
F1Tmm
192
71.25
70.79
72.65
73.12
69.59
70.35
70.20
67.77
67.76
69.13
70.12
64.12
67.85
66.82
720
68.07
66.85
69.99
69.08
65.97
66.37
67.74
65.82
64.30
65.60
66.00)
53.98
Avg.
70.63
69.28
71.95
71.86
68.71
68866
69.29
67.998
67.46
68.30
69.06
61.56
96
73.17
68.88
68.19
69.06
70.16
68.86
70.87
67.40
65.30
68.09
68.35
66.87
192
68.93
68.20
67.68
68.36
69.02
67.07
69.13
67.04
65.10
68.74
66.63
62.62
66.29
65.29
68.04
66.39
67.60
720
67.28
63.72
66.53
54.49
67.01
66.59
66.41
66.87
63.58
65.53
63.74
59.32
Avg.
69.22
67.00
67.17
64.26
68.54
66.95
68.61
66.93
64.57
67.49
65.32
62.77
96
69.40
67.86
69.05
70.85
68.75
67.60
70.63
68.32
67.95
67.05
69.89
63.18
192
67.01
65.30
68.85
67.61
66.49
65.24
68.02
66.02
63.87
65.91
68.33
55.52
67.78
63.95
720
65.74
62.27
68.81
65.87
63.53
61.78
68.38
68.22
60.84
63.25
64.99
58.82
Avg.
66.87
64.91
68.76
67.78
65.99
64.11
68.70
66.63
63.68
65.20
67.32
58.71
96
78.74
76.23
79.37
78.38
75 69.9
79.22
78.48
77.14
73.91
75.31
78.83
77,74
Weathet
192
77.08
75.84
78.27
77.38
75.15
78.17
77.93
76.43
72.51
74.95
76.77
76.11
336
76.59
78.34
76.18
75.20 15.20
77.77
75.45
75.11
72.29
75.52
74.92
720
71.24
76.73
78.79
73.21
76.23
77.69
75.11
74.39
71.53
75.46
74.68
73.19
Avg.
77.41
76.09
78.69
76.29
75.57
78.21
76.74
75:77
72.56
75.31
76.27
75.49
96
51.16
48.16
48.21
49.17
49.17
48.62
48.42
48.34
49.91
49.26
46.44
49.92
Exchange
192
52.71
51.21
52.08
49.42 .9.42
50.82
48.32
48. 14
50.06
51.36
49.07
44.38
50.28
336
52.51
49.90
50.67
50.84
45.24
44.33
51.42
51.26
49.55
42.65
720
59.49
49.45
58.88
55.37
50.65
41.11
54.49
56.49
51.28
$1.87
49.05
$7.09
Avg.
53.97
49.68
51.94
51.16
50.37
45.82
48.85
51.58
50.95
49.94
45.63
51.96
24
81.93
77.39
56.69
80.51
75.38
64.57
72.90
58.62
56.41
$4,92
54.82
67.21
66.19
71.94
56.94
$7.17
48
84.81
80.11
57.76
84.14
79.48
68.97
71.55
57.71
55.39
57.70
63.00)
75.39
60
83.37
78.71
59.47
81.37
76.99
69.97
71.50
61.61
53.23
58.84
60.56
74.62
Avg.
83.55
79.14
57.26
82.31
77.79
67.43
71.97
58.72
55.15
57.16
60.03
71.93
96
90.83
89.85
89.94
90.34
89.67
89.35
89.74
88.73
85.90
86.93
88.62
88.67
Fllctriiity
192
90.52
89.32
336
89.84
89.12
89.51
89.19
89.22
87.57
89.08
88.33
84.97
88.46
88.91
87.48
720
88.74
87.66
88.36
88.26
87.86
86.91
88.01
88.00
83.76
87,34
87.89
87.14
Avg.
90.40
89.59
89.58
89.74
89.56
88.43
89.38
88.56
85.48
87.96
88.97
88.03
96
88.42
87.664
85.63
88.38
86.43
86.94
85.71
80.00
85.37
86.65
Traffic
192
336
89.24
88.59
89.07
89.86
88.08
89.98
88.72
82.61
89,22
88.02
720
89.71
89.10
90.48
90.68
88.
88.91
89.65
85.30
89.59
89.26
Avg.
89:07
88.51
88.21
89.48
87.69
88. 65.65
87.97
82.37
88.08
87.79
29

---

G
Broader Impacts
This paper copes with the general analysis of real-world mixed time series, which under-explored in literature but frequently encountered in various practical applications. such industrial main- tenance, healthcare monitoring, and financial analysis. Since previous studies struggle address the opatial-tempore heterogeneity problem for real-world MiTS, we present MiTSformer that fun- damentally reveals and recovers the latent continuous variables for discrete variables to facilitate exploiting the intricate spatial-temporal patterns within MiTS, thereby being amenable to various analysis task. Our model achieves the state-of-the-arr performance on five mainstream tasks that cover 30+ real-world datasets from diverse application domains. Therefore, the proposed model makes promising to tackle real-world MITS analysis tasks, which helps our society make better decisions. Our paper mainly focuses on scientific research and has no obvious negative social impact.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have clearly claimed our contributions in the end of the introduction from the perspective of (1) Problem Formulation, (2) Technical Design, and (3) Empirical Evaluation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have analyzed the limitations our proposal in Appendex D Also, we investigated the computational efficiency of our approach in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
31

---

Answer: [NA]
Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided the details regarding reproducibility. including computa- tional platforms, dataset descriptions, model configurations, task task-specific pipelines, hyperpa- rameters settings, and pseudo-code of our approach in Appendix [A] Also, We have released the source code on Anonymous Github as stated in the Abstract. The download links of the datasets and pre-processing functions are also included in the project homepage.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
32

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The datasets used in our paper are benchmarks and are publicly available. We have provided the relevant references and hyperlinks of datasets. Also, We have released the source code Anonymous Github as stated in the Abstract. The download links of the datasets and pre- pre-procesiing functions are also included in the project homepage.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. ec/
bublic/guides/ Staeees//ooeSuubbssssooooooo
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided all the experimental implementation details in Appendix A including (1) dataset descriptions and splits of training/validationneeel sets (Table 5);
(2) Experiment configurations and hyperparameter settings of our approach (Table 6) and baselines (Table 7).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All experiments are repeated three times and we have reported the average results. Also, we have investigated the sensitivity our approach Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
33

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have clearly stated the experimental platforms (e.g.. GPUs and CPUs) and software (the version of Python and Pytorch) used in our paper Appendix A and we have analyzed the computational efficiency of our approach in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't it make it into the paper).
paper (e.g., preliminary or failed experiments that NeurlPS Code of Ethics https: httpr//s.....pbbbceaccaaaceeeaaa
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the
Answer: [Yes]
Justification: anonymity.
We have reviewed the NeurIPS Code of Ethics and we have ensured to preserve
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
34

---

We have discussed both potential positive societal impacts and negative societal impacts of work in Appendix G
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the original paper (or URL) of the code package or dataset that is used in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
35

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pace se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have released the source code on Anonymous Github stated in the Abstract. The instructions for running our codes are provided on the project homepage at Anonymous Github. If our paper is accepted, we commit to releasing the relevant code as open-source and will provide detailed documentation support its use and replication of our results.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
36

---

Answer: [NA]
Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:
Justification:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
37


Reviews:
[{"review_id": "A2K390JjNW", "reviewer": "Reviewer_GdPs", "summary": "The MiTSformer framework includes Latent Continuity Recovery, which recovers latent continuous variables (LCVs) from discrete variables (DVs) using multi-scale temporal context and adversarial guidance, and Spatial-Temporal Attention Blocks, which capture dependencies within and across LCVs and continuous variables (CVs) through self- and cross-attention mechanisms. The paper explores general mixed time series analysis, addressing spatial-temporal heterogeneity. MiTSformer adapts to recover LCVs and capture spatial-temporal dependencies, demonstrating state-of-the-art performance across five tasks on 34 datasets.", "questions": "The paper claims that using MiTSformer allows for freedom from the strict limitations of mixed naive Bayes models and variational inference previously used to match the distributions of continuous variables (CVs) and discrete variables (DVs). However, the reasons for this claim are not clearly explained. A detailed explanation is needed to justify why MiTSformer is less restrictive. Additionally, a performance comparison between MiTSformer and these traditional methods would be beneficial to substantiate its effectiveness.\n\nSecondly, did you compare the latent continuous variables (LCVs) obtained from randomly sampled DVs in the forecasting tasks with the actual CV information? Such an analysis would be interesting and could provide deeper insights into the model's performance.", "limitations": "One limitation mentioned is that the framework cannot be directly applied to categorical discrete variables. In practice, many time series datasets combine categorical data with time series data, such as event data and product sales demand. This limitation is significant since these types of datasets are common in real-world applications.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a fresh problem definition by addressing the heterogeneity between continuous variables (CVs) and discrete variables (DVs) in mixed time series analysis.  The MiTSformer framework effectively recovers latent continuous variables (LCVs) from DVs and captures spatial-temporal dependencies, offering a balanced and comprehensive modeling approach. The framework is highly effective in handling various types of variables, providing clear insights into their relationships, which is invaluable for designing robust foundation models for time series.", "weaknesses": "Firstly, the experimental setup relies on converting continuous variables (CVs) to discrete variables (DVs) for experimentation, rather than using datasets with naturally mixed CVs and DVs. This may introduce unnecessary complexity and potential information loss, weakening the paper's motivation. Secondly, the complexity of the framework, which includes the recovery network and adversarial learning components, may pose challenges for practical implementation and scalability. The claim that the framework effectively recovers the inherent continuous nature of DVs and maintains temporal similarity through adversarial learning lacks robust verification and evidence. The improvements attributed to L_Dis are minimal, as shown in Table 4's ablation study, and further analytical reasons should be provided to support these findings.\n\nMiTSformer", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer GdPs for the favorable recognition of our work's problem setting, effectiveness, and invaluable contribution to designing robust time series foundation models. We deeply appreciate your detailed and perceptive feedback. Rest assured, we are committed to addressing your concerns and improving our work.\n\n***W1: Experimental settings of DVs***\n\nThanks for raising this insightful point. Actually, in the real world, many DVs originate from latent CVs due to constraints like measurement and storage limitations. We elaborate on this in $\\underline{\\text{Line } 47 \\sim 58 \\text{ and Figure 2}}$ in our paper, which may interest you. Our conversion process simulates the generation process of DVs and discretizes variables while preserving their inherent coupling relationships and properties. This operation also allows us to compare the recovered LCVs with the actual ones, which are investigated in the responses to **W3&Q2** for you. In light of your feedback, we will add relevant clarifications to our paper.\n\n***W2: Practical implementation and scalability***\n\nWe appreciate your valuable insights and are glad to address your concerns below.\n* **Necessity**: We would like to emphasize that **mixed time series modeling differs largely from typical time series tasks due to the variable heterogeneity**. Each module in our MiTSformer is essential to address this problem. Specifically, the recovery network is a lightweight module but is indispensable for recovering LCVs. The ablation study displayed in $\\underline{\\text{Table 4 on Page 9}}$ also justifies its effectiveness.\n* **Computational Efficiency**: We analyzed computational effort in $\\underline{\\text{Appendix B on Page 20}}$, demonstrating the great efficiency of MiTSformer. Additionally, we conducted an experiment for the recovery network based on ETTh1, showing that the computational costs brought by the recovery network are minimal, while the performance improvements are significant. \n\n||MiTSformer|w/o Recovery Network|\n|:-|:-|:-|\n|MAE|0.442 |0.626|\n|Time Costs (s/iter)|0.065|0.060|\n|Memory Footprint (Mb)|1397|1336|\n\n* **Pre-training for Real-world Applications**: We can employ self-supervised pre-training for the backbone of MiTSformer, whose parameters can be fixed during deployment and we can fine-tune the task head if needed, ensuring both efficiency and scalability.\n\nThanks for the opportunity to clarify these issues and we will add them to our paper.\n\n***W3&Q2: Verification of LCV recovery***\n\nThis is an invaluable and interesting suggestion. As suggested, we have included the relevant visualizations and quantitative evaluations in $\\underline{\\text{Figure 1 and Table 1 of global response PDF}}$, showing MiTSformer achieves accurate recovery of LCVs for DVs, which re-establishs their continuous and fine-grained temporal variations. In light of your suggestion, we will add relevant results and discussions to our paper.\n\n***W4: Effectiveness of the discrimination loss***\n\nThanks for your rigorous concerns. we design the variable modality discrimination loss $\\mathcal{L} _{\\mathrm{Dis}}$ to align the feature distributions of CVs and LCVs, facilitating their correlation modeling. The discrimination loss works collaboratively with other loss terms and we discussed their synergy in $\\underline{\\text{Section 3.5 on Page 6}}$. The performance decrease without the discrimination loss is not substantial because smoothness loss $\\mathcal{L} _{\\mathrm{Smooth}}$ and reconstruction loss $\\mathcal{L} _{\\mathrm{Rec}}$ can promote the recovery of LCVs to some extent. However, incorporating $\\mathcal{L} _{\\mathrm{Dis}}$ can further provide a consistent performance gain throughout all datasets for various tasks.\n\n***Q1: Comparison with mixed naive Bayes (NB) and variational inference (VI)-based  methods***\n\nThanks for your constructive feedback. We would like to resolve your concerns regarding the following aspects: \n* **Theoretically**, mixed NB and VI methods are typically designed for tabular data [1][2]. They **struggle with time series**, and often **rely on certain assumptions** like conditional independence [3], **limiting their ability to model the correlations of DVs and CVs** [2]. In contrast, MiTSformer leverages the temporal adjacencies to achieve latent continuity recovery, making it **capable of handling time series data** and effectively **capturing inherent nonlinear correlations**.\n\n* **Empirically**, we compare MiTSformer against typical mixed NB-based models HVM [1] and VI-based methods VAMDA [2] on mixed time series classification datasets and summarized the results in $\\underline{\\text{Table 1 of global response PDF}}$, showing MiTSformer consistently outperforms HVM and VAMDA. We promise to include these results and discussions in our paper.\n\n***About the limitations:***\n\nThanks for highlighting this. The original expressions in Appendix D may have caused some misunderstanding. We would like to clarify that our framework is designed for **DVs with numerical magnitude** by effectively recovering their LCVs. Actually, categorical variables can be divided into those **with numerical magnitude (e.g., ordinal categories)** and those **without numerical magnitude (i.e., nominal categories such as gender)**. Our framework is effective for the former, which is frequently encountered in real-world time series. For the event data and sales demand that you have mentioned, if categorical data represent varying levels of a quantity (e.g., sales volume), MiTSformer can also effectively handle it. We will revise the relevant clarifications to avoid any confusion.\n\n*Reference*:\n\n[1]. Hybrid variable monitoring: An unsupervised process monitoring framework with binary and continuous variables. Automatica, 2023.\n\n[2]. A flexible probabilistic framework with concurrent analysis of continuous and categorical data. IEEE TII, 2023.\n\n[3]. Naive Bayesian classification of structured data. Machine learning, 2004", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nZZeUPePuC", "reviewer": "Reviewer_3NoE", "summary": "The paper \"Addressing Spatial-Temporal Heterogeneity: General Mixed Time Series Analysis via Latent Continuity Recovery and Alignment\" introduces MiTSformer, a framework designed to address the challenges of mixed time series (MiTS) data, which include both continuous variables (CVs) and discrete variables (DVs). The framework recovers latent continuous variables (LCVs) behind DVs to ensure sufficient and balanced spatial-temporal modeling. MiTSformer employs hierarchical aggregation of temporal context and adversarial learning to align DVs with CVs. The framework is validated on five MiTS analysis tasks, showing state-of-the-art performance across multiple datasets.", "questions": "How does MiTSformer handle highly non-linear temporal patterns in both CVs and DVs?\nCan the framework be extended to handle non-binary discrete variables, and if so, how?\nWhat are the computational requirements for training MiTSformer on large-scale datasets?\nHow does the framework perform in real-time applications where data arrives sequentially?", "limitations": "The paper presents a compelling and innovative approach to handling mixed time series data. However, the complexity of the method, data dependency, and scalability issues suggest that further development and validation are needed to ensure its broad applicability and practical utility.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novel Approach: The introduction of latent continuous variables (LCVs) to handle the spatial-temporal heterogeneity in mixed time series is innovative and addresses a significant gap in current methodologies.\nComprehensive Framework: MiTSformer is versatile, capable of handling various tasks such as classification, regression, anomaly detection, imputation, and long-term forecasting.\nRobust Performance: The framework demonstrates superior performance across a wide range of datasets and tasks, indicating its robustness and effectiveness.\nDetailed Analysis: The paper provides thorough empirical evaluations and ablation studies, which validate the effectiveness of the proposed method.", "weaknesses": "Complexity: The framework's complexity may pose implementation challenges for practitioners, potentially limiting its accessibility and usability.\nData Dependency: The performance heavily relies on the quality and diversity of the training data, which may limit its applicability in scenarios with limited data availability.\nScalability: While the method shows promising results, its scalability to very large datasets or real-time applications is not fully demonstrated.\nLimited Explanation of Hyperparameters: The paper could benefit from a more detailed explanation of the choice and tuning of hyperparameters, which is crucial for replication and practical application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate Reviewer 3NoE's positive acknowledgment of our methodology's innovation, comprehensiveness, effectiveness, and experimental robustness. We are especially grateful for the constructive and insightful feedback provided. Rest assured, we are dedicated to addressing your concerns.\n\n***W1: Implementation and usability***\n\nThanks for kindly raising this concern and we would like to address it for you below:\n* **Necessity of Each Module**: We would like to emphasize that **mixed time series modeling differs significantly from typical time series tasks due to variable heterogeneity**. Thereby, each component in MiTSformer is essential to address it via latent continuity recovery and alignment. Our experiments, particularly the ablation studies displayed in $\\underline{\\text{Table 4 on Page 9}}$ further justify this point.\n* **User Accessibility**: Like most deep time series models, MiTSformer can be trained and deployed in an end-to-end manner. Also, **we have tried our best to make our framework accessible to practitioners by providing **open-source and well-organized codes and scripts**  with sufficient implementation details** ($\\underline{\\text{Appendix A on Pages } 13\\sim 19}$) to facilitate ease of use.\n\nIn light of your feedback, we will add relevant clarifications to our paper.\n\n***W2: Data dependency***\n\nYou are right that the performance of MiTSformer, like most deep learning models, benefits from high-quality training data. However, we can empower MiTSformer's robustness under data limitations with advanced techniques such as **Data Augmentation** [1] and **Self-supervised Pre-Training** [2]. We promise to update these discussions in our paper.\n\n***W3: Scalability to very large datasets***\n\nThis is an interesting question. We would like to elaborate it with the following points:\n\n1. **Computational Efficiency**: As demonstrated in the computational efficiency analysis in $\\underline{\\text{Appendix B on Page 20}}$, MiTSformer exhibits great efficiency concerning memory footprint and training time, which ensures the feasibility for very large datasets.\n2. **Fast Attention**: MiTSformer adopts self- and cross-attention to model variable correlations, which is the primary source of computational load. When encountering datasets with a large number of variables, we can adopt efficient linear attentions, e.g., Flowformer [3] to further reduce the computational burdens.\n3. **Efficient Pre-training and Fine-tuning**: We can empower MiTSformer with efficient pre-training and fine-tuning frameworks like LoRA [4] to further enhance the scalability.\n   \nEnlightened by your feedback, we will include these discussions in our paper.\n\n***W4:  Explanation of Hyperparameters***\n\nWe highly agree that hyperparameters are key issues for MiTSformer. Owing to page constraints, we had to provide the details of the hyperparameter setting in $\\underline{\\text{Appendix A on Pages 13-15}}$, and hyperparameter sensitivity analysis in $\\underline{\\text{Appendix C on Pages 20-22}}$. In summary, we find that MiTSformer is relatively stable in the selection of $d_{model}$ and $l_{layer}$. Also, MiTSformer is quite robust to the weights of loss items (i.e., $\\lambda_1$, $\\lambda_2$, and $\\lambda_3$), and moderate weights (e.g., $0.3 \\sim 1.0$) would bring optimal performance. Also, we promise to further emphasize hyperparameter analysis in the main text.\n\n***Q1: Handle non-linear patterns***\n\nMiTSformer effectively handles highly non-linear temporal patterns in both CVs and DVs via Spatial-temporal attention blocks as depicted in $\\underline{\\text{Figure 5 on Page 6}}$, where self-attention and cross-attention can model the nonlinear spatial-temporal correlations within LCVs and CVs and across LVs and CVs, respectively. \n\n***Q2: Handle non-binary discrete variables***\n\nThis is an insightful and interesting question. A quick answer is \"yes\". Actually, DVs can take on multiple states ($\\geq 2$) that reflect the magnitude and can be directly input into the recovery network to obtain LCVs outputs. It is noted that more states in a DV imply richer information granularity. We chose the binary scenarios in our experimental settings as they are the most challenging and commonly encountered in the real world.\n\nAlso, we conducted experiments on classification datasets with various numbers of discrete states in $\\underline{\\text{Table 2 of global response PDF}}$, showing that as the number of states increases, accuracy improves due to the richer information. The above results and discussions will be included in our paper.\n\n\n***Q3： Computational efficiency***\n\nWe highly agree that computational complexity is a key issue for MiTSformer's practicality, and we analyzed it in terms of training time and memory cost, which are included in $\\underline{\\text{Appendix B on Page 20}}$. In general, MiTSformer maintains great performance and efficiency. Enlightened by your feedback, we promise to further emphasize the analysis of computational efficiency in the main text.\n\n***Q4: Real-time applications***\n\nThanks for your valuable question. MiTSformer can process sequentially arrived data in a similar way to most time series models. For example, well-trained MiTSformer can be deployed in real-time systems. When new data sequentially arrive, they are appended to the historical data to form a full sequence, which is then fed into the well-trained model to generate online predictions. \n\n\n*Reference*：\n\n[1]. Self-supervised contrastive representation learning for semi-supervised time-series classification. IEEE TPAMI, 2023.\n\n[2]. TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling. ICML, 2024.\n\n[3]. Flowformer: Linearizing transformers with conservation flows. ICML, 2022.\n\n[4]. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UWazmCBm0A", "reviewer": "Reviewer_ZwAv", "summary": "•\tThis paper introduces a type of spatial-temporal heterogeneity caused by the gap between continuous variables (CVs) and discrete variables (DVs). \n•\tThe author introduces latent continuous variables to create a unified continuous numerical space for both CVs and DVs, with the aim to address the heterogeneity caused by the gap between these types of variables.\n•\tThe Latent Continuity Recovery architecture is innovative.", "questions": "1.How does the proposed method scale with large datasets or in real-time applications? what is the computational complexity of the proposed method? \n2. Could you explain more explicit and detailed connection between the mathematical formulas and the LCV transformation process?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The introduction of latent continuous variables (LCVs) to bridge the gap between DVs and CVs is an innovative approach. The proposed method for latent continuity recovery through adaptive and hierarchical aggregation of multi-scale adjacent context information is a creative combination of existing ideas.\n\nThe code is open-sourced, clean, and well-organized, demonstrating a high standard of technical implementation.\n\nThe proposed solution addresses a fundamental issue in spatial-temporal modeling, making it highly relevant for various applications in fields such as precipitation, temperature, humidity, etc.", "weaknesses": "1. The presentation for the transformation process for LCVs can be improved for better clarity. A more explicit and detailed connection between the mathematical formulas and the LCV transformation process would strengthen the clarity and comprehensiveness of the paper.  \n2. The formulation and application of the regularization term to encourage smoothness across time need to be better explained.\n3. The experiment is mostly comprehensive, but the analytical explanation of latent variables can be strengthened. This analysis could be interesting to broad audience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer ZwAv for the positive evaluation of our work's innovativeness, creativity, and technical quality. Your detailed and insightful comments have helped us improve our work substantially. We hope your concerns are addressed.\n\n***W1&Q3: Mathematical formulas and the LCV Recovery***\n\nThanks for your kind valuable suggestion. We are glad to provide a more detailed and explicit explanation:\n\n1. **LCV Recovery Network**: Our LCV recovery is based on a recovery network composed of residual dilated convolutional neural networks, which hierarchically aggregate multi-scale temporal context information to transform DVs into LCVs. Also, we provided the detailed network structure in $\\underline{\\text{Appendix A.3 on Page 13}}$.\n2. **Mathematical Formulation**: The recovery network receives a DV as input and outputs its LCV, as depicted in $\\underline{\\text{Figure 4 on Page 5}}$. This process can be mathematically described as\n$$\nx^{LC} = \\mathrm{Rec}\\text{-}\\mathrm{Net}(x^{D}) = h_{n}\n$$\nwhere the residual dilated convolutional network is defined by the iterative process as \n$$\nh_{i} = \\mathrm{Conv}^{d_{i}}(h_{i-1}) + h_{i-1}, \\quad \\text{for} \\; i = 1, 2, \\ldots, n\n$$\nIn this formulation, $h_{i}$ represents the output of the $i$-th residual block, $\\mathrm{Conv}^{d_{i}}$ denotes the convolution operation along the temporal axis to aggregate contextual information with dilation rate $d_{i}$. The final output, $x^{LC}$, is the result after $n$ residual blocks. Additionally, we perform z-score normalization for each  $x^{LC} \\in \\mathbb{R}^{1 \\times T} $ to ensure training stability as\n$$\nx^{LC} = \\frac{x^{LC}-\\mathrm{Mean}(x^{LC})}{\\mathrm{Std}(x^{LC})}\n$$\nwhere $\\mathrm{Mean}$ and $\\mathrm{Std}$ denote the mean and the standard deviation along the time axis, respectively.\nAccording to your suggestion, we promise to include these detailed formulations in our submission to enhance clarity and comprehensiveness.  \n\n***W2: Better explaining the smoothness regularization term***\n\nWe appreciate your constructive feedback. We would like to clarify this issue in detail:\n\nInspired by the temporal autocorrelation nature of continuous variables, we devise the smoothness constraint loss $\\mathcal{L} _{\\mathrm{smooth}}$ to regularize the LCV recovery process. Mathematically, we aim to enforce the smoothness of adjacent points of LCVs as\n\n$$\n\\sum_{t=1}^{T-1}{\\mathrm{Abs}( x_{t+1}^{D}-x_{t}^{D}) \\cdot (x_{t+1}^{LC}-x_{t}^{LC}) ^2}\n$$\n\nwhere $\\mathrm{Abs}\\left( x_{t-1}^{D}-x_{t}^{D} \\right)$ denotes the absolute difference between two adjacent time points of DVs, whose value can be $ \\\\{0, 1 \\\\}$ ( $1$ indicates \"sudden change\" and $0$ indicates \"no sudden change\"). For computational efficiency, we introduce a constant-valued smoothness matrix $\\boldsymbol{S}$ to achieve the above objective by multiplying it with $x^D$ and $x^{LC}$ as\n\n$$\n\\mathcal{L} _{\\mathrm{smooth}}=\\left\\| \\mathrm{Abs}\\left( \\boldsymbol{S}x^D \\right) \\otimes \\left( \\boldsymbol{S}x^{LC} \\right) \\right\\| _{2}^{2}\n$$\n\nwhere $\\otimes$ denotes the Hadamard product operation.  \nThanks for highlighting this issue, we promise to better explain the smoothness regularization term in our paper.\n \n\n***W3: Strengthening the analytical explanation of LCVs***\n\nThanks for your kind valuable suggestion. For a better investigation of LCV recovery, we have provided both visualization and quantitative analysis of the recovered LCVs and their actual LCVs. You can refer to $\\underline{\\text{Figure 1  and Table 1 of global response PDF}}$ for the results. \n\nAs presented, MiTSformer achieves accurate recovery of LCVs for DVs, which re-establishs their continuous fine-grained and informative temporal variation patterns. By recovering LCVs, we reduce the information granularity gap between DVs and CVs and align their distributions, which facilitates sufficient and balanced correlation modeling of mixed variables and further enhances the performance of various tasks. We promise to add relevant results and discussions to our paper to further validate the effectiveness and necessity of LCV recovery.\n\n\n***Q1: Scale with large datasets or in real-time applications***\n\nThanks for your insightful questions. We would like to address your concerns from the following aspects.\n\n* **For Scaling with Large Datasets**:  First, MiTSformer exhibits great computational efficiency as demonstrated in $\\underline{\\text{Appendix B on Page 20}}$. Secondly, we can empower MiTSformer with efficient linear self-attention techniques, e.g., Flowformer [1] to further reduce the computational burdens. Furthermore, we can leverage efficient pre-training and fine-tuning frameworks like LoRA [2] for scalability at deployment.\n* **For Real-Time Applications:** MiTSformer can process real-time mixed time series data in a similar way to most time series models. In real-time applications, well-trained MiTSformer can be deployed in online systems. When new data points arrive, they can be appended to the historical data to form a full sequence and then fed into MiTSformer to generate online predictions. \n\nAccording to your feedback, we will supplement the above discussions in our paper.\n\n***Q2: Computational complexity of MiTSformer***\n\nThanks for your invaluable feedback. We highly agree that computational complexity is a key issue for investigating the MiTSformer's practicality, and we analyzed it in terms of training time and memory cost, which are included in $\\underline{\\text{Appendix B on Page 20}}$. In general, MiTSformer maintains great performance and efficiency, especially for the training time. Enlightened by your feedback, we promise to further emphasize computational complexity analysis in the main text of our paper.\n\n\n*Reference:*\n\n[1]. Flowformer: Linearizing transformers with conservation flows. ICML, 2022.\n\n[2]. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vWhZdLStCP", "reviewer": "Reviewer_nZaK", "summary": "The authors propose to model mixed time series with both continuous variables and discrete variables by constructing latent continuous variables (LCVs) from discrete variables (DVs). Several super-supervised learning constraints are proposed to help improve the effectiveness of LCVs as well as the co-learning of LCVs and CVs with both cross-attention and self-attention modules.", "questions": "The paper lacks evidence to support the effectiveness of the proposed self-supervised objective functions. There is no theoretical or strong intuition behind them. For example, why does Temporal Adjacent Smoothness Constraint matter? I know from experiments it seems necessary, but intuitively and by commonsense, some discrete variables could indeed have sudden changes. In your example of meteorology, rainfall could indeed suddenly stop at a time point because the cloud moves from where the measurement device is. How do you defend your model in case where a DV is indeed subject to sudden changes by its nature? Moreover, for this constraint, do we have to use this specific formulation, because minimizing every two adjacent points could be too strong? How about K-lipschitz continuity as constraint, where k can be determined by the CVs? I would love to see some discussion on why you select this formulation specifically.\n\nIn addition, the adversarial framework that discriminates between CV and LCV seems weak. There is no guarantee that it will work in theory, as GAN-like structures are unstable. Even if the optimization is stable, in what condition would a model think \"I cannot discriminate between CV and LCV\"? What specific properties are in CVs that LCV could learn to trick a discriminator? For facial recognition, we can tell if the object in an image is like a real human face or not, based on whether some details are distorted. For time series, how to interpret that? If at some time points, the time series are experiencing abnormal variations, that does not mean it is “fake”. What specific properties are you trying to look into when you design this loss function? If you just want the LCVs to reflect some of the variations in the CVs, you could design some simpler constraints just like your Temporal Adjacent Smoothness Constraint, correct? I would love to see some discussions.", "limitations": "I do not see limitations are discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clarity in paper-writing. Both figures and languages are easy to understand and well-polished. \n2. Originality. The idea is novel, easy to follow, and most importantly, it makes sense. \n3. Quality. The paper is conducted on various datasets for various tasks to support its claim as a general model. \n4. Code is released and sufficient training details are provided.", "weaknesses": "1. The paper lacks evidence to support the effectiveness of the proposed self-supervised objective functions. There is no theoretical or strong intuition behind them. \n\n1.1 For example, why does Temporal Adjacent Smoothness Constraint matter? I know from experiments it seems necessary, but intuitively and by commonsense, some discrete variables could indeed have sudden changes. In your example of meteorology, rainfall could indeed suddenly stop at a time point because the cloud moves from where the measurement device is. How do you defend your model in case where a DV is indeed subject to sudden changes by its nature? Moreover, for this constraint, do we have to use this specific formulation, because minimizing every two adjacent points could be too strong? How about K-lipschitz continuity as constraint, where k can be determined by the CVs? I would love to see some discussion on why you select this formulation specifically.\n\n1.2 In addition, the adversarial framework that discriminates between CV and LCV seems weak. There is no guarantee that it will work in theory, as GAN-like structures are unstable. Even if the optimization is stable, in what condition would a model think \"I cannot discriminate between CV and LCV\"? What specific properties are in CVs that LCV could learn to trick a discriminator? For facial recognition, we can tell if the object in an image is like a real human face or not, based on whether some details are distorted. For time series, how to interpret that? If at some time points, the time series are experiencing abnormal variations, that does not mean it is “fake”. What specific properties are you trying to look into when you design this loss function? If you just want the LCVs to reflect some of the variations in the CVs, you could design some simpler constraints just like your Temporal Adjacent Smoothness Constraint, correct? I would love to see some discussions.\n\n2. Some issues with presentation. Variable z is first used in Eq.2 and then in Eq.4, but it was never explained in text. Usually, the audience is much more interested in the error bars instead of showing both MAE and MSE, as both metrics are extremely similar. Please adjust accordingly. \n\n3. Some issues with the experimental setting. The reconstruction loss is essential, but the key issue is, is the generated LCVs actually correct? Wouldn't it be wonderful if you could test on some datasets where some DVs are discretized from CVs, and you can try to recover the CVs and plot the results? If the accuracy of actually recovering CVs is not guaranteed, we do not need to have to “recover” a continuous time series, but we can obtain some latent embedding from the DV, which should also work well by intuition. Therefore, this experiment should be added as a demonstration that recovering LCV is possible. It would be a prerequisite to prove it is helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate Reviewer nZaK's positive acknowledgment of our work's originality, clarity, and quality. We are especially grateful for the detailed and insightful feedback provided. Rest assured, we are dedicated to addressing your concerns and enhancing our work.\n\n***W1.1&Q1: Smoothness constraints and K-Lipschitz continuity***\n\nThanks for your insightful comments. These are all great points. We would like to address them with the following aspects: \n\n* **Purpose of the Smoothness Constraint:** Consider an ideal continuous variable should be **be equipped with interpretable autocorrelation or smooth variations**, we design $\\mathcal{L} _{\\mathrm{Smooth}}$ to promote the recovered LCVs to achieve this. Also, we implement $\\mathcal{L} _{\\mathrm{Smooth}}$ with an adjustable coefficient $\\lambda _1 $. If a DV is known to commonly undergo sudden changes, $\\lambda _1 $ can be reduced accordingly.\n* **Collaboration and Mutual Restraint with Other Losses**: Essentially, the smoothness loss works synergistically with other losses, such as reconstruction and task losses. While the smoothness loss aims to promote smooth changes, it is balanced by the other losses to prevent over-smoothing, which could negatively impact the performance of reconstruction and downstream tasks. That is to say, our model can adaptively learn a proper smoothness degree. We elaborated this point in $\\underline{\\text{Section 3.5 on Page 6}}$ of our paper, which may be of interest to you. Also, the sensitivity analysis of $\\lambda _{1}$ presented in $\\underline{\\text{Figure 17 on Page 22}}$ verifies its robustness. \n* **K-Lipschitz Continuity as Constraints**: We greatly appreciate your invaluable suggestion of using K-Lipschitz continuity, which is **a wonderful idea and allows us to use CVs as guidance to determine the smoothness degree**. We can formulate it as $\\mathcal{L}_{\\mathrm{smooth}}^{K\\text{-Lipschitz}}$ = \n\n$\\sum_{t=1}^{T-1} \\max( 0,|x_{t}^{LCV}-x_{t+1}^{LCV}|-K) $,\n\nThis term penalizes the consecutive points that exceed the bound $K$, which is determined by CVs. Empirically, we conducted experiments on the ETTh1&h2 dataset to verify its effectiveness as in $\\underline{\\text{Table 4 in global response PDF}}$, which shows comparable performance to our original formulation. In light of your suggestions, we promise to add the relevant results discussions to our paper.\n\n\n***W1.2&Q2: About the adversarial discrimination framework***\n\nThank you for raising these excellent points. We are glad to address your concerns as follows:\n\n* **Discriminating Features in Time Series**: \nFor the recovered LCVs, we believe that if they present **similar temporal variation features (e.g.,  autocorrelation, periodicity) and statistical features (e.g., distributions)** like those in CVs, they can trick the discriminator. Also, we have provided visualization evidence in $\\underline{\\text{Figure 1 in global response PDF}}$, showing the LCVs accurately recover their actual temporal variations and distributions. \n\n* **Learning to Align Distributions**: Actually, due to the difference in information granularity and distributions between CVs and DVs, directly modeling mixed variables would inevitably cause errors. The objective of our adversarial framework is to **promote aligning the distributions of LCVs and CVs**, akin to Domain Adaptation [1]. This alignment facilitates to model correlation across DVs (represented by LCVs) and CVs, which is crucial for downstream tasks. \n\n* **About Anomaly Samples**: We agree that time series may experience abnormal variations, specifically for anomaly detection tasks. Here we would like to clarify that during the training phase, we focus on leveraging normal samples to capture the typical distributions and temporal variations, which is in line with previous works [2]. In the testing phase when anomaly may occur, the discriminator is discarded and irrelevant to the LCV recovery process.\n\n* **Training Stability**: We ensure the overall training stability by incorporating other supervision signals, e.g., smoothness loss and task loss, whose synergy and mutual restraint have been discussed in $\\underline{\\text{Section 3.5 on Page 6}}$.\n \n\n***W2: Presentation issues***\n\nThanks for your scientific rigor. Actually, we explain $z$ in lines 194~195 of our paper and we will further emphasize it. Also, in light of your suggestion, the error bars of MiTSformer will be included in our paper. You can also refer to $\\underline{\\text{Table 4 in global response PDF}}$ for relevant results.\n\n\n***W3: The effectiveness of LCV recovery***\n\nThank you for the opportunity to clarify these points and improve our paper. We would like to address your concerns with the following aspects:\n\n1. **Visualization Evidence**: As suggested, we have visualized the recovered LCVs and the actual LCVs, together with quantitative deviation analysis. Both results are included in $\\underline{\\text{Figure 1 and Table 1 in global response PDF}}$, showing that MiTSformer achieves accurate recovery of LCVs for DVs, which re-establishs their latent fine-grained and informative temporal variations.\n2. **Necessity of LCV Recovery**: By recovering LCVs, we reduce the information granularity gap between DVs and CVs and align their distributions, enabling us to flexibly and reliably model correlations across CVs and DVs for various tasks.\n3. **Directly Modeling DVs**: We conducted ablation experiments ($\\underline{\\text{Row 5 in Table 4 on Page 9}}$) where we directly fused the embeddings of CVs and DVs without LCVs recovery. As presented, the performance in both forecasting ($\\mathrm{MAE}: 0.433 \\rightarrow 0.529$ ) and classification tasks ($\\mathrm{Acc}: 71.9 \\rightarrow 69.3$ ) significantly decreased, demonstrating the necessity and effectiveness of LCV recovery.\n\n*Reference:*\n\n[1]. Unsupervised domain adaptation by backpropagation. ICML, 2015\n\n[2]. Anomaly transformer: Time series anomaly detection with association discrepancy. ICLR, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Summary of Revisions and Global Response
We sincerely thank all the reviewers for their insightful reviews and valuable comments, which are instructive for us to improve our paper further.

**Pioneering the exploration of the mixed time series analysis**, this paper proposes a task-general framework MiTSformer by recovering and aligning the latent continuity of mixed variables for sufficient and balanced spatial-temporal modeling, being amenable to various tasks. Experimentally, **MiTSformer establishes SOTA performance on 34 public benchmarks covering five mixed time series analysis tasks**.

We are delighted that the reviewers generally held positive opinions of our paper, in that the problem definition is "**fresh**", "**novel**" and is a "**fundamental issue**", the proposed method "**addresses a significant gap**", and is "**creative**", "**innovative**", and "**invaluable for designing robust time series foundation models**",  the empirical evaluation is "**thorough**", "**comprehensive**" and "**robust**", and the presentation is "**clear**", "**easy to understand**".

The reviewers also raised insightful and constructive concerns. We made every effort to address all the concerns by providing detailed clarifications, sufficient evidence, requested results, and in-depth analysis. Here is the summary of the major revisions:

* **Add in-depth investigation of LCV recovery (Reviewers GdPs, ZwAv, and nZaK)**.  We have supplemented the visualization and quantitative evaluations to compare the recovered LCVs with their actual ones, demonstrating the necessity and effectiveness of LCV recovery.
* **Add comparison with traditional mixed data modeling approaches  (Reviewer GdPs):** We have supplemented the comparison with traditional mixed data modeling methods, i.e., mixed naive Bayesian-based and variational inference-based methods, with MiTSformer both theoretically and empirically.
* **Add experiments for DVs with multiple discrete states (Reviewer 3NoE):** We've analyzed and conducted experiments to demonstrate MiTSformer's capability to process multi-state DVs effectively.
* **Illustrate the applications for large datasets and real-time deployments (Reviewers 3NoE and ZwAv):** We've analyzed the feasibility and efficiency of MiTSformer to tackle real-world large datasets and real-time processing. Also, we've provided relevant instructions and guidance.
* **Clarify some key concepts (Reviewers 3NoE and nZaK):** We have clarified key aspects of MiTSformer, such as the mathematical formulations of LCV recovery, the implementation and effectiveness of smoothness constraints, and the adversarial framework.
* **Resolve some presentation issues (Reviewer nZaK):** We resolved presentation issues by polishing equations and adding error bars of experiments for enhanced clarity.
  
The valuable suggestions from four reviewers are very helpful for us to improve our paper. We'd be very happy to answer any further questions.
Looking forward to the reviewer's feedback.

**The mentioned Tables and Figures are included in the following PDF file.**
* **Figure 1**: Visualization of the recovered LCVs and the actual LCVs for Reviewers GdPs, ZwAv, and nZaK.
* **Tabel 1**: Quantitative deviations between the recovered LCVs and the actual LCVs for Reviewers GdPs, ZwAv, and nZaK.
* **Tabel 2**: Performance comparison with mixed NB- and VI-based methods for Reviewer GdPs.
* **Table 3**: Results under the different number of DV's states for Reviewer 3NoE.
* **Table 4**: Results of K-Lipschitz Continuity-based smoothness constraint for Reviewer nZaK.
* **Table 5**: Robust analysis of MiTSformer with error bars for Reviewer nZaK.

Meta-review: The paper proposes MiTSformer, a general framework for mixed time-series consisting of both discrete variables (DVs) and continuous variables (CVs). Key components include the recovery of latent continuous variables (LCVs) from discrete variables (DVs) and the exchange of information between LCVs/CVs with self- and cross-attention blocks. The training process is guided by self-supervised smoothness/adversarial/reconstruction losses as well as task-specific supervision.

The reviews of this paper are cautiously positive after the rebuttal phase. All reviewers acknowledge the practical relevance and limited exploration of mixed time-series. The concept of latent continuity recovery with the help of a residual dilated CNN, in particular, received attention from the reviewers and was praised for its effectiveness and innovation. The reviewers also commended the paper for its comprehensive evaluation on a variety of datasets and tasks, including ablation studies of all relevant network components, and the availability of high-quality code and training details.

The concerns raised by the reviewers broadly fall into the categories presentation, technical design, and experiments. While the paper is generally well presented, the technical presentation of the latent continuity recovery, including the introduction of the smoothness loss, is at times difficult to follow. Multiple reviewers were concerned about the complexity of the MiTSformer framework, especially with regard to reimplementation and reproduction, and the lack of intuition behind the individual objective functions. The authors defended their design choices well during the rebuttal phase, including an alternative smoothness loss based on K-Lipschitz continuity. The experiments, while generally comprehensive, notably lacked comparisons between recovered and actual LCVs, as well as tests with multi-state DVs, both of which the authors provided in their rebuttal. The reviewers also suggested additional baseline comparisons, including HVM and VAMDA, which were shown to be inferior to MiTSformer in the rebuttal as well.

With all major weaknesses addressed and the insight that the scores might also have been influenced by misunderstandings that the authors were able to clarify during the discussion period, the paper is recommended for acceptance as a poster, with the expectation that all relevant discussions and results from the rebuttal period will be integrated into the camera-ready version.
Decision: Accept (poster)