# SMATE: Semi-Supervised Spatio-Temporal Representation
Learning on Multivariate Time Series

Jingwei Zuo, Karine Zeitouni and Yehia Taher

DAVID Lab, University of Versailles, Université Paris-Saclay, Versailles, France

Email: {jingwei.zuo, karine.zeitouni, yehia.taher}@uvsq.fr

***Abstract***—Learning from Multivariate Time Series (MTS) has attracted widespread attention in recent years. In particular, label shortage is a real challenge for the classification task on MTS, considering its complex dimensional and sequential data structure. Unlike self-training and positive unlabeled learning that rely on distance-based classifiers, in this paper, we propose SMATE, a novel semi-supervised model for learning the interpretable Spatio-Temporal representation from weakly labeled MTS. We validate empirically the learned representation on 30 public datasets from the UEA MTS archive. We compare it with 13 state-of-the-art baseline methods for fully supervised tasks and four baselines for semi-supervised tasks. The results show the reliability and efficiency of our proposed method.

***Index Terms***—Neural Networks, Multivariate Time Series, Semi-supervised Learning, Representation Learning

## I. INTRODUCTION

Most Multivariate Time Series (MTS) data, such as sensor readings, are labeled during the data collection process. The post-labeling of MTS is much more costly than classic data (e.g., images, text, etc.) due to the low interpretability over the real-valued sequence, leading to a considerable constraint for MTS classification in real-life scenarios.

Weakly supervised learning becomes an alternative option to fully supervised algorithms. The previous studies on weak-label Time Series (TS) learning are usually based on self-training [1] or Positive Unlabeled Learning [2] with a carefully designed distance measure [3] or stopping criterion [4] to learn the pseudo-labels. Besides, they mostly focus on the Univariate Time Series (UTS) with the One-Nearest-Neighbor (1NN) classifier on raw data space, which is widely outpaced by today's advanced approaches [5], such as Deep Neural Networks (DNNs) [6] or ensemble methods [5].

From Univariate Time Series (UTS) to Multivariate Time Series (MTS), traditional methods usually combine the compact and effective features from different variables, such as Shapelet Ensemble [7], [8], global discriminative patterns [9], or bag-of-patterns [10], [11]. However, the predefined features usually fail to capture MTS essentials: the temporal dependency and the interactions of multiple variables (i.e., *spatial interactions*<fnref n="1" />). Recently, some deep learning-based methods were proposed to capture the MTS features with various network structures [12], [13], showing promising performance on MTS classification tasks. However, the above-mentioned methods are mostly fully supervised, and rarely consider the label shortage issue when building the MTS classifier.

The recent research turns to Representation Learning [14] when handling weakly labeled MTS, which allows learning low dimensional embeddings in an unsupervised manner, such as using triplet loss [15] to form the embedding space, then even an SVM classifier is powerful enough on the learned representation [15]. However, existing techniques suffer from three major issues. First, the interactions between the MTS variables are generally computed on the entire 1-D series, ignoring the fact that the local spatial interactions at the sub-sequence level may evolve in the dynamic sequence, that is *spatial dynamics*. Second, the representation learned in a pure unsupervised manner depends mostly on the loss function selection. As no label information is utilized to learn the representation [15], there is a risk that it deviates from the true features, thus affecting the classifier performance. Third, they rather employ deep learning as a blind box and do not focus on the interpretability of the learned representation.

Therefore, to handle both the MTS complex structure and the label shortage problem, we propose SMATE, Semi-supervised Spatio-temporal representation learning on Multivariate Time Series. An auto-encoder based structure allows mapping the MTS samples from raw features space $\mathcal{X}$ to low dimensional embedding space $\mathcal{H}$. A Spatial Modeling Block combined with a multi-layer convolutional network captures the spatial dynamics, whereas a GRU-based structure extracts the temporal dynamic features. Thereby, SMATE is capable of compressing the essential Spatio-temporal characteristics of MTS samples into low-dimensional embeddings. On top of this embedding space $\mathcal{H}$, we propose a semi-supervised three-step regularization process to bring the model to learn class-separable representations, where both the labeled and unlabeled samples contribute to the model's optimization. This regularization process comes with the capability of visualization at each step, making SMATE interpretable.

We summarize the paper's main contributions as follows:

* **Spatio-temporal dynamic features in MTS:** We claim and demonstrate that the temporal dependency and the evolution of the spatial interactions (*spatial dynamics*) are important for building a reliable MTS embedding.

* **Weak supervision of representation learning:** With limited labeled data, SMATE can learn reliable class-separable MTS representations for downstream tasks, such as MTS classification (MTSC).

* **Interpretable MTS embedding learning:** SMATE allows for visual interpretability, not only from the class-

<footnote n="1">¹We use the term *spatial* in this paper to represent the variable axis.</footnote>
separable representations but also in each step of the semi-supervised regularization process.

* **Extensive experiments on the MTS datasets:** The experiments are carried out on 30 MTS datasets from different application domains. The detailed evaluation with 13 supervised baselines and four semi-supervised work is provided, which shows the effectiveness and the efficiency of SMATE over the state of the art.

The rest of this paper starts with a review of the most related work. Then, we formulate the problems of the paper. Later, we present in detail our proposal SMATE, which is followed by the experiments on real-life datasets and the conclusion.

## II. RELATED WORK

### A. Multivariate Time Series Representation Learning

**Definition 1 (multivariate time series):** A multivariate time series (MTS) $\mathbf{x} \in \mathbb{R}^{T \times M}$ is a sequence of real-valued vectors: $\mathbf{x}=(x_1, x_2, ..., x_i, ..., x_T)$, where $x_i \in \mathbb{R}^M$, $M$ is the variable number. When $M = 1$, we call it *univariate time series* (UTS).

A natural way to learn MTS representation is to extend the approaches developed earlier on UTS [16]–[18]. For instance, [7] combines Shapelet representation from different variables to build an ensemble-like learner. Similarly, SMTS [10] and WEASEL+MUSE [11] adopt the symbolic features with the Bag-of-Patterns concept to model the variable’s relationship.

Different from the interpretable feature-based representations, various deep learning models are proposed to capture the spatial interactions in MTS. MC-DCNN [13] extracts firstly 1D-CNN features from each variable, then combines them with a Fully Connected (FC) Layer. Whereas the authors in [19] abandon the combination strategy but apply directly 1D-CNN to all variables. The 2D-CNN features with the cross-attention mechanism in CA-SFCN [6] enhanced the dependencies captured by 1D-CNN on both temporal and spatial axes. Last but not least, the hybrid LSTM-CNN structure is capable of extracting both local and long-term features. Various work, such as the Squeeze-and-Excitation block in LSTM-FCN [12] or the multi-view learning-like module in TapNet [20], enhanced the hybrid structure via modeling the spatial interactions. However, the interactions are generally computed at the sequence level, ignoring the fact that the local spatial interactions at the sub-sequence level may evolve in the dynamic sequence, i.e., *spatial dynamics*. Moreover, they are all fully supervised, requiring huge labels during the training process. Also, the learned representations are result-oriented (e.g., pursuing higher accuracy), with less focus on the interpretability, considered by the data mining community.

### B. Semi-supervised Learning on Time Series

The pioneering work [1], [3] on Semi-supervised TS Learning are based on self-training or Positive Unlabeled Learning [2] with the Nearest-Neighbor (1NN) classifier and a carefully designed distance, such as DTW [1] or DTW-D [3], and optimized stopping criterion [4] for importing the pseudolabels. Those work are followed by [21] for wider contexts.

Though not mentioned in their papers, the self-training framework is extensible from the UTS to the MTS setting by using an adapted distance [22], such as DTW<sub>D</sub> or DTW<sub>A</sub>. However, under more complex scenarios nowadays, such as 30 UEA MTS datasets [23] collected from different domains, the distance-based classifiers show limited performance [5].

Learning MTS representations [14] in a weakly supervised setting draws much attention recently. Unsupervised Scalable Representation Learning (USRL) described in [15] combines causal dilated convolutions with triplet loss for contrastive learning. Similarly, authors in [21] adopt Multi-Task Learning (MTL) to learn the self-supervised UTS features from an auxiliary forecasting task. The recent work Semi-TapNet [20] proposes an Attentional Prototype Network to learn from the unlabeled samples. However, the above-mentioned approaches do not explore thoroughly both the labeled samples and the *Spatio-temporal dynamics* in MTS. We show in Table I the comparison of the methods for learning MTS representation.

<table><caption>TABLE I: Existing methods for learning MTS Representation</caption><thead><tr><th></th><th>SMTS</th><th>MUSE</th><th>Shapelet</th><th>USRL</th><th>TapNet</th><th>MLSTM<br>FCN</th><th>CA-<br>SFCN</th><th>INN-<br>DTW</th><th>SMATE</th></tr></thead><tbody><tr><td>Temporal Dynamics</td><td>-</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>✓</td><td>-</td><td>-</td><td>✓</td></tr><tr><td>Spatial Dynamics</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>-</td><td>✓</td></tr><tr><td>Interpretable Representation</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>✓</td></tr><tr><td>Label Shortage</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>✓</td></tr></tbody></table>

## III. PROBLEM FORMULATION

In this section, we formulate the *Spatio-temporal dynamics* learning and semi-supervised classification problems for MTS. Table II summarizes the notations used in the paper.

<table><caption>TABLE II: Notation</caption><thead><tr><th>Notation</th><th>Description</th></tr></thead><tbody><tr><td>D, D<sub>t</sub>, D<sub>u</sub></td><td>dataset, labeled portion, unlabeled portion</td></tr><tr><td>T, M, N</td><td>MTS length, variable numbers, dataset size</td></tr><tr><td>L, D</td><td>embedding length, embedding dimension size</td></tr><tr><td>m, P</td><td>temporal window size, embedding pool size</td></tr><tr><td>x, h</td><td>MTS instance, latent embedding</td></tr><tr><td>s</td><td>variable/spatial interaction</td></tr><tr><td>θ</td><td>general parameters to be optimized</td></tr></tbody></table>

### A. Spatio-temporal Representation for MTS

The Spatio-temporal modeling of MTS requires considering both the temporal dependency $p(x_{t'}|x_t)$ ($t' > t$) and the spatial interactions between the variables. Previous studies [12], [20] usually consider the spatial interactions at the sequence level: $s = \{\mathbf{x}^i \boxtimes \mathbf{x}^j\} \in \mathbb{R}^M$, where $\mathbf{x}^i, \mathbf{x}^j \in \mathbb{R}^{T \times 1}$, $\boxtimes$ indicates the interactions between the variables. However, the local spatial interactions at the subsequence level $s_t = \{\mathbf{x}^i_{t-m/2, t+m/2} \boxtimes \mathbf{x}^j_{t-m/2, t+m/2}\} \in \mathbb{R}^M$ may evolve in the dynamic sequence, where $m$ is the window size. To illustrate, given the system status at time $t$ in MTS, it is not only decided by the local value $x_t \in \mathbb{R}^M$ given a temporal status, but also by its neighbor values $[x_{t-m/2} : x_{t+m/2}] \in \mathbb{R}^{M \times m}$, which brings a spatial correlation matrix on temporal neighbors and spatial variables given a spatial status $s_t \in \mathbb{R}^M$.

Therefore, given a sample $\mathbf{x} \in \mathbb{R}^{T \times M}$ in raw space $\mathcal{X}$, the Spatio-temporal representation learning for MTS is to learn a low-dimensional representation $\mathbf{h} \in \mathbb{R}^{L \times D}$ on embedding
space $\mathcal{H}$, integrating both temporal dynamic $p(x_{t'}|x_t)$ and spatial dynamic $p(s_{t'}|s_t)$. The item *dynamic* refers to the unstable system status within the evolving multivariate sequential data.

## B. Semi-Supervised Learning on MTS

**Definition 2 (weak-label MTS dataset):** A weakly labeled MTS dataset $\mathcal{D} = \{\mathcal{D}_l, \mathcal{D}_u\}$ contains both labeled and unlabeled MTS samples:

$$ \mathcal{D}_l = \{\mathbf{x}_i, y_i\}_{i=1}^{N^*r}, \quad \mathcal{D}_u = \{\hat{\mathbf{x}}_i\}_{i=1}^{N^*(1-r)} $$

$r$ ($0 \le r \le 1$) indicates the ratio of the labeled samples in $\mathcal{D}$ of size $N$, $y_i$ is the annotation of the labeled instance $\mathbf{x}_i$.

The semi-supervised MTS learning aims at training a classifier to predict successfully the label of a testing MTS sample, adopting the supervised training from $\mathcal{D}_l$ and further unsupervised adjustment/optimization from $\mathcal{D}_u$.

## IV. PROPOSAL: SMATE

In this section, we introduce SMATE, which captures the essential characteristics of MTS and allows learning an interpretable representation space in a semi-supervised manner.

### A. Global Structure of SMATE

SMATE is based on an asymmetric auto-encoder structure, integrating three key components: Spatio-temporal dynamic encoder, sequential decoder, and semi-supervised three-step regularization of the embedding space.

As shown in Figure 1, a two-channel encoder explores both spatial and temporal dynamics and embeds the input MTS into a low-dimensional representation space, where the embedded samples are sparsely distributed with the reconstruction-based optimization. On the unsupervised embedding space, a three-step regularization is applied to learn class-separable embeddings. The class centroids are regularized by labeled and unlabeled samples, showing interpretability over the representation space. Finally, the model is jointly optimized by the reconstruction objective and the regularization objective.

### B. Spatial Modeling Block (SMB)

Firstly, we introduce a novel module, Spatial Modeling Block (SMB), to capture the spatial interactions at subsequence levels. As shown in Figure 2, SMB takes as input an MTS representation $h = \{h_i\}_{i=1}^T \in \mathbb{R}^{T \times d}$ ($d = M$ for the first block in spatial encoding channel), followed by a one-dimensional average pooling layer on each variable $h^j \in \mathbb{R}^{T \times 1}$, encoding the temporal neighbors into the horizontal system status $s_H(i) = \text{avg}([h_{i-m/2} : h_{i+m/2}])$, where $i$ is the time tick, $m$ is the window size. Then the Fully Connected (FC) layers allow firstly the interaction of the horizontal system status $s_H$ in the vertical direction via a low-dimensional compression $s_V \in \mathbb{R}^{T \times d'}$, then remapping it to the initial data space to decide the spatial interaction weights at each one-dimensional segment. We define the spatial interactions $s = \{s_i\}_{i=1}^T$, where $s_i \in \mathbb{R}^d$, representing the interaction weights for the vector $h_i \in \mathbb{R}^d$. The output of SMB is described by $h'=h \odot s$, with the calibrated weights for each 1-D TS segment, where $\odot$ is the element-wise multiplication.

### C. Spatio-Temporal Encoding on MTS

Given $\mathbf{x} \in \mathbb{R}^{T \times M}$, the low-dimensional representation $\mathbf{h} \in \mathbb{R}^{L \times D}$ embeds the Spatio-temporal features of $\mathbf{x}$ by a neural network-based function $f_\theta(\mathbf{x})$. The low-dimensional embedding brings dramatic improvement on both the efficiency and accuracy for classification tasks [20], as the classifier is not impaired by the redundant information in raw data.

As shown in Figure 1, for the temporal channel, among different variants of the recurrent neural networks (RNN), we specifically consider Gated Recurrent Units (GRUs) that mitigate the vanishing gradient problem. An update gate $z_t$ and a reset gate $r_t$ control the hidden state $h_t \in \mathbb{R}^{d_g}$ with the observation $x_t \in \mathbb{R}^M$ and the previous hidden state $h_{t-1} \in \mathbb{R}^{d_g}$, where $d_g$ is the output dimension of GRUs. The update functions are as follows:

$$ r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r) \quad (1) $$

$$ z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z) \quad (2) $$

$$ h_t = (1-z_t) \odot h_{t-1} + z_t \odot \tanh(W_h x_t + U_h(h_{t-1} \odot r_t) + b_h) \quad (3) $$

where $W_x, U_x, b_x$ ($x \in [r, z, h]$) are model parameters, $\sigma$ is the sigmoid function, $\odot$ is the element-wise multiplication. Three GRUs are cascaded with a 1-D pooling layer to output $\mathbf{h}(\mathbf{T}) \in \mathbb{R}^{L \times d_g}$, where $L=T/\mathcal{P}$, $\mathcal{P}$ is the pool/sampling size.

For the spatial channel, we define the convolutional module:

$$ \mathbf{h}'(l) = \text{SMB}(\mathbf{h}(l)) \quad (4) $$

$$ \mathbf{h}(l+1) = \text{ReLU}(\mathbf{BN}(W \otimes \mathbf{h}'(l) + b)) \quad (5) $$

where $l$ ($0 \le l < 3$) is the module number, $\mathbf{h}(0)=\mathbf{x} \in \mathbb{R}^{T \times M}$, $\mathbf{h}(l) \in \mathbb{R}^{T \times d_c}$, $d_c$ is the filter number, $W$ is the 1-D convolutional kernel of size $m$, $\odot$ is the convolution operator. Within each of the three modules, the SMB firstly calibrates the interaction weights for each 1-D segment and outputs $\mathbf{h}' \in \mathbb{R}^{T \times d}$. Then a 1-D convolutional layer concatenated with Batch Normalization and Rectified Linear Units (ReLU) is deployed. Similar to the temporal channel, a 1-D pooling layer is applied after the last convolutional block to output $\mathbf{h}(\mathbf{S}) \in \mathbb{R}^{L \times d_c}$. Finally, we output the combined spatial and temporal features $h_{\text{concat}} = \text{concat}(\mathbf{h}(\mathbf{T}), \mathbf{h}(\mathbf{S})) \in \mathbb{R}^{L \times (d_g+d_c)}$ and apply two FC layers to get the Spatio-temporal embedding $\mathbf{h} \in \mathbb{R}^{L \times D}$. The matrix representation captures the Spatio-temporal features and facilitates the MTS restoration.

### D. Joint Model Optimization

As shown in Figure 1, since the representation learned via an autoencoder-based structure generally has a sparse distribution of class-specific samples [14], the unsupervised training derived from the reconstruction objective does not consider thoroughly the inner relation between class-specific samples but focus on the restoration performance from the embeddings. To address the issue, we propose a joint model optimization that integrates the temporal reconstruction and the three-step regularization objectives. Specifically, the joint optimization combines both the labeled and unlabeled samples to learn the class-specific clusters on the embedding space.

Firstly, we define the temporal reconstruction loss as:

$$ L_R = \sum_t \|x_t - \tilde{x}_t\|_2 \quad (6) $$
<figure><img src="image_1.png" /><figcaption>Fig. 1: Model Structure of SMATE</figcaption></figure>

where $x_t, \tilde{x}_t \in \mathbb{R}^M$, corresponding to the observations in the raw and reconstructed MTS instances $x$ and $\tilde{x}$.

Then, the three-step regularization approaches the embeddings within the class-specific clusters to the virtual class centroids, which are trained progressively.

**Step 1 Supervised Centroids Initialization:** The class centroids are initialized by the class-specific embeddings. Given the labeled training set $\mathcal{D}_l = \{X^k\}_{k=1}^K$ where $K$ is the class number, $X^k \in \mathbb{R}^{N_k \times T \times M}$ is a sample collection of class $k$, $N_k$ is the sample number of class $k$. Then the embedding set $H^k = f_\theta(X^k) \in \mathbb{R}^{N_k \times L \times D}$ initializes the class centroid $\mathbf{c}_k$ by:

$$ \mathbf{c}_k = \text{mean}(H^k), \quad \mathbf{c}_k \in \mathbb{R}^{L \times D} \qquad (7) $$

**Step 2 Supervised Centroids Adjustment:** Once the centroids are initialized, we can make the supervised adjustment since the distance-based class probability allows to assess the contribution of individual samples on the centroid's decision. In other words, the centroid $\mathbf{c}_k$ is affected by larger contribution weights brought by nearby samples of class $k$. Let $\mathbf{x}_i^k \in \mathbb{R}^{T \times M}$ be a time series of class $k$, we define the weight of $\mathbf{x}_i^k$ to $\mathbf{c}_k$ as the inverse Euclidean Distance (ED) between the embedding $\mathbf{h}_i^k = f_\theta(\mathbf{x}_i^k) \in \mathbb{R}^{L \times D}$ and the centroid $\mathbf{c}_k$:

$$ W_{k,i} = 1 - \frac{ED(\mathbf{h}_i^k, \mathbf{c}_k)}{\sum_{j=1}^{K} ED(\mathbf{h}_i^k, \mathbf{c}_j)} \qquad (8) $$

Then the class centroid $\mathbf{c}_k$ can be adjusted accordingly by the labeled samples within the class-specific cluster:

$$ \mathbf{c}_k = \sum_{i=1}^{N_k} W_{k,i} \cdot \mathbf{h}_i^k, \quad \mathbf{h}_i^k \in H^k \qquad (9) $$

**Step 3 Unsupervised Centroids Adjustment:** Apart from the reconstruction-based optimisation, the unlabeled sample $\tilde{\mathbf{x}}_i$ in $\mathcal{D}_u = \{\tilde{\mathbf{x}}_i\}_{i=1}^{N_k(1-r)}$ is capable of adjusting the centroid $\mathbf{c}_k$ via the propagated label from the distance-based class probability:

$$ \hat{p}_{\theta}(y = k | \tilde{\mathbf{x}}_i) = 1 - \frac{ED(f_{\theta}(\tilde{\mathbf{x}}_i), \mathbf{c}_k)}{\sum_{j=1}^{K} ED(f_{\theta}(\tilde{\mathbf{x}}_i), \mathbf{c}_j)} \qquad (10) $$

The unlabeled sample $\tilde{\mathbf{x}}_i$ will be then integrated into the class-specific cluster with the highest probability. We can further adjust the class centroid $\mathbf{c}_k$ considering the unlabeled samples:

$$ \mathbf{c}_k = \frac{N_k}{N_k + \hat{N}_k} \sum_{i=1}^{N_k} W_{k,i} \cdot \mathbf{h}_i^k + \frac{\hat{N}_k}{N_k + \hat{N}_k} \sum_{i=1}^{\hat{N}_k} \hat{p}_{k,i} \cdot \hat{\mathbf{h}}_i^k \qquad (11) $$

<figure><img src="image_2.png" /><figcaption>Fig. 2: The Spatial Modeling Block(SMB)</figcaption></figure>

where $\hat{\mathbf{h}}_i^k = f_\theta(\hat{\mathbf{x}}_i^k)$, $\hat{N}_k$ is the number of samples of class $k$ in $\mathcal{D}_u$ with the propagated label.

The class centroids are initialized and adjusted by both labeled and unlabeled samples, from which we formalize the regularization loss derived from the labeled samples:

$$ L_{\text{Reg}}(\theta) = - \sum_{k} \log W_{\theta}(y = k | \mathbf{x}) \qquad (12) $$

As both the reconstruction and regularization losses are normalized, we define the global optimization objective as:

$$ \min_{\theta}(L_R + \lambda L_{\text{Reg}}) \qquad (13) $$

where $\lambda \ge 0$ is a hyperparameter that balances the two losses. Importantly, $L_{\text{Reg}}$ is included such that the embedding process not only serves to reduce the dimensions – it is actively conditioned to facilitate the encoder in learning class-separable embeddings. In practice, SMATE is not sensitive to $\lambda$; then for all the experiments, we set $\lambda$ to 1.

## V. EXPERIMENTS

### A. Experimental setup

The model<fnref n="2" /> was trained using the Adam optimizer on a single Tesla V100 GPU of 32 Go memory with CUDA 10.2. We train an SVM classifier with radial basis function kernel on the learned embedding space, which is evaluated on the newly released UEA archive [23]<fnref n="3" /> for supervised analysis, where the datasets {ArticularyWordR., Epilepsy, Heartbeat, SelfRegulationSCP1} from four different domains are adopted for semi-supervised study.

### B. Classification Performance Evaluation

We use the accuracy as the default metric [5] for the supervised tasks. We also report the number of Win/Ties and the average rank [20] of different methods.

#### 1) Comparison Methods:

* Distance-based Nearest Neighbor (1NN) on non-normalized (non-norm) or normalized (norm) MTS [22]: 1NN-ED; 1NN-DTW_I; 1NN-DTW_D; 1NN-DTW_A.

* Bag-of-patterns classifier. WEASEL+MUSE [11].

* Deep Learning-based classifier. USRL [15]; TapNet [20]; MLSTM-FCN [12]; CA-SFCN [6]; SMATE_NR: SMATE without supervised Regularization, instead, a Softmax layer is applied on the embedding.

<footnote n="2">²The source code: https://github.com/JingweiZuo/SMATE</footnote>

<footnote n="3">³The datasets can be found in www.timeseriesclassification.com</footnote>
2) *Results Analysis:* Table III shows the results comparison with the baselines. “N/A” indicates the model is not applicable due to memory overflow. Overall, SMATE defends its reliability with 11 Wins/Ties and the highest average rank of 3.85 among all the baselines. The current state-of-the-art deep learning methods (TapNet, CA-SFCN) and the powerful data mining method (WEASEL+MUSE) have close ranks (4.73/5.45/4.66). Besides, USRL and SMATE<sub>NR</sub> perform much worse than SMATE with the same SVM classifier, confirming the reliability of our supervised regularization on the embedding space. Moreover, SMATE achieves the best performance among the baselines on all the datasets of EEG/MEG applications [23] (*FaceDetection*, *FingerMovements*, *HandMovementDirection*, *MotorImagery*, *SelfRegulationSCP1*, *SelfRegulationSCP2*), where the signals (i.e., variables) generally have strong and dynamic dependencies with each other. The spatial dynamic interactions could be essential characteristics that SMATE has successfully captured.

### C. Semi-supervised Classification Performance

For semi-supervised tasks, we evaluate the classifier's accuracy at different supervision levels. For comparison, we applied one classic model **1NN-DTW-D** [3] and three recently proposed semi-supervised deep learning models: **USRL** [15], **Semi-TapNet** [20] and **MTL** [21]. Since **1NN-DTW-D** and **MTL** are initially designed for UTS, we adapt them by:

* Adopting $DTW_D$ [22] as distance in **1NN-DTW-D**.

* Updating the MTS network optimization metrics in **MTL**.

The results in Figure 3 shows the superiority of SMATE over the baselines. In a Motion Recognition task (*Articulary-WorkR*.), from 10% labeled training set to fully labeled one, the accuracy of SMATE varies only by 0.046, compared to INN-DTW-D (0.264), USRL (0.286), Semi-TapNet (0.151) and MTL (0.225), showing that SMATE is capable of learning a class-separable representation under weak supervision. This conclusion is also demonstrated in EEG/MEG applications (*SelfRegulationSCP1*), with 10% labeled samples, SMATE obtained a higher accuracy (0.781) than fully supervised 1NN-DTW-D (0.775), USRL (0.771), Semi-TapNet (0.739) and MTL (0.730). In an Audio Spectra task (*Heartbeat*), though the fully supervised accuracy of SMATE (0.741) is not as good as Semi-TapNet (0.751), the weakly supervised SMATE with 10% labeled samples performs the best among all the models, indicating the reliability of the semi-supervised model.

### D. Visualization & Interpretation of the Representation Space

Apart from the thorough exploration of the weakly labeled samples, the representation space learned via SMATE shows good interpretability compared to the traditional Deep Learning models [6], [12], [13], [19].

We show in Figure 4 that t-SNE visualization of the representation space for the *Epilepsy* dataset, which contains four human activities: *Walking while gesturing*, *Walking slowly*, *Walking fast*, *Walking normally*. The 137 samples in the training set are projected to the representation space with 10% labels adopted for training. We show respectively the

space visualization at each regularization step. The results suggest that the representation space is interpretable for not only the effect of the weak supervision at each step but also the classification results. For instance, in Figure 4d, three samples of *Walking while gesturing* stay close to the class centroid of *Walking fast*, which may lead to the misclassification of certain samples in the two classes. To improve the classifier, more labels of the two classes in the training set can be added.

### E. Performance of Spatial Modeling Block (SMB)

To validate the Spatial Modeling Block (SMB), firstly, we compare the classification accuracy of SMATE with or without integrating SMB on the 27 datasets that SMATE has successfully executed. Then we rebuild SMATE by replacing SMB with the following modules in the state-of-the-art work which learn the variable relationships of MTS: **Random Dimension Permutation (RDP)** in TapNet [20] and **Squeeze-and-Excitation (SE)** in MLSTM-FCN [12]. Briefly, SMATE-SMB achieves [17 Wins|8 Ties|2 Losses] to SMATE-NonSMB, indicating that SMB contributes to a better MTS representation. Besides, SMB performs better than other modules: [14 Wins|8 Ties|5 Losses] to SE, [12 Wins|9 Ties|6 Losses] to RDP. RDP performs relatively better than SE, as a set of grouped variables produced by RDP provides various MTS views, allowing exploring the interactions between the subsets of all variables more thoroughly. However, extra parameters for variable groups are introduced. SE is a parameter-free module but considers each variable has a unique and stable state when interacting with others, which ignores the dynamic features in time series. SMB answers both the questions of the parameter-free settings and the dynamic interactions. The results show that capturing the spatial dynamic interactions at the sub-sequence level performs better than modeling the variable interactions at the sequence level [12], [20].

## VI. CONCLUSION

In this paper, we proposed SMATE, to learn the Spatial-temporal representation on weakly-labeled multivariate time series. The weak supervision on the embedding space allows building a reliable classifier, which is extremely valuable in real-life scenarios with label shortage issues. The results show that the evolving variable interactions (i.e., *spatial dynamics*) play an essential role in modeling multivariate time series. Moreover, SMATE allows for visual interpretability in both the learned representation and the semi-supervised representation learning process. Our future work will be oriented towards extending SMATE to support multivariate time series with missing values and unequal length in more realistic scenarios.

## ACKNOWLEDGEMENTS

This research was supported by DATAIA convergence institute as part of the *Programme d'Investissement d'Avenir (ANR-17-CONV-0003)* operated by DAVID Lab, UVSQ, Université Paris-Saclay. The authors would like to thank Anthony Bagnall and his team for providing the community with valuable datasets and source codes in the UEA & UCR Time Series Classification Repository. The authors would also like to thank Eamonn Keogh for his careful review and remarks.
<table><caption>TABLE III: Performance Comparison for MTS classification over UEA MTS archive</caption><thead><tr><th>Dataset</th><th>SMATE</th><th>SMATE<sub>NR</sub></th><th>USRL</th><th>TapNet</th><th>MLSTM<br>FCN</th><th>CA-SFCN</th><th>WEASEL<br>+MUSE</th><th>INN-ED</th><th>INN-<br>DTW<sub>I</sub></th><th>INN-<br>DTW<sub>D</sub></th><th>INN-ED<br>(norm)</th><th>INN-DTW<sub>I</sub><br>(norm)</th><th>INN-DTW<sub>D</sub><br>(norm)</th><th>INN-DTW<sub>A</sub><br>(norm)</th></tr></thead><tbody><tr><td>ArticularyWordR.</td><td><b>0.993</b></td><td>0.987</td><td>0.987</td><td>0.987</td><td>0.973</td><td>0.97</td><td>0.99</td><td>0.97</td><td>0.98</td><td>0.987</td><td>0.97</td><td>0.98</td><td>0.987</td><td>0.987</td></tr><tr><td>ArtialFibrillation</td><td>0.133</td><td>0.133</td><td>0.133</td><td><b>0.333</b></td><td>0.267</td><td><b>0.333</b></td><td><b>0.333</b></td><td>0.267</td><td>0.267</td><td>0.2</td><td>0.267</td><td>0.267</td><td>0.22</td><td>0.267</td></tr><tr><td>BasicMotions</td><td><b>1</b></td><td><b>1</b></td><td><b>1</b></td><td><b>1</b></td><td>0.95</td><td><b>1</b></td><td><b>1</b></td><td>0.675</td><td><b>1</b></td><td>0.975</td><td>0.97</td><td>0.975</td><td><b>1</b></td><td><b>1</b></td></tr><tr><td>CharacterTrajectories</td><td>0.984</td><td><b>0.997</b></td><td>0.994</td><td><b>0.997</b></td><td>0.985</td><td>0.988</td><td>0.99</td><td>0.964</td><td>0.969</td><td>0.99</td><td>0.969</td><td>0.989</td><td>0.989</td><td>0.989</td></tr><tr><td>Cricket</td><td>0.986</td><td>0.968</td><td>0.986</td><td>0.986</td><td>0.958</td><td>0.972</td><td>0.99</td><td>0.944</td><td>0.986</td><td>0.944</td><td>0.986</td><td>1</td><td>1</td><td>1</td></tr><tr><td>DuckDuckGese</td><td>N/A</td><td>N/A</td><td><b>0.675</b></td><td>0.575</td><td><b>0.675</b></td><td>N/A</td><td>0.575</td><td>0.275</td><td>0.55</td><td>0.6</td><td>0.55</td><td>0.6</td><td>0.567</td><td>N/A</td></tr><tr><td>EigenWorms</td><td>N/A</td><td>N/A</td><td>0.878</td><td>0.489</td><td>0.504</td><td>N/A</td><td><b>0.89</b></td><td>0.55</td><td>0.603</td><td>0.618</td><td>0.549</td><td>N/A</td><td>0.619</td><td>N/A</td></tr><tr><td>Epilepsy</td><td>0.964</td><td>0.946</td><td>0.957</td><td>0.971</td><td>0.761</td><td>0.986</td><td><b>1</b></td><td>0.667</td><td>0.978</td><td>0.964</td><td>0.666</td><td>0.978</td><td>0.964</td><td>0.979</td></tr><tr><td>EthanolConcentration</td><td><b>0.981</b></td><td>0.904</td><td>0.88</td><td>0.904</td><td>0.941</td><td>0.856</td><td>0.964</td><td>0.93</td><td>0.93</td><td>0.93</td><td>0.93</td><td>0.93</td><td>0.93</td><td>0.93</td></tr><tr><td>FaceDetection</td><td>0.399</td><td>0.373</td><td>0.236</td><td>0.323</td><td>0.373</td><td>0.323</td><td><b>0.43</b></td><td>0.293</td><td>0.304</td><td>0.323</td><td>0.293</td><td>N/A</td><td>0.323</td><td>0.316</td></tr><tr><td>FingerMovements</td><td><b>0.647</b></td><td>0.556</td><td>0.528</td><td>0.556</td><td>0.545</td><td>0.545</td><td>0.519</td><td>0.513</td><td>0.529</td><td>0.519</td><td>0.5</td><td>0.529</td><td>0.529</td><td>0.529</td></tr><tr><td>HandMovementD.</td><td><b>0.62</b></td><td>0.55</td><td>0.54</td><td>0.53</td><td>0.58</td><td>0.49</td><td>0.55</td><td>0.52</td><td>0.53</td><td>0.55</td><td>0.52</td><td>0.53</td><td>0.509</td><td>0.509</td></tr><tr><td>Handwriting</td><td><b>0.554</b></td><td>0.365</td><td>0.27</td><td>0.378</td><td>0.365</td><td>0.365</td><td>0.279</td><td>0.306</td><td>0.231</td><td>0.278</td><td>0.306</td><td>0.231</td><td>0.224</td><td>0.224</td></tr><tr><td>Heartbeat</td><td>0.421</td><td>0.335</td><td>0.533</td><td>0.357</td><td>0.286</td><td>0.322</td><td>0.371</td><td>0.509</td><td><b>0.607</b></td><td>0.2</td><td>0.316</td><td>0.286</td><td>0.601</td><td>0.601</td></tr><tr><td>Heartbeat</td><td>0.741</td><td>0.615</td><td>0.737</td><td>0.751</td><td>0.663</td><td><b>0.756</b></td><td>0.62</td><td>0.659</td><td>0.717</td><td>0.619</td><td>0.658</td><td>0.717</td><td>0.571</td><td>0.571</td></tr><tr><td>InsertWingbeat</td><td>N/A</td><td>N/A</td><td>0.16</td><td><b>0.208</b></td><td>0.167</td><td>N/A</td><td>0.128</td><td>N/A</td><td>0.115</td><td>0.128</td><td>N/A</td><td>N/A</td><td>N/A</td><td>N/A</td></tr><tr><td>JapuNelVowels</td><td>0.965</td><td>0.924</td><td>0.867</td><td><b>0.989</b></td><td>0.965</td><td>0.975</td><td>0.924</td><td>0.949</td><td>0.924</td><td>0.924</td><td>0.949</td><td>0.949</td><td>0.959</td><td>0.959</td></tr><tr><td>Libras</td><td>0.849</td><td>0.834</td><td>0.867</td><td>0.85</td><td>0.856</td><td>0.89</td><td>0.833</td><td><b>0.894</b></td><td>0.872</td><td>0.833</td><td><b>0.894</b></td><td>0.87</td><td>0.879</td><td>0.879</td></tr><tr><td>LSST</td><td>0.582</td><td>0.568</td><td>0.558</td><td>0.568</td><td>0.373</td><td><b>0.674</b></td><td>0.59</td><td>0.456</td><td>0.575</td><td>0.551</td><td>0.456</td><td>0.575</td><td>0.551</td><td>0.551</td></tr><tr><td>MotorImagery</td><td><b>0.59</b></td><td><b>0.59</b></td><td>0.54</td><td><b>0.59</b></td><td>0.51</td><td>N/A</td><td>0.51</td><td>0.39</td><td>N/A</td><td>0.5</td><td>N/A</td><td>0.5</td><td>0.5</td><td>0.5</td></tr><tr><td>NATOPS</td><td>0.922</td><td>0.87</td><td>0.944</td><td>0.939</td><td>0.889</td><td>0.956</td><td>0.87</td><td>0.973</td><td>0.945</td><td>0.883</td><td>0.85</td><td>0.883</td><td>0.883</td><td>0.883</td></tr><tr><td>PEMS-SF</td><td><b>0.803</b></td><td>0.744</td><td>0.688</td><td>0.751</td><td>0.699</td><td>N/A</td><td>0.705</td><td>0.734</td><td>0.711</td><td>0.705</td><td>0.734</td><td>0.711</td><td>0.73</td><td>0.73</td></tr><tr><td>PenDigits</td><td>0.98</td><td>0.98</td><td>0.98</td><td><b>0.983</b></td><td>0.98</td><td>0.975</td><td>0.948</td><td>0.973</td><td>0.939</td><td>0.977</td><td>0.973</td><td>0.939</td><td>0.977</td><td>0.977</td></tr><tr><td>Phoneme</td><td>0.177</td><td>0.19</td><td>0.862</td><td><b>0.246</b></td><td>0.175</td><td>0.11</td><td>0.19</td><td>0.104</td><td>0.151</td><td>0.151</td><td>0.104</td><td>0.151</td><td>0.151</td><td>0.151</td></tr><tr><td>RacketSports</td><td>0.849</td><td>0.816</td><td>0.862</td><td>0.868</td><td>0.803</td><td>0.875</td><td><b>0.934</b></td><td>0.868</td><td>0.842</td><td>0.803</td><td>0.868</td><td>0.842</td><td>0.803</td><td>0.858</td></tr><tr><td>SelfRegulationSCP1</td><td><b>0.887</b></td><td>0.874</td><td>0.771</td><td>0.739</td><td>0.874</td><td>0.734</td><td>0.71</td><td>0.771</td><td>0.765</td><td>0.775</td><td>0.771</td><td>0.765</td><td>0.775</td><td>0.786</td></tr><tr><td>SelfRegulationSCP2</td><td><b>0.567</b></td><td>0.539</td><td><b>0.556</b></td><td>0.55</td><td>0.472</td><td>N/A</td><td>0.48</td><td>0.483</td><td>0.533</td><td>0.539</td><td>0.483</td><td>0.533</td><td>0.539</td><td>0.539</td></tr><tr><td>SpokenArabicDigits</td><td>0.979</td><td>0.967</td><td>0.956</td><td>0.983</td><td>0.99</td><td>0.982</td><td>0.967</td><td>0.967</td><td>0.96</td><td>0.963</td><td>0.967</td><td>0.959</td><td>0.963</td><td>0.963</td></tr><tr><td>StandWalkJump</td><td><b>0.533</b></td><td>0.4</td><td>0.4</td><td>0.4</td><td>0.67</td><td>0.2</td><td>0.333</td><td>0.2</td><td>0.333</td><td>0.2</td><td>0.333</td><td>0.2</td><td>0.333</td><td>0.2</td></tr><tr><td>UWave GestureLibrary</td><td>0.897</td><td>0.869</td><td>0.884</td><td>0.894</td><td>0.891</td><td>0.8</td><td><b>0.916</b></td><td>0.881</td><td>0.868</td><td>0.903</td><td>0.81</td><td>0.868</td><td>0.903</td><td>0.9</td></tr></tbody><tfoot><tr><td>Avg. Rank</td><td>3.85</td><td>6.19</td><td>5.9</td><td>4.73</td><td>7.33</td><td>5.45</td><td>4.66</td><td>9.3</td><td>7.43</td><td>6.37</td><td>9.37</td><td>7.88</td><td>6.83</td><td>6.21</td></tr><tr><td>Wins (Ties)</td><td><b>11</b></td><td><b>3</b></td><td>6</td><td>5</td><td>2</td><td>5</td><td>8</td><td>0</td><td>2</td><td>2</td><td>0</td><td>2</td><td>1</td><td>2</td></tr></tfoot></table>

<figure><img src="image_3.png" /><figcaption>(a) ArticularyWordR. (Motion)</figcaption></figure>

<caption>(b) Epilepsy (Human Activity)</caption>

<caption>(c) Heartbeat (Audio Spectra)</caption>

<caption>(d) SelfRegulationSCP1 (EEG/MEG)</caption>

<figure><img src="image_4.png" /><figcaption>(a) Without any regularization</figcaption></figure>

<caption>(b) Regularization step 1</caption>

<caption>(c) Regularization step 2</caption>

<caption>(d) Regularization step 3</caption>

<caption>Fig. 4: The t-SNE visualization of the representation space for the *Epilepsy* dataset. We set the supervised ratio to 0.1. The colors of the embeddings represent their inherent labels, which are not fully adopted for training. The class centroids are marked by ✗.</caption>

on the preliminary version of this paper, as well as Nicoleta Preda and Zaineb Chelly for their suggestions.

REFERENCES

[1] L. Wei and E. Keogh, "Semi-supervised time series classification," in *KDD*, 2006, p. 748-753.

[2] M. N. Nguyen, X. L. Li, and S. K. Ng, "Positive unlabeled learning for time series classification," in *IJCAI*, 2011, p. 1421-1426.

[3] Y. Chen, B. Hu, E. Keogh, and G. E. Batista, "DTW-D: Time Series Semi-Supervised Learning from a Single Example," in *KDD*, 2013, p. 383-391.

[4] C. A. Ratanamahatana and D. Wanichsan, "Stopping Criterion Selection for Efficient Semi-supervised Time Series Classification," *Soft. Eng., Arti. Intel., Net. & Para./Distri. Comp.*, pp. 1-14, 2008.

[5] A. P. Ruiz, M. Flynn, J. Large, M. Middlehurst, and A. Bagnall, "The great multivariate time series classification bake off: a review and experimental evaluation of recent algorithmic advances," *Data Mining and Knowledge Discovery*, vol. 35, no. 2, pp. 401-449, 2021.

[6] Y. Hao and H. Cao, "A New Attention Mechanism to Classify Multi-variate Time Series," in *IJCAI*, 2020, pp. 1999-2005.

[7] M. S. Cetin, A. Mueen, and V. D. Calhoun, "Shapelet ensemble for multi-dimensional time series," in *SDM*, 2015, pp. 307-315.

[8] R. Mousheimish, Y. Taher, and K. Zeitouni, "Automatic Learning of Predictive CEP Rules: Bridging the Gap between Data Mining and Complex Event Processing," in *DEBS*, 2017, p. 158-169.

[9] A. Dorle, F. Li, W. Song, and S. Li, "Learning Discriminative Virtual Sequences for Time Series Classification," in *CIKM*, 2020, p. 2001-2004.

[10] M. Gokce Baydogan, G. Runger, M. G. Baydogan, and G. Runger, "Learning a symbolic representation for multivariate time series classification," *Data Min Knowl Disc*, vol. 29, pp. 400-422, 2015.

[11] P. Schäfer and U. Leser, "Multivariate Time Series Classification with WEASEL+MUSE," in *AALTD*, 2018.

[12] F. Karim, S. Majumdar, H. Darabi, and S. Harford, "Multivariate Istm-fcn for time series classification," *Neural Networks*, vol. 116, 2019.

[13] Y. Zheng, Q. Liu, E. Chen, Y. Ge, and J. L. Zhao, "Time series classification using multi-channels deep convolutional neural networks," in *WAIM*, 2014, pp. 298-310.

[14] Y. Bengio, A. Courville, and P. Vincent, "Representation learning: A review and new perspectives," *IEEE TPAMI*, pp. 1798-1828, 2013.

[15] J.-Y. Franceschi, A. Dieuleveut, and M. Jaggi, "Unsupervised Scalable Representation Learning for Multivariate Time Series," in *NeurIPS'19*.

[16] J. Zuo, K. Zeitouni, and Y. Taher, "Exploring interpretable features for large time series with se4tec," in *EDBT*, 2019, pp. 606-609.

[17] J. Zuo, K. Zeitouni, and Y. Taher, "Incremental and Adaptive Feature Exploration over Time Series Stream," *IEEE BigData'19*, pp. 593-602.

[18] J. Zuo, K. Zeitouni, and Y. Taher, "iSETS: Incremental Shapelet Extraction from Time Series Stream," *ECML-PKDD*, pp. 790-793, 2019.

[19] J. Yang, M. N. Nguyen, P. P. San, X. L. Li, and S. Krishnaswamy, "Deep convolutional neural networks on multichannel time series for human activity recognition," in *IJCAI*, 2015, p. 3995-4001.

[20] X. Zhang, Y. Gao, J. Lin, and C.-T. Lu, "TapNet: Multivariate Time Series Classification with Attentional Prototypical Network," in *AAAI'20*.

[21] S. Jawed, J. Grabocka, and L. Schmidt-Thieme, "Self-Supervised Learning for Semi-Supervised Time Series Classification," in *PAKDD'20*.

[22] M. Shokoohi-Yekta, B. Hu, H. Jin, J. Wang, and E. Keogh, "Generalizing DTW to the multi-dimensional case requires an adaptive approach HHS Public Access," *Data Min Knowl Discov*, vol. 31, no. 1, pp. 1-31, 2017.

[23] A. Bagnall, H. A. Dau, J. Lines, M. Flynn, J. Large, A. Bostrom, P. Southam, and E. Keogh, "The uea multivariate time series classification archive, 2018," *arXiv preprint arXiv:1811.00075*, 2018.