# Learning Multivariate Time Series with Unsupervised Representation Methods using Simulated Datasets

Thabang Lebese, Cécile Mattrand, David Clair, Jean-Marc Bourinet

<figure><img src="image_1.png" /></figure>

## To cite this version:

Thabang Lebese, Cécile Mattrand, David Clair, Jean-Marc Bourinet. Learning Multivariate Time Series with Unsupervised Representation Methods using Simulated Datasets. ..., Apr 2023, Louvain-la-Neuve, Belgium. 2023. [hal-04442065](https://hal.science/hal-04442065v1)

HAL Id: hal-04442065

https://hal.science/hal-04442065v1

Submitted on 6 Feb 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Learning Multivariate Time Series with Unsupervised Representation Methods using Simulated Datasets

## 1. Overview

* **Unsupervised representation learning** is useful for extracting representations from unlabeled multivariate time-series

* Extracted representations can match **underlying patterns** or states

* We explore **3 unsupervised** learning SOTAs: Triplet Loss, TNC and CPC

* We evaluate each on three **simulated datasets**

* Evaluations are on two downstream tasks: **clustering** and **classification**

## 2. Motivation

* Multivariate time series are mostly unlabeled, potentially long and of unequal lengths in the same dataset

* Modeling MTS requires significant time and expertise

* Unsupervised representation learning helps with understanding of generating processes and boosts performance of subsequent ML tasks

* Objectives if this work:

1) Implement and reproduce three unsupervised learning SOTAs (Triplet Loss¹, TNC² and CPC³) techniques on three simulated datasets

2) Demonstrate their strengths, weaknesses and determine suitability of each

* We demonstrate their practicality in various scenarios

## 3. Baselines & training setup

* All baselines use a Deep **BiRNN** as encoder backbone

* CPC uses **Gated Recurrent Units (GRU)** as a regressor

* Across baselines:

* All **latent space dimensionality** set to 10

* **Window size** set to 50 across baselines

* We perform a 60/20/20 - **train/validation/test** data split accordingly

* Maximum **training epochs** limited to (140, 400, 500) for (TNC, CPC, Triplet loss)

* Cross baseline **hyper-parameter** tuning:

* Window size selection

* PU learning weight parameter (TNC)

* No parameter tuning for the latent space

## 4. Methods

<figure><img src="image_3.png" /></figure>

## 5. Datasets

<table><caption>Table I: Signal distributions for three simulated trivariate time series. Simulation 1 covers states 0-3, simulation 2 covers 0-4, and simulation 3 covers 0-9.</caption><thead><tr><th colspan="2">TEN states</th><th colspan="10">simulation 3</th></tr><tr><th colspan="2">FIVE states</th><th colspan="10">simulation 2</th></tr><tr><th colspan="2">FOUR states</th><th colspan="10">simulation 1</th></tr><tr><th>Features</th><th>State 0</th><th>State 1</th><th>State 2</th><th>State 3</th><th>State 4</th><th>State 5</th><th>State 6</th><th>State 7</th><th>State 8</th><th>State 9</th></tr></thead><tbody><tr><td>Feature 0</td><td>GP(periodic)</td><td>NARMA<sub>α</sub></td><td>GP(Sq.Exp)</td><td>NARMA<sub>β</sub></td><td>GP(Matern)</td><td>GP(Matern)</td><td>NARMA<sub>β</sub></td><td>GP(Sq.Exp)</td><td>NARMA<sub>α</sub></td><td>GP(Periodic)</td></tr><tr><td>Feature 1</td><td>GP(periodic)</td><td>NARMA<sub>α</sub></td><td>GP(Sq.Exp)</td><td>NARMA<sub>β</sub></td><td>GP(Matern)</td><td>GP(Matern)</td><td>NARMA<sub>β</sub></td><td>GP(Sq.Exp)</td><td>NARMA<sub>α</sub></td><td>GP(Periodic)</td></tr><tr><td>Feature 2</td><td>GP(Sq.Exp)</td><td>NARMA<sub>β</sub></td><td>GP(Periodic)</td><td>NARMA<sub>α</sub></td><td>GP(Matern)</td><td>GP(Matern)</td><td>NARMA<sub>α</sub></td><td>GP(Periodic)</td><td>NARMA<sub>β</sub></td><td>GP(Sq.Exp)</td></tr></tbody></table>

## 6. Results

<table><caption>Table II: Baseline performance for classification (accuracy and AUPRC) and clustering (Silhouette and DBI) across all datasets.</caption><thead><tr><th rowspan="2"></th><th colspan="4">Triplet Loss</th><th colspan="4">TNC</th><th colspan="4">CPC</th></tr><tr><th>δ</th><th>AUPCR</th><th>Accuracy</th><th>Silhouette</th><th>DBI</th><th>AUPCR</th><th>Accuracy</th><th>Silhouette</th><th>DBI</th><th>AUPCR</th><th>Accuracy</th><th>Silhouette</th><th>DBI</th></tr></thead><tbody><tr><td>Sim. 1</td><td>50</td><td>0.920</td><td>73.104</td><td>0.477</td><td>0.804</td><td><strong>0.998</strong></td><td><strong>97.167</strong></td><td><strong>0.750</strong></td><td><strong>0.330</strong></td><td>0.871</td><td>57.400</td><td>0.504</td><td>0.795</td></tr><tr><td>Sim. 2</td><td>50</td><td>-</td><td>82.033</td><td>0.484</td><td>0.866</td><td>-</td><td><strong>97.517</strong></td><td><strong>0.636</strong></td><td><strong>0.616</strong></td><td>-</td><td>65.183</td><td>0.492</td><td>0.757</td></tr><tr><td>Sim. 3</td><td>50</td><td>0.790</td><td>75.470</td><td>0.426</td><td>1.130</td><td><strong>0.949</strong></td><td><strong>90.180</strong></td><td><strong>0.648</strong></td><td><strong>0.509</strong></td><td>0.432</td><td>43.290</td><td>0.356</td><td>0.903</td></tr><tr><td>Avg.</td><td></td><td>0.855</td><td>76.869</td><td>0.462</td><td>0.933</td><td><strong>0.9735</strong></td><td><strong>94.9547</strong></td><td><strong>0.678</strong></td><td><strong>0.485</strong></td><td>0.6515</td><td>55.291</td><td>0.451</td><td>0.818</td></tr></tbody></table>

### Our evaluation metrics: KPIs

* **Clusterability:** TNC outperforms other two models

* These scores are evaluated on top of k - means⁴ clusters

* On **Silhouette score** and **DBI**: TNC has best overall scores

* Overall performance decreases with dataset complexity and length

* Triplet Loss has a better Silhouette score than CPC, but CPC has better DBI than Triplet Loss

* **Classifiability:** TNC outperforms other two models

* We use **AUPCR** and **Accuracy**: overall best scores are with TNC

* Overall performance diminishes with dataset complexity and length

* Although Triplet Loss has better scores for both AUPCR and accuracy over CPC

<figure><img src="image_4.png" /><figcaption>Two-dimensional t-SNE representation of the learned representations for five states simulation dataset across all baselines.</figcaption></figure>

<figure><img src="image_5.png" /></figure>

* **Triplet Loss**¹ uses **contrastive** learning by sampling a **tuple of triplets** $(x^{pos}, x^{ref}, x_k^{neg})$

* There is weight sharing between similar $(x^{pos}, x^{ref})$ and non-similar $(x_k^{neg}, x^{ref})$ time series samples

* We sample multiple negative $x_k^{neg}$ randomly chosen independently

* We optimize the objective:

$$ \mathcal{L} = -\log \sigma \left( f_{\theta}(x^{ref})^T f_{\theta}(x^{pos}) \right) - \frac{1}{K} \sum_{k=1}^{K} \log \sigma \left( -f_{\theta}(x^{ref})^T f_{\theta}(x_k^{neg}) \right) $$

* **TNC**² combines **contrastive** learning with **Positive Unlabeled (PU)**

* Learning

* It has 3 components:

1) Temporal Neighborhood: determining regions with similar underlying states

2) Encoder: a sequential deep/shallow neural network

3) Discriminator: a multi-headed binary classifier

* TNC optimizes the objective:

$$ \mathcal{L} = -\mathbb{E}_{W_t \sim X} \left[ \mathbb{E}_{W_t \sim N_t} \left[ \log (D(Z_t, Z)) \right] + \mathbb{E}_{W_k \sim \bar{N}_t} \left[ w_t \log (D(Z_t, Z)) + (1 - w_t) \log (1 - D(Z_t, Z)) \right] \right] $$

* **CPC**³ combines **predictive coding** with **contrastive** learning.

* The objective is to minimize mutual information between $x_{t+1}$ and contextual latent states $c_t$

* Here components are jointly optimized via an **InfoNCE loss**

1) Encoder: a sequential deep/shallow neural network

2) Auto-regressor: constructing contexts from encodings

* CPC Optimizes the objective:

$$ \mathcal{L} = -\mathbb{E}_X \left[ \log \frac{f_k(x_{t+k}, c_t)}{\sum_{x_j \in X} f_k(x_j, c_t)} \right] $$

## 7. Conclusion

* **TNC** outperformed two other methods on **investigated** KPI scores

* **Weight parameter** in TNC is important for the loss and representation quality

* **Window size** selection is crucial for performance across baselines

* TNC is a **better option** and worthy of **further consideration**

## 8. Future work

* Evaluate TNC on real MVT car data with 2.5 years of driving

* Explore scenarios such as increasing number of features to incorporate different sensory data

* Combine all strategies to learn larger datasets of multiple vehicles

## References

1. Representation with Triplet loss [J.Y. Franceschi, et.al. (2020)]

2. Representation Temporal Neighborhood Coding [S. Tonekaboni, et.al. (2021)]

3. Representation with Contrastive Predictive Coding [A. van den Oord, et.al. (2019)]

4. K-means: Classification and analysis of multivariate observations [J. MacQueen, (1967)]

5. t-SNE: Visualizing data using t-SNE [L. Van der Maaten, et.al. (2008)]

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>