# Correlation Study between Time Series Entropies and Training Efficiency of Autoencoder Models

Dylan Baptiste, Ramla Saddem, François Foyer, Alexandre Philippot

► To cite this version:

Dylan Baptiste, Ramla Saddem, François Foyer, Alexandre Philippot. Correlation Study between Time Series Entropies and Training Efficiency of Autoencoder Models. International Conference on Smart-Digital-Green Technologies and Artificial Intelligence Sciences, Galala University, Dec 2024, Galala City, Egypt. [hal-04994611](https://hal.science/hal-04994611v1)

HAL Id: hal-04994611

https://hal.science/hal-04994611v1

Submitted on 2 May 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Correlation Study between Time Series Entropies and Training Efficiency of Autoencoder Models

Dylan Baptiste*†, Ramla Saddem*, François Foyer†, Alexandre Philippot*

* Université de Reims Champagne-Ardenne, CRESTIC, Reims, France

† Seckiot, Paris, France

{dylan.baptiste, ramla.saddem, alexandre.philippot}@univ-reims.fr, francois.foyer@seckiot.fr

***Abstract***—Today, data exploitation has become possible thanks to various artificial intelligence techniques. However, the questionable quality of some data can compromise the reliability of these models. In some contexts, a large number of features are available, making it challenging to select the most relevant ones for model development. Several entropy measures have been proposed in the literature as a means of quantifying the complexity of time series. In this paper, we propose to investigate the relationship between certain time series entropy measures (Approximate Entropy, Sample Entropy, Fuzzy Entropy, Bubble Entropy, Entropy of Entropy, Phase Entropy, Spectral Entropy and Slope Entropy) and the results of training an autoencoder—a deep learning architecture—in order to estimate the quality of model outputs in advance.

***Index Terms***—Entropy, Machine Learning, Autoencoder, Time Series

## I. INTRODUCTION

The increasing availability of time series data across different fields has led to a growing demand for methods capable of analyzing and extracting meaningful patterns from this type of data [1], [2]. Time series are typically characterized by their temporal dependencies, non-linearity, and noise, which can make traditional modeling techniques insufficient. Recently, autoencoder-based models have gained popularity due to their capacity to learn latent representations [3], [4] that capture the essential features of time series data [5]. However, one challenge remains: the efficiency of training autoencoders is often dependent on the underlying structure and complexity of the data. To better understand this relationship, entropy measures have been proposed [6]–[15] as a means to quantify the complexity of time series. Entropy in time series data can be thought of as a measure of uncertainty, unpredictability or disorder in the temporal sequence. In [16] the authors use entropy feature selection method on medical time series data, they achieve an average reduction of 3.10 times in the number of model parameters for the same performance. There are several entropy measures each capturing different aspects of complexity. The hypothesis of this study is that the complexity of a time series, as quantified by its entropy, may directly indicate how an autoencoder training session should perform. This relationship should be visible with a positive correlation

between the entropy measures and the reconstruction error of the autoencoder model. If some entropy measures are strongly correlated, this can lead to the anticipation of results before launching training sessions, which are heavy and time-consuming calculations in deep learning, and can be used for feature selection, or any other form of decision-making before modeling deep learning solutions.

In this paper, we investigate the correlation between time series entropy and the training efficiency of autoencoder models on a dataset of network traffic captures from industrial control systems. We first describe the autoencoder model and all the entropy measures used with a brief description of each. Then, we present the dataset used and the data preparation process. Next, the methodology used to calculate the entropy measures and train the autoencoder model is described. Finally, we present the experimental results and discuss the implications of the findings.

## II. AUTOENCODER ARCHITECTURE

Autoencoders are neural network models used primarily for representation learning [17]. They were then used as dimentinality reduction framework [3], but can be also use in various other tasks such as anomaly detection, data denoising, generative model, recommendation system and much more with some slight architecture modifications.

These models consist generally of two main parts: an encoder *E*, which compresses the input data $x \in \mathbb{R}^n$ into a lower-dimensional representation named the latent-space $z \in \mathbb{R}^d$ with $d < n$, and a decoder *D*, which produces a reconstruction $\hat{x} \in \mathbb{R}^n$ of the input data from the latent-space representation. The training process minimizes a reconstruction error between the input data and the output data of the model. As part of a first-year thesis at the cybersecurity company Seckiot [18], we investigate whether entropy, a measure of uncertainty or disorder in the data, correlates with the efficiency of training autoencoders in terms of their ability to minimize reconstruction error.

The autoencoder used in this study consists of an input layer, multiple hidden layers in both the encoder and decoder parts, the latent space which is modeled with a single layer, and an output layer. The Figure 1 shows the architecture of the autoencoder model used.

The work presented in this article was carried out as part of collaboration between CReSTIC Labs and Seckiot funded under the ANRT (Association Nationale de la Recherche et de la Technologie), the national association for Research and Technology.
<figure><img src="image_2.png" /><figcaption>Fig. 1. Autoencoder architecture</figcaption></figure>

## III. DATASET AND DATA PREPARATION

The dataset utilized for this study originates from the 4SICS dataset [19], which contains network traffic captures (pcap files) of multiple industrial control systems (ICS). While this dataset has been processed for various cybersecurity applications, it was selected here primarily to provide sufficient data for experimentation with real world time series data in Operational Technology (OT) environments. The pcap files were pre-processed into the Seckiot cybersecurity platform [18], which provides cartography and analysis of network traffic in OT environments. The tool allows to do protocol resolution and pacquet deduplication from pcap files and the saving of the data in time series database (Elasticsearch). Network packet exchange between assets on different network levels was captured then resampled to 1-second intervals to calculate the volume of data exchanged between assets on each levels. The Figure 2 illustrates the network layers L2, L3 and L4 where the data was captured.

<figure><img src="image_3.png" /><figcaption>Fig. 2. L2, L3 and L4 network layers illustrated</figcaption></figure>

All time series are then normalized using the standardization method :

$$x = \frac{x - \mu}{\sigma}$$

where $\mu$ is the mean and $\sigma$ is the standard deviation of the time series to be able to compare training losses between different time series.

## IV. ENTROPY ANALYSIS METHODOLOGY

In this section the methodology used to calculate the entropy measures and the training of the autoencoder model is described. A description of the entropy measures used and the detail of the autoencoder model is provided.

### A. Entropy Measures

Entropy is a measure of uncertainty or disorder in a system, and can be used to quantify the complexity of time series data. In this study, we consider multiple time series entropy measures to capture different aspects of complexity in the time series data.

* Approximate Entropy (ApEn) measures the regularity or predictability of a time series by quantifying the likelihood that similar patterns of observations remain consistent over time. A lower ApEn indicates more regularity, while a higher value suggests increased complexity or irregularity. [6]

* Sample Entropy (SampEn) improves upon ApEn by reducing bias and eliminating self-matching during comparisons. It evaluates the similarity between sequences of data points, where lower values indicate more predictability and higher values reflect greater complexity. [7]

* Fuzzy Entropy (FuzzyEn) extends SampEn by incorporating fuzzy set theory, introducing a degree of fuzziness in pattern similarity to better handle noisy and imprecise data, making it more robust in detecting subtle patterns. [8], [9]

* Bubble Entropy (BubbEn) measures complexity based on bubble sorting, which rearranges data points in a specific order. It quantifies the amount of effort (or swaps) needed to organize a sequence, linking complexity to the sorting process. [10]

* Entropy of Entropy (EnofEn) is a meta-level entropy that examines the distribution of entropies across different time windows, providing an overall complexity assessment of a system by analyzing the variability in individual entropy measures. [11]

* Phase Entropy (PhasEn) assesses complexity by analyzing the phase space trajectories of the time series, often used to understand dynamical systems' temporal behaviors and predictability. [12]

* Spectral Entropy (SpecEn) measures the distribution of power across frequency components in a signal. Low SpecEn indicates dominance of specific frequencies, while high SpecEn suggests a more uniform distribution, often associated with noise. [13], [14]

* Slope Entropy (SlopEn) quantifies variability by analyzing the slopes between successive points. It captures irregularities in the rate of change, useful for detecting shifts or transitions in the system's dynamics. [15]

Let $X$ be the set of time series data, and $E$ be the set of the eight entropy measures presented above. Let $x_i \in X$ and $e_j \in E$. The entropy measures are defined as

$$e_j : \mathbb{R}^n \mapsto \mathbb{R}^+$$

For each time series $x_i \in X$, the entropy measures $e_j \in E$ are calculated as $v_{ij} = e_j(x_i)$.

### B. Autoencoder Training

The autoencoder model is trained on the time series data to learn a latent space representation that captures the essential
features of the data. Each time series is split by a sliding window of fixed size, these windows are used as input and output data for the autoencoder model. For each time series the same model architecture with the same metaparameters is used. Between each training session the wheights are reinitialized to get a fair comparison between the different time series. After each training session, the mean reconstruction error of the last 10 epochs is calculated on the training and validation datasets. These values are used later to compute the correlation with the entropy measures.

We fix the number of hidden layers to 2 for both the encoder and decoder, with 64 and 32 neurons respectively for the encoder and 32 and 64 neurons for the decoder. The latent space has 16 neurons. We use the ReLU activation function for all layers except the latent space and output layer, where the linear activation function is used. This architecture is chosen arbitrarily to balance the training time and the performance of the model on the dataset. The Adam optimizer [20] with default parameters is used. As for the reconstruction error, we use the Mean Squared Error (MSE) loss function, defined as

$$ \mathcal{L} = \frac{1}{n} \sum_{i=1}^{n} (x_i - \hat{x}_i)^2 $$

where $x_i$ is the input data and $\hat{x}_i$ is the reconstructed one. The window size chosen to split the time series is fixed to 120 data points, which means with a sampling rate of 1 second, all windows cover a 2 minutes period. The dataset is split into 75% for training and 25% for validation. One of the perspectives is to vary all these parameters to see if they have an impact on the results.

## V. EXPERIMENTAL RESULTS

The experimental results show the correlation between the entropy measures obtained in the procedure described in IV-A and the mean reconstruction error of the autoencoder model described in IV-B. The correlation coefficient is calculated using the Pearson correlation coefficient formula:

$$ r_{ij} = \frac{\sum_{i=1}^{n} (v_{ij} - \bar{v}_{ij})(\mathcal{L}_i - \bar{\mathcal{L}})}{\sqrt{\sum_{i=1}^{n} (v_{ij} - \bar{v}_{ij})^2} \sqrt{\sum_{i=1}^{n} (\mathcal{L}_i - \bar{\mathcal{L}})^2}} $$

where $v_{ij}$ is the entropy measure j of the time series i, $\mathcal{L}_i$ is the mean reconstruction error of the time series i, n is the number of time series, and $\bar{v}_{ij}$ and $\bar{\mathcal{L}}$ are the means of the entropy measures and the mean reconstruction error respectively.

The results show in Figure 3 present the correlation between the entropy measures and the mean reconstruction error of the last 10 epoch of the autoencoder model with the training and the validation dataset. The correlation between the entropy measures and the reconstruction errors are relatively poor, indicating that the complexity of the time series data, as quantified by the entropy measures on this specific dataset, does not directly correlate well with the efficiency of training the autoencoder model. The correlation with the validation dataset does not exceed 0.5 with a maximum of 0.49 for the

BubbEn. On the training dataset the correlation are higher in average with a maximum of 0.67 for the EnofEn. It is worth noting that PhasEn exhibits a negative correlation with the reconstruction error on the validation and training datasets. The correlation between the different entropy measures is also shown. The results indicate that some entropy measures are more correlated with each other than others, suggesting that they capture similar aspects of complexity in the time series data. For example we show a correlation of $r = 0.93$ between ApEn and FuzzEn, which is expected since FuzzEn is an extension of ApEn [8]. In contrast, the correlation between FuzzEn and PhasEn is $r = -0.09$, this near 0 value suggesting that FuzzEn and PhasEn capture different aspects of complexity in the time series data.

## VI. DISCUSSION

The results of this study indicate that the complexity of time series data, as quantified by entropy measures, does not extremely correlate with the efficiency of training autoencoder models as first expected. This suggests that the entropy measures used may not be suitable for predicting the performance of autoencoder models on this specific dataset. They could may be multiple reasons for this result, the first one is that all time series data of the dataset are not equal in time, leading to a bias during the training of the autoencoder model. In fact if a time series is shorter than another, the model will have less data to learn from and will be more efficient to reconstruct the data. A second reason is that the entropy measures used may not capture the relevant aspects of complexity in the time series data that affect the training of autoencoder models. Another cause of this low correlation is information redundancy: some time series may be complex in terms of entropy, but have repetitive patterns or redundancy, which makes them easier to reconstruct by the autoencoder. So, even with high perceived complexity, the model can achieve low reconstruction error. This notion of redundancy could be measured by autocorrelation calculations on the data. It could also explain why EnofEn is one of the most correlated measures. It is worth than Apen, SampEn and FuzzEn have a strong correlation between them, which is expected since FuzzEn and SampEn are successive improvements of ApEn. We plan to explore other entropy measures or complexity measures like hurst exponent or fractal dimension [21] which are not entropies but are also used to quantify the complexity of time series data. Lyapunov exponent [22] could also be used to quantify the chaotic behavior of the time series data. We also want to examine the impact of fixed parameters like the sampling rate (fixed to 1s here), the window size (fixed to 120 data point here) and some hyperparameters of the autoencoder model to see if they have an impact on the results.

## VII. CONCLUSION

In this paper, we have investigated the correlation between time series entropy measures and the training efficiency of autoencoder models. The results of the experiment indicate that the complexity of time series data, as quantified by
<figure><img src="image_4.png" /><figcaption>Fig. 3. Correlation between entropy measures and mean reconstruction error</figcaption></figure>

entropy measures listed in IV-A, does not correlate well with
the efficiency of training autoencoder models. This suggests
that the entropy measures used may not capture entirely the
relevant aspects of complexity in the time series data that
affect the training of autoencoder models. For future work,
we plan to explore other entropy measures that the ones
used in this study and add other dataset. In the experimental
setup some variables are parametrized arbitrarily, like the time
sampling (1 second) and the window size (120). There is also
numerous other parameters related to the autoencoder model
like number of layers, number of neurons per layer, activation
function, optimizer, loss function, latent space dimension, etc.

REFERENCES

[1] G. E. Box, G. M. Jenkins, G. C. Reinsel, and G. M. Ljung, *Time series analysis: forecasting and control*. John Wiley & Sons, 2015.

[2] R. H. Shumway, D. S. Stoffer, and D. S. Stoffer, *Time series analysis and its applications*. Springer, 2000, vol. 3.

[3] G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” *Science*, vol. 313, no. 5786, pp. 504–507, 2006. [Online]. Available: https://www.science.org/doi/abs/10.1126/science.1127647

[4] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, “Extracting and composing robust features with denoising autoencoders,” in *Proceedings of the 25th international conference on Machine learning*, 2008, pp. 1096–1103.

[5] M. Memarzadeh, B. Matthews, and T. Templin, “Multiclass anomaly detection in flight data using semi-supervised explainable deep learning model,” *Journal of Aerospace Information Systems*, vol. 19, no. 2, pp. 83–97, 2022. [Online]. Available: https://doi.org/10.2514/1.I010959

[6] S. M. Pincus, “Approximate entropy as a measure of system complexity.” *Proceedings of the National Academy of Sciences*, vol. 88, no. 6, pp. 2297–2301, 1991. [Online]. Available: https://www.pnas.org/doi/abs/10.1073/pnas.88.6.2297

[7] J. S. Richman and J. R. Moorman, “Physiological time-series analysis using approximate entropy and sample entropy,” *American journal of physiology-heart and circulatory physiology*, vol. 278, no. 6, pp. H2039–H2049, 2000.

[8] W. Chen, Z. Wang, H. Xie, and W. Yu, “Characterization of surface emg signal based on fuzzy entropy,” *IEEE Transactions on Neural Systems and Rehabilitation Engineering*, vol. 15, no. 2, pp. 266–272, 2007.
[9] W. Chen, J. Zhuang, W. Yu, and Z. Wang, "Measuring complexity using fuzzyen, apen, and sampen," *Medical Engineering & Physics*, vol. 31, no. 1, pp. 61–68, 2009. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1350453308000726

[10] G. Manis, M. Aktaruzzaman, and R. Sassi, "Bubble entropy: An entropy almost free of parameters," *IEEE Transactions on Biomedical Engineering*, vol. 64, no. 11, pp. 2711–2718, 2017. [Online]. Available: https://ieeexplore.ieee.org/document/7842617

[11] C. F. Hsu, S.-Y. Wei, H.-P. Huang, L. Hsu, S. Chi, and C.-K. Peng, "Entropy of entropy: Measurement of dynamical complexity for biological systems," *Entropy*, vol. 19, no. 10, 2017. [Online]. Available: https://www.mdpi.com/1099-4300/19/10/550

[12] A. Rohila and A. Sharma, "Phase entropy: A new complexity measure for heart rate variability," *Physiological measurement*, vol. 40, no. 10, p. 105006, 2019.

[13] G. Powell and I. Percival, "A spectral entropy method for distinguishing regular and irregular motion of hamiltonian systems," *Journal of Physics A: Mathematical and General*, vol. 12, no. 11, p. 2053, 1979.

[14] T. Inouye, K. Shinosaki, H. Sakamoto, S. Toi, S. Ukai, A. Iyama, Y. Katsuda, and M. Hirano, "Quantification of eeg irregularity by use of the entropy of the power spectrum," *Electroencephalography and clinical neurophysiology*, vol. 79, no. 3, pp. 204–210, 1991.

[15] D. Cuesta-Frau, "Slope entropy: A new time series complexity estimator based on both symbolic patterns and amplitude information," *Entropy*, vol. 21, no. 12, 2019. [Online]. Available: https://www.mdpi.com/1099-4300/21/12/1167

[16] Y. Huang, Y. Zhao, A. Capstick, F. Palermo, H. Had-dadi, and P. Barnaghi, "Analyzing entropy features in time-series data for pattern recognition in neurological conditions," vol. 150, p. 102821. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0933365724000630

[17] D. E. Rumelhart and J. L. McClelland, *Learning Internal Representations by Error Propagation*, 1987, pp. 318–362.

[18] Seckiot. [Online]. Available: https://www.seckiot.fr/

[19] SCADA / ICS PCAP files from 4SICS. Netresec. [Online]. Available: https://www.netresec.com/?page=PCAP4SICS

[20] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," 2017. [Online]. Available: https://arxiv.org/abs/1412.6980

[21] H. E. Hurst, "Long-term storage capacity of reservoirs," *Transactions of the American Society of Civil Engineers*, vol. 116, no. 1, pp. 770–799, 1951. [Online]. Available: https://ascelibrary.org/doi/abs/10.1061/TACEAT.0006518

[22] A. Wolf, J. B. Swift, H. L. Swinney, and J. A. Vastano, "Determining lyapunov exponents from a time series," *Physica D: Nonlinear Phenomena*, vol. 16, no. 3, pp. 285–317, 1985. [Online]. Available: https://www.sciencedirect.com/science/article/pii/0167278985900119