# Unsupervised Anomaly Detection in NSL-KDD Using β-VAE: A Latent Space and Reconstruction Error Approach

Dylan Baptiste, Ramla Saddem, Alexandre Philippot, François Foyer

► To cite this version:

Dylan Baptiste, Ramla Saddem, Alexandre Philippot, François Foyer. Unsupervised Anomaly Detection in NSL-KDD Using β-VAE: A Latent Space and Reconstruction Error Approach. 2025 15th France-Japan & 13th Europe-Asia Congress on Mechatronics (MECATRONICS) / 23rd International Conference on Research and Education in Mechatronics (REM), Dec 2025, Saint-Ouen-sur-Seine, France. pp.1-6, 10.1109/MECATRONICS-REM67547.2025.11349488. hal-05520998

HAL Id: hal-05520998

https://hal.science/hal-05520998v1

Submitted on 20 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Unsupervised Anomaly Detection in NSL-KDD
Using β-VAE: A Latent Space and Reconstruction
Error Approach

Dylan Baptiste*†, Ramla Saddem*, Alexandre Philippot*, François Foyer†

*Université de Reims Champagne-Ardenne, CRESTIC, Reims, France

Email: {dylan.baptiste, ramla.saddem, alexandre.philippot}@univ-reims.fr

†Seckiot, Paris, France

Email: francois.foyer@seckiot.fr

**Abstract**—As Operational Technology increasingly integrates with Information Technology, the need for Intrusion Detection Systems becomes more important. This paper explores an unsupervised approach to anomaly detection in network traffic using β-Variational Autoencoders on the NSL-KDD dataset. We investigate two methods: leveraging the latent space structure by measuring distances from test samples to the training data projections, and using the reconstruction error as a conventional anomaly detection metric. By comparing these approaches, we provide insights into their respective advantages and limitations in an unsupervised setting. Experimental results highlight the effectiveness of latent space exploitation for classification tasks.

**Index Terms**—Autoencoder, Deep Learning, Unsupervised Learning, Anomaly Detection, Cybersecurity, Intrusion Detection System.

## I. INTRODUCTION

The increasing integration of Operational Technology (OT) with Information Technology (IT) systems has led to a growing need for intrusion detection systems (IDS) in industrial environments. Anomaly detection is a crucial component of IDS, as it enables the identification of malicious activities that deviate from normal behavior. In this context, the NSL-KDD dataset [1] is a widely used benchmark for evaluating the performance of anomaly detection algorithms.

In this paper, we explore an unsupervised approach to anomaly detection in network traffic using β-Variational Autoencoders (β-VAE) [2]. β-VAEs are a deep learning model that can learn a low-dimensional representation of the input data, known as the latent space. By leveraging the latent space structure, we aim to detect anomalies in network traffic without the need for labeled data.

We investigate two complementary methods for anomaly detection using β-VAEs: measuring the distances from test samples to the projections of the training data in the latent space, and using the reconstruction error as a conventional anomaly detection metric. By comparing these approaches, we

provide insights into their respective advantages and limitations in an unsupervised setting.

The remainder of this paper is organized as follows: Section II provides definitions and formalizations of the key concepts in this study, such as the β-VAE model and the NSL-KDD dataset. Section III presents related work in the field of anomaly detection and the use of the latent space in the autoencoder framework. Section IV describes the methodology used in this study, including the VAE architecture and the anomaly detection methods. Section VI presents the experimental results and discusses the performance of the proposed methods. Finally, Section VII concludes the paper and outlines directions for future work.

## II. DEFINITIONS

### A. β-VAE Model

The β-VAE architecture is composed of an encoder $q$ that maps the input data $x$ to the latent space $z$ and a decoder $p$ that reconstructs the initial data from the latent representation.

The original training objective of a VAE is to maximize the evidence lower bound (ELBO), which for a β-VAE becomes

$$ \mathbb{E}_{q_{\phi}(z|x)}[\log p_{\theta}(x|z)] - \beta D_{KL}(q_{\phi}(z|x)\|p(z)) \quad (1) $$

where $\theta$ and $\phi$ are the parameters of the decoder and encoder networks, respectively, and $z$ is the latent variable; $q_{\phi}(z|x)$ is the approximate posterior distribution of $z$ given input $x$, $p_{\theta}(x|z)$ is the data likelihood given the latent variable, and $D_{KL}$ is the Kullback-Leibler divergence [3] between the approximate posterior and the prior distribution $p(z)$. The β term controls the balance between latent space regularization and reconstruction fidelity. Higher values of β enforce greater disentanglement but may compromise reconstruction accuracy. The model is trained by minimizing the loss function (2) with a stochastic gradient descent algorithm.

In practice, the model is trained by minimizing the negative ELBO, which is written as

$$ - \mathbb{E}_{q_{\phi}(z|x)}[\log p_{\theta}(x|z)] + \beta D_{KL}(q_{\phi}(z|x)\|p(z)) \quad (2) $$

The work presented in this article was carried out as part of collaboration between CReSTIC Labs and Seckiot funded under the ANRT (Association Nationale de la Recherche et de la Technologie), the national association for Research and Technology.
<table><caption>TABLE I
CATEGORIES OF ATTACKS IN NSL-KDD</caption><thead><tr><td>DoS</td><td>Probe</td><td>U2R</td><td>R2L</td></tr></thead><tbody><tr><td>neptune, smurf, back, teardrop, pod, land, apache2, mailbomb, processtable, udpstorm, worm</td><td>ipsweep, nmap, portsweep, satan, mscan, saint</td><td>buffer_overflow, loadmodule, perl, rootkit, http tunnel, ps, sqlattack, xterm</td><td>ftp_write, guess_password, imap, multihop, phf, spy, warezclient, warezmaster, snmpgetattack, snmpguess, xlock, xsnoop</td></tr></tbody></table>

A reparameterization trick [4] is applied to sample $z$ from the latent distribution $q_{\phi}(z|x)$, allowing gradients to be back-propagated through the stochastic sampling process.

In the rest of this study the reconstruction error will be denoted as $\mathcal{L}_{\text{rec}} = -\mathbb{E}_{q_{\phi}(z|x)}[\log p_{\theta}(x|z)]$ and the KL divergence as $\mathcal{L}_{\text{KL}} = D_{KL}(q_{\phi}(z|x) || \mathcal{N}(0, I))$, where $\mathcal{N}(0, I)$ is the standard normal distribution.

## B. NSL-KDD Dataset

The NSL-KDD dataset is a benchmark dataset for evaluating the performance of intrusion detection systems. It is a modified version of the KDD Cup 1999 dataset, which contains network traffic data. The NSL-KDD dataset consists of 41 features, including 34 continuous and 7 categorical features. The dataset contains five classes of network traffic: normal, denial of service (DoS), probe, user-to-root (U2R), and remote-to-local (R2L). Table I shows the categories of attacks in the NSL-KDD dataset [1].

## III. RELATED WORK

Anomaly detection in unsupervised settings has been the subject of numerous approaches based on autoencoders (AE) and their variants, with some recent studies focusing on leveraging the latent space. Previous works on unsupervised anomaly detection have explored various methods to handle high-dimensional or highly nonlinear data.

Already in 2007, [5] proposed a fault detection method for industrial processes based on the k-nearest neighbors (k-NN) rule, using only data from normal operation. This approach addresses the absence of anomalous training data by modeling the distribution of distances between normal samples and their nearest neighbors. Anomalies are then identified as samples whose distance exceeds a threshold derived from this distribution. In this work, the authors clearly explain the principle that will be revisited later in works exploiting approaches generating richer spaces, such as the latent spaces of AEs or their variants.

Hybrid models combining AEs or their variants with neighborhood techniques have been developed to enhance anomaly detection on high-dimensional data [6], [7], and [8]. These methods highlighted the advantage of nonlinear representation while leveraging distance measures in the latent space.

Other works focused on industrial applications and monitoring systems have also highlighted the interest of this approach, demonstrating that combining reconstruction error with latent space distance analysis can yield competitive or

even superior performance compared to traditional anomaly detection methods [9] and [10].

Subsequently, works introduced the use of models based on Variational Autoencoders (VAE), and more specifically β-VAE, to achieve a more structured and interpretable organization of the latent space. These studies suggest that considering the distribution of latent variables can contribute to finer anomaly detection, whether through reconstruction error measures or distances in the latent space [11]. In a similar context, [12] aimed at estimating a confidence measure through the exploitation of projections in the latent space and Mahalanobis distance has been presented to enhance intrusion detection on datasets like NSL-KDD.

Other contributions have sought to constrain the latent space to promote the emergence of clusters with similar behaviors. For example, [13] aims to limit the reconstruction capacity of AEs during training using an additional constraint that acts as a regularization on the latent space. In [14], compact clustering methods in the latent space were developed in a semi-supervised framework, allowing for the grouping of projections of samples with the same label, attracting unlabeled projections in the space and thus better identifying deviations. These techniques illustrate the interest of latent structure for separating normal data from anomalies.

The work proposed in this article aligns with the previously mentioned studies. We leverage the structure of the latent space of a β-VAE for anomaly detection in network traffic, using the NSL-KDD dataset. We formalize and compare the two methods (reconstruction error and distance in the latent space) for anomaly detection, highlighting their respective advantages and disadvantages. We show that exploiting the latent space yields results comparable to those of reconstruction error, while providing better interpretability of the results. We also observe the impacts of certain parameters on the performance of both methods, particularly β and k, the number of neighbors considered for calculating the distance in the latent space.

## IV. METHODOLOGY

In this study, we focus on the binary classification task of detecting normal traffic and anomalies in an unsupervised setting; in the analysis, we will also present results per attack classes and categories.

### A. Data Preprocessing

We exploit an unsupervised learning approach, using only the labeled normal data from the training dataset. We have
therefore re-divided NSL-KDD presented in section II-B into 3 parts:

* The anomalous dataset, which includes all attack data from the training and test datasets. This dataset is named $X_{attack}$.

* The training dataset, which includes only the labeled normal data from the training dataset. This dataset is named $X_{train}$.

* The test dataset, which includes only the labeled normal data from the test dataset. This dataset is named $X_{test}$.

The NSL-KDD dataset includes a mix of categorical, boolean, and continuous features. To prepare the data for training the β-VAE model, we first applied one-hot encoding to the 3 categorical features (protocol_type, service, and flag), converting them into binary vectors. The 4 boolean features (land, logged_in, is_guest_login, and is_host_login) were encoded as binary values (0 or 1). The remaining 33 features are continuous and were standardized using the mean and standard deviation computed from the training set $X_{train}$, ensuring all features operate on a comparable scale, which is essential for stable model training.

No feature selection was performed; we retained all features except for the difficulty attribute, which is not relevant to our study. Class labels were excluded during training to maintain an unsupervised learning setting.

## B. Model Architecture

We use a β-VAE architecture as described in section II-A. The encoder and decoder networks each comprise three fully connected hidden layers: 64, 32, and 16 neurons for the encoder, and 16, 32, and 64 neurons for the decoder. The encoder outputs the mean and log variance through two separate fully connected layers with 8 neurons each, which represent the parameters of the Gaussian distribution in this 8-dimensional latent space. The decoder takes the sampled latent variable and reconstructs the input data. For the stochastic gradient descent algorithm, we use the Adam optimizer [15] with a learning rate of 0.001 and a batch size of 2048.

During reconstruction, the total reconstruction loss is computed as a linear combination of three loss functions, each tailored to the nature of the feature:

* **Categorical features:** softmax activation with categorical cross-entropy loss

$$ \mathcal{L}_{\text{cat}} = -\frac{1}{n} \sum_{i=1}^{n} \sum_{j=1}^{m} x_{ij} \log(\hat{x}_{ij}) \quad (3) $$

where n is the number of samples, m is the number of categories for a given feature, $x_{ij}$ is a binary indicator (0 or 1) that the $i^{th}$ sample belongs to category j, and $\hat{x}_{ij}$ is the predicted probability for category j in sample i.

* **Boolean features:** sigmoid activation with binary cross-entropy loss

$$ \mathcal{L}_{\text{bool}} = -\frac{1}{n} \sum_{i=1}^{n} [x_i \log(\hat{x}_i) + (1-x_i) \log(1-\hat{x}_i)] \quad (4) $$

where $x_i$ is the true binary value and $\hat{x}_i$ is the predicted probability for the $i^{th}$ sample.

* **Continuous features:** linear activation with mean squared error (MSE)

$$ \mathcal{L}_{\text{cont}} = \frac{1}{n} \sum_{i=1}^{n} (x_i - \hat{x}_i)^2 \quad (5) $$

where $x_i$ and $\hat{x}_i$ are the true and reconstructed continuous values for the $i^{th}$ sample.

The total reconstruction loss $\mathcal{L}_{\text{rec}}$ is defined as a linear combination of the three components

$$ \mathcal{L}_{\text{rec}} = \mathcal{L}_{\text{cat}} + \mathcal{L}_{\text{bool}} + \mathcal{L}_{\text{cont}} \quad (6) $$

The β-VAE loss $\mathcal{L}$ is then the combination of $\mathcal{L}_{\text{rec}}$ and the KL divergence term, as defined in Equation (2) and weighted by the β parameter

$$ \mathcal{L} = \mathcal{L}_{\text{rec}} + \beta \mathcal{L}_{\text{KL}} \quad (7) $$

## V. β-VAE EXPLOITATION FOR CLASSIFICATION

Anomaly detection can be approached in two distinct ways within the framework of our β-VAE model: through reconstruction error or by analyzing the latent space. Each of these methods allows classifying data as anomalies or normal data, but based on different criteria. In both cases, we evaluate performance using false positive rate (FPR) and true positive rate (TPR) for different thresholds, with the Area Under the Receiver Operating Characteristic curve (AUROC) as the performance metric.

### A. Anomaly detection based on reconstruction error

The first anomaly detection approach relies on reconstruction error or by analyzing the latent space. Each of these methods allows classifying data as anomalies or normal data, but based on different criteria. In both cases, we evaluate performance using false positive rate (FPR) and true positive rate (TPR) for different thresholds, with the Area Under the Receiver Operating Characteristic curve (AUROC) as the performance metric.

Once the reconstruction error is calculated, a threshold is set to distinguish normal data from anomalous data. Data points for which the error exceeds this threshold are considered anomalous, while those with an error below the threshold are classified as normal. This detection approach is named $\mathcal{L}_{\text{rec}}$-classification in the rest of this work.

The Algorithm 1 implements $\mathcal{L}_{\text{rec}}$-classification.

### B. Anomaly detection based on latent space

The second approach involves leveraging the latent space of the β-VAE model to detect anomalies. The idea is to project the normal data from the training set $X_{train}$ into the latent space using the encoder $q_\phi$ of the β-VAE model.

This approach will be referred to as $\mathcal{Z}_k$-classification, where k is an integer representing the number of neighbors to consider for calculating the average Euclidean distance. We denote
**Algorithm 1 $\mathcal{L}_{rec}$-classification**

**Require:** $x$ a sample to classify, $(q_{\phi}, p_{\theta})$: a trained $\beta$-VAE, $\tau$: the threshold

**Ensure:** $y$: classification label: normal or anomaly

1: $z \sim q_{\phi}(z|x) \quad \triangleright \text{Encoder}$

2: $\hat{x} \leftarrow p_{\theta}(z) \quad \triangleright \text{Decoder}$

3: **if** $\mathcal{L}_{rec}(x, \hat{x}) > \tau$ **then**

4: $y \leftarrow$ anomaly

5: **else**

6: $y \leftarrow$ normal

7: **end if**

8: **return** $y$

$\mathcal{Z}_k^X(x)$ as the average Euclidean distance between $z$ (the projection of $x$) and the $k$ nearest neighbors of the projections of $X$. This average is calculated using the formula (8).

$$ \mathcal{Z}_k^X(x) = \frac{1}{k} \sum_{j=1}^{k} \|z - z'_{(j)}\|_2 \quad (8) $$

with $z \sim q_{\phi}(z|x)$ and $z'_{(j)}$ the $j$-th nearest neighbor of $z$ in the set of projections of $X$.

Similarly to the reconstruction error-based method, if an average distance exceeds a threshold, the data point will be considered anomalous; otherwise, it will be classified as normal.

The Algorithm 2 implements the $\mathcal{Z}_k$-classification.

**Algorithm 2 $\mathcal{Z}_k$-classification**

**Require:** $x$ a sample to classify, $(q_{\phi}, p_{\theta})$: a trained $\beta$-VAE, $X_{train}$: the training dataset, $k$: the number of neighbors, $\tau$: the threshold

**Ensure:** $y$: classification label: normal or anomaly

1: $Z_{train} \leftarrow \{z_i \sim q_{\phi}(z|x_i), \forall x_i \in X_{train}\}$

2: $z \sim q_{\phi}(z|x)$

3: Find the $k$ nearest neighbors $z'_{(1)}, \dots, z'_{(k)}$ of $z$ in $Z_{train}$

4: **if** $\mathcal{Z}_k^{X_{train}}(x) > \tau$ **then**

5: $y \leftarrow$ anomaly

6: **else**

7: $y \leftarrow$ normal

8: **end if**

9: **return** $y$

## VI. EXPERIMENTAL RESULTS

First, we present the performance of the two methods, $\mathcal{L}_{rec}$-classification and $\mathcal{Z}_k$-classification, on the binary classification of normal versus anomalous traffic. As stated in Section V, we evaluate performance using AUROC. To assess stability, we ran both methods over four runs with different seeds. The AUROC is computed for each run and then averaged over the four runs.

Figure 1 and Table II show the mean AUROC across $\beta$ and $k$, averaged over four runs. We have tested the $\mathcal{Z}_k$-classification with $k$ values of 1, 100, 150, 200, 250, 300,

400, 500, 1000, 2000, 3000, 4000, and 5000. The $\beta$ parameter was tested with values of 0, 0.00001, 0.0001, 0.001, 0.01, 0.1, and 0.5. In Table II, bold values represent the best mean result per $\beta$. Underlined values indicate cases where the mean AUROC with $\mathcal{Z}_k$-classification outperforms $\mathcal{L}_{rec}$-classification for a specific $\beta$.

In general, increasing $k$ improves the AUROC for the $\mathcal{Z}_k$-classification method. Results show that $\mathcal{Z}_k$-classification can outperform $\mathcal{L}_{rec}$-classification in some cases with large value of $k$. For $\mathcal{L}_{rec}$-classification, the best mean AUROC is achieved with $\beta = 0$, and this method appears relatively insensitive to $\beta$; AUROC ranges from 0.962 to 0.968. With this $\beta$ setting, $\mathcal{L}_{rec}$-classification is outperformed by $\mathcal{Z}_k$-classification for $k \ge 200$. For $\mathcal{Z}_k$-classification, the best mean AUROC is obtained with $\beta = 10^{-5}$ and $k = 5000$.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Mean AUROC of $\mathcal{L}_{rec}$-classification and $\mathcal{Z}_k$-classification with variables $\beta$ and $k$</figcaption></figure>

For the rest of the result we choose to focus on the results of a model trained with $\beta = 10^{-5}$ and $k = 5000$. The performance of this model is highlighted in Figure 2, Figure 3 as ROC curves analysis and in Figure 4 to show the distribution of data classified by both methods.
<table><caption>TABLE II
MEAN AUROC OF $\mathcal{L}_{rec}$-CLASSIFICATION AND $\mathcal{Z}_k$-CLASSIFICATION WITH VARIABLES $\beta$ AND $k$</caption><thead><tr><th rowspan="2">β</th><th colspan="12">AUROC (%)</th><th rowspan="2">L<sub>rec</sub></th></tr><tr><th>Z<sub>1</sub></th><th>Z<sub>100</sub></th><th>Z<sub>150</sub></th><th>Z<sub>200</sub></th><th>Z<sub>250</sub></th><th>Z<sub>300</sub></th><th>Z<sub>400</sub></th><th>Z<sub>500</sub></th><th>Z<sub>1000</sub></th><th>Z<sub>2000</sub></th><th>Z<sub>3000</sub></th><th>Z<sub>4000</sub></th><th>Z<sub>5000</sub></th></tr></thead><tbody><tr><td>0</td><td>94.11</td><td>96.43</td><td>96.63</td><td>96.76</td><td><u>96.89</u></td><td><u>96.99</u></td><td><u>97.09</u></td><td><u>97.14</u></td><td><u>97.40</u></td><td><u>97.48</u></td><td><u>97.56</u></td><td><u>97.66</u></td><td><b><u>97.70</u></b></td><td>96.78</td></tr><tr><td>0.00001</td><td>94.49</td><td><u>96.79</u></td><td><u>97.03</u></td><td><u>97.16</u></td><td><u>97.25</u></td><td><u>97.32</u></td><td><u>97.46</u></td><td><u>97.52</u></td><td><u>97.68</u></td><td><u>97.75</u></td><td><u>97.81</u></td><td><u>97.87</u></td><td><b><u>97.90</u></b></td><td>96.23</td></tr><tr><td>0.0001</td><td>94.28</td><td><u>96.60</u></td><td><u>96.76</u></td><td><u>96.96</u></td><td><u>97.12</u></td><td><u>97.19</u></td><td><u>97.26</u></td><td><u>97.29</u></td><td><u>97.52</u></td><td><u>97.65</u></td><td><u>97.70</u></td><td><b><u>97.73</u></b></td><td><u>97.73</u></td><td>96.52</td></tr><tr><td>0.001</td><td>93.51</td><td>95.81</td><td>96.20</td><td>96.47</td><td><u>96.66</u></td><td><u>96.71</u></td><td><u>96.66</u></td><td>96.58</td><td><u>96.69</u></td><td><u>96.80</u></td><td><u>96.82</u></td><td><u>96.85</u></td><td><b><u>96.86</u></b></td><td>96.61</td></tr><tr><td>0.01</td><td>93.47</td><td>96.16</td><td>96.37</td><td><u>96.48</u></td><td><u>96.57</u></td><td><u>96.64</u></td><td><u>96.71</u></td><td>96.76</td><td><b><u>96.96</u></b></td><td><u>96.85</u></td><td><u>96.86</u></td><td><u>96.85</u></td><td><u>96.82</u></td><td>96.44</td></tr><tr><td>0.1</td><td>91.05</td><td>93.52</td><td>93.85</td><td>94.11</td><td>94.31</td><td>94.46</td><td>94.64</td><td>94.76</td><td>95.14</td><td>95.32</td><td>95.35</td><td>95.35</td><td>95.32</td><td><b><u>96.48</u></b></td></tr><tr><td>0.5</td><td>75.08</td><td>84.26</td><td>85.28</td><td>86.01</td><td>86.59</td><td>87.06</td><td>87.81</td><td>88.37</td><td>89.88</td><td>90.93</td><td>91.35</td><td>91.56</td><td>91.67</td><td><b><u>96.28</u></b></td></tr></tbody></table>

<figure><img src="image_3.png" /><figcaption>Fig. 2. ROC curves for the binary classification task with $\mathcal{L}_{rec}$-classification and $\mathcal{Z}_k$-classification</figcaption></figure>

Figure 2 shows the ROC curves for the two methods on the binary classification task.

<figure><img src="image_4.png" /><figcaption>Fig. 3. ROC curves on $X_{test}$ and $X_{attack}$ with $\mathcal{L}_{rec}$-classification and $\mathcal{Z}_{5000}$-classification, per attack class</figcaption></figure>

Figure 3 shows the ROC curves for the two anomaly detection methods per attack class (Probe, DoS, U2R, and

R2L) described in Section II-B. Some attack classes are more difficult to detect than others.

<figure><img src="image_5.png" /><figcaption>Fig. 4. Distribution of $\mathcal{Z}_{5000}$-classification and $\mathcal{L}_{rec}$-classification on $X_{test}$ and $X_{attack}$. Blue points are classified as normal, purple as **Probe**, orange as **DoS**, green as **U2R**, and red as **R2L**. The distribution of each category is represented as a density on the opposing axes.</figcaption></figure>

Figure 4 shows the distribution of data classified by both methods. Both approaches achieve excellent results. The normal distribution is clearly separated from the attack distribution in both methods. We can also see that sometimes certain normal data are well classified by one method and not the other. This suggests the two methods are not redundant and can be complementary. In fact, it is possible to implement an adaptive thresholding mechanism that considers both methods to classify the data.

## VII. CONCLUSION AND PERSPECTIVES

This work studied unsupervised anomaly detection on NSL-KDD with a $\beta$-VAE, by comparing two decision signals based on different principles: reconstruction-based scoring ($\mathcal{L}_{rec}$-classification) and latent space distance-based scoring computed as the mean Euclidean distance to the $k$ nearest
neighbors ($Z_k$-classification). We showed that latent distance can match or surpass reconstruction error depending on $\beta$ and $k$, while the two signals remain complementary for some samples.

The latent-space method enables incremental learning. Because decisions rely on reference embeddings, the model can be updated online by appending new normal and labeled abnormal projections without retraining the $\beta$-VAE. This makes it possible to adapt to evolving operating conditions and to progress from anomaly detection to behavior classification: clusters of latent patterns corresponding to distinct operating modes can be tracked and labeled over time, enabling fine-grained classification of behaviors in addition to binary anomaly flags.

Beyond Euclidean distance, we can also consider a Mahalanobis score in latent space. Rather than sampling $z$ from $q_{\phi}(z|x)$, we simply use the encoder outputs as a deterministic embedding, the mean $\mu(x)$ and evaluate a Mahalanobis distance to the normal reference statistics estimated on $Z_{train}$ [16]. The advantage of Mahalanobis distance over Euclidean distance is that it takes into account the covariance structure of the data, which can be particularly useful in high-dimensional spaces where features may be correlated, as discussed in [12].

Future work will focus on fusing reconstruction-based and latent-based scores via calibrated or learned aggregation; implementing and benchmarking Mahalanobis-based detectors in the latent space; broadening the evaluation to diverse datasets and operating conditions; and leveraging incremental learning to track latent clusters and enable behavior-aware intrusion analysis.

## REFERENCES

[1] M. Tavallaee, E. Bagheri, W. Lu, and A. A. Ghorbani, "A detailed analysis of the KDD CUP 99 data set," in *2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications*, Jul. 2009, pp. 1-6.

[2] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner, "beta-vae: Learning basic visual concepts with a constrained variational framework," in *International conference on learning representations*, 2017.

[3] S. Kullback and R. A. Leibler, "On information and sufficiency," *The Annals of Mathematical Statistics*, vol. 22, no. 1, pp. 79-86, 1951.

[4] D. P. Kingma and M. Welling, "Auto-encoding variational bayes," 2022.

[5] Q. He and J. Wang, "Fault detection using the k-nearest neighbor rule for semiconductor manufacturing processes," *IEEE Transactions on Semiconductor Manufacturing*, vol. 20, no. 4, pp. 345-354, 2007.

[6] H. Song, Z. Jiang, A. Men, and B. Yang, "A Hybrid Semi-Supervised Anomaly Detection Model for High-Dimensional Data," *Computational Intelligence and Neuroscience*, vol. 2017, no. 1, p. 8501683, 2017.

[7] J. Guo, G. Liu, Y. Zuo, and J. Wu, "An Anomaly Detection Framework Based on Autoencoder and Nearest Neighbor," in *2018 15th International Conference on Service Systems and Service Management (ICSSSM)*, Jul. 2018, pp. 1-6.

[8] F. Angiulli, F. Fassetti, and L. Ferragina, "LatentOut: An unsupervised deep anomaly detection approach exploiting latent space distribution," *Machine Learning*, vol. 112, no. 11, pp. 4323-4349, Nov. 2023.

[9] Z. Zhang, T. Jiang, S. Li, and Y. Yang, "Automated feature learning for nonlinear process monitoring – An approach using stacked denoising autoencoder and k-nearest neighbor rule," *Journal of Process Control*, vol. 64, pp. 49-61, Apr. 2018.

[10] R. Corizzo, M. Ceci, and N. Japkowicz, "Anomaly Detection and Repair for Accurate Predictions in Geo-distributed Big Data," *Big Data Research*, vol. 16, pp. 18-35, Jul. 2019.

[11] S. Ramakrishna, Z. Rahiminasab, G. Karsai, A. Easwaran, and A. Dubey, "Efficient Out-of-Distribution Detection Using Latent Space of $\beta$-VAE for Cyber-Physical Systems," *ACM Trans. Cyber-Phys. Syst.*, vol. 6, no. 2, Apr. 2022.

[12] I. Pitsiorlas, G. Arvanitakis, and M. Kountouris, "Trustworthy Intrusion Detection: Confidence Estimation Using Latent Space," *2024 22nd International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt)*, pp. 92-98, 2024.

[13] M. Astrid, M. Z. Zaheer, and S. Lee, "Constricting Normal Latent Space for Anomaly Detection with Normal-only Training Data," in *5th Workshop on practical ML for limited/low resource settings*, 2024.

[14] K. Kamnitsas, D. Castro, L. L. Folgoc, I. Walker, R. Tanno, D. Rueckert, B. Glocker, A. Criminisi, and A. Nori, "Semi-Supervised Learning via Compact Latent Space Clustering," in *Proceedings of the 35th International Conference on Machine Learning*, ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. PMLR, 10-15 Jul 2018, pp. 2459-2468.

[15] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in *3rd International Conference on Learning Representations, ICLR 2015*, San Diego, CA, USA, May 7-9, 2015, Conference Track *Proceedings*, Y. Bengio and Y. LeCun, Eds., 2015. [Online]. Available: http://arxiv.org/abs/1412.6980

[16] G. J. McLachlan, "Mahalanobis distance," *Resonance*, vol. 4, no. 6, pp. 20-26, 1999.