# Retrieval Augmented Deep Anomaly Detection for Tabular Data

Hugo Thimonier, Fabrice Popineau, Arpad Rimmel, Bich-Liên Doan

► To cite this version:

Hugo Thimonier, Fabrice Popineau, Arpad Rimmel, Bich-Liên Doan. Retrieval Augmented Deep Anomaly Detection for Tabular Data. CIKM '24: The 33rd ACM International Conference on Information and Knowledge Management, Oct 2024, Boise ID, United States. pp.2250 - 2259, 10.1145/3627673.3679559. hal-05238746

HAL Id: hal-05238746

https://hal.science/hal-05238746v1

Submitted on 3 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

# Retrieval Augmented Deep Anomaly Detection for Tabular Data

Hugo Thimonier

hugo.thimonier@centralesupelec.fr

Université Paris-Saclay, CNRS, CentraleSupélec,
Laboratoire Interdisciplinaire des Sciences du Numérique
Gif-sur-Yvette, France

Arpad Rimmel

arpad.rimmel@centralesupelec.fr

Université Paris-Saclay, CNRS, CentraleSupélec,
Laboratoire Interdisciplinaire des Sciences du Numérique
Gif-sur-Yvette, France

Fabrice Popineau

fabrice.popineau@centralesupelec.fr

Université Paris-Saclay, CNRS, CentraleSupélec,
Laboratoire Interdisciplinaire des Sciences du Numérique
Gif-sur-Yvette, France

Bich-Liên Doan

bich-lien.doan@centralesupelec.fr

Université Paris-Saclay, CNRS, CentraleSupélec,
Laboratoire Interdisciplinaire des Sciences du Numérique
Gif-sur-Yvette, France

## Abstract

Deep learning for tabular data has garnered increasing attention in recent years, yet employing deep models for structured data remains challenging. While these models excel with unstructured data, their efficacy with structured data has been limited. Recent research has introduced retrieval-augmented models to address this gap, demonstrating promising results in supervised tasks such as classification and regression. In this work, we investigate using retrieval-augmented models for anomaly detection on tabular data. We propose a reconstruction-based approach in which a transformer model learns to reconstruct masked features of normal samples. We test the effectiveness of KNN-based and attention-based modules to select relevant samples to help in the reconstruction process of the target sample. Our experiments on a benchmark of 31 tabular datasets reveal that augmenting this reconstruction-based anomaly detection (AD) method with sample-sample dependencies via retrieval modules significantly boosts performance. The present work supports the idea that retrieval module are useful to augment any deep AD method to enhance anomaly detection on tabular data. Our code to reproduce the experiments is made available on GitHub.

## CCS Concepts

* Security and privacy → Intrusion/anomaly detection and malware mitigation; * Computing methodologies → Semi-supervised learning settings; Neural networks.

## Keywords

Anomaly Detection, Tabular Data, Deep Learning

## ACM Reference Format:

Hugo Thimonier, Fabrice Popineau, Arpad Rimmel, and Bich-Liên Doan. 2024. Retrieval Augmented Deep Anomaly Detection for Tabular Data. In *Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM '24), October 21–25, 2024, Boise, ID, USA*. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3627673.3679559

<figure><img src="image_3.png" /></figure>

This work is licensed under a Creative Commons Attribution
International 4.0 License.

CIKM '24, October 21–25, 2024, Boise, ID, USA
© 2024 Copyright held by the owner/author(s).
ACM ISBN 979-8-4007-0436-9/24/10
https://doi.org/10.1145/3627673.3679559

## 1 Introduction

Semi-supervised anomaly detection (AD) consists in learning to characterize a normal distribution using a dataset only composed of samples belonging to the normal<fnref n="1" /> class, in order to identify in a separate dataset the samples that do not belong to this normal distribution, namely anomalies. This class of algorithms is often used when the imbalance between classes is too severe, causing standard supervised approaches to fail [39]. Examples of such applications are cyber intrusion detection [2], fraud detection on credit card payment [15, 35], or tumor detection on images [37]. On the contrary, unsupervised anomaly detection refers to identifying anomalies in a dataset without using labeled training data. These algorithms aim to discover patterns or structures in the data and flag instances that deviate significantly from these patterns. The application of such an approach usually includes detecting mislabeled samples or removing outliers from a dataset that may hinder a model’s training process.

While deep learning methods have become ubiquitous and are widely used in the industry for various tasks on unstructured data, relying on deep models for tabular data remains challenging. Indeed, Grinsztajn et al. [11] discuss how the inherent characteristics of tabular data make this type of data challenging to handle by standard deep models. Hence, recent research on deep learning for structured data [1, 9, 17, 29, 32] has been oriented towards proposing novel training frameworks, regularization techniques or architectures tailored for tabular data. Similarly, general AD methods appear to struggle with tabular data, while the best-performing AD algorithms on tabular data involve accounting for the particular structure of this data type. For instance, [3, 22, 30, 34] put forward self-supervised anomaly detection algorithms targeted for tabular data that significantly outperform general methods on most tested datasets.

In particular, recent research has emphasized the pivotal role of combining feature-feature and samples-sample dependencies in fostering deep learning model’s performance on tabular data [9, 17, 32]. Following these recent findings, we investigate the benefits of including **external retrieval modules** to leverage sample-sample dependencies to augment existing AD methods. External retrieval modules can be considered instrumental as they **can augment any existing model** that may only rely on feature-feature dependencies.

<footnote n="1">¹The term normal relates to the concept of normality, in opposition to anomaly.</footnote>
In contrast, models that rely on internal retrieval mechanisms are
bound to some inductive biases and cannot be used to learn all
possible tasks that may be relevant for anomaly detection.

Leveraging both types of dependencies is critical to detect all
types of anomalies effectively and can increase consistency across
datasets, as we empirically show in section 5.1. Han et al. [13]
categorize anomalies in tabular data into 4 families of anomalies
which require different types of dependencies to be correctly iden-
tified. First, *dependency anomalies* explicitly refers to samples that
do not follow the dependency structure that *normal* data follow
require feature-feature dependencies to be efficiently identified.
Second, *global anomalies* refer to unusual data points that deviate
significantly from the norm. Relying on both dependencies should
improve a model’s capacity to detect these anomalies. Third, *local
anomalies* that refer to the anomalies that are deviant from their
local neighborhood can only be identified by relying on sample-
sample dependencies. Finally, *clustered anomalies*, also known as
group anomalies, are composed of anomalies that exhibit similar
characteristics. This type of anomaly requires feature-feature de-
pendencies to be identified.

We test the relevance of external modules by employing trans-
formers [36] in a mask-reconstruction framework to construct an
anomaly score as it was proven to offer strong anomaly detection
performance [34]. We implement several external retrieval methods
to augment the vanilla transformer and evaluate the performance
of each approach on an extensive benchmark of tabular datasets.

We empirically show that the tested approaches incorporating
retrieval modules to account for the sample-sample relations out-
perform the vanilla transformer that only attends to feature-feature
dependencies. Furthermore, we propose an empirical experiment
to account for the pertinence of combining dependencies, showing
that detecting some types of anomalies can require a particular
type of dependency.

The present work offers the following contributions:

* We propose an extensive evaluation of retrieval-based meth-
  ods for AD on tabular data.

* We empirically show that augmenting existing AD methods with a retrieval module to leverage sample-sample dependencies can help improve detection performance.

* We compare our approach to existing methods found in the literature and observe that our method obtains competitive performance metrics.

* We provide an explanation as to why combining dependen-
  cies leads to better identification of anomalies in tabular
  data.

## 2 Related works

Ruff et al. [25] discuss how anomaly detection bears several denom-
inations that more or less designate the same class of algorithms:
anomaly detection, novelty detection, and outlier detection. The lit-
erature comprises 4 main classes of anomaly detection algorithms:
density estimation, one-class classification, reconstruction-based,
and self-supervised algorithms.

*Density estimation.* It is often seen as the most direct approach to
detecting anomalies in a dataset. The density estimation approach
consists in estimating the *normal* distribution and flagging low

probability samples under this distribution as an anomaly. Existing
methods include using Copula as the COPOD method proposed
in [18], Local Outlier Factor (LOF) [5], Energy-based models [41]
flow-based models [20].

*Reconstruction-based methods.* Other anomaly detection methods focus on learning to reconstruct samples belonging to the *normal* distribution. In inference, the capacity of the model to reconstruct an unseen sample is used as a measure of anomalousness. The more capable a model is to reconstruct a sample, the more likely the sample is to belong to the *normal* distribution seen in training. Such approach include methods involving autoencoders [6, 16], diffusion models [38, 42], GANs [27] or attention-based models [34].

*One-Class Classification.* One-class classification describes the task of identifying anomalies without directly estimating the *normal* density. This class of algorithm involves discriminative models that directly estimate a decision boundary. In [26, 28, 33], the authors propose algorithms that estimate the support of the *normal* distribution, either in the original data space or in a latent space. During inference, one flags the samples outside the estimated support as anomalies. Other one-class classification methods include tree-based approaches such as isolation forest (IForest) [21], extended isolation forest [14], RRCF [12] and PIDForest [8]. Other methods include approaches to augment existing one-class classification methods with a classifier by generating synthetic anomalies during training, such as DROCC [10].

*Self-Supervised Approaches.* Given the recent successes of self-supervision for many tasks, researchers have also investigated using self-supervised methods for anomaly detection. [3] and [22] propose transformation based anomaly detection methods for tabular data. The former relies on a classifier's capacity to identify which transformation was applied to a sample to measure anomalousness, while the latter relies on a contrastive approach. Similarly, [30] also proposes a contrastive approach to flag anomalies by learning feature-feature relation for normal samples. Parallel to this line of work, [24, 31] have focused on proposing self-supervised approaches for representation learning tailored for anomaly detection.

*Retrieval modules.* Retrieval modules have gained attention in recent years in many fields of machine learning. For instance, [4] introduces a retrieval module to foster the scalability and efficiency of diffusion models. Parallel to that, [19] introduced retrieval for cross-task generalization of large language models, and [7] introduced it to enhance prompt learning. Finally, retrieval methods have been increasingly used to increase the performance of deep models for tabular data. For instance, [17] and [32] introduced internal retrieval modules in deep architecture for supervised tasks on tabular data, while [34] relied on internal retrieval modules for anomaly detection. Finally, [9] investigated using external retrieval modules to augment an MLP for supervised tasks on tabular data.

## 3 Method

### 3.1 Learning Objective

Let $\mathcal{D}_{train} = \{\mathbf{x}_i \in \mathbb{R}^d\}_{i=1}^n$ represent the training set composed
of *n* normal samples with *d* features. The standard approach to
<figure><img src="image_4.png" /><figcaption>Figure 1: Forward pass for sample z, see section 3.4 for more detail on training procedure. In the case of no retrieval module, the prediction for a sample z consists of the upper part of the figure with $\lambda = 0$.</figcaption></figure>

anomaly detection involves learning some function $\phi_{\theta}: \mathbb{R}^d \to \mathcal{Z}$ by minimizing a loss function $\mathcal{L}$. The chosen loss function and the space $\mathcal{Z}$ will vary according to the class of the considered anomaly detection algorithm. Nevertheless, the overall aim of $\mathcal{L}$ is to characterize the distribution of the samples in the training set as precisely as possible. Depending on the chosen AD algorithm, the obtained representation $\phi_{\theta}(\mathbf{x})$ of sample $\mathbf{x}$ can be used directly or indirectly to compute an anomaly score.

Formally, the training objective can be summarized as follows

$$ \min_{\theta \in \Theta} \sum_{\mathbf{x} \in \mathcal{D}_{train}} \mathcal{L}(\mathbf{x}, \phi_{\theta}(\mathbf{x})), \quad (1) $$

where $\mathcal{L}(\mathbf{x}, \phi_{\theta}(\mathbf{x}))$ will vary according to the chosen task. For a reconstruction-based method, $\mathcal{Z}$ can be the original data space and $\mathcal{L}$ can be the squared $l_2$-norm of the difference between the original sample $\mathbf{x}$ and its reconstructed counterpart, $\mathcal{L}(\mathbf{x}, \phi_{\theta}(\mathbf{x})) = \|\mathbf{x} - \phi_{\theta}(\mathbf{x})\|^2$.

Introducing an external retrieval module permits keeping the original objective unchanged while augmenting $\phi_{\theta}$ with sample-sample dependencies through non-parametric mechanisms. The model involves non-parametric relations as it leverages the entire training dataset to make its prediction. Hence, the model can con- jointly attend to feature-feature and sample-sample interactions to optimize its objective.

Formally, instead of minimizing the loss as described in eq. (1), the parameters $\theta$ of the function $\phi_{\theta}$ are optimized to minimize the loss function as follows

$$ \min_{\theta \in \Theta} \sum_{\mathbf{x} \in \mathcal{D}_{train}} \mathcal{L}(\mathbf{x}, \phi_{\theta}(\mathbf{x}; \mathcal{D}_{train})). \quad (2) $$

Nevertheless, not all approaches to AD may benefit from such non-parametric mechanisms. Some pretext tasks involving sample-sample dependencies may lead to degenerate solutions, e.g. approaches based on contrastive learning such as the approaches of [22] or [30]. However, reconstruction-based AD methods appear as a natural class of algorithms that may benefit from these non-parametric mechanisms.

### Mask Reconstruction.
In the mask reconstruction context, we empirically investigate the pertinence of external retrieval modules, as detailed in section 3.2. Our approach includes stochastic masking which consists in masking each entry in a sample vector $\mathbf{x} \in \mathbb{R}^d$ with probability $p_{mask}$ while setting as the objective task the prediction of the masked-out features from the unmasked features. Formally, we sample $\mathbf{m} \in \mathbb{R}^d$ a binary mask vector taking value 1 when the corresponding entry in $\mathbf{x}$ is masked, 0 otherwise. This mask $\mathbf{m}$ is then used to construct $\mathbf{x}^m, \mathbf{x}^o \in \mathbb{R}^d$ representing respectively the masked and unmasked entries of sample $\mathbf{x}$. $\mathbf{x}^m, \mathbf{x}^o$ are obtained as follows,

$$ \begin{aligned} \mathbf{x}^m &= \mathbf{m} \odot \mathbf{x} \\ \mathbf{x}^o &= (\mathbf{1}_d - \mathbf{m}) \odot \mathbf{x}, \end{aligned} \quad (3) $$

where $\mathbf{1}_d$ is the $d$-dimensional unit vector.

A model $\phi_{\theta}: \mathbb{R}^d \times \{0, 1\}^d \to \mathbb{R}^d$ is trained to reconstruct the mask features $\mathbf{x}^m$ from its unmasked counterpart $\mathbf{x}^o$ and the mask vector $\mathbf{m}$. By construction, $\phi_{\theta}(\mathbf{x}^o; \mathbf{m})$ only has non-zero values for corresponding masked features in $\mathbf{m}$.

In the present work, we evaluate the benefit of replacing the traditional reconstruction learning objective defined as

$$ \min_{\theta \in \Theta} \sum_{\mathbf{x} \in \mathcal{D}_{train}} d(\mathbf{x}^m, \phi_{\theta}(\mathbf{x}^o; \mathbf{m})), \quad (4) $$

where $d(.,,.)$ is a distance measure; with its equivalent augmented with a retrieval module as follows

$$ \min_{\theta \in \Theta} \sum_{\mathbf{x} \in \mathcal{D}_{train}} d(\mathbf{x}^m, \phi_{\theta}(\mathbf{x}^o; \mathbf{m}, \mathcal{D}_{train}^o)), \quad (5) $$

where $\mathcal{D}_{train}^o = \{\mathbf{x}_i^o \in \mathbb{R}^d\}_{i=1}^n$. In inference, $\mathcal{D}_{train}^o$ is replaced by $\mathcal{D}_{train}$ in which none of the features of the training samples are masked.

## 3.2 Retrieval methods

Let $z$ denote the sample of interest for which we wish to reconstruct its masked features $\mathbf{z}^m$ given its observed counterpart $\mathbf{z}^o$. Let $C$
<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /><figcaption>Figure 2: For each of the 31 datasets on which models were evaluated, we report the average F1-score over 20 runs for 20 different seeds. We refer readers to Thimonier et al. [34] for details on the obtained metrics and the hyperparameters used for each method. For both figures, the model displayed on the far left is the worst-performing model for the chosen metric, and the one on the far right is the best-performing model. We also highlight the metric of the best-performing model in bold.</figcaption></figure>

denote the candidate samples from the training set from which k *帮手* are to be retrieved, and *H* the retrieved *助手*, *H* ⊆ *C*.

We consider several external retrieval modules that rely on similarity measures to identify relevant samples to augment the encoded representation of the sample of interest *z*. It involves placing a retrieval module after the transformer encoder and before the output layer, as shown in figure 1. We investigate in section 5 the impact of modifying the location of the retrieval module and consider placing it before the encoder as an alternative.

For each method, the retrieval module consists in selecting the top-k elements that maximize a similarity measure S(·, ·) and use a value function to obtain representations of the chosen samples V(·, ·) to be aggregated with sample z.

**KNN-based module.** First, we consider a simple method that identifies the k most relevant samples in *C* using a KNN approach. For- mally, the similarity and value functions are defined as follows

$$ 
\begin{aligned}
S(\mathbf{z}, \mathbf{x}) &= -\|\mathbf{h}_z - \mathbf{h}_x\| \\
\mathcal{V}(\mathbf{z}, \mathbf{x}) &= \mathbf{h}_x,
\end{aligned} 
\quad (6) 
$$

where $\mathbf{h}_x$ and $\mathbf{h}_z$ denote the representations of respectively sample x and z and $||·||$ is the $l_2$-norm.

**Attention-based modules.** Second, we consider attention mechanisms to select *H*. We consider three types of attention inspired by those proposed in [9]. First, the vanilla attention (later referred to as v-attention), where the score and value function used to select the retrieved samples are defined as

$$ 
\begin{aligned}
S(\mathbf{z}, \mathbf{x}) &= W_Q(\mathbf{h}_z)^\top W_K(\mathbf{h}_x) \\
\mathcal{V}(\mathbf{z}, \mathbf{x}) &= W_V(\mathbf{h}_x).
\end{aligned} 
\quad (7) 
$$

where $W_Q$, $W_K$ and $W_V$ are learned parameters. Second, we also consider another type of attention module, later referred to as attention-bsim, which involves replacing the score

function defined in eq. (7) as follows

$$ 
\begin{aligned}
S(\mathbf{z}, \mathbf{x}) &= -\|W_K(\mathbf{h}_z) - W_K(\mathbf{h}_x)\|^2 \\
\mathcal{V}(\mathbf{z}, \mathbf{x}) &= W_V(\mathbf{h}_x).
\end{aligned} 
\quad (8) 
$$

Third, we consider attention-bsim-bval a modification of the value function in eq. (8) as

$$ 
\begin{aligned}
S(\mathbf{z}, \mathbf{x}) &= -\|W_K(\mathbf{h}_z) - W_K(\mathbf{h}_x)\|^2 \\
\mathcal{V}(\mathbf{z}, \mathbf{x}) &= T(W_K(\mathbf{h}_z) - W_K(\mathbf{h}_x)),
\end{aligned} 
\quad (9) 
$$

where $T(·) = \text{LinearWithoutBias} \circ \text{Dropout} \circ \text{ReLU} \circ \text{Linear}(·)$.

**Aggregation.** The retrieval modules necessitate aggregating the obtained retrieved representations $V(\mathbf{z}, \mathbf{x})$ with the representation of the sample of interest $\mathbf{z}$. We aggregate the value of the selected top-k *助手*, to be fed to the final layer

$$ \tilde{\mathbf{h}}_z = (1 - \lambda) \cdot \mathbf{h}_z + \lambda \cdot \frac{1}{k} \sum_{x \in \mathcal{H}} \mathcal{V}(\mathbf{z}, \mathbf{x}). \quad (10) $$

where $\lambda \in [0, 1)$ is a hyperparameter.

## 3.3 Anomaly score

We construct an anomaly score to assess whether a test sample belongs to the normal distribution or should be considered an anomaly. As a reconstruction-based method, our anomaly score is directly obtained from the optimized loss during training: the better the trained model reconstructs a sample, the more likely the sample is to be normal. Indeed, since the model has exclusively seen normal samples during training, it should be less able to reconstruct anomalies correctly since they stem from a different distribution. On the contrary, unseen normal samples should be well reconstructed. We rely on the squared $l_2$-norm of the difference between the reconstructed sample and the original sample for numerical features, while we use the cross-entropy loss function for categorical features.
<table><caption>**Table 1:** Comparison of transformer-based methods. We observe that the external retrieval module attention-bsim significantly improves the AD performance of the vanilla transformer by 4.3% regarding the F1-Score and 1.2% for the AUROC.</caption><thead><tr><th></th><th>Transformer</th><th>+KNN</th><th>+v-att.</th><th>+att-bsim</th><th>+att-bsim-bval</th></tr></thead><tbody><tr><td>F1-Score (↑)</td><td>56.2</td><td>56.1</td><td>55.7</td><td>58.6</td><td>53.9</td></tr><tr><td>AUROC (↑)</td><td>83.4</td><td>83.1</td><td>83.1</td><td>84.4</td><td>82.1</td></tr></tbody></table>

We rely on a mask bank composed of *m* *d*-dimensional masks to construct the anomaly score. We apply each mask to each validation sample and reconstruct the masked features to compute the reconstruction error for each mask. Thus, each validation sample is masked and reconstructed *m* times. The anomaly score is constructed as the average reconstruction error over the *m* masks. To construct the mask bank, we fix the maximum of features to be masked simultaneously *r* and construct $m = \sum_{k=1}^{r} \binom{d}{k}$ masks. Choosing deterministic masks instead of random masks to create the mask bank used for inference is beneficial for two reasons. First, since the model will reconstruct all features at least once, it increases the likelihood of identifying different types of anomalies. Indeed, anomalies that deviate from the normal distribution due to a single feature would only be identified if the corresponding mask hiding this feature would be included. Second, this approach ensures that all samples are masked identically to build the anomaly score. We investigate the impact of constructing a random mask bank instead of a deterministic mask bank in section 5.5. We use the whole unmasked training set<fnref n="2" /> $C = \mathcal{D}^{train}$ to predict the masked features of each sample for each of the *m* masked vectors and construct the anomaly score for a validation sample *z* as

$$ \text{AD-Score}(z; \mathcal{D}^{train}) = \frac{1}{m} \sum_{k=1}^{m} \mathcal{L}(z^{(k)}; \mathcal{D}^{train}), \quad (11) $$

where $\mathcal{L}(z^{(k)}; \mathcal{D}^{train})$ designates the loss for the sample *z* with mask *k*.

## 3.4 Training pipeline

Let $x \in \mathcal{X} \subseteq \mathbb{R}^d$ be a sample with *d* features, which can be either numerical or categorical. Let *e* designate the hidden dimension of the transformer. The training pipeline consists of the following steps:

**Masking.** We sample from a Bernoulli distribution with probability $p_{mask}$ whether each of the *d* features is masked.

$$ \text{mask} = (m_1, \dots, m_d), $$

where $m_j \sim \mathcal{B}(1, p_{mask}) \forall j \in [1, \dots, d]$ and $m_j = 1$ if feature *j* is masked.

**Encoding.** For numerical features, we normalize to obtain 0 mean and unit variance, while we use one-hot encoding for categorical features. At this point, each feature *j* for *j* $\in [1, 2, \dots, d]$ has an $e_j$-dimensional representation, $\text{encoded}(x_j) \in \mathbb{R}^{e_j}$, where $e_j = 1$ for numerical features and for categorical features $e_j$ corresponds to its cardinality. We then mask each feature according to the sampled mask vector and concatenate each feature representation with the

corresponding mask indicator function. Hence, each feature *j* has an ($e_j + 1$)-dimensional representation

$$ ((1 - m_j) \cdot \text{encoded}(x_j), m_j) \in \mathbb{R}^{e_j+1}, $$

where $x_j$ is the *j*-th features of sample *x*.

**In-Embedding.** We pass each of the features encoded representations of sample *x* through learned linear layers Linear($e_j + 1, e$). We also learn *e*-dimensional index and feature-type embeddings as proposed in [17]. Both are added to the embedded representation of sample *x*. The obtained embedded representation is thus of dimension $d \times e$

$$ h_x = (h_x^1, h_x^2, \dots, h_x^d) \in \mathbb{R}^{d \times e}, $$

and $h_x^j \in \mathbb{R}^e$ corresponds to the embedded representation of feature *j* of sample *x*.

**Transformer Encoder.** The embedded representation obtained from the previous step is then passed through a transformer encoder. The output of the transformer $h_x$ is of dimension $d \times e$.

**Out-Embedding.** The output of the transformer, $h_x \in \mathbb{R}^{d \times e}$ is then used to compute an estimation of the original *d*-dimensional vector. To obtain the estimated feature *j*, we take the *j*-th *d*-dimensional representation which is output by the transformer encoder, $h_{x_j} \in \mathbb{R}^d$, and pass it through a linear layer Linear($e, e_j$), where $e_j$ is 1 for numerical features and the cardinality for categorical features.

**External Retrieval Modules.** During training, for a batch $\mathcal{B}$ composed of *b* samples, for each sample $x \in \mathcal{B}$, the entire batch serves as the candidates *C*. In inference, a random subsample of the training set is used as *C*. Both for training and inference, when possible memory-wise, we use as $\mathcal{B}$ and *C* the entire training set. As input, the retrieval module receives a $\mathbb{R}^{d \times e}$ representation for each sample. Operations described in eq. (6), (7), (8) and (9) are performed on the flattened representations of samples $x, h_{\text{flatten}}^x \in \mathbb{R}^{d \times e}$. After selecting $\mathcal{H} \subseteq \mathcal{C}$, each sample is transformed back to its original dimension to allow aggregation as described in eq. (10).

## 4 Experiments

**Datasets.** We experiment on an extensive benchmark of tabular datasets following previous work [30, 34]. The benchmark is comprised of two datasets widely used in the anomaly detection literature, namely Arrhythmia and Thyroid, a second group of datasets, the "Multi-dimensional point datasets", obtained from the Outlier Detection DataSets (ODDS)<fnref n="3" /> containing 28 datasets. We also include three real-world datasets from [13]: fraud, campaign, and backdoor.

<footnote n="2">²For large datasets, we resort to a random subsample of the training set for computational reasons.</footnote>

<footnote n="3">³http://odds.cs.stonybrook.edu/</footnote>
<table><caption>**Table 2:** Comparison of the F1-score (↑) of transformer+attention-bsim across values of k. Here -1 stands for $H = C$. Some values are N/A either because it is not relevant to compute or when there are not enough samples in the training set for the selected value of k.</caption><thead><tr><th>k</th><th>0</th><th>5</th><th>25</th><th>50</th><th>200</th><th>500</th><th>-1</th></tr></thead><tbody><tr><td colspan="8">transformer+attention-bsim</td></tr><tr><td>Abalone</td><td>42.5±7.8</td><td>53.0±6.4</td><td><b>54.9±5.4</b></td><td><b>55.0±5.4</b></td><td>52.0±5.6</td><td>54.0±6.5</td><td>53.0±5.7</td></tr><tr><td>Satellite</td><td>65.6±3.3</td><td>71.5±2.4</td><td>71.3±1.3</td><td>71.2±1.6</td><td>70.8±1.8</td><td>71.2±1.7</td><td><b>71.9±1.5</b></td></tr><tr><td>Lympho</td><td>88.3±7.6</td><td>91.7±8.3</td><td><b>93.3±8.2</b></td><td>91.7±8.3</td><td>N/A</td><td>N/A</td><td>90.0±8.1</td></tr><tr><td>Satimage</td><td>89.0±4.1</td><td>88.8±3.8</td><td>88.4±3.8</td><td>89.4±4.2</td><td>88.8±4.3</td><td>89.1±4.3</td><td><b>93.2±1.7</b></td></tr><tr><td>Thyroid</td><td>55.5±4.8</td><td><b>56.9±5.3</b></td><td>55.9±5.2</td><td>56.3±5.2</td><td>56.4±5.2</td><td>55.9±5.6</td><td>55.8±6.3</td></tr><tr><td>Cardio</td><td>81.0±4.1</td><td>81.2±1.6</td><td><b>81.9±1.4</b></td><td><b>81.9±1.4</b></td><td><b>81.9±1.4</b></td><td><b>81.9±1.4</b></td><td>80.6±2.4</td></tr><tr><td>Ionosphere</td><td>88.1±2.8</td><td>89.4±5.0</td><td>90.2±4.5</td><td>89.8±4.3</td><td>N/A</td><td>N/A</td><td><b>91.7±2.1</b></td></tr><tr><td>mean</td><td>70.3</td><td>76.1</td><td><b>76.6</b></td><td>76.5</td><td>70.0</td><td>70.4</td><td><b>76.6</b></td></tr><tr><td>mean std</td><td>4.9</td><td>4.7</td><td>4.3</td><td>4.3</td><td>3.7</td><td>3.9</td><td>4.0</td></tr></tbody></table>

*Experimental Settings.* Following previous work in the AD literature, [3, 44], we construct the training set with a random subsample of the normal samples representing 50% of the normal samples, we concatenate the 50% remaining with the entire set of anomalies to constitute the validation set. Similarly, we fix the decision threshold for the AD score such that the number of predicted anomalies equals the number of existing anomalies.

To evaluate to which extent sample-sample dependencies are relevant for anomaly detection, we compare models that attend to relations between samples to the vanilla transformer model. We compare the different methods discussed using the F1-score (↑) and AUROC (↑) metrics following the literature. For each dataset, we report an average over 20 runs for 20 different seeds; we report in Table 1 the average F1 and AUROC.

We considered three regimes for the transformer dimensions depending on the dataset size. The transformer encoder comprises 2 or 4 layers with 4 attention heads and hidden dimension $e \in \{8, 16, 32\}$ for smaller to larger datasets. We train the transformer with a mask probability $p_{mask}$ set to 0.25 or 0.15 and rely on the LAMB optimizer [40] with $\beta = (0.9, 0.999)$ and also included a Lookahead [43] wrapper with slow update rate $\alpha = 0.5$ and $k = 6$ steps between updates. We also include a dropout regularization with $p = 0.1$ for attention weights and hidden layers. We ensure that during training, all samples from a batch are not masked simultaneously so that the retrieval module receives encoded representations of unmasked samples as it will in the inference stage. We considered the same transformer architecture and hyperparameters for the same dataset for each considered approach. For external retrieval modules, we chose for simplicity $\lambda = 0.5$ for aggregation as detailed in eq. (10). We study in section 5.3 the effect of varying the value of $\lambda$ on the model's performance. For the KNN module, we set $k = 5$ as the cardinality of $H$ since KNN-based anomaly detection methods [23] with $k = 5$ have shown strong anomaly detection performance [30]. In contrast, for the attention modules, we set $H = C$ and use the attention weights to compute a weighted mean to be aggregated as in eq. (10). We further discuss the choice of $k$ in section 5.2. Finally, depending on the dataset, we trained the model until the loss stopped improving for 50 or 100 consecutive epochs. Each

experiment in the present work can be replicated using the code made available on github<fnref n="4" />.

*Results.* As reported in Table 1, we observe that not all retrieval modules induce a significant boost in anomaly detection performance. We observe that only the transformer augmented by the attention-bsim module performs significantly better than the vanilla transformer. Indeed, augmenting the vanilla transformer with the retrieval module detailed in eq. (8) allows to increase the average F1-score by 4.3% and AUROC by 1.2%. This result is all the more interesting since it contradicts the results obtained for the supervised classification and regression tasks investigated in previous work [9] where the module that obtains the best performance is attention-bsim-bval. However, let us mention that the attention-bsim-bval module involved in our work is not identical to the one put forward in [9] as it does not involve any label.

For completeness, we compare the different architectures proposed in the present work to existing methods in the literature. We rely on the experiments conducted in [30, 34] for the metrics of the competing methods. We display in figure 2 the comparison to existing methods. We compare our methods to recent deep methods, namely GOAD [3], DROCC [10], NeuTraL-AD [22], the contrastive approach proposed in [30] and NPT-AD [34]. We also compare to classical non-deep methods such as Isolation Forest [21], KNN [23], RRCF [12], COPOD [18] and PIDForest [8]. We refer the reader to [34] for the detail on the F1-score per dataset.

# 5 Discussion

## 5.1 Why combine dependencies?

To account for the fact that the retrieval-augmented transformer outperforms the vanilla transformer, we hypothesize that different types of anomalies require different dependencies to identify them. In this section, we provide a simple example to demonstrate this statement. Consider a simple three dimensional data space, $x = (x_1, x_2, x_3) \in \mathbb{R}^3$, in which the relation between the features of

<footnote n="4">⁴https://github.com/hugothimonier/Retrieval-Augmented-Deep-Anomaly-Detection-for-Tabular-Data</footnote>
<table><caption>**Table 3: Share (%) of each class correctly identified (↑). Average over 5 data splits. The Table should be read as follows: On average, the transformer correctly classified 78.5% of type 1 anomalies as anomalies.**</caption><thead><tr><th></th><th>Normal</th><th>Anomalies (type 1)</th><th>Anomalies (type 2)</th></tr></thead><tbody><tr><th>Mask-KNN</th><td>93.0% (±0.4)</td><td>100.0%(±0.0)</td><td>77.3%(±4.4)</td></tr><tr><th>Transformer</th><td>91.4% (±1.4)</td><td>78.5%(±1.0)</td><td>100.0%(±0.0)</td></tr><tr><th>+att-bsim</th><td>94.6% (±0.5)</td><td>88.0%(±0.8)</td><td>100.0%(±0.0)</td></tr></tbody></table>

normal sample are defined as follows,

$$ 
\begin{aligned}
x_2 &= \alpha_1 + \beta_1 x_1 + \varepsilon \\
x_3 &= \alpha_2 + \beta_2 x_2^2 + \varepsilon,
\end{aligned} 
\quad (12) $$

where $\varepsilon$ is some white noise and $(\alpha_1, \alpha_2, \beta_1, \beta_2) \in \mathbb{R}^2$ are scalars.

Let us consider two types of anomalies as shown in Figure 3. First anomalies of type 1, in which the relations between features are identical to the ones given in eq. (12) but in a different subspace. Now consider type 2 anomalies, for which the values of the generating feature $x_1$ are in the same subspace as normal samples, the relation between $x_1$ and $x_2$ is the same, but the parameters of the relation between $x_2$ and $x_3$ differ. Type 1 (resp. 2) anomalies are akin to local anomalies (resp. dependency anomalies) discussed in [13].

To test our hypothesis, we compare the retrieval augmented transformer to the vanilla transformer and Mask-KNN, a reconstruction-based technique introduced in [34], that relies on KNN imputation to reconstruct masked features. Mask-KNN (resp. the transformer) can be considered approximately equivalent to the retrieval augmented transformer without considering the feature-feature dependencies (resp. the sample-sample dependencies).

In the present framework, models only leveraging inter-feature relations, such as the vanilla transformer, may have limited capacities to identify anomalies if they satisfy the same relations as given in eq. (12) but in a different subspace, i.e., anomalies of type 1. Similarly, a model that only relies on inter-sample relations, e.g., Mask-KNN, would struggle to correctly identify anomalies of type 2 as they lie in a subspace close to normal samples.

**Synthetic Dataset.** We effectively test this hypothesis by constructing a synthetic three-dimensional dataset where the features of normal samples satisfy the relations described in eq. (12). We also construct two types of anomalies where type 1 anomalies follow the same inter-feature relation as normal samples but with values in a non-overlapping interval as those of the normal class. Type 2 anomalies are constructed to be close to the normal population but with inter-feature relation differing from eq. (12). This synthetic dataset is displayed in figure 3. The normal population comprises 1000 samples, and we generate 200 anomalies for each type. We use half of the normal samples to train models and use the rest merged with the anomalies as the validation set. We compare the capacity of Mask-KNN, the vanilla transformer, and the retrieval-augmented transformer to identify anomalies and display the obtained results in Table 3. We consider the same mask bank, $\{(1, 0, 0), (0, 1, 0), (0, 0, 1)\}$, for all three approaches. We use

<figure><img src="image_7.png" /><figcaption>**Figure 3: Anomalies of type 1 (●) require inter-sample dependencies to be correctly identified with high probability. Anomalies of type 2 (●) on the other hand require inter-feature dependencies to be correctly identified.**</figcaption></figure>

the same strategy for selecting the decision threshold as detailed in section 4.

**Results.** We observe in Table 3 that the vanilla transformer struggles to detect type 1 anomalies in comparison with other methods, as they explicitly require inter-sample dependencies to be identified. Nevertheless, it correctly identifies 91.4% of normal samples on average and perfectly detects type 2 anomalies. Conversely, Mask-KNN obtains significantly lower performance than competing methods for type 2 anomalies while correctly identifying type 1 anomalies and normal samples. Notice how both methods cannot correctly identify anomalies from one of the two anomaly types despite using a perfectly separable dataset. On the contrary, the retrieval augmented transformer can better identify both types of anomalies as it can leverage inter-sample and inter-feature dependencies. This experiment provides information as to why combining dependencies for anomaly detection is relevant: it allows the detection of most anomaly types while other approaches are confined to some anomaly categories.

## 5.2 Analysis of the effect of the number of helpers *k* on the performance

Note that we randomly selected a subset of the dataset from the benchmark for the ablation studies conducted hereafter. We use this subset of datasets in each experiment for computational reasons and to avoid cherry-picking the hyperparameters.

We investigate the impact of varying *k*, the number of helpers, for both the transformer augmented by attention-bsim and the KNN module since they are the two best-performing retrieval-augmented models. We keep hyperparameters constant and only make *k* vary between runs. We report in Table 2 the obtained results for both architectures for values of *k* $\in \{0, 5, 25, 50, 200, 500, -1\}$, where -1 implies that $H = C$.
<table><caption>**Table 4:** Comparison of the performance of the transformer + attention-bsim model for different retrieval module architecture based on the F1-score (↑).</caption><thead><tr><th>Retrieval<br>Aggregation</th><th>post-enc<br>post-enc</th><th>post-emb<br>post-enc</th><th>post-emb<br>post-emb</th></tr></thead><tbody><tr><td>Abalone</td><td>53.0±5.7</td><td>47.0±7.6</td><td>30.5±14</td></tr><tr><td>Satellite</td><td>71.9±1.5</td><td>60.5±0.9</td><td>65.1±2.3</td></tr><tr><td>Lympho</td><td>90.0±8.1</td><td>91.6±8.3</td><td>84.2±11.1</td></tr><tr><td>Satimage</td><td>93.2±1.7</td><td>50.8±17.0</td><td>65.9±15.3</td></tr><tr><td>Thyroid</td><td>55.8±6.3</td><td>55.2±5.9</td><td>58.1±5.0</td></tr><tr><td>Ionosphere</td><td>91.7±2.1</td><td>88.2±1.4</td><td>91.3±2.0</td></tr><tr><td>mean</td><td>75.9</td><td>65.6</td><td>65.9</td></tr><tr><td>mean std</td><td>4.2</td><td>6.9</td><td>8.3</td></tr></tbody></table>

A noticeable trend is that both architectures obtain the best results with moderate numbers of helper, i.e. $k \in \{25, 50\}$, and have worse performance for smaller values of $k$. This observation suggests that performance displayed in table 1 could be improved for optimized values of $k$.

## 5.3 Analysis of the effect of the value of $\lambda$

We analyze the performance variation for the best-performing external retrieval module, namely transformer+attention-bsim, for varying values of $\lambda$. We compute the average F1-score over 10 runs for each value of $\lambda$ in $\{0.1, 0.2, \dots, 0.7\}$ while keeping the same hyperparameters. We report the results in Table 5.

We observe a slight variation of the average metric across the datasets when setting lambda values from 0.1 to 0.7. The maximum value is obtained for 0.5, but the difference with other values of $\lambda$ is non-significant. Nevertheless, we observe significant differences between the obtained results for isolated datasets for different values of $\lambda$. This observation supports the idea that an optimal value of $\lambda$ exists for each dataset, which may differ between datasets.

## 5.4 Location of the retrieval module

We investigate the impact of the retrieval module's location on the retrieval-augmented models' performance. To do so, we focus on the transformer+attention-bsim model since it has shown to be the best-performing retrieval-based method. We compare three different architectures:

* (post-enc, post-enc) for post-encoder location and post-encoder aggregation: the architecture detailed in figure 1,

* (post-emb, post-enc), the architecture in which the retrieval module is located after the embedding layer, but the aggregation is still located after the encoder,

* (post-emb, post-emb) the architecture where both retrieval and aggregation are located after the embedding layer.

We do not report the results for the cardio dataset since it failed to converge for the (post-emb, post-enc) and (post-emb, post-emb) architectures and output NaN values in inference. We used the same hyperparameters for all three architectures. For (post-emb, post-enc) and (post-emb, post-emb) architectures we report the average over 10 runs. We display the results in Table 4.

<table><caption>**Table 5:** Comparison of transformer+attention-bsim across values of $\lambda$ based on the F1-Score (↑).</caption><thead><tr><th>λ</th><th>0</th><th>0.1</th><th>0.2</th><th>0.3</th><th>0.4</th><th>0.5</th><th>0.6</th><th>0.7</th></tr></thead><tbody><tr><td>Abalone</td><td>42.5</td><td>44.3</td><td>47</td><td>47.3</td><td>46.5</td><td>53</td><td>46.5</td><td>45.8</td></tr><tr><td>Satellite</td><td>65.6</td><td>72</td><td>72</td><td>72.5</td><td>73</td><td>71.9</td><td>73.5</td><td>73.4</td></tr><tr><td>Lympho</td><td>88.3</td><td>90.8</td><td>90.8</td><td>91.7</td><td>92.5</td><td>90</td><td>91.7</td><td>90</td></tr><tr><td>Satim.</td><td>89</td><td>94.5</td><td>94.6</td><td>94.1</td><td>94.1</td><td>93.2</td><td>93.8</td><td>94.1</td></tr><tr><td>Thyroid</td><td>55.5</td><td>57.2</td><td>57.4</td><td>56.6</td><td>56.6</td><td>55.8</td><td>57.1</td><td>57.3</td></tr><tr><td>Cardio</td><td>68.8</td><td>80.9</td><td>80.6</td><td>80.9</td><td>80.9</td><td>80.9</td><td>80.9</td><td>80.7</td></tr><tr><td>Ionosp.</td><td>88.1</td><td>90.3</td><td>92.1</td><td>92.2</td><td>91.2</td><td>91.7</td><td>92.3</td><td>91.7</td></tr><tr><td>mean</td><td>71.1</td><td>75.7</td><td>76.4</td><td>76.5</td><td>76.4</td><td>76.6</td><td>76.5</td><td>76.1</td></tr></tbody></table>

We observe that the (post-enc, post-enc) architecture obtains the highest mean and lowest mean standard deviation over the tested datasets by a sizable margin. The transformer encoder's expressiveness allows for better representations of a data sample than the embedding layer and may account for such results. Indeed, this shows that the retrieval modules that receive the embedded representation as inputs are less able to select the relevant sample to foster mask reconstruction and anomaly detection performance. Moreover, since the encoder and retrieval modules are trained jointly, the retrieval module can help the encoder converge to a state that favors sample representations that allow relevant clusters to be formed.

## 5.5 Random mask bank

We also investigate the impact of constructing a random mask bank for inference instead of selecting a deterministic mask bank as discussed in section 3.3. We construct for inference a random mask bank composed of the same number of masks as for the deterministic mask bank and use the same probability $p_{mask}$ as used to train the model. We compare the performance of the transformer model+attention-bsim based on the F1-score for the two set-ups and display the results in Table 6. We observe that the deterministic mask bank detects anomalies better on most tested datasets. When computing the anomaly score with the deterministic mask bank, the model obtains an average F1-score of 76.6 over the 7 datasets, while with the random mask bank, the model obtains 65.4. Moreover, as expected, we also observed a significantly larger standard deviation between runs. We might expect the standard deviation to decrease as the number of masks increases, which would induce significant computational overhead.

# 6 Limitations and Conclusion

*Limitations.* As with most non-parametric models that leverage the training set in inference, our retrieval-augmented models display a higher complexity than parametric approaches. These approaches can scale well for datasets with a reasonable number of features $d$; however, for large values of $d$, these models incur a high memory cost.

*Conclusion.* In this work, we have proposed an extensive investigation into external retrieval to augment reconstruction-based
<table><caption>**Table 6:** Comparison of the performance of the transformer + attention-bsim model based on the F1-Score (↑) for two mask bank set-ups. The same model was used for inference in both set-ups. We report an average over 10 different splits of the data.</caption><thead><tr><th>Mask bank</th><th>random</th><th>deterministic</th></tr></thead><tbody><tr><td>Abalone</td><td>43.0±14.9</td><td><strong>53.0±5.7</strong></td></tr><tr><td>Satellite</td><td>58.4±5.5</td><td><strong>71.9±1.5</strong></td></tr><tr><td>Lympho</td><td>86.7±8.5</td><td><strong>90.0±8.1</strong></td></tr><tr><td>Satimage</td><td>47.7±20.5</td><td><strong>93.2±1.7</strong></td></tr><tr><td>Thyroid</td><td><strong>55.6±6.1</strong></td><td><strong>55.8±6.3</strong></td></tr><tr><td>Cardio</td><td><strong>81.3±1.6</strong></td><td><strong>80.6±2.4</strong></td></tr><tr><td>Ionosphere</td><td>85.2±4.3</td><td><strong>91.7±2.1</strong></td></tr><tr><td>mean</td><td>65.4</td><td><strong>76.6</strong></td></tr><tr><td>mean std</td><td>8.8</td><td><strong>4.0</strong></td></tr></tbody></table>

anomaly detection methods for tabular data. We have shown that augmenting existing AD methods using attention-based retrieval modules can help foster performance by allowing the model to attend to sample-sample dependencies. Indeed, our experiments involving an extensive benchmark of tabular datasets demonstrate the effectiveness of retrieval-based approaches since the architecture involving the attention-bsim module surpasses the vanilla transformer by a significant margin. We also provide a first explanation as to why combining both types of dependencies can be critical to obtaining consistent performance across datasets: different types of anomalies require different types of dependencies to be efficiently detected.

*Future Work.* Overall, our work showed that the best-performing attention-based retrieval mechanism relies on other forms of attention than vanilla attention. Parallel to that, models like NPT-AD have shown strong performance for anomaly detection on tabular data and rely on standard multi-head self-attention through the Attention Between Datapoints (ABD) mechanism, close to the v-attention module, to leverage inter-sample relations. Our findings may invite further research on modifying the ABD mechanism involved in NPTs to improve their AD performance. Finally, the use of external retrieval modules proved effective for the task of anomaly detection using mask reconstruction. The proposed external retrieval module could also be easily added to existing deep-AD methods to test whether they may prove relevant for other pretext tasks for anomaly detection on tabular data.

## Acknowledgment

This work was granted access to the HPC resources of IDRIS under the allocation 2023-101424 made by GENCI. This research publication is supported by the Chair "Artificial intelligence applied to credit card fraud detection and automated trading" led by Centrale-Supelec and sponsored by the LUSIS company.

## References

[1] Sercan Ö. Arik and Tomas Pfister. 2021. TabNet: Attentive Interpretable Tabular Learning. In *Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence*,

AAAI 2021, Virtual Event, February 2-9, 2021. AAAI Press, 6679–6687. https://doi.org/10.1609/aaai.v35i8.16826

[2] Rachid Ben Said, Zakaria Sabir, and Iman Askertz. 2023. CNN-BiLSTM: A Hybrid Deep Learning Approach for Network Intrusion Detection System in Software-Defined Networking With Hybrid Feature Selection. *IEEE Access* 11 (2023), 138732–138747. https://doi.org/10.1109/AACCESS.2023.3340142

[3] Liron Bergman and Yedid Hoshen. 2020. Classification-Based Anomaly Detection for General Data. In *International Conference on Learning Representations*. https://openreview.net/forum?id=H1LK_Btvs

[4] Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer. 2022. Retrieval-Augmented Diffusion Models. In *Advances in Neural Information Processing Systems*, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 15309–15324. https://proceedings.neurips.cc/paper_files/paper/2022/file/62868cc2fc1eb5cdf321d05b4b88510c-Paper-Conference.pdf

[5] Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng, and Jörg Sander. 2000. LOF: Identifying Density-Based Local Outliers. *SIGMOD Rec.* 29, 2 (may 2000), 93–104. https://doi.org/10.1145/335191.335388

[6] Xiaoran Chen and Ender Konukoglu. 2018. Unsupervised Detection of Lesions in Brain MRI using constrained adversarial auto-encoders. In *Medical Imaging with Deep Learning*. https://openreview.net/forum?id=H1nGLZ2oG

[7] Xiang Chen, Lei Li, Ningyu Zhang, Xiaozhuan Liang, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022. Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learning. In *Advances in Neural Information Processing Systems*, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 23908–23922. https://proceedings.neurips.cc/paper_files/paper/2022/file/97011c648eda678424f9292dadeae72e-Paper-Conference.pdf

[8] Parikshit Gopalan, Vatsal Sharan, and Udi Wieder. 2019. PIDForest: Anomaly Detection and Certification via Partial Identification. In *Neural Information Processing Systems*. https://api semanticscholar org/ CorpusID:202766416

[9] Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. 2023. TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023. arXiv:2307.14338 [cs.LG]

[10] Sachin Goyal, Aditi Raghunathan, Moksh Jain, Harsha Vardhan Simhadri, and Prateek Jain. 2020. DROCC: Deep Robust One-Class Classification. In *Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119)*, Hal Daum III and Aarti Singh (Eds.). PMLR, 3711–3721. https://proceedings.mlr press/v119/goyal20c.html

[11] Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. 2022. Why do tree-based models still outperform deep learning on typical tabular data?. In *Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track*. https://openreview.net/forum?id=Fp7_phQszn

[12] Sudipto Guha, Nina Mishra, Gourav Roy, and Okke Schrijvers. 2016. Robust Random Cut Forest Based Anomaly Detection on Streams. In *International Conference on Machine Learning*.

[13] Songqiao Han, Xiyang Hu, Hailiang Huang, Minqi Jiang, and Yue Zhao. 2022. ADBench: Anomaly Detection Benchmark. In *Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track*. https://openreview.net/forum?id=foA_SFQ9zo0

[14] Sahand Hariri, Matias Carrasco Kind, and Robert J. Brunner. 2021. Extended Isolation Forest. *IEEE Transactions on Knowledge and Data Engineering* 33, 4 (2021), 1479–1489. https://doi.org/10.1109/TKDE.2019.2947676

[15] Waleed Hilal, S. Andrew Gadsden, and John Yawney. 2022. Financial Fraud: A Review of Anomaly Detection Techniques and Recent Advances. *Expert Systems with Applications* 193 (2022), 116429. https://doi.org/10.1016/j.eswa.2021.116429

[16] Ki Hyun Kim, Sangwoo Shim, Yongsub Lim, Jongseob Jeon, Jeongwoo Choi, Byungchan Kim, and Andre S. Yoon. 2020. RaPP: Novelty Detection with Reconstruction along Projection Pathway. In *International Conference on Learning Representations*.

[17] Jannik Kossen, Neil Band, Clare Lyle, Aidan Gomez, Tom Rainforth, and Yarin Gal. 2021. Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning. In *Advances in Neural Information Processing Systems*, A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (Eds.). https://openreview.net/forum?id=WRXzOa2z5T

[18] Zheng Li, Yue Zhao, Nicola Botta, Cezar Ionescu, and Xiyang Hu. 2020. COPOD: Copula-Based Outlier Detection. In *2020 IEEE International Conference on Data Mining (ICDM)*. IEEE. https://doi.org/10.1109/icdm50108.2020.00135

[19] Bill Yuchen Lin, Kangmin Tan, Chris Miller, Beiwen Tian, and Xiang Ren. 2022. Unsupervised Cross-Task Generalization via Retrieval Augmentation. In *Advances in Neural Information Processing Systems*, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 22003–22017. https://proceedings.neurips.cc/paper_files/paper/2022/file/8a0d3ae989a382ce6e50312bc35bf7e1-Paper-Conference.pdf

[20] Boyang Liu, Pang-Ning Tan, and Jiayu Zhou. 2022. Unsupervised Anomaly Detection by Robust Density Estimation. *Proceedings of the AAAI Conference on Artificial Intelligence* 36, 4 (Jun. 2022), 4101–4108. https://doi.org/10.1609/aaai.v36i4.20328
[21] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. 2008. Isolation Forest. In *2008 Eighth IEEE International Conference on Data Mining*. 413–422. https://doi.org/10.1109/ICDM.2008.17

[22] Chen Qiu, Timo Pfoermer, Marius Kloft, Stephan Mandt, and Maja Rudolph. 2021. Neural Transformation Learning for Deep Anomaly Detection Beyond Images. In *Proceedings of the 38th International Conference on Machine Learning*, ICML 2021, 18–24 July 2021, Virtual Event (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 8703–8714. http://proceedings.mlr.press/v139/qiu21a.html

[23] Sridhar Ramaswamy, Rajeev Rastogi, and Kyuseok Shim. 2000. Efficient Algorithms for Mining Outliers from Large Data Sets. In *Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data*, May 16–18, 2000, Dallas, Texas, USA, Weidong Chen, Jeffrey F. Naughton, and Philip A. Bernstein (Eds.). ACM, 427–438. https://doi.org/10.1145/342009.335437

[24] Tal Reiss and Yedid Hoshen. 2023. Mean-Shifted Contrastive Loss for Anomaly Detection. In *Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence (AAAI’23/IAAI’23/EAAI’23)*. AAAI Press, Article 240, 8 pages. https://doi.org/10.1609/aaai.v37i2.25309

[25] Lukas Ruff, Jacob R. Kauffmann, Robert A. Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G. Dietterich, and Klaus-Robert Müller. 2021. A Unifying Review of Deep and Shallow Anomaly Detection. *Proc. IEEE* 109, 5 (May 2021), 756–795. https://doi.org/10/gjmk3g arXiv:2009.11732

[26] Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoai Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. 2018. Deep One-Class Classification. In *Proceedings of the 35th International Conference on Machine Learning* (Proceedings of Machine Learning Research, Vol. 80). PMLR, 4393–4402. http://proceedings.mlr.press/v80/ruff18a.html

[27] Thomas Schlegl, Philipp Seeböck, Sebastian M. Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. 2017. Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery. In *Information Processing in Medical Imaging*, Marc Niethammer, Martin Styner, Stephen Aylward, Hongtu Zhu, Ipek Oguz, Pew-Thian Yap, and Dinggang Shen (Eds.). Springer International Publishing, Cham, 146–157.

[28] Bernhard Schölkopf, Robert Williamson, Alex Smola, John Shawe-Taylor, and John Platt. 1999. Support Vector Method for Novelty Detection. In *Proceedings of the 12th International Conference on Neural Information Processing Systems* (Denver, CO) (NIPS’99). MIT Press, Cambridge, MA, USA, 582–588.

[29] Ira Shavitt and Eran Segal. 2018. Regularization Learning Networks: Deep Learning for Tabular Datasets. In *Advances in Neural Information Processing Systems*, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2018/file/500e75a036dc2d7d2fec5da1b71d36cc-Paper.pdf

[30] Tom Shenkar and Lior Wolf. 2022. Anomaly Detection for Tabular Data with Internal Contrastive Learning. In *International Conference on Learning Representations*.

[31] Kihyuk Sohn, Chun-Liang Li, Jinsung Yoon, Minho Jin, and Tomas Pfister. 2021. Learning and Evaluating Representations for Deep One-Class Classification. In *International Conference on Learning Representations*. https://openreview.net/forum?id=HCSgyPUfeDj

[32] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein. 2021. SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training. CoRR abs/2106.01342 (2021).

arXiv:2106.01342 https://arxiv.org/abs/2106.01342

[33] David Tax and Robert Duin. 2004. Support Vector Data Description. *Machine Learning* 54 (01 2004), 45–66. https://doi.org/10.1023/B:MACH.0000008084.60811.49

[34] Hugo Thimonier, Fabrice Popineau, Arpad Rimmel, and Bich-Liën Doan. 2024. Beyond Individual Input for Deep Anomaly Detection on Tabular Data. In *Proceedings of the 41st International Conference on Machine Learning* (Proceedings of Machine Learning Research, Vol. 235), Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (Eds.). PMLR, 48097–48123. https://proceedings.mlr.press/v235/thimonier24a.html

[35] Hugo Thimonier, Fabrice Popineau, Arpad Rimmel, Bich-Liën Doan, and Fabrice Daniel. 2023. Comparative Evaluation of Anomaly Detection Methods for Fraud Detection in Online Credit Card Payments. arXiv:2312.13896 [cs.LG]

[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In *Advances in Neural Information Processing Systems*, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

[37] Qi Wei, Yinhao Ren, Rui Hou, Bibo Shi, Joseph Y. Lo, and Lawrence Carin. 2018. Anomaly detection for medical images based on a one-class classification. In *Medical Imaging 2018: Computer-Aided Diagnosis*, Nicholas Petrick and Kensaku Mori (Eds.), Vol. 10575. International Society for Optics and Photonics, SPIE, 105751M. https://doi.org/10.1117/12.2293408

[38] Julia Wolleb, Florentin Bieder, Robin Sandkühler, and Philippe C. Cattin. 2022. Diffusion Models for Medical Anomaly Detection. In *Medical Image Computing and Computer Assisted Intervention – MICCAI 2022*, Linwei Wang, Qi Dou, P. Thomas Fletcher, Stefanie Speidel, and Shuo Li (Eds.). Springer Nature Switzerland, Cham, 35–45.

[39] Sun Yanmin, Andrew Wong, and Mohamed S. Kamel. 2011. Classification of imbalanced data: a review. *International Journal of Pattern Recognition and Artificial Intelligence* 23 (11 2011), 687–719. https://doi.org/10.1142/S0218001409007326

[40] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2020. Large Batch Optimization for Deep Learning: Training BERT in 76 minutes. In *International Conference on Learning Representations*. https://openreview.net/forum?id=Syx4wnEtvH

[41] Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang. 2016. Deep Structured Energy Based Models for Anomaly Detection. In *Proceedings of The 33rd International Conference on Machine Learning* (Proceedings of Machine Learning Research, Vol. 48), Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, New York, USA, 1100–1109. https://proceedings.mlr.press/v48/zhai16.html

[42] Hui Zhang, Zheng Wang, Zuxuan Wu, and Yu-Gang Jiang. 2023. DiffusionAD: Norm-guided One-step Denoising Diffusion for Anomaly Detection. arXiv:2303.08730 [cs.CV]

[43] Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton. 2019. Lookahead Optimizer: k steps forward, 1 step back. In *Advances in Neural Information Processing Systems*, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2019/file/90fd4f88f588ae64038134f1eeaa023f-Paper.pdf

[44] Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. 2018. Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection. In *International Conference on Learning Representations*.