# A Robust Autoencoder Ensemble-Based Approach for Anomaly Detection in Text

Jeremie Pantin, Christophe Marsala

<figure><img src="image_1.png" /></figure>

## To cite this version:

Jeremie Pantin, Christophe Marsala. A Robust Autoencoder Ensemble-Based Approach for Anomaly Detection in Text. International Conference on Knowledge-Based and Intelligent Information & Engineering Systems, Sep 2025, Osaka, Japan. pp.1986-1995, 10.1016/j.procs.2025.09.319. hal-05482115

HAL Id: hal-05482115

https://hal.science/hal-05482115v1

Submitted on 28 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
29th International Conference on Knowledge-Based and Intelligent Information & Engineering Systems (KES 2025)

# A Robust Autoencoder Ensemble-Based Approach for Anomaly Detection in Text

Jeremie Pantin<sup>a,*</sup>, Christophe Marsala<sup>b</sup>

<sup>a</sup>Université Caen Normandie, ENSICAEN, CNRS, Normandie Univ, GREYC UMR 6072, F-14000, France

<sup>b</sup>Sorbonne Université, CNRS, LIP6, F-75005 Paris, France

## Abstract

Despite the maturity of anomaly detection in structured and image data, anomaly detection in text remains surprisingly underexplored. Existing methods struggle with two key challenges: manifold collapse in high-dimensional language embeddings, and semantic entanglement that obscures contextual anomalies. We address both with the Robust Local AutoEncoder (RLAE), which combines robust subspace recovery with local geometry preservation. It allows to learn disentangled and anomaly-sensitive representations. We extend this to RoSAE, an ensemble of randomly pruned RLAEs that enhances robustness through architectural diversity. We introduce TAC which is the first benchmark framework to distinguish anomaly types in text, using topic hierarchies to separate independent from contextual anomalies for fair and reproducible evaluation. TAC resolves inconsistencies in prior benchmarks. RoSAE consistently outperforms state-of-the-art baselines across eight diverse corpora (6 more than state-of-the-art approaches), particularly under challenging contextual settings. Our results highlight the importance of both local structure and ensemble diversity in textual anomaly detection, and position RoSAE as a robust, scalable, and efficient method for tackling both independent and contextual anomalies.

© 2025 The Authors. Published by Elsevier B.V.

This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0)

Peer-review under responsibility of the scientific committee of the KES International.

**Keywords:** Autoencoder; Anomaly detection; Natural language processing; Unsupervised learning; Text mining

## 1. Introduction

Anomaly Detection (AD) [7, 28] and Outlier Detection (OD) [1, 35] are tasks aiming to estimate whether an observation is normal or not. These tasks have been recently applied to text to highlight particular or abnormal document in a corpus. In this context, it is still a flourishing and promising field, and nowadays the most popular approaches perform one-class classification (OCC) to discern outliers (anomalies) from non-outliers (inliers) [13, 16, 19]. Also, self-supervised approaches are often preferred [11, 24, 29]. It should be noted that self-supervised approaches are

<footnote>* Corresponding author.
E-mail address: jeremie.pantin@unicaen.fr</footnote>
<figure><img src="image_3.png" /><figcaption>Fig. 1: Examples of different kinds of anomalies.</figcaption></figure>

not limited to text, and applications with large language model for vision have also been proposed [38]. On the other hand, ensemble-based approaches are not present in the literature, despite their great success on different types of data [6, 9]. Recent approaches introduce the use of Multi-Head Self-Attention [11, 24, 29] to learn a representation from pre-trained language models as BERT [12, 26].

An underlying problem of AD lies in the identification of different kinds of anomalies. To illustrate, we present in Fig. 1 the case of a corpus with 3 highlighted topics: *Medical*, *Culture* and *Politic*. Under each topic, textual documents are represented with coloured rectangles: gray and green coloured are inliers and red coloured are anomalies. Instances of such document are shown beneath (abstract of articles from sources like Reuters, BBC, ...). Two types of anomaly are presented: independent (cultural-related document in medical feed) and contextual (political in culture feed).

These different kinds of anomalies are usually not considered in recent works when building a textual AD model. While independent, contextual and collective anomalies have been identified for other modalities [1, 7], it is not the case for text: text anomalies are only considered as documents from one topic appearing in the wrong topic subset. However anomalies can have multiple forms, and independent anomalies are easier to find than contextual.

In this paper we introduce a novel ensemble-based approach that performs contextual anomaly detection. Compared to the most recent approaches, we first focus on the identification of two types of anomalies, independent and contextual, and we introduce the Textual Anomaly Contamination (TAC), a novel experimental setup. Secondly, our main contribution lies in the introduction of a new approach performing anomaly detection based on an ensemble of robust autoencoders: the Robust Subspace Local Recovery AutoEncoder (RLAE). This autoencoder is confronted with contaminated corpora (trained on both inliers and anomalies) and discriminates inliers from anomalies under the assumption that one distribution is highly contaminated. Next, our Robust Subspace Local Recovery Autoencoder Ensemble (RoSAE) represents inliers in a low-dimensional subspace and learns several manifolds.

## 2. Related Works

As a background knowledge, a *corpus* consists of individual *documents*, each associated with one or more *topics* that reflect its semantic content. A *topic* denotes the main subject of a document [4], while *subtopics* capture secondary themes. For example, in the Reuters press releases, a document tagged with the topic *Sports* may include subtopics such as *Football*, *Tennis*, or broader contextual elements like *World Cup* or *Finance*.

### 2.1. AutoEncoder

AutoEncoders (AE) have been widely used for anomaly or outlier detection with high-dimensional data [9, 17] and have been successful with other kinds of data [3, 10, 19]. We can find different approaches relying on them for building ensemble methods [6, 9]. In the context of textual data, the risk of using autoencoders with language models leads to the apparition of degenerate solution in the learning step. Thus, robust properties are needed for handling
<figure><img src="image_4.png" /><figcaption>Fig. 2: Overview of our contributions, including contamination setup (left part) and our proposed approach RoSAE (right). For RoSAE, *n* RLAE models are trained with different pruned neurons. The anomaly score is then computed with the median of all reconstructed loss of each detectors.</figcaption></figure>

phenomenon like manifold collapse [33]. Such issue can be mitigated using robust projections in the learned space of
the autoencoder (eigenfunction decomposition [5], ...).

## 2.2. Manifold learning

Robust Subspace Recovery [20, 25] is a robust manifold learning technique that maps inlier distributions in a subspace where anomalies/outliers are at the edge. Based on this, an autoencoder has been proposed to encode observation from the input space in a robust subspace [19]. One problem of such an autoencoder is the difficulty to address scenarios with features that present similar range values (problem of entanglement) [34]. Entanglement is especially more difficult with contextual anomalies where documents share similar semantic properties. This problem can be tackled using a local projection from the nearest neighbours in the latent representation space. Several manifold learning approaches like locally linear embedding (LLE) [27], neighbourhood components analysis (NCA) [15] or soft nearest neighbours (SNN) [14] can help to mitigate it.

## 2.3. Kinds of Anomaly

A large number of contributions have settled anomaly/outlier taxonomies [1, 29, 35] and different types of anomalies have been studied in the literature: *independent* anomaly, *contextual* anomaly, and *collective* anomaly. It is clear that several types of anomaly could coexist within the same corpus. As said earlier, a topic describes the main subject of a document. Subtopics are often used to associate a document with other subjects that could appeared in the document. Depending on the document, there may be multiple subtopics within a broader category. For instance, the *Sports* topic encompasses the *Football* and the *Tennis* topics. Consequently, if we take this hierarchical structure into account, a form of contextual anomaly appears [13].

## 2.4. Benchmark Strategies

Benchmark process for textual anomaly detection (AD) requires careful consideration to ensure fair and meaningful comparisons across models particularly with respect to anomaly types. Most existing works simulate anomalies by contaminating corpora that are initially anomaly-free. Three main contamination strategies have emerged.

First is the *Manual contamination*, used in early works [16, 23], which involves selecting anomalies manually into corpora with unrelated topics (making the task simple). The second approach is a *Topic-based contamination* that is widely adopted in recent literature [13, 19], consisting to select one topic as the inlier class and treats all others as anomalies. However, it ignores topic hierarchies, often misclassifying semantically related documents. The third strategy is the *Hierarchy-aware contamination* [11, 24, 29] that preserves the ontology of topic relations by merging
documents from subtopics into coherent parent categories (e.g., Football and Tennis as Sports). In this case, contextual anomalies are not considered.

Beyond contamination, training protocols also vary. Some models are trained only on inliers and evaluated on contaminated sets, while others are trained and tested on mixed data, requiring disentanglement in latent space. In this work, we adopt and generalize the third approach by introducing the TAC procedure (Section 4), which enables consistent and reproducible contamination across multiple datasets and anomaly types.

### 3. RoSAE: Robust Subspace Local Recovery Autoencoder Ensemble

Robust subspace recovery autoencoders have successfully tackled AD in text corpora, they lack locality and geometry awareness for mitigating disentanglement and manifold collapse from transformer-based language models. We propose to tackle these problems by introducing an ensemble (RoSAE) of Robust Subspace Local Recovery AutoEncoder (RLAE) which integrates locality in the latent representation through a local neighbouring method (Figure 2).

#### 3.1. Randomly Connected One-Class Autoencoder

To improve generalization and reduce overfitting, we adopt a stochastic architecture in which each autoencoder in the ensemble has randomly pruned connections. This strategy introduces architectural diversity, increasing robustness in the face of noisy or highly variable data, a common scenario in textual anomaly detection. Let $x_i \in \mathbb{R}^D$ denote an input instance from a corpus of $N$ documents. A standard autoencoder consists of an encoder $\mathcal{E}$ that maps $x_i$ to a latent representation $z_i = \mathcal{E}(x_i) \in \mathbb{R}^e$, and a decoder $\mathcal{D}$ that reconstructs the input as $\hat{x}_i = \mathcal{D}(z_i)$. In our design, each base autoencoder applies random pruning to a subset of its internal connections. Let $\theta \in [0.2, 0.5]$ be a pruning rate drawn uniformly at random. For each instance, a fraction $\theta$ of connections between selected layers are randomly disabled before training. This creates a structurally different sub-network at each instantiation, following the spirit of architectural bagging.

This technique is inspired by prior ensemble-based anomaly detection methods [9], but differs in two ways. First, we apply random pruning with variable sparsity rates per autoencoder, increasing variance and reducing redundancy. Secondly, unlike methods that allow arbitrary overlap of layer connectivity, we ensure controlled and non-overlapping sparsification between specific layers to maintain structural diversity. This strategy yields a population of diverse base detectors, each trained independently on the same data but with a distinct functional bias. Their outputs are later aggregated (see Section 3.4).

#### 3.2. Robust Subspace Recovery Layer

A common challenge in autoencoder-based anomaly detection is that reconstructions may be overly influenced by irrelevant or noisy features, especially in high-dimensional input spaces. The Robust Subspace Recovery (RSR) layer [19, 20] addresses this issue by projecting the encoded representations into a lower-dimensional subspace that suppresses outliers and enhances the structure of inlier distributions. Formally, let $\mathcal{E}$ be the encoder network, which maps an input instance $x_i \in \mathbb{R}^D$ to a latent vector $z_i = \mathcal{E}(x_i) \in \mathbb{R}^e$. The RSR layer then applies a linear transformation $\mathbf{A} \in \mathbb{R}^{D \times e}$, where $d < e$, yielding a compact representation $\hat{z}_i = \mathbf{A}z_i \in \mathbb{R}^d$. The decoder $\mathcal{D}$ attempts to reconstruct the input via $\hat{x}_i = \mathcal{D}(\hat{z}_i)$. The reconstruction loss is defined using an $\ell_{2,p}$ norm (with $p > 0$) to ensure robustness to outliers:

$$L_{AE}^{p}(\mathcal{E}, \mathbf{A}, \mathcal{D}) = \sum_{i=1}^{N} \|\mathbf{x}_i - \hat{\mathbf{x}}_i\|_{2}^{p}. \qquad (1)$$

To guide the subspace learning, the RSR layer is regularized with two objectives: i) the projection $\mathbf{A}$ should approximately preserve the latent representation $\mathbf{z}_i$; ii) the rows of $\mathbf{A}$ should be approximately orthonormal to prevent
degeneracies. These are captured in the following regularization term:

$$L_{RSR}^{q}(\mathbf{A}) = \sum_{i=1}^{N} \|\mathbf{z}_{i} - \mathbf{A}^{\top}(\mathbf{A}\mathbf{z}_{i})\|_{2}^{q} + \|\mathbf{A}\mathbf{A}^{\top} - \mathbf{I}_{d}\|_{F}^{q}, \quad (2)$$

where $\|\cdot\|_F$ is the Frobenius norm, and $\mathbf{I}_d$ is the $d \times d$ identity matrix. The first term enforces subspace reconstruction fidelity, while the second encourages orthonormality of the projection. The full loss for the robust autoencoder with subspace regularization becomes:

$$L_{RSRAE}(\mathcal{E}, \mathbf{A}, \mathcal{D}) = L_{AE}^{p}(\mathcal{E}, \mathbf{A}, \mathcal{D}) + \lambda_{1} L_{RSR}^{q}(\mathbf{A}), \quad (3)$$

with $\lambda_1 > 0$ controlling the strength of the RSR regularization. While the RSR layer improves robustness and generalization, it operates under a global linearity assumption. This can suppress important variations in the local structure of the latent space, particularly when detecting contextual anomalies, where anomalies are subtle and semantically similar to inliers. In such cases, the RSR projection may flatten informative distinctions. To overcome this, we introduce a complementary term that explicitly preserves local geometry, detailed next.

### 3.3. Robust Local Embedding

To address the limitations of global linearity in RSR, we introduce a Local Neighbourhood Embedding (LNE) term that preserves local structure in the latent space. Our goal is to ensure that data points with similar semantic content remain close after encoding, which is crucial for identifying anomalies that deviate contextually rather than globally. This term is inspired by Locally Linear Embedding (LLE) [27, 8], which assumes that the local geometry around each input can be approximated by a linear combination of its nearest neighbors. In our setting, LNE acts as a regularizer that aligns the geometry of the input space with that of the learned subspace. Let $\{\mathbf{x}_i\}_{i=1}^N \subset \mathbb{R}^D$ denote the input data. For each $\mathbf{x}_i$, we define $N_i$ as the set of indices of its k-nearest neighbors (excluding itself). We first compute reconstruction weights:

$$\min_{\mathbf{w}_i} \left\| \mathbf{x}_i - \sum_{j \in N_i} w_{ij} \mathbf{x}_j \right\|_2^2 \quad \text{subject to} \quad \sum_{j \in N_i} w_{ij} = 1. \quad (4)$$

These weights capture the local relationships in the input space. We then enforce that these relationships are preserved in the encoded subspace. Let $\mathbf{z}_i = \mathbf{A} \cdot \mathcal{E}(\mathbf{x}_i)$ be the projected embedding. The LNE loss is given by:

$$L_{LNE}(\mathbf{A}) = \sum_{i=1}^{N} \sum_{j \in N_i} w_{ij} \|\mathbf{z}_i - \mathbf{z}_j\|_2^2 = \sum_{i=1}^{N} \sum_{j \in N_i} w_{ij} \|\mathbf{A}\mathbf{x}_i - \mathbf{A}\mathbf{x}_j\|_2^2. \quad (5)$$

This term encourages the latent space to reflect the same local structure as the input space. As a result, contextual anomalies, which often appear normal globally but tackle local semantic coherence, become more separable. We now define the complete objective for our Robust Local Autoencoder (RLAE), which combines reconstruction, robust projection, and local embedding:

$$L_{RLAE}(\mathcal{E}, \mathbf{A}, \mathcal{D}) = L_{AE}^{1}(\mathcal{E}, \mathbf{A}, \mathcal{D}) + \lambda_{1} L_{RSR}^{1}(\mathbf{A}) + \lambda_{2} L_{LNE}(\mathbf{A}), \quad (6)$$
<figure><img src="image_5.png" /><figcaption>Fig. 3: Learned manifold for a one-class autoencoder (AE), a robust autoencoder (RSR), and our RLAE on the IMDB corpus. Due to minimal token variation, disentanglement problem is **challenging**. Red points are anomalies, blue are inliers (ranked via ROC-AUC). AE exhibits manifold collapse; RSR prevents it but yields entangled manifolds. RLAE avoid both phenomenon, and achieves clearer and structured separation.</figcaption></figure>

where $\lambda_1$ and $\lambda_2$ are hyperparameters balancing the global robustness and local consistency terms. The LNE term provides geometric context to RLAE, making it more sensitive to anomalies that arise from nuanced semantic shifts. By preserving neighborhood topology in the latent space, the model becomes better equipped to detect deviations that are invisible to global metrics alone.

The effectiveness of our proposed RLAE model in preserving local geometry is visually demonstrated in Figure 3, which shows a contaminated setup of IMDB corpus. IMDB is a particularly **challenging** corpus due to the subtle semantic overlap between anomalous and inlier documents. The first panel (left) displays the ground truth separation with a t-SNE representation, followed by three model projections: standard autoencoder (AE), RSR autoencoder (RSR), and our full RLAE model. We observe that while both AE and RSR struggle to clearly disentangle anomalies from inliers, often collapsing them into overlapping regions, RLAE produces a more structured latent space. This separation illustrates how the LNE term enhances the model’s ability to preserve semantic locality, making it more sensitive to contextual shifts. In particular, the IMDB projection demonstrates that the LNE component effectively resolves cases where global projection fails. We also note that learning on 2d-manifold is only for illustration purpose.

### 3.4. Ensemble Learning and Text Representation

In ensemble methods, a combination of several models, also called *base detectors*, is often more robust than usage of a single model [9]. In RoSAE, we use the loss reconstruction of each autoencoders and then normalise each base detector scores through standard deviation of one unit. We then aggregate for each observation with median value. Regarding text representation, recent works have recorded their results on BERT-based language models. The RoSAE model is not based on the self-attention mechanism, such as for [24, 29], instead we use the RoBERTa Sentence Transformer from [26]. This choice is motivated with Sentence Transformer being more performant for encoding long text than its original counterpart.

## 4. Textual Anomaly Contamination (TAC)

In textual anomaly detection literature, each approach introduces their own method to build a synthetic contaminated corpus. In this context we observe that a taxonomy of anomaly is not considered, making it even harder to perform a fair comparison between approaches. In this paper, we propose a novel contamination algorithm for performing contamination with independent and contextual anomalies (Figure 2).

Independent anomalies represent observations that lack any meaningful relationship with other topics. Specifically, anomaly topics and inlier topics have different hierarchical parents within the category structure. Let a corpus X with the topic set $\mathcal{Y}$, a labelled document of a corpus $(x, y) \in X \times \mathcal{Y}$ and $\zeta$ the inlier category. The corresponding inlier subset is noted $X_{\zeta} \subseteq X$. Let $\mathcal{A}$ be the subset of all anomalies such as $\mathcal{A} \subset X$. We have: $\mathcal{A} = X \setminus X_{\zeta}$.
**Algorithm 1 TAC: Textual Anomaly Contamination (Independent)**

**Require:** Inlier topic $\zeta$, corpus X, split size $l$, contamination rate $v$

**Ensure:** $0 < l \le N$

$c \leftarrow lv$

$i \leftarrow 0$

Initialize empty matrix Z

$$ \mathcal{A} \leftarrow \{x_j \times y_j \in X \times Y | \forall j \in [0, N], y_j \neq \zeta\} $$

$$ X_{\zeta} \leftarrow \{X \setminus \mathcal{A}\} $$

**while** $|Z| < c$ **do**

if Parent($y_i$) $\neq$ Parent($\zeta$) **then**

Append($x_i$, $y_i$) to Z

**end if**

$i \leftarrow i + 1$

**end while**

Fill Z with $X_{\zeta}$ until $|Z| = l$

**return** Shuffle(Z)

► Inlier Matrix

► Set = instead of $\neq$ for contextual

Regarding $\mathcal{A}$, we can make the distinction with two different constraints. We note $P(y)$ the direct parent of $y$ in a given hierarchy. First, an observation $x_i$ is considered to be an anomaly if its parent topic is different of inlier parent topics:

$$ \mathcal{A}_p(\zeta) = \{a \mid P(\zeta) \neq P(y), (a, y) \in \mathcal{A} \times Y\} \quad (7) $$

The second constraint corresponds to documents that do not lie in $X_{\zeta}$ but share the same parent topic as $\zeta$. These observations are identified as another kind of anomaly: contextual anomaly. We denote $\mathcal{A}_c(\zeta) = \{a \mid P(\zeta) = P(y), (a, y) \in \mathcal{A} \times Y\}$. Our proposed TAC for independent anomalies is presented in Algorithm 1. Three hyper-parameters are used: the contamination rate $v$, length of the contaminated split $l$ and the inlier topic noted $\zeta$.

# 5. Experiments

## 5.1. Data

Recent works [16, 19, 22, 29] use classification datasets such as Reuters-21578 and 20 Newsgroup with a dedicated preparation for comparing their models. For all corpora we apply data contamination of independent anomalies and contextual anomalies with TAC (Algorithm 1). To be fair with each methods and each corpora, we set the size of the preparation subset to the maximum number of available document, and results are averaged over 10 of runs. The train split of each corpus is used for training, and test split for evaluation. In a first step, we set $v = 0.10$. in this section, corpora with topic hierarchies are referred as semantic corpora.

**Semantic corpora.** We apply TAC on four semantic corpora. For 20 Newsgroups, we group subtopics into seven coarse-grained categories: computer, for sale, motors, politics, religion, science, and sports. In Reuters-21578, we discard multi-labeled documents and reorganize the remaining topics into four high-level categories commodities, finance, metals, and energy, following the taxonomy of [32]. We select the eight most populated topics for evaluation. For DBpedia 14 [36], we derive six parent classes using the provided ontology. Finally, Web of Science [18], which includes a three-level hierarchical taxonomy, is reduced to seven top-level categories; ambiguous subtopics linked to multiple parents are resolved by assigning them to the larger parent.

**Spam and sentiment analysis.** We use Enron dataset and SMS Spam [2] dataset. We prepare a train and test split before contaminating ham in spam, and then spam in ham. In a similar process, IMDB [21] and SST2 [31] are used for benchmarking independent contamination with more difficult scenarios.
<table><caption>Table 1: Independent and contextual contamination results with $v = 0.10$. AUC is the default metric for independent contamination and AP is added for contextual contamination. Embedding is Distill RoBERTA and each result is performed on test split with TAC. Red is for the best performing model, and blue for the second one. * FATE uses few-shot learning and has awareness of 5 anomalies. † recorded results on corresponding papers.</caption><thead><tr><th></th><th>Model</th><th>20New.</th><th>Reuters</th><th>WOS</th><th>DBpedia</th><th>Enron</th><th>SMS Sp.</th><th>IMDB</th><th>SST2</th></tr></thead><tbody><tr><td rowspan="4">Original<sup>†</sup></td><td>OCSVM</td><td>0.715</td><td>0.932</td><td>-</td><td>-</td><td>-</td><td>-</td><td><span style="color:red;">0.511</span></td><td>-</td></tr><tr><td>RSRAE</td><td><span style="color:blue;">0.844</span></td><td>0.868</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>CVDD</td><td>0.822</td><td><span style="color:blue;">0.951</span></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>FATE<sup>*</sup></td><td><span style="color:red;">0.870</span></td><td><span style="color:red;">0.968</span></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td rowspan="6">Independent</td><td>OCSVM</td><td>0.884<sup>±0.06</sup></td><td>0.851<sup>±0.11</sup></td><td>0.876<sup>±0.07</sup></td><td><span style="color:blue;">0.962</span><sup>±0.03</sup></td><td><span style="color:blue;">0.687</span><sup>±0.18</sup></td><td>0.683<sup>±0.25</sup></td><td>0.539<sup>±0.16</sup></td><td>0.575<sup>±0.06</sup></td></tr><tr><td>RandNet</td><td>0.622<sup>±0.12</sup></td><td>0.689<sup>±0.18</sup></td><td>0.636<sup>±0.16</sup></td><td>0.575<sup>±0.20</sup></td><td>0.466<sup>±0.18</sup></td><td>0.687<sup>±0.13</sup></td><td>0.503<sup>±0.06</sup></td><td>0.486<sup>±0.05</sup></td></tr><tr><td>RSRAE</td><td>0.769<sup>±0.10</sup></td><td>0.768<sup>±0.13</sup></td><td>0.731<sup>±0.13</sup></td><td>0.634<sup>±0.12</sup></td><td>0.518<sup>±0.11</sup></td><td>0.642<sup>±0.10</sup></td><td><span style="color:blue;">0.540</span><sup>±0.06</sup></td><td><span style="color:blue;">0.577</span><sup>±0.06</sup></td></tr><tr><td>CVDD</td><td>0.781<sup>±0.06</sup></td><td><span style="color:blue;">0.895</span><sup>±0.11</sup></td><td><span style="color:red;">0.948</span><sup>±0.04</sup></td><td>0.958<sup>±0.04</sup></td><td>0.629<sup>±0.17</sup></td><td><span style="color:blue;">0.717</span><sup>±0.12</sup></td><td>0.513<sup>±0.11</sup></td><td>0.562<sup>±0.05</sup></td></tr><tr><td>FATE<sup>*</sup></td><td><span style="color:red;">0.911</span><sup>±0.02</sup></td><td><span style="color:red;">0.926</span><sup>±0.10</sup></td><td>0.918<sup>±0.08</sup></td><td>0.951<sup>±0.12</sup></td><td>0.484<sup>±0.20</sup></td><td>0.683<sup>±0.18</sup></td><td>0.517<sup>±0.23</sup></td><td>0.529<sup>±0.04</sup></td></tr><tr><td>RLAE</td><td>0.792<sup>±0.07</sup></td><td>0.829<sup>±0.16</sup></td><td>0.843<sup>±0.09</sup></td><td>0.840<sup>±0.08</sup></td><td>0.611<sup>±0.14</sup></td><td>0.657<sup>±0.16</sup></td><td>0.521<sup>±0.06</sup></td><td>0.542<sup>±0.04</sup></td></tr><tr><td rowspan="4">Contextual</td><td>RoSAE</td><td><span style="color:blue;">0.907</span><sup>±0.05</sup></td><td><span style="color:red;">0.926</span><sup>±0.11</sup></td><td><span style="color:blue;">0.939</span><sup>±0.04</sup></td><td><span style="color:red;">0.983</span><sup>±0.01</sup></td><td><span style="color:red;">0.713</span><sup>±0.11</sup></td><td><span style="color:red;">0.813</span><sup>±0.11</sup></td><td><span style="color:red;">0.560</span><sup>±0.05</sup></td><td><span style="color:red;">0.594</span><sup>±0.04</sup></td></tr><tr><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Model</td><td colspan="2">20New.</td><td colspan="2">Reuters</td><td colspan="2">WOS</td><td colspan="2">DBpedia</td></tr><tr><td></td><td>AP</td><td>AUC</td><td>AP</td><td>AUC</td><td>AP</td><td>AUC</td><td>AP</td><td>AUC</td></tr><tr><td>OCSVM</td><td>0.220<sup>±0.16</sup></td><td>0.681<sup>±0.12</sup></td><td>0.471<sup>±0.23</sup></td><td>0.755<sup>±0.12</sup></td><td>0.599<sup>±0.22</sup></td><td>0.889<sup>±0.10</sup></td><td>0.566<sup>±0.23</sup></td><td>0.882<sup>±0.08</sup></td></tr><tr><td>RandNet</td><td>0.164<sup>±0.10</sup></td><td>0.579<sup>±0.14</sup></td><td>0.337<sup>±0.26</sup></td><td>0.662<sup>±0.20</sup></td><td>0.147<sup>±0.07</sup></td><td>0.544<sup>±0.15</sup></td><td>0.319<sup>±0.15</sup></td><td>0.694<sup>±0.12</sup></td></tr><tr><td>RSRAE</td><td>0.177<sup>±0.07</sup></td><td>0.584<sup>±0.10</sup></td><td>0.340<sup>±0.22</sup></td><td>0.729<sup>±0.15</sup></td><td>0.214<sup>±0.13</sup></td><td>0.623<sup>±0.13</sup></td><td>0.260<sup>±0.14</sup></td><td>0.660<sup>±0.12</sup></td></tr><tr><td>CVDD</td><td>0.131<sup>±0.15</sup></td><td>0.613<sup>±0.11</sup></td><td>0.481<sup>±0.29</sup></td><td>0.824<sup>±0.16</sup></td><td>0.623<sup>±0.15</sup></td><td><span style="color:blue;">0.916</span><sup>±0.08</sup></td><td>0.646<sup>±0.19</sup></td><td><span style="color:blue;">0.919</span><sup>±0.09</sup></td></tr><tr><td>FATE<sup>*</sup></td><td><span style="color:blue;">0.301</span><sup>±0.24</sup></td><td><span style="color:blue;">0.689</span><sup>±0.12</sup></td><td><span style="color:blue;">0.536</span><sup>±0.20</sup></td><td><span style="color:red;">0.867</span><sup>±0.05</sup></td><td><span style="color:red;">0.628</span><sup>±0.13</sup></td><td>0.913<sup>±0.08</sup></td><td><span style="color:red;">0.784</span><sup>±0.14</sup></td><td><span style="color:red;">0.951</span><sup>±0.08</sup></td></tr><tr><td>RLAE</td><td>0.201<sup>±0.10</sup></td><td>0.642<sup>±0.12</sup></td><td>0.420<sup>±0.24</sup></td><td>0.757<sup>±0.21</sup></td><td>0.334<sup>±0.15</sup></td><td>0.738<sup>±0.12</sup></td><td>0.368<sup>±0.14</sup></td><td>0.747<sup>±0.09</sup></td></tr><tr><td>RoSAE</td><td><span style="color:red;">0.325</span><sup>±0.14</sup></td><td><span style="color:red;">0.718</span><sup>±0.09</sup></td><td><span style="color:red;">0.609</span><sup>±0.11</sup></td><td><span style="color:red;">0.880</span><sup>±0.07</sup></td><td><span style="color:red;">0.687</span><sup>±0.07</sup></td><td><span style="color:red;">0.921</span><sup>±0.03</sup></td><td><span style="color:red;">0.840</span><sup>±0.08</sup></td><td><span style="color:red;">0.951</span><sup>±0.02</sup></td></tr></tbody></table>

## 5.2. Preparation

We use TAC for independent and contextual contamination, and on each candidate inliers with $v = 0.1$. The maximum amount of available documents is retrieved [D]. All results are compared against Area Under Curve ROC (AUC) and Average Precision (AP) metrics. For every dataset, we train and record test results on each possible class and average them for presentation. The total number of runs for each corpus and each split contamination is 10. Thus, for 20 Newsgroups with independent contamination we perform a total of $20 \times 10$ train/test procedure.

We propose to compare our approach against OCSVM[30], RSRAE[19], an autoencoder ensemble model RandNet [9], CVDD [29] and FATE [11]. For OCSVM we use the implementation from PyOD[37] and for RSRAE we use our own PyTorch implementation. All our results approximate theirs. We also display CVDD results with several adaptations. In the original paper, CVDD learns on all the training corpus (inlier only) instead of the training split only. We use their implementation, knowing the approach has advantage against other baseline models. The same comment goes for FATE, which additionally use few-shot learning and benefits from anomaly awareness.

For RoSAE and RandNet we set the number of base predictors to 20. Our code is using the PyOD base implementation [37]. We propose an implementation based on the *BaseDetector* of PyOD so each of the compared model can be accordingly tested. We use the pruning methods from PyTorch for creating randomly connected autoencoders. Additionally, we also set hyperparameters $\lambda_1 = 0.1$ and $\lambda_2 = 0.05$. For avoiding manifold collapse problem and generates solutions, we advise that $\lambda_2 < \lambda_1$. We set the latent dimension of the LNE layer to 32. On the other hand, we set the epoch number to 50 and random connection probability between [0.2, 0.5].

## 5.3. Results

*Experiment.* We assess RoSAE on independent contamination, contextual contamination, and robustness under very limited scenario. Results are reported in Table 1. RoSAE matches or exceeds recent methods such as FATE, particularly on non-hierarchical corpora like Enron and SST2. This suggests strong generalization to diverse text distributions beyond structured topic hierarchies. Our approach also outperforms all baselines on both AUC and AP, with an average AP gain of +9% over the next best model. It significantly improves over RLAE alone, confirming the benefit of ensemble smoothing and local geometry regularization. The performance gap is especially pronounced on challenging corpora. Under low contamination ($v = 0.05$), RoSAE maintains low variance and narrow score ranges across runs
<figure><img src="image_6.png" /><figcaption>Fig. 4: Results of RoSAE with contextual contamination on size of ensemble, on different values for hyper-parameter K and for lambda.</figcaption></figure>

highlighting its resilience to sampling noise and overfitting. The ensemble effectively reduces both bias and variance. In Section 3.3, the Figure 3 visualizes 2D embeddings on IMDB. Standard AEs fail to separate anomalies, and RSR alone yields dense but entangled representations. RLAE better separates anomalies from inliers, validating its ability to disentangle contextual cues through robust and localized manifold learning.

**Ablation.** We analyze the sensitivity of RoSAE to key hyperparameters. As shown in Figure 4a, performance improves with the number of base detectors up to 30, beyond which gains diminish while computational cost increases; thus, 20 detectors offer a balanced trade-off. Figure 4c highlights the effect of the neighborhood size *k* in RLAE: while smaller *k* values tend to yield better results, RoSAE remains robust across a range, confirming the stability of local embeddings. Finally, Figure 4b shows the impact of the locality weight $\lambda_2$ under fixed $\lambda_1 = 0.1$; larger $\lambda_1$ values help stabilize training, allowing $\lambda_2$ to fine-tune local structure without inducing overfitting. These findings support our design choices and demonstrate that RoSAE maintains strong performance across a wide hyperparameter space. These findings confirm RoSAE's efficacy in detecting subtle textual anomalies and performance across various settings.

**Discussion.** RoSAE effectiveness in contextual anomaly detection stems from its integration of global robustness (via RSR), local geometry preservation (via LNE), and ensemble variance reduction. Contextual anomalies, semantically plausible inconsistent documents, are difficult to detect using global models alone. RoSAE addresses this by learning structured and disentangled latent spaces where such anomalies become geometrically separable. This is visually evident in Figure 3, where RLAE produces clearer separation between inliers and anomalies, unlike standard AEs or RSRAE. Quantitatively, Table 1 confirms RoSAE's dominance: it ranks first (blue) or second (red) in all datasets, and achieves the lowest or near-lowest variance across runs, reflecting both accuracy and stability. Its advantage is especially strong on high-entropy corpora like SST2 and Enron, where subtle context shifts define anomalies. Additionally, RoSAE can be trained efficiently on a CPU, whereas recent competing methods typically require GPU resources. Furthermore, our results underscore that most state-of-the-art approaches focus on independent contamination (first part of Table 1) and exhibit lower robustness compared to RoSAE, particularly in contextual settings.

## 6. Conclusion

We introduced RoSAE, a robust autoencoders ensemble that leverages subspace recovery, local geometry preservation, and architectural diversity for detecting subtle contextual anomalies in text. We also proposed TAC, a contamination framework that explicitly distinguishes between independent and contextual anomalies. TAC reveals that different anomaly types exist, vary in detection difficulty, and benefit from separate evaluation, enabling more interpretable and fair comparisons across models. Together, RoSAE and TAC advance the state-of-the-art in textual anomaly detection by improving both detection performance and result interpretability/reproducibility.

## References

[1] Aggarwal, C.C., 2017. Outlier analysis. 2 ed., Springer.

[2] Almeida, T.A., Hidalgo, J.M.G., Yamakami, A., 2011. Contributions to the study of sms spam filtering: New collection and results, in: ACM Symposium on Document Engineering (DOCENG’11).
[3] Beggel, L., Pfeiffer, M., Bischl, B., 2020. Robust anomaly detection in images using adversarial autoencoders, in: Brefeld, U., Fromont, E., Hotho, A., Knobbe, A., Maathuis, M., Robardet, C. (Eds.), Machine Learning and Knowledge Discovery in Databases, Springer. pp. 206–222.

[4] Bejan, M., Manolache, A., Popescu, M., 2023. AD-NLP: A benchmark for anomaly detection in natural language processing, in: Bouamor, H., Pino, J., Bali, K. (Eds.), Proc. of the 2023 Conf. on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Singapore. pp. 10766–10778.

[5] Bengio, Y., Delalleau, O., Roux, N.L., Paiement, J.F., Vincent, P., Ouimet, M., 2004. Learning Eigenfunctions Links Spectral Embedding and Kernel PCA. Neural Computation 16, 2197–2219.

[6] Briskilal, J., Subalalitha, C., 2022. An ensemble model for classifying idioms and literal texts using bert and roberta. Information Processing and Management 59, 102756.

[7] Chandola, V., Banerjee, A., Kumar, V., 2009. Anomaly detection: A survey. ACM Computing Surveys 41, 1–58.

[8] Chen, J., Liu, Y., 2011. Locally linear embedding: a survey. Artificial Intelligence Review 36, 29–48.

[9] Chen, J., Sathe, S., Aggarwal, C., Turaga, D., 2017. Outlier detection with autoencoder ensembles, in: SIAM Int. Conf. data mining, pp. 90–98.

[10] Chen, Z., Yeo, C.K., Lee, B.S., Lau, C.T., 2018. Autoencoder-based network anomaly detection, in: Wireless Telecommunications Symposium (WTS), pp. 1–5. doi:10.1109/WTS.2018.8363930.

[11] Das, A.S., Ajay, A., Saha, S., Bhuyan, M., 2023. Few-shot anomaly detection in text with deviation learning, in: Int. Conf. on Neural Information Processing, Springer. pp. 425–438.

[12] Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2019. BERT: Pre-training of deep bidirectional transformers for language understanding, in: Conf. of the North American Chapter of the Association for Computational Linguistics.

[13] Fouché, E., Meng, Y., Guo, F., Zhuang, H., Böhm, K., Han, J., 2020. Mining text outliers in document directories, in: IEEE Int. Conf. on Data Mining (ICDM), IEEE, pp. 152–161.

[14] Frosst, N., Papernot, N., Hinton, G., 2019. Analyzing and improving representations with the soft nearest neighbor loss, in: Chaudhuri, K., Salakhutdinov, R. (Eds.), Proc. of the 36th Int. Conf. on Machine Learning, PMLR. pp. 2012–2020.

[15] Goldberger, J., Hinton, G.E., Roweis, S., Salakhutdinov, R.R., 2004. Neighbourhood components analysis. Advances in neural information processing systems 17.

[16] Kannan, R., Woo, H., Aggarwal, C.C., Park, H., 2017. Outlier Detection for Text Data. SDM Int. Conf. on Data Mining 17, 489–497.

[17] Kieu, T., Yang, B., Guo, C., Jensen, C.S., 2019. Outlier detection for time series with recurrent autoencoder ensembles, in: 28th Int. Joint Conf. on Artificial Intelligence, IJCAI-19, pp. 2725–2732. doi:10.24963/ijcai.2019/378.

[18] Kowsari, K., Brown, D.E., Heidarysafa, M., Meimandi, K.J., Gerber, M.S., Barnes, L.E., 2017. Hdltex: Hierarchical deep learning for text classification, in: 16th IEEE Int. Conf. on machine learning and applications (ICMLA), IEEE, pp. 364–371.

[19] Lai, C.H., Zou, D., Lerman, G., 2020. Robust Subspace Recovery Layer for Unsupervised Anomaly Detection. ICLR.

[20] Lerman, G., Maunu, T., 2018. An overview of robust subspace recovery. Proc. of the IEEE 106, 1380–1410.

[21] Maas, A.L., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., Potts, C., 2011. Learning word vectors for sentiment analysis, in: Pr49th Annual Meeting of the Assoc. for Comput. Linguistics: Human Language Technologies, ACL, Portland, Oregon, USA. pp. 142–150.

[22] Mahapatra, A., Srivastava, N., Srivastava, J., 2012. Contextual Anomaly Detection in Text Data. Algorithms 5, 469–489.

[23] Manevitz, L.M., Yousef, M., 2001. One-Class SVMs for Document Classification. JMLR 2, 139–154.

[24] Manolache, A., Brad, F., Burceanu, E., 2021. DATE: Detecting anomalies in text via self-supervision of transformers, in: Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL. pp. 267–277.

[25] Rahmani, M., Atia, G.K., 2017. Randomized robust subspace recovery and outlier detection for high dimensional data matrices. IEEE Transactions on Signal Processing 65, 1580–1594. doi:10.1109/TPS.2016.2645515.

[26] Reimers, N., Gurevych, I., 2019. Sentence-bert: Sentence embeddings using siamese bert-networks, in: Conf. on Empirical Methods in Natural Language Processing, ACL.

[27] Roweis, S.T., Saul, L.K., 2000. Nonlinear dimensionality reduction by locally linear embedding. science 290, 2323–2326.

[28] Ruff, L., Kauffmann, J.R., Vandermeulen, R.A., Montavon, G., Samek, W., Kloft, M., Dietterich, T.G., Müller, K.R., 2021. A Unifying Review of Deep and Shallow Anomaly Detection. Proc. of the IEEE 109, 756–795.

[29] Ruff, L., Zemlyanskiy, V., Vandermeulen, R., Schnake, T., Kloft, M., 2019. Self-attentive, multi-context one-class classification for unsupervised anomaly detection on text, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy. pp. 4061–4071. URL: https://aclanthology.org/P19-1398, doi:10.18653/v1/P19-1398.

[30] Schölkopf, B., Platt, J.C., Shawe-Taylor, J., Smola, A.J., Williamson, R.C., 2001. Estimating the Support of a High-Dimensional Distribution. Neural Computation 13, 1443–1471. doi:10.1162/089976601750264965.

[31] Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A., Potts, C., 2013. Recursive deep models for semantic compositionality over a sentiment treebank, in: Conf. on Empirical Methods in Natural Language Processing, ACL, Seattle, Washington, USA. pp. 1631–1642.

[32] Toutanova, K., Chen, F., Popat, K., Hofmann, T., 2001. Text classification in a hierarchical mixture model for small training sets, in: 10th Int. Conf. on Information and knowledge management, pp. 105–113.

[33] Wang, P.Z., Wang, W.Y., 2019. Riemannian normalizing flow on variational Wasserstein autoencoder for text modeling, in: Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, Minneapolis, Minnesota. pp. 284–294. doi:10.18653/v1/N19-1025.

[34] Wang, X., Chen, H., Tang, S., Wu, Z., Zhu, W., 2022. Disentangled representation learning. arXiv preprint arXiv:2211.11695.

[35] Zhang, J., 2013. Advancements of Outlier Detection: A Survey. ICST Transactions on Scalable Information Systems 13.

[36] Zhang, X., Zhao, J., LeCun, Y., 2015. Character-level convolutional networks for text classification. Advances in neural inf. proc. systems 28.

[37] Zhao, Y., Nasrullah, Z., Li, Z., 2019. Pyod: A python toolbox for scalable outlier detection. JMLR 20, 1–7.

[38] Zhou, Q., Pang, G., Tian, Y., He, S., Chen, J., 2023. Anomalyclip: Object-agnostic prompt learning for zero-shot anomaly detection, in: The Twelfth Int. Conf. on Learning Representations.