<figure><img src="image_1.png" /></figure>

HAL
open science

# Unsupervised Outlier Detection Using Memory and Contrastive Learning

Ning Huyan, Dou Quan, Xiangrong Zhang, Xuefeng Liang, Jocelyn

Chanussot, Licheng Jiao

► To cite this version:

Ning Huyan, Dou Quan, Xiangrong Zhang, Xuefeng Liang, Jocelyn Chanussot, et al. Unsupervised Outlier Detection Using Memory and Contrastive Learning. IEEE Transactions on Image Processing, 2022, 31, pp.6440-6454. 10.1109/TIP.2022.3211476. hal-03932521

HAL Id: hal-03932521

https://hal.science/hal-03932521v1

Submitted on 25 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
arXiv:2107.12642v1 [cs.CV] 27 Jul 2021

# Unsupervised Outlier Detection using Memory and Contrastive Learning

Ning Huyan¹ Dou Quan¹ Xiangrong Zhang*¹ Xuefeng Liang¹ Jocelyn Chanussot²
Licheng Jiao¹

¹School of Artificial Intelligence, Xidian University, Shaanxi, China

²Research center of Inria Grenoble-Rhone-Alpes, France

xrzhang@mail.xidian.edu.cn, xliang@xidian.edu.cn, dquan@stu.xidian.edu.cn

## Abstract

Outlier detection is one of the most important processes taken to create good, reliable data in machine learning. The most methods of outlier detection leverage an auxiliary reconstruction task by assuming that outliers are more difficult to be recovered than normal samples (inliers). However, it is not always true, especially for auto-encoder (AE) based models. They may recover certain outliers even outliers are not in the training data, because they do not constrain the feature learning. Instead, we think outlier detection can be done in the feature space by measuring the feature distance between outliers and inliers. We then propose a framework, MCOD, using a memory module and a contrastive learning module. The memory module constrains the consistency of features, which represent the normal data. The contrastive learning module learns more discriminating features, which boost the distinction between outliers and inliers. Extensive experiments on four benchmark datasets show that our proposed MCOD achieves a considerable performance and outperforms nine state-of-the-art methods.

## 1. Introduction

Outliers, also named anomalies, are unknown samples that deviate significantly from the distribution of normal samples (inliers). Outliers could be the lesions in medical images [26], frauds of banking transactions [39], and abnormal behavior in video surveillance [24, 32]. Detecting outliers is a crucial issue and important in everyday life. Since outliers rarely appear and are unpredictable, outliers and inliers are commonly unlabeled. The tasks of outlier detection (OD) in practice are unsupervised, which aims at identifying the commonalities within the unlabeled data to facilitate outlier detection [3]. The challenge of OD is that the data distribution of inliers and outliers can be extremely complicated, especially, they are unlabeled [15].

In the era of deep learning, many studies tried to leverage

<figure><img src="image_3.png" /><figcaption>Figure 1. The upper one is an AE based module, which detects outliers according to the reconstruction error in the pixel space. The lower one is our MCOD, which detects outliers in the feature space using self-supervised contrastive learning and a memory model.</figcaption></figure>

an auxiliary deep reconstruction model for OD. In particular, auto-encoder (AE) [1] is a widely used model [22, 33], as shown in Fig. 1, AE based methods determine if a sample is outlier or not according to the reconstruction error in the pixel-space, because they assume that inliers are easier to be reconstructed than outliers. However, it is not always true. The reason behind this fact is that AE models mainly focus on the quality of sample reconstruction, do not care about if the encoded features solely represent the inlier rather than outliers. Specifically, most AE based methods do not constrain the consistency of encoded feature for intra-class samples and the discrimination for inter-class samples during training process. Consistency denotes the features of samples in the same class are close, and discrimination means the features in different classes are distinct. Due to the lack of these two constraints, the encoded feature in AE may include the shared features between inliers and outliers that induce a small reconstruction error for outliers. The research [22] that also reported the outliers could be well reconstructed, even if the model was only trained on inliers.

In our study, we find that the outlier can be detected in the feature space, not necessary to leverage the reconstruction model. To this end, the extracted features must be consistent for inliers, meanwhile, discriminating to separate outliers from inliers. Following this idea, we propose an un-
supervised outlier detection method using memory and con-
trastive learning (MCOD) shown in Fig. 2. Firstly, we pro-
pose a memory module [7, 8, 21] to learn the consistent fea-
tures of inliers. It has three operations (writing, forgetting,
and reading) and a memory slot. The memory slots store
the memory prototype of each class, which is the predicted
cluster center of the class. The writing computes memory
prototypes and updates them into memory slots. More im-
portantly, we propose a forgetting operation that adds per-
turbations to prototypes to suppress the outlier prototypes
in memory slots. The reading constrains the feature of sam-
ple to be close to its corresponding prototype. Therefore,
MCOD is able to learn the consistent features. Secondly,
we design a self-supervised contrastive learning module to
learn the discriminating features from unlabeled inliers and
outliers. It has two symmetrical encoders in which the sam-
ple label is obtained by self-augmentation.

The main contributions of this paper are concluded as
follows:

* We propose an outlier detector framework (MCOD), which learns the discriminating feature by means of self-supervised contrastive learning and the consistent feature using memory mechanism.
* We conduct the outlier detection in the feature space without reconstructing the original samples.
* MCOD is trained by an end-to-end mode. Extensive experiments on four benchmarks demonstrate that MCOD outperforms nine SOTA methods.

## 2. Related work

**Outlier detection.** The mainstream approaches of out-
lier detection (OD) are unsupervised frameworks, which as-
sume that outliers are minority and significantly deviated
from inliers [3]. Generally, they can be grouped into two
categories: *discrimination-based* and *reconstruction-based*
methods. The discrimination-based OD methods detect out-
liers according to the hyperplane or hypersphere in the fea-
ture space, in which inliers are distributed in the hyper-
sphere and outliers are outside. The representative methods
are one-class SVM (OC-SVM) [28], support vector data de-
scriptor (SVDD) [29] and the deep version of SVDD (deep
SVDD) [23], which detect outliers by minimizing the vol-
ume of the hypersphere of the inliers. However, they have
less ability of learning the distribution of complex high-
dimensional data.

The reconstruction-based OD methods often employ
auto-encoders to detect outliers by measuring the recon-
struction errors [6, 12, 33, 38, 41]. They think that outliers
are more difficult to be reconstructed than inliers because
outliers are the minority. Zong et al. [41] integrated AE
and a density estimation model based on Gaussian Mixture

Model (GMM) for a better inlier reconstruction. Zhou et al. [38] proposed a deep auto-encoder (RDAE) associated with the robust principal component analysis (RPCA) to discard the outlier in the feature space. Lai et al. [12] followed this idea and designed a robust subspace recovery layer (RSR) for AE. Unlike these methods, the work [33] gradually separated inliers and outliers by alternatively minimizing reconstruction errors according to labeled inliers. All above methods made great efforts to achieve a better reconstruction, even relied on inliers only for training. Nevertheless, they neglected to constrain the features to solely represent inliers. Therefore, such AE based methods may reconstruct outliers to some extent.

**Self-supervised contrastive learning.** Self-supervised method is able to learn feature representation of unlabeled data by means of supervision. The method [30] applied the self-supervised learning to build pseudo labels using different pretext tasks and utilized the uncertainty of discriminating model to detect outliers. Instead of matching predictions to pseudo labels, contrastive learning [5, 9, 10] both maximizes the agreement between a sample and its augmentations, and the differences between different samples. To effectively enlarge the number of negative samples and maintain consistency of samples across preceding several preceding mini-batches, MoCo [9] proposed a dynamic dictionary as a queue that enqueue current mini-batch samples coming from an updated encoder and dequeue the earliest sample. In this work, we apply contrastive learning and queue to learn the discriminating features of unlabeled data, which facilitates cluster predictions for subsequently memory module learning [37].

**Memory mechanism.** Recently, memory mechanism has demonstrated a remarkable effectiveness on many tasks [14, 25, 40]. It has also been applied for anomaly detection problems [7, 21]. However, their feature learning is based on auto-encoder and semi-supervised architecture, whose training data are inliers only. With appropriate reading and writing operations, the memory module is able to learn the prototypes that represent inliers and ignore outliers. Due to this property, we leverage memory mechanism to achieve the consistent features from data mixed with unlabeled inliers and outliers.

## 3. The proposed framework

The framework of our proposed unsupervised outlier de-
tector MCOD is shown in Fig. 2. The contrastive learning
takes charge of learning the discriminating features. The
memory module is responsible for learning the consistent
prototypes of inliers. More details are given in the follow-
ing sections.
<figure><img src="image_4.png" /><figcaption>Figure 2. Schematic illustration of MCOD. It consists of two symmetrical encoders, one queue, memory slots, and three operations. f, z, c are outputs of feature layer, embedding layer and relevancy prediction layer, respectively. In the training process, two encoders extract features from the input sample and its augmentation and perform contrastive learning. The reading operation selects the corresponding memory prototypes for input samples, and the writing operation updates memory prototypes. The forgetting operation adds perturbations to prototypes to suppress the outlier prototypes in memory slots.</figcaption></figure>

## 3.1. Problem formulation

Let us consider a dataset $X \subset \mathcal{X}$, where $\mathcal{X}$ is the data space spanned by all samples in the space. The dataset is mixed up with both inliers $X_{in}$ and outliers $X_{out}$, and satisfy $X = X_{in} \cup X_{out}$ and $X_{in} \cap X_{out} = \emptyset$. The number of outliers $X_{out}$ is much smaller than the number of $X_{in}$. The goal of outlier detection is to separate $X_{out}$ from data $X$. Generally, the feature of outliers $X_{out}$ should be out of the feature distribution of inliers $X_{in}$. Thus, it is reasonable to detect outliers in the feature space. Our goal is to design a framework to learn the consistent feature of the entire data distribution to represent inliers, and the discriminating features to separate outliers from inliers.

## 3.2. Contrastive learning for discriminating feature

Since the data are unlabeled, we leverage a self-supervised contrastive learning to learn the discriminating features. Usually, contrastive learning framework has two encoders, which are contrastively optimized by sample pairs. The framework tries to maximize the feature similarity of positives (a sample and its self-argumentations) and the feature difference of negatives (two different samples).

In MCOD, a sample is augmented by various transformations, including random flipping, random rotating, random crop, random contrast changes, and Gaussian blur. Then, we have a self-labeled pair $(x_i^q, x_i^k)$. We design two symmetrical encoders as shown in the contrastive learning block of Fig. 2. The input of the first encoder $E_q$ is $x_i^q$, and the input of the second encoder $E_k$ is $x_i^k$. The features $f_i^q$ and $f_i^k$ of the sample-pair are formulated as

$$f_i^q = E_q(x_i^q), \quad f_i^k = E_k(x_i^k), \qquad (1)$$

To keep more information in features, we build a non-linear embedding layer $g_q$ after encoders $E_q$, and then conduct contrastive learning using these embedding representations,

$$z_i^q = g_q(f_i^q), \quad z_i^k = g_k(f_i^k), \qquad (2)$$

where $z_i^q$ and $z_i^k$ are the embedding representations of image $x_i^q$ and $x_i^k$ respectively.

To better optimize the embedding representations from contrastive learning, we follow the idea of [9] that expands the set of negative samples from preceding mini-batches by create a queue. The queue has a capacity of 4096, and will be filled up after first few iterations. Afterwards, the new embeddings in the later mini-batches are added to the queue (enqueue), meanwhile, the earliest embeddings are removed (dequeue). To optimize the embedding representations from contrastive learning, we adopt the InfoNCE loss [4, 9, 20]

$$\mathcal{L}_z = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(z_i^q \cdot z_i^k / \tau_z)}{\sum_{z_j^k \in \mathcal{Z}} \exp(z_i^q \cdot z_j^k / \tau_z)}, \qquad (3)$$

where $N$ indicates the size of mini-batch, $\mathcal{Z} = \{z_i^k\} \cup \{z\}$, $z$ represents embedding stored in the queue, $\tau_z$ is the temperature parameter.

## 3.3. Memory module for consistent feature

As outliers are significantly deviated from inliers, they should be separated according to the distance in the feature space. We need a consistent feature to represent inliers, which can be considered as the cluster center of sample features in one class, called prototypes. Therefore, we design
a memory module which maintain explicit memory slots to store the renewable prototypes as shown in the memory model block of Fig. 2. The features of inputs are constrained to be similar with their corresponding prototypes.

**Memory relevancy prediction.** To compute a memory prototype, we need to predict the relationship between a sample and each cluster center. Then, we add a clustering layer, $h(\cdot)$, after the embedding layer, $g(\cdot)$, which is a fully connected layer for relation estimation. Its output is a probability,

$$c_i^q = h_q(z_i^q), \quad c_i^k = h_k(z_i^k), \qquad (4)$$

where $c_i^q = \{c_{ij}^q\}$ and $c_i^k = \{c_{ij}^k\}$, $j = 1...K$ represent the probabilities of the samples $x_i^q$ and $x_i^k$ belonging to the $j_{th}$ memory prototype, respectively. We call them as the memory relevancy prediction. $K$ is the number of memory prototypes. A sample is expected to be close to its corresponding prototype, but distant from other prototypes. In other words, the sample should locate in one cluster, but not other clusters. This can be achieved by InfoNCE loss using $K$ predictions as samples,

$$\mathcal{L}_c = -\frac{1}{K} \sum_{i=1}^{K} \log \frac{\exp(v_i^q \cdot v_i^k / \tau_c)}{\sum_{j=1}^{K} \exp(v_i^q \cdot v_j^k / \tau_c)}, \quad (5)$$

where $v_i^q \in \{c_{ij}^q\}$, $v_i^k \in \{c_{ij}^k\}$, $j = 1...K$. $\tau_c$ is a temperature parameter. $v_i^q$ and $v_i^k$ are normalized before the optimization.

Moreover, we add a regularization to prevent the model falling into a local minimum where the majority gathers to the minority clusters [37]. It is formulated as

$$\mathcal{L}_r = \frac{1}{N} \sum_{j=1}^{K} \left( \sum_{i=1}^{N} c_{ij}^{q} \right)^2, \qquad (6)$$

**Memory reading.** To ensure the consistency of features, we need to minimize the distance between the sample feature and its corresponding memory prototype. To this end, we apply a reading operation with assistance of the memory relevancy prediction.

Given memory prototypes $M = \{m_1, ..., m_j\} \in \mathbb{R}^{K \times d}$, finding the most similar prototype can be achieved by a hard assignment memory reading $r(\cdot)$,

$$\hat{f}_i^q = r(c_{ij}^q, m_j) = \arg \max_{m_j \in M} (c_{ij}^q), \qquad (7)$$

where $\hat{f}_i^q$ is the read memory prototype of sample $x_i^q$ from memory slots, and $c_{ij}^q$ is the memory relevancy prediction between $\hat{f}_i^q$ and $m_j$. Considering that samples usually share

certain features across a few relevant clusters, we adopt a soft assignment memory reading,

$$\hat{f}_i^q = r(c_{ij}^q, m_j) = \sum_{j=1}^{K} \varphi(c_{ij}^q) m_j, \qquad (8)$$

where $\varphi(\cdot)$ is the softmax function.

In training process, our goal is to minimize the distance between $\hat{f}_i^q$ and $\hat{f}_i^q$ under current memory prototypes and cluster prediction. The feature consistency loss based on the Euclidean distance is

$$\mathcal{L}_m = \frac{1}{N} \sum_{i=1}^{N} \|\hat{f}_i^q - f_i^q\|^2, \qquad (9)$$

By this way, $\mathcal{L}_m$ can push $f_i^q$ closer to its corresponding memory prototype.

**Memory writing.** To find better consistent features, memory prototypes should be updated during the training. Thus, we employ memory writing operation to generate new memory prototypes according to the features and their cluster predictions stored in the queue. The writing operation, $w(\cdot)$, is

$$m_j = w(f_i^{queue}, c_{ij}^{queue}) = \sum_{i=1}^{N_Q} \varphi(c_{ij}^{queue}) f_i^{queue}, \quad (10)$$

where $N_Q = 4096$ is the number of features stored in the queue. We update the memory in the queue at each training iteration, which can keep memory prototypes more stable.

**Memory forgetting.** Due to the training data are mixed with outliers, the learned memory prototypes may also include outlier features and result in a worse performance. Then, we propose a memory forgetting operator to add perturbations to the memory prototypes using the attention mechanism. The attention weight of each memory prototype is determined by the number of related support samples.

$$n_j = \sum_{i=1}^{N_Q} \mathbb{I}_{\max}(c_{ij}), \qquad (11)$$

where $\mathbb{I}_{\max}(\cdot)$ is 1 when $c_{ij}$ is the maximal output value for sample $x_i$, otherwise is 0. $n_j$ is the number of the support samples of the $j_{th}$ memory prototype, and $n_1 + n_2 + ... + n_K = N_Q$. The smaller $n_j$ is, the more likely the prototype belongs to outliers. Thus, we add the Gaussian noise to suppress the these prototypes in the memory slots. The forgetting operation is formulated as

$$m_j = m_j + \eta_j, \qquad (12)$$

where $\eta_j \sim \mathcal{N}(0, \sigma_j)$ and $\sigma_j = 1 - n_j/N_Q$.

The memory forgetting operation can force the outliers to keep distant from memory prototypes.
**Learning and inference.** The overall objective function of MCOD can be formulated as

$$ \mathcal{L} = \mathcal{L}_z + \mathcal{L}_c + \mathcal{L}_m + \lambda \mathcal{L}_r, \quad (13) $$

where $\lambda$ is a weight parameter of $\mathcal{L}_r$.

In the training process, $E_q$ plays as a general network, and $E_k$ plays as a momentum network. Their parameters are gradually updated as

$$ \theta^k := \alpha \theta^k + (1 - \alpha) \theta^q. \quad (14) $$

where $\theta^q$ and $\theta^k$ are the parameters set of ($E_q, g_q, h_q$) and ($E_k, g_k, h_k$) respectively. $\alpha \in [0, 1)$ is momentum coefficient which is empirically set to be $\alpha = 0.999$ in all our experiment. The training procedure of MCOD is summarized in Algorithm 1.

In the inference process, the reading operation selects the corresponding memory prototypes of samples. The outliers can be detected according to the distance between their feature and the read memory prototype.

**Algorithm 1: Training algorithm of MCOD**

**Input:** Training samples $X \in \mathcal{X}$, the number of memory prototypes $K$, temperature parameters $\tau_z$ and $\tau_c$

**Output:** A trained MCOD with parameters sets $\Theta$ for each iteration do

**Step1:** Choose a random mini-batch of samples and generate augmentations for the samples;

**Step2:** Calculate $\mathcal{L}_z, \mathcal{L}_c$ and $\mathcal{L}_r$ by the outputs $z$ and $c$ using Eq. 3, Eq. 5 and Eq. 6;

**Step3:** Add current $f_q, z_q$ and $c_q$ into the queue and remove the earliest ones;

**Step4:** Conduct reading operation $r(\cdot)$ based on current memory prototypes $M$ using Eq. 8 and calculate $\mathcal{L}_m$ in Eq. 9;

**Step5:** Conduct writing operation $w(\cdot)$ using Eq. 10 to update memory prototypes $M$;

**Step6:** Conduct forgetting operation defined in Eq. 12.

end

# 4. Experiment

To demonstrate the superiority of our MCOD, we compare the MCOD with the baselines OC-SVM [27] and nine state-of-the-arts including CAE [19], DRAE [33], DSEBM [36], RDAE [38], MOGAL [17], RSRAE [12], RSRAE+ [12] SSD-IF [30] and E3Out [30]. The experiments are conducted on four benchmarks: MNIST [13], Fashion-MNIST [34], CIFAR10 [11], and CIFAR100 [11]. MCOD is implemented on NVIDIA GTX 2080.

## 4.1. Dataset

MNIST [13] contains 70K gray-scale images of ten digits from “0” to “9” size of 28 x 28.

Fashion-MNIST(F-MNIST) [34] contains 70K clothing and accessories gray-scale images with size of 28 x 28.

CIFAR10 [11] contains 60K images with size of 32x32, which are grouped into ten classes.

CIFAR100 [11] contains the same images in CIFAR10, but has 100 fine-grained classes or 20 coarse-grained classes. Following the settings in papers [30, 31]. we select 20 coarse-grained classes in our experiment.

## 4.2. Evaluation metrics

To evaluate the outlier detection performance, we adopt the Area under the Receiver Operating Characteristic curve (AUROC) and Area under the Precision-Recall curve (AUPR) as evaluation metrics [2]. The higher AUROC and AUPR represent a better performance. The AUPR scores usually are computed in following ways: taking inliers as positive samples (AUPR-IN) and taking outliers as positive samples (AUPR-OUT). All experiments are performed independently for five times and report the average results.

## 4.3. Implementation details

To test the effectiveness of our MCOD, we follow the standard experimental setting in previous studies [31, 33, 38]. Taking the images from the same semantical category as inliers and the random samples from other categories as outliers. We carry out the cross-validation, each category plays as inliers in turn, and report the average performance.

We use Wide ResNet (WRN) [35] as our backbone network with the widen factor as 4. The dimensions of encoded features and embedding representations are 256 and 128, respectively. The model is optimized by Adam optimizer with $\beta = (0.9; 0.999)$. The mini-batch size is 256. The learning rate is 0.0001 and the weight decay is 0.0005. For MNIST and F-MNIST, the temperature parameters $\tau_z$ and $\tau_c$ are both set as 1. For CIFAR10 and CIFAR100, they are set as 10 and 0.01, respectively. $\lambda$ is 0.05 in all experiments. By default, the number of prototypes stored in the memory is 10. In the training process, we warm-up our network without using memory module in first 100 epochs, afterwards, include memory in later 100 epochs.

## 4.4. Performance comparison

We compare the MCOD with the state-of-the-arts including seven AE based methods: CAE [19], DRAE [33], DSEBM [36], RDAE [38], MOGAL [17], RSRAE [12], RSRAE+ [12] and two self-supervised based methods: SSD-IF [30] and E3Out [30]. SSD-IF shares the same self-supervised discriminating network with E3Out, but uses the Isolation Forest (IF) [16] as detector. The proportions of
<table><caption>Table 1. Comparison with nine SOTA methods on four datasets with two different proportions of outliers (p = 0.1 and p = 0.2). The best performance is shown in bold font.</caption><thead><tr><th>Dataset</th><th>p</th><th>CAE [19]</th><th>DRAE [33]</th><th>DSEBM [36]</th><th>RDAE [38]</th><th>MOGAL [17]</th><th>RSRAE [12]</th><th>RSRAE+ [12]</th><th>SSD-IF [30]</th><th>E3Out [30]</th><th>MCOD</th></tr></thead><tbody><tr><td colspan="12" style="text-align:center; font-weight:bold;">AUROC%</td></tr><tr><td>MNIST</td><td>0.1</td><td>68.0±0.7</td><td>66.9±2.6</td><td>60.5±8.5</td><td>71.8±1.0</td><td>30.9±1.4</td><td>84.8±0.9</td><td>73.9±1.4</td><td>93.8±0.4</td><td>94.9±0.1</td><td><strong>96.2±2.1</strong></td></tr><tr><td>MNIST</td><td>0.2</td><td>64.0±1.0</td><td>67.3±2.1</td><td>56.3±6.5</td><td>67.0±0.7</td><td>37.8±2.4</td><td>78.9±1.7</td><td>74.6±0.8</td><td>90.5±0.0</td><td>92.9±0.1</td><td><strong>94.1±3.3</strong></td></tr><tr><td>F-MNIST</td><td>0.1</td><td>70.3±0.9</td><td>67.1±1.8</td><td>53.2±3.0</td><td>75.3±1.2</td><td>22.8±1.3</td><td>78.3±2.0</td><td>84.6±1.4</td><td>91.0±0.0</td><td>93.5±0.1</td><td><strong>98.7±0.6</strong></td></tr><tr><td>F-MNIST</td><td>0.2</td><td>64.4±1.8</td><td>65.7±1.9</td><td>53.1±5.6</td><td>70.9±1.1</td><td>34.0±2.9</td><td>74.5±3.1</td><td>82.1±1.3</td><td>87.7±0.0</td><td>92.1±0.1</td><td><strong>94.8±0.9</strong></td></tr><tr><td>CIFAR10</td><td>0.1</td><td>55.8±1.4</td><td>56.0±0.1</td><td>60.2±1.2</td><td>55.4±0.6</td><td>56.2±0.7</td><td>56.6±1.1</td><td>55.5±2.6</td><td>64.3±0.0</td><td>84.7±0.2</td><td><strong>87.6±4.1</strong></td></tr><tr><td>CIFAR10</td><td>0.2</td><td>54.7±1.2</td><td>55.6±0.3</td><td>61.4±0.4</td><td>54.2±0.9</td><td>55.7±1.3</td><td>55.6±1.7</td><td>57.4±1.2</td><td>58.9±0.1</td><td>80.3±0.2</td><td><strong>85.3±1.7</strong></td></tr><tr><td>CIFAR100</td><td>0.1</td><td>51.2±0.4</td><td>51.0±0.6</td><td>59.2±1.0</td><td>55.6±0.8</td><td>53.2±1.0</td><td>57.1±0.8</td><td>57.7±0.5</td><td>56.4±0.0</td><td>81.3±0.2</td><td><strong>87.7±4.2</strong></td></tr><tr><td>CIFAR100</td><td>0.2</td><td>54.4±0.9</td><td>55.5±0.3</td><td>57.9±1.0</td><td>54.9±0.6</td><td>53.1±1.5</td><td>56.3±0.7</td><td>56.2±0.8</td><td>53.2±0.1</td><td>79.1±0.2</td><td><strong>84.7±1.2</strong></td></tr><tr><td colspan="12" style="text-align:center; font-weight:bold;">AUPR-IN %</td></tr><tr><td>MNIST</td><td>0.1</td><td>92.0±0.3</td><td>93.0±0.9</td><td>91.6±2.3</td><td>93.1±0.3</td><td>78.8±3.4</td><td>97.4±0.2</td><td>95.7±0.4</td><td>99.2±0.2</td><td><strong>99.4±0.0</strong></td><td>99.4±0.5</td></tr><tr><td>MNIST</td><td>0.2</td><td>82.7±0.5</td><td>86.6±1.2</td><td>81.2±2.8</td><td>89.2±0.7</td><td>70.6±1.2</td><td>91.3±0.9</td><td>91.1±0.3</td><td>97.3±0.0</td><td><strong>98.1±0.0</strong></td><td>97.9±1.4</td></tr><tr><td>F-MNIST</td><td>0.1</td><td>94.3±0.3</td><td>93.9±0.4</td><td>88.9±0.8</td><td>95.8±0.3</td><td>74.8±1.5</td><td>96.2±0.4</td><td>97.5±0.3</td><td>98.5±0.0</td><td>99.0±0.0</td><td><strong>99.8±0.1</strong></td></tr><tr><td>F-MNIST</td><td>0.2</td><td>85.3±0.8</td><td>86.9±0.9</td><td>79.6±2.4</td><td>89.2±0.7</td><td>66.6±1.4</td><td>90.4±1.2</td><td>93.4±0.7</td><td>95.5±0.0</td><td>97.4±0.1</td><td><strong>98.4±0.5</strong></td></tr><tr><td>CIFAR10</td><td>0.1</td><td>91.0±0.2</td><td>90.7±0.1</td><td>92.3±0.3</td><td>90.7±0.2</td><td>91.1±0.2</td><td>91.4±0.2</td><td>91.2±0.6</td><td>93.6±0.1</td><td>97.7±0.0</td><td><strong>98.0±0.9</strong></td></tr><tr><td>CIFAR10</td><td>0.2</td><td>81.6±0.7</td><td>81.7±0.2</td><td>85.2±0.1</td><td>81.0±0.4</td><td>82.0±0.6</td><td>82.1±0.8</td><td>83.4±0.6</td><td>84.1±0.0</td><td>93.5±0.1</td><td><strong>94.2±1.3</strong></td></tr><tr><td>CIFAR100</td><td>0.1</td><td>90.3±0.1</td><td>90.9±0.3</td><td>92.2±0.3</td><td>90.9±0.3</td><td>90.4±0.4</td><td>91.6±0.2</td><td>91.9±0.1</td><td>91.6±0.0</td><td>97.2±0.0</td><td><strong>98.1±0.8</strong></td></tr><tr><td>CIFAR100</td><td>0.2</td><td>81.7±0.5</td><td>81.8±0.2</td><td>83.7±0.6</td><td>81.5±0.3</td><td>80.9±0.6</td><td>82.7±0.3</td><td>83.0±0.5</td><td>81.8±0.0</td><td>93.1±0.1</td><td><strong>95.0±0.8</strong></td></tr><tr><td colspan="12" style="text-align:center; font-weight:bold;">AUPR-OUT %</td></tr><tr><td>MNIST</td><td>0.1</td><td>32.9±0.7</td><td>30.5±2.0</td><td>23.0±5.3</td><td>35.8±0.8</td><td>15.2±3.8</td><td>45.4±2.2</td><td>30.6±1.5</td><td>68.4±0.5</td><td>71.0±0.2</td><td><strong>85.5±5.1</strong></td></tr><tr><td>MNIST</td><td>0.2</td><td>40.7±1.0</td><td>42.5±2.3</td><td>32.3±6.0</td><td>43.2±0.8</td><td>28.0±2.7</td><td>53.0±1.6</td><td>47.8±1.4</td><td>70.1±0.0</td><td>76.3±0.1</td><td><strong>88.5±4.9</strong></td></tr><tr><td>F-MNIST</td><td>0.1</td><td>29.3±0.1</td><td>25.5±1.1</td><td>19.7±2.3</td><td>31.7±1.4</td><td>14.8±2.1</td><td>37.0±4.5</td><td>55.0±2.4</td><td>70.1±0.0</td><td>77.2±0.1</td><td><strong>90.4±3.7</strong></td></tr><tr><td>F-MNIST</td><td>0.2</td><td>36.8±1.3</td><td>36.6±2.5</td><td>31.7±4.8</td><td>41.4±0.9</td><td>28.3±2.9</td><td>46.3±5.1</td><td>63.5±1.3</td><td>71.5±0.0</td><td>82.0±0.2</td><td><strong>82.6±2.0</strong></td></tr><tr><td>CIFAR10</td><td>0.1</td><td>14.4±0.7</td><td>14.7±0.1</td><td>14.7±0.5</td><td>14.9±0.3</td><td>13.6±0.4</td><td>14.0±0.6</td><td>13.1±1.0</td><td>18.8±0.0</td><td>45.7±0.3</td><td><strong>62.3±6.4</strong></td></tr><tr><td>CIFAR10</td><td>0.2</td><td>25.5±0.6</td><td>26.8±0.2</td><td>27.8±0.3</td><td>25.7±0.7</td><td>25.0±0.9</td><td>25.8±1.2</td><td>25.3±0.8</td><td>27.7±0.0</td><td>54.5±0.2</td><td><strong>73.0±5.7</strong></td></tr><tr><td>CIFAR100</td><td>0.1</td><td>14.5±0.6</td><td>15.0±0.1</td><td>16.2±0.5</td><td>15.0±0.4</td><td>12.6±0.5</td><td>14.1±0.4</td><td>14.5±0.3</td><td>13.4±0.0</td><td>37.0±0.4</td><td><strong>61.4±5.5</strong></td></tr><tr><td>CIFAR100</td><td>0.2</td><td>25.6±0.6</td><td>27.0±0.2</td><td>27.8±0.6</td><td>26.5±0.4</td><td>24.4±0.9</td><td>25.2±0.4</td><td>25.8±0.5</td><td>23.1±0.0</td><td>49.9±0.3</td><td><strong>66.3±2.8</strong></td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>Figure 3. Comparison of changes of AUROC (% vertical axis) with varied outlier proportion p (horizontal axis) from 0.05 to 0.25. The step is 0.05.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 4. T-SNE visualization of learned features of digits 1 to 10 in MNIST (first line) and class 1 to 10 in F-MNIST (second line). Blue and red dots represent inliers and outliers in the feature space.</figcaption></figure>
outliers in datasets are set to be $p = 0.1$ and $p = 0.2$. The numerical results of AUROC, AUPR-IN and AUPR-OUT are listed in Table 1.

We can see that our MCOD outperforms most other methods on all datasets in terms of all evaluation metrics. Specially, when $p = 0.1$, MCOD achieves a considerable improvement on AUROC (MNIST 28.2%, F-MNIST 28.4%, CIFAR10 31.8% and CIFAR100 36.5%) compared with CAE. For the simpler dataset MNIST, MCOD outperforms six AE based methods (DRAE 29.3%, DSEBM 35.7%, RDAE 24.4%, MOGAL 65.3%, RSRAE 11.4%, RSRAE+ 22.3%). For complicated datasets (CIFAR10 and CIFAR100), our superiority becomes more significant. These results demonstrate that learning consistent and discriminating features does facilitate the performance of outlier detection.

Compared to self-supervised based methods, in terms of the metric AUROC and $p = 0.1$, MCOD outperforms SSD-IF (MNIST 2.4%, F-MNIST 7.7%, CIFAR10 23.3% and CIFAR100 31.3%) and E3Out (MNIST 1.3%, F-MNIST 5.2%, CIFAR10 2.9% and CIFAR100 6.4%), respectively. Although it performs slightly worse than E3Out on MNIST in terms of AUPR-IN, MCOD achieves better scores on F-MNIST, CIFAR10, and CIFAR100. E3Out aims at recognizing the types of sample augmentations, and assumes that the augmentations of inlier samples could be recognized to their corresponding types with a high confidence, but the augmentations of outlier samples could not. Such mechanism of recognizing different types of sample augmentations makes the learned features of outlier and inlier less discriminating. For the simpler dataset MNIST, E3Out achieves a higher AUPR-IN score, but a much lower AUPR-OUT score. For the complicated datasets CIFAR10 and CIFAR100, E3Out commonly performs worse than MCOD in terms of both AUPR-IN and AUPR-OUT scores. This indicates that E3Out is inclined to recognize the simple outliers as inliers and mistakes complicated inlier data as outliers.

Unlike AUPR-IN, AUPR-OUT chooses outliers as positive samples, which is a more intuitive metric of outlier detection. MCOD outperforms all other methods, especially, is better than E3Out with remarkable 14.5% and 12.2% gains on MNIST. When $p = 0.2$, the performances of CAE, DSEBM, RDAE, and RSRAE drop dramatically. By contrast, the performance of MCOD is still stable. This significant advantage on AUPR-OUT demonstrates the superiority of our MCOD on learning more discriminating features of outliers. Other methods focus on learning a better fitting model for inlier because inlier is the majority, but consider less on the discrimination of features. By contrast, our MCOD takes both sides into account. The memory model learns the consistent feature to fit inliers. The contrastive learning model forces the features of inlier and outlier more distinct. Therefore, MCOD demonstrates more

<figure><img src="image_7.png" /><figcaption>Figure 5. AUROC (% vertical axis) with (W/Forg.) and without forgetting (W/O Forg.) operations with varied outlier proportion $p$ (horizontal axis).</figcaption></figure>

performance gains on AUPR-OUT against AUPR-IN.

Figure 10 shows the visualization of learned features of inliers and outliers in MNIST and F-MNIST by t-SNE [18]. We can see that outliers are grouped together and keep a distance to inliers in most cases. This intuitive result illustrates the effectiveness of our MCOD. The features of inliers are consistent, meanwhile, the features of outliers are distinct from inliers. It also proves that the outlier detection can be done in the feature space.

## 4.5. Varied proportions of outliers

As outlier proportion $p$ can severely affect the performances of OD methods, we test MCOD in terms of AUROC with varied $p = [0.05, 0.1, 0.15, 0.2, 0.25]$. We also compare it with eight competing methods, including two baselines (OC-SVM and MOCO-OC), four AE-based methods (CAE, CAE-IF, DRAE, RDAE), and two self-supervised methods (SSD-IF and E3Out). OC-SVM utilizes the raw data as input. Instead, MOCO-OC uses features learned by MoCo [9] as the input for OC-SVM. CAE-IF uses IF as the outlier detector. The AUROC curves of performance are shown in Fig. 3.

We can see our MCOD demonstrates an obvious superiority on most outlier proportions, except that E3Out and SSD-IF show a better AUROC score only when $p = 0.05$. With the outlier proportion increasing, the gap between MCOD and others is gradually enlarged. On MNIST and F-MNIST, the comparable methods are E3Out and SSD-IF. On complicated datasets CIFAR10 and CIFAR100, only E3Out is comparable with MCOD. This result indicates that self-supervised methods are superior to AE based methods. And the contrastive learning can contribute
<figure><img src="image_8.png" /><figcaption>Figure 6. AUROC, AUPR-IN, AUPR-OUT (% vertical axis) with varied number of memory prototypes (horizontal axis) [5, 10, 20, 30, 50].</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 7. AUROC (% vertical axis) with varied temperature parameters $\tau_z$ (horizontal axis) and $\tau_c$ (different color bins) [0.01, 0.05, 0.1, 0.5, 1, 10, 100].</figcaption></figure>

more performance gain. The possible reason could be following. As $p$ increases, outliers (randomly sampled data) are more likely to join to AE model training. Thus, AE based methods result in a better outlier reconstruction. For the self-supervised method, E3Out is based on the accuracy of recognizing the types of sample augmentations. With more outliers being involved, E3Out may predict the type of outlier augmentation with a less uncertainty (higher confidence), which degrade the performance of outlier detection. Although our MCOD also faces more outliers, the memory model constrains the consistency of inlier features. It makes MCOD detect outlier easier by measuring the distance between samples and their corresponding prototypes.

## 4.6. Ablation study

**Evaluation of memory forgetting operation.** Since the forgetting operation plays a crucial role of suppressing the outlier prototypes in memory slots, we report the AUROC scores with (W/ Forg.) and without forgetting operations (W/O Forg.) when vary $p$ from 0.05 to 0.5 in Fig. 5. We can see MCOD (W/ Forg.) shows a significant superiority than MCOD (W/O Forg.) on more complicated datasets CIFAR10 and CIFAR100 when $p < 0.3$. But MCOD (W/ Forg.) quickly downgrades when $p \ge 0.3$, because it breaks the assumption that outliers are minority. For the simpler datasets MNIST and F-MNIST, we can observe a similar trend. Since images in MNIST and F-MNIST are simpler, the memory model is easy to learn the consistent features of inliers. Thus, our proposed forgetting operator demonstrates less advantages. Images in CIFAR10 and CIFAR100 are complex and diverse. Such random outliers in the data disrupt the consistent features learning. However, our proposed forgetting operation can alleviate this influence, and demonstrates an obvious performance gain on CIFAR10 and CIFAR100 when $p < 0.3$.

**Evaluation on varied memory prototype numbers.** Since the memory prototypes is essential for learning the consistent feature of inlier, we report performance of MCOD with varied number of memory prototypes [5, 10, 20, 30, 50] in terms of AUROC, AUPR-IN, and AUPR-OUT on MNIST, F-MNIST, CIFAR10, and CIFAR100 dataset. The experiments is conducted with outliers proportion $p = 0.1$. Figure 6 illustrates that MCOD is insensitive to the change of prototype numbers. Considering both performances and efficiency, we select the prototypes number as 10.

**Evaluation on varied temperature parameters.** We report the AUROC scores with different temperature parameters $\tau_z$ and $\tau_c$ in Eq. 3 and 5. Each of them is set as [0.01, 0.05, 0.5, 1, 10, 100] and the results are reported in Fig. 7. According to this result, we set $\tau_z = 1$ and $\tau_c = 1$ for MNIST and F-MNIST, and $\tau_z = 10$ and $\tau_c = 0.01$ for CIFAR10 and CIFAR100.

## 5. Conclusion

In this paper, we proposed an unsupervised outlier detection framework using a memory module with specially designed reading, writing, and forgetting operations and a contrastive learning module. Unlike conventional AE based methods detecting outliers according to the reconstruction errors in the pixel space, our proposed MCOD distinguishes outliers and inliers in the feature space. The experimental results on four benchmark datasets, which have various data distributions, have demonstrated the superiority of our MCOD compared to nine state-of-the-art methods. The de-
tailed analysis shows that the memory module does learn
consistent features for inliers, and the contrastive learning
module extracts the discriminating features to separate out-
liers from inliers. Our future work will investigate prior
features of inliers to strengthen the feature consistency.

References

[1] Hervé Bourlard and Yves Kamp. Auto-association by multi-layer perceptrons and singular value decomposition. *Biological cybernetics*, 59(4-5):291-294, 1988. 1

[2] Kendrick Boyd, Kevin H Eng, and C David Page. Area under the precision-recall curve: point estimates and confidence intervals. In *ECML-PKDD*, pages 451–466. Springer, 2013. 5

[3] Raghavendra Chalapathy and Sanjay Chawla. Deep learning for anomaly detection: A survey. preprint arXiv:1901.03407, 2019. 1, 2

[4] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. preprint arXiv:2003.04297, 2020. 3

[5] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In CVPR, volume 1, pages 539–546, 2005. 2

[6] Simao Eduardo, Alfredo Nazábal, Christopher KI Williams, and Charles Sutton. Robust variational autoencoders for outlier detection and repair of mixed-type data. In AISTATS, pages 4056–4066, 2020. 2

[7] Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In *ICCV*, pages 1705–1714, 2019. 2

[8] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. preprint arXiv:1410.5401, 2014. 2

[9] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9729–9738, 2020. 2, 3, 7

[10] Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. *IEEE TPAMI*, 2020. 2

[11] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 5

[12] Chieh-Hsin Lai, Dongmian Zou, and Gilad Lerman. Robust subspace recovery layer for unsupervised anomaly detection. preprint arXiv:1904.00152, 2019. 2, 5, 6

[13] Yann LeCun, LD Jackel, Leon Bottou, A Brunot, Corinna Cortes, J Denker, Harris Drucker, I Guyon, UA Muller, Edvard Sackinger, et al. Comparison of learning algorithms for handwritten digit recognition. In ICANN. 5

[14] Sangho Lee, Jinyoung Sung, Youngjae Yu, and Gunhee Kim. A memory network approach for story-based temporal summarization of 360 videos. In CVPR, pages 1410–1419, 2018. 2

[15] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. preprint arXiv:1706.02690, 2017. 1

[16] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In *ICDM*, pages 413–422, 2008. 5

[17] Yezheng Liu, Zhe Li, Chong Zhou, Yuanchun Jiang, Jian-
shan Sun, Meng Wang, and Xiangnan He. Generative ad-
versarial active learning for unsupervised outlier detection.
*IEEE TKDE*, 2019. 5, 6

[18] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. *JMLR*, 9(Nov):2579–2605, 2008. 7

[19] Jonathan Masci, Ueli Meier, Dan Cireșan, and Jürgen Schmidhuber. Stacked convolutional auto-encoders for hierarchical feature extraction. In *ICANN*, 5, 6

[20] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. preprint arXiv:1807.03748, 2018. 3

[21] Hyunjong Park, Jongyoun Noh, and Bumsub Ham. Learning memory-guided normality for anomaly detection. In CVPR, pages 14372–14381, 2020. 2

[22] Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. Ogan: One-class novelty detection using gans with constrained latent representations. In CVPR, pages 2898–2906, 2019. 1

[23] Lukas Ruff, Robert A. Vandermeulen, Nico Görnitz, Lucas Deecke, Shoaib A. Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In ICML, volume 80, pages 4393–4402, 2018. 2

[24] Mohammad Sabokrou, Mohammad Khalooei, Mahmood Fathy, and Ehsan Adeli. Adversarially learned one-class classifier for novelty detection. In CVPR, pages 3379–3388, 2018. 1

[25] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. One-shot learning with memory-augmented neural networks. preprint arXiv:1605.06065, 2016. 2

[26] Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In IPMI. 1

[27] Bernhard Schölkopf, John C. Platt, John Shawe-Taylor, Alex J. Smola, and Robert C. Williamson. Estimating the support of a high-dimensional distribution. Neural Computation, 13(7):1443–1471, 2001. 5

[28] Bernhard Schölkopf, Robert C Williamson, Alexander J Smola, John Shawe-Taylor, John C Platt, et al. Support vector method for novelty detection. In NIPS, volume 12, pages 582–588, 1999. 2

[29] David M. J. Tax and Robert P. W. Duin. Support vector data description. *Mach. Learn.*, 54(1):45–66, Jan 2004. 2

[30] Siqi Wang, Yijie Zeng, Xinwang Liu, Sihang Zhou, En Zhu, Marius Kloft, Jianping Yin, and Qing Liao. Self-supervised deep outlier removal with network uncertainty and score refinement. 2020. 2, 5, 6

[31] Siqi Wang, Yijie Zeng, Xinwang Liu, En Zhu, Jianping Yin, Chuanfu Xu, and Marius Kloft. Effective end-to-end unsupervised outlier detection via inlier priority of discriminative network. In NIPS, pages 5962–5975, 2019. 5

[32] Ziming Wang, Yuexian Zou, and Zeming Zhang. Cluster attention contrast for video anomaly detection. In *ACMM*, pages 2463–2471, 2020. 1
[33] Yan Xia, Xudong Cao, Fang Wen, Gang Hua, and Jian Sun. Learning discriminative reconstructions for unsupervised outlier removal. In *ICCV*, pages 1511–1519, 2015. 1, 2, 5, 6

[34] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. *preprint arXiv:1708.07747*, 2017. 5

[35] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. *preprint arXiv:1605.07146*, 2016. 5

[36] Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang. Deep structured energy based models for anomaly detection. *preprint arXiv:1605.07717*, 2016. 5, 6

[37] Huasong Zhong, Chong Chen, Zhongming Jin, and Xian-Sheng Hua. Deep robust clustering by contrastive learning. *preprint arXiv:2008.03030*, 2020. 2, 4

[38] Chong Zhou and Randy C Paffenroth. Anomaly detection with robust deep autoencoders. In *ACM SIGKDD*, pages 665–674, 2017. 2, 5, 6

[39] Xun Zhou, Sicong Cheng, Meng Zhu, Chengkun Guo, Sida Zhou, Peng Xu, Zhenghua Xue, and Weishi Zhang. A state of the art survey of data mining-based fraud detection and credit scoring. *MATEC Web of Conferences*, 189:03002, 01 2018. 1

[40] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dmgan: Dynamic memory generative adversarial networks for text-to-image synthesis. In *CVPR*, pages 5802–5810, 2019. 2

[41] Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencodding gaussian mixture model for unsupervised anomaly detection. In *ICLR*, 2018. 2
# Supplementary

## A. Evaluation on more datasets

To evaluate the robustness of MCOD, we test it on two more challenging datasets: STL-10 and Tiny Imagenet. They both have less training data compared to the four datasets in main manuscript. Moreover, the inliers and outliers in Tiny Imagenet are much more diverse, which raises the difficulty of outlier detection.

STL-10 contains 500 training images, 800 test images per class (10 classes) with size of 96 × 96. In our experiment, we combine training and test images together to get 1.3K images per class. The images from the same semantic category are taken as inliers and the random samples from other categories are considered as outliers.

Tiny Imagenet contains 200 classes of images from a distinct subset of Imagenet with size of 64 × 64. We select the first 20 categories with 500 images per category as inliers respectively. The outliers are from other 199 categories besides of the current inliers category.

The data setting is as the same as that in the main manuscript. The temperature parameters $\tau_z$ and $\tau_c$ are set as 10 and 0.01 for both STL-10 and Tiny Imagenet. Other implementation details are set as the same as in Section 4.3 of the main manuscript. We run each scenario 5 times and report the average results in table 2, when $p = 0.1$ and $p = 0.2$. MCOD demonstrates a considerable effectiveness on both datasets. In terms of AUROC, MCOD achieves 94.8% ($p = 0.1$) and 89.0% ($p = 0.2$) on STL-10. For more complicated outliers MCOD gets 81.8% ($p = 0.1$) and 79.2% ($p = 0.2$) and 81.8% on Tiny Imagenet. When the proportion of outliers increases, the performance of MCOD on these two datasets decreases slightly (89.0% on STL-10 and 79.2% on Tiny Imagenet) which shows the consistent tendency as discussed in the main manuscript.

## B. The visualization of feature distribution of MNIST and F-MNIST

<table><caption>Table 2. Results of MCOD on STL-10 and Tiny Imagenet</caption><thead><tr><th>Dataset</th><th>p</th><th>AUROC%</th><th>AUPR-IN%</th><th>AUPR-OUT%</th></tr></thead><tbody><tr><td rowspan="2">STL-10</td><td>0.1</td><td>94.8</td><td>99.3</td><td>70.8</td></tr><tr><td>0.2</td><td>89.0</td><td>96.4</td><td>68.8</td></tr><tr><td rowspan="2">Tiny Imagenet</td><td>0.1</td><td>81.8</td><td>97.0</td><td>48.8</td></tr><tr><td>0.2</td><td>79.7</td><td>92.6</td><td>59.3</td></tr></tbody></table>

To illustrate that the learned features can be utilized to detect outliers, we calculate the differences between the extracted features of each inlier/outlier sample and the corresponding prototype read from memory defined in Eq. 9, and normalized the difference in a range [0, 1]. Bigger value means a smaller difference. The histogram of the difference distribution is plotted in Fig. 8 and Fig. 9, which have 100

bins. Each bin represents the number of outliers and inliers who have similar difference value. Inliers are shown in green bins and outliers are shown in red bins.

We can see an the obvious different distribution of outliers and inliers, which illustrates that it is feasible to detect outliers in the feature space. Moreover, one can see the distribution of outliers has a large variation, but the variation of inliers is rather small. The reason is that the outliers are from different classes, different from the memory prototypes. Thanks to the consistency learning, the learned features of inliers have a pretty high similarity.

## C. TSNE visualization on CIFAR10 and CIFAR100

The visualization of learned features on CIFAR10 and CIFAR100 are shown in Fig. 10. Specifically, (a) shows the visualizations of t-SNE of CIFAR10. 10 classes are given according to top-to-bottom and left-to-right order. (b) shows the visualizations of t-SNE of CIFAR100. 20 classes are given according to top-to-bottom and left-to-right order. Blue dots are inliers and red dots are outliers. In most cases, outliers can be well separated from inliers.
<figure><img src="image_10.png" /><figcaption>Figure 8. Feature difference distribution of ten classes in MNIST. The horizontal axis is the difference value and the vertical axis is the number of samples in each bin.</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>Figure 9. Feature difference distribution of ten classes in F-MNIST. The horizontal axis is the difference value and the vertical axis is the number of samples in each bin.</figcaption></figure>
<figure><img src="image_12.png" /><figcaption>Figure 10. T-SNE visualization of features $f$ on CIFAR10 (a) (from class 1 to class 10) and CIFAR100 (b) (from class 1 to class 20). Blue and red dots represent inliers and outliers in feature space.</figcaption></figure>