Fast Adaptive Anomaly Detection

Abstract: The ability to detect anomaly has long been recognized as an inherent human ability, yet to date, practical AI solutions to mimic such capability have been lacking.This lack of progress can be attributed to several factors.  To begin with, the distribution of “abnormalities” is intractable.   Anything outside of a given normal population is by definition an anomaly. This explains why a large volume of workin this area has been dedicated to modeling the normal distribution of a given task followed by detecting deviations from it. This direction is however unsatisfying as it would require modeling the normal distribution of every task that comes along, which includes tedious data collection.  In this paper, we report our work aiming to handle these issues. To deal with the intractability of abnormal distribution, we leverage Energy Based Model (EBM). EBMs learn to associates low energies to correct values and higher energies to incorrect values.  As its core, the EBM em-ploys Langevin Dynamics (LD) in generating these incorrect samples based on an iterative optimization procedure, alleviating the intractable problem of modeling the world of anomalies.  Then, in order to avoid training an anomaly detector for every task, we utilize an adaptive sparse coding layer. Our intention is to design a plug and play feature that can be used to quickly update what is normal during inference time. Lastly, to avoid tedious data collection, this mentioned update of the sparse coding layer needs to be achievable with just a few shots. Here, we employ a meta learning scheme that simulates such a few shot setting during training. We support our findings with strong empirical evidence.

Reviews and author responses:

 Dear reviewer **d5Wi** and reviewer **Mzn6**,

Thank you again for your time reviewing our paper. We have tried hard to respond to your comments. As the final deadline of the discussion period is approaching in one day, we would like to know if our responses have addressed your concerns. And please do not hesitate to let us know if you have any further concerns that we can use this last chance to address.

Thank you,

Authors


 Dear reviewers,

Once again, we would like to express our gratitude to all reviewers for their supportive and constructive feedback. 

We have addressed the following comments and updated our manuscript accordingly:
- As suggested by Reviewer **4fuZ**, we have performed experiments with contaminated data, and presented the results in Appendix Section B.3 in the revision.
- As suggested by Reviewer **egaq**, we have presented further discussions and results on incorporating temporal information to the video anomaly detection experiments, and presented the results in Appendix Section B.4 in the revision.
- We have clarified both the task novelty and the technical novelty of our paper in response to reviewer **Mzn6**’s comments. 
- We have presented further clarifications on the terms ‘Normal’ and ‘Abnormal’, and discussed the advantages compared to other methods including non-deep-learning methods in response to the comments of Reviewer **d5Wi**.
- We have corrected typos, improved the equations, and included explanations to the metrics as suggested by the reviewers. 

As the final deadline for the discussion period is approaching, we sincerely hope that reviewers will read our response again to see if there are any further concerns that we can address.

Thank you,

Authors


 We thank the feedback from the reviewer and we respectfully disagree with the reviewer regarding the novelty of our paper.

**Regarding the task novelty**, to the best of our knowledge, we are the first to address both anomaly detection and localization under the fast adaptation scenario. This new problem setting could be significant for practical applications and we demonstrate that in the paper with two challenging real-world datasets. For this novel task, we specifically show in Table 1 and Table 2 that, although our method is only adapted to each novel category of objects or scene with **a few samples**, it is able to deliver performance that is at least comparable to that of those methods trained with **massive samples**.

**Regarding technical novelty:**

- The term energy-based model (EBM) in our paper refers to a track of research that falls under probabilistic models that are attracting increasing attention in recent years [a,b,c]. These models utilize Langevin dynamics and Contrastive divergence as the standard ways to conduct optimization. Our method is the first to explore and extend such an EBM with fast adaptation capability, which allows the modeled density to rapidly adapt to new tasks / concepts.

- To the best of our knowledge, we are the first to present an adaptive dictionary that leverages meta-learning, and is able to achieve rapid dictionary updates through simple plugging in of features as the dictionary atoms. The smooth shrinkage function and the sparse coding with receptive fields are also new techniques that we propose to improve the sparse coding in deep models. (If the reviewer can suggest any related or similar works, we would be happy to discuss further.)

**Regarding the correctness of the paper**, we would appreciate it if the reviewer could provide more details about the claims that the reviewer considered are not well-supported so that we could have further discussions.


 We appreciate the constructive feedback from the reviewer. We have made significant efforts to address the reviewer concerns below. We hope the reviewer will take these into account when making a final decision regarding our paper.

**1. How the technique in Section 3.1 improves the robustness of the network on different types of objects**

We believe the concern is mainly referring to Section 3.1.1, where we enlarge the receptive field of the adaptive sparse coding layer to improve the robustness. As discussed in this section, for the feature extractor, a small receptive field hardly captures enough contextual information, while a large receptive field would make feature maps spatially coarse (hard to spot small anomaly regions). Thus applying a feature extractor without carefully tuning the receptive field of each layer might result in inconsistent performance. Instead, we enlarge the receptive field on the sparse coding layer (effectively capturing more contextual information while maintaining the ability to spot small anomaly regions) to make the model more robust when different objects are involved. 
 
**2. Typos**

We have fixed the typos in the revised version.

**3. The caption of Table 1**

Thanks for pointing this out. In fact, we consider the ‘Category’ column as the first column in Table 1, therefore Col 2-5 refer to performance with AE (SSIM), AE (MSE), AnoGAN, and VE-VAE, which covers the column of AE (SSIM). We hope this resolves the reviewer's concern.

**4. Leveraging temporal information**

We thank the reviewer for this constructive suggestion. The energy-based structure is agnostic to the network architectures and we could utilize other backbones (e.g. 3D CNN) to incorporate temporal information. We see no obstacle in applying spatial-temporal anomaly detection and will keep this as one of the potential future directions. We present preliminary results of leveraging temporal information in the revised **Appendix Section B.4**, where we utilize 3D convolutions with 5 frames as the input and achieve improved performance on 3 out of 4 video datasets. 

**5. Novelty of this paper**

The key innovation we are presenting in the paper is anomaly detection under the fast adaptation scenario. Regarding technical novelty, we are the first to extend the energy-base model with fast adaptation ability, which allows the learned density to quickly adapt to a new task without re-training the EBM.


 We thank the reviewer for the positive and constructive feedback, especially the suggestion about including contaminated training data.

In this paper, we follow the common practice in the field of anomaly detection in order to ensure fair comparisons are made, but we do agree with the reviewer that contaminated normal samples could be an issue for real-world use cases. To address this concern, we provide additional experiments with training data contaminated with 0 to 10% abnormal samples. For more details, please refer to **Appendix Section B.3**. Our method demonstrates robustness against a reasonable amount of data contamination. 

 We thank the reviewer for the constructive feedback as well as the recognition of the novelty and contribution of our work. We address the reviewer's concerns below.

**1. Experimental analysis is limited to image and video datasets**

We appreciate the comments. In this work, we mainly focus on anomaly detection from the Computer Vision perspective, which is challenging due to the high-dimensional input space and diverse patterns of anomaly. Extension to other modalities is definitely one of the doors we are hoping to open with this research.

**2. Definition of mIoU**

We have included the definition of mIoU metric in the revised paper (at the end of the first paragraph of Sec 5.1).

**3. The limited scope of competitor methods**

As the reviewer already mentioned, to the best of our knowledge, we are not able to find existing methods that are designed for anomaly detection under the fast adaptation scenario.

We have tried extending several existing few-shot density estimation methods including [a,b,c] to the same adaptive scenario like ours. Unfortunately, they failed to produce satisfying results on the real-world datasets we used. Combining MAML with the Autoencoder (MAML-AE) is the only extension that we found could deliver decent performance.

We agree with and thank the reviewer for correctly pointing out that the contribution of this paper lies in the capability of fast adapting to new tasks with only a few samples.

___

[a] Towards a Neural Statistician, ICLR 2017.

[b] Conditional neural processes, ICML 2018.

[c] SSD: A Unified Framework for Self-Supervised Outlier Detection, ICLR 2021.

 We appreciate the feedback which raises an important question on what is normal and abnormal. We present our response as bullet points below and hope these could address the reviewer’s concerns about our paper.

___

**Comment:**

“The paper claims a "principled" approach to detecting rare samples compared to normal ones. One would expect that abnormality implies improbability. The "principle" here is that something abnormal is something whose probability is small. The paper never avails itself of any discussion of what it means to be "anomalous" in terms of probability. Similarly, neither does the evaluation section mention anything about false positive or false negative detection rates.

As an example of this confusion, the use of the term "normal" in the introduction could refer to a normal, that is Gaussian distribution of the non-anomalous population. Fitting a normal distribution to samples, then looking for samples that fall on the tails of the distribution is a conventional method to detect anomalies. Since this is not what the paper proposes, the authors may want to clarify this at the outset. However, it raises the speculation whether the mean squared error expressed in Equation 8 is just implicitly an appeal to a normal distribution. A quadratic error term is consistent with the use of a Gaussian (normal) distribution.”

**Response:**

**“Normal” and “Abnormal" in terms of probability.** Our perspective in this paper is that the definitions of normal and abnormal are not easily represented by any specific quantity or probability. Normal samples are subjective to the use cases and vary from task to task. For example, in industrial inspection, normal samples are defined as the set of defect-free industrial components and anything which can not fit into this set would be considered as abnormal (objects with defects), while in autonomous driving, one can imagine that anomalies are foreground objects. It is therefore quite important that any approach to anomaly detection needs to be based on a “support set of normal samples” from which anomalies can be well defined, which is the essence of what is proposed in our paper and was clearly stated. Along this line, our paper looks at two critical components: (1) how do we represent the “support set” efficiently for which we proposed an energy based model and meta learning, and (2) how do we onboard a new support set rapidly for which we proposed a sparse coding approach. Regarding the evaluation metric, false positive and false negative detection rates can vary significantly w.r.t. the threshold picking. We follow the common practice of anomaly detection and report AUC-ROC which is threshold-invariant, therefore offering a comprehensive measurement of the model performance.

**The term “normal”**. The term ‘normal’ used in our paper is not directly related to the Normal (Gaussian) distribution. The samples from real-world datasets are high-dimensional, and existing distributions are unable to adequately describe them. This is one of the reasons why data-driven attributes of deep learning-based methods are much more appropriate here as we have shown in the paper.

___

**Comment:**

“Honestly I find it hard to tell if the evaluation presented in the paper makes the case for improvement over other deep learning methods, or more conventional image analysis methods. It appears the main take-away from the evaluation section is the acceptable detection ability on limited number of samples. How does this method stack up with previous methods when given comparably large numbers of training samples?”

**Response:**

We emphasize that this paper is addressing few shot adaptive anomaly detection, not when a large number of training samples are available which is outside the scope of our paper. There has been no previous work in anomaly detection in a few shot settings in industrial inspection as far as we know. Therefore to compare and get a sense of the progress this paper made, we provided performance on supervised methods (Table 1 for industrial inspection) as well as a strong baseline, MAML-AE. Comparison with the former is not apple to apple, but it helps to shed light on our approach. For the video surveillance task, there is an early work on fast adaptation, r-GAN, which we also proved in Table 2 is outperformed by our method. 

Although we are addressing fast adaptive anomaly detection, nevertheless, in Appendix Section B.3, we have shown that applying the conventional method based on image difference did not achieve satisfying results. We also provide the performance of [a] (non-deep learning method) under the fast adaptation scenario on the CUHK dataset, which achieves an **AUC-ROC of 59.45**. The performance is significantly degraded compared with **75.32** achieved by our method.

___

[a] Online detection of unusual events in videos via dynamic sparse coding, CVPR 2011


This paper presents a model for detecting irregularities in images, such as would occur in manufacturing acceptance testing.  As a form of anomaly detection, the claims made are for a principled modeling approach and an adaptive algorithm quick to learn from just a few samples.   Based on "energy based models" for modeling data densities, the paper claims improvements to avoid the need for retraining for new tasks, such as, instead of synthesizing negative samples rom noise, using a more targeted method of “learning from
inpainting” operation to learn anomalies. 

 The paper claims a "principled" approach to detecting rare samples compared to normal ones.  One would expect that abnormality implies improbability.  The "principle" here is that something abnormal is something whose probability is small.  The paper never avails itself of any discussion of  what it means to be "anomalous" in terms of probability.  Similarly neither does the evaluation section mention anything about false positive or false negative detection rates. 

As an example of this confusion, the use of the term "normal" in the introduction could refer to a normal, that is Gaussian distribution of the non-anomalous population. Fitting a normal distribution to samples, then looking for samples that fall on the tails of the distribution is a conventional method to detect anomalies.  Since this is not what the paper proposes, the authors may want to clarify this at the outset.  However it raises the speculation whether the mean squared error expressed in Equation 8 is just implicitly an appeal to a normal distribution. A quadratic error term is consistent with the use of a Gaussian (normal) distribution. 

Honestly I find it hard to tell if the evaluation presented in the paper makes the case for improvement over other deep learning methods, or more conventional image analysis methods.  It appears the main take-away from the evaluation section is the acceptable detection ability on limited number of samples.   How does this method stack up with previous methods when given comparably large numbers of training samples?   In short, the term 'anomaly detection' has been applied to a wide range of unsupervised techniques, with little in common among them.  This literature, using deep learning methods that have no connection to the existing literature that is more than a decade old, calls into question whether the proposed methods are competitive with previous methods that do not originate by applying deep learning techniques.  One really has to look at the history of the field to claim to have made an advance. 

My apologies for a shallow review due to a lack of familiarity with this branch of the field, however there are some basic questions that need to be raised about the lack of connection of the field with the vast work that precedes the current trend in deep learning to really understand the significance of the work. 

The paper proposes a framework for anomaly detection and localization that allows fast adaptation to new tasks. 
Specifically, the authors propose an energy-based model (EBM) with an adaptive sparse coding layer directly trained with normal features of a target task.

A meta-learning process is followed to extract common knowledge across tasks enabling few shots adaptation. 
Shrinkage functions, sparse coding with large receptive fields, and learning by inpainting are introduced to improve and accelerate the EBM
training.  Strengths:

- The method adapts to new tasks autonomously;
- Ablation study and comparison with supervised models;
- Exhaustive implementation details in the Appendix section.

Weaknesses: 

- Experimental analysis is limited to image and video datasets;
- The mIoU metric is never defined in the paper;
- Fairly limited scope of competitor methods (which could depend on the limited suitability of other methods in the adopted setting). The paper proposes an interesting anomaly detection approach and an experimental evaluation that also involves fully supervised models.

Results are competitive, and some of them appear close to the upper bound performances provided by supervised alternatives.

The authors also provide an exhaustive description of the architecture and hyperparameters in the Appendix section.

I think the main merit of the proposed method is the ability to achieve satisfactory anomaly detection performance without large availability of normal samples while adapting to new tasks.

This paper proposes an image classification system, where a set of normal patterns are stored as a "dictionary," and the degree of deviation is used as the anomaly score. 
 
The overall architecture seems to feature a conventional deep encoder and a pattern matching module that compares the encoded latent vector against the pattern dictionary. The pattern matching is done by solving the lasso regression problem. 

The authors propose a certain online learning approach, following existing few-shot learning methods, and also a synthetic sample generation approach using random perturbation combined with a gradient method.  This paper can be viewed as a proposal of a new image anomaly detection system that combines existing methods. The proposed "adaptive" dictionary-based approach is not new. The use of a deep encoder for image analysis is, of course, not new. The term "energy-based" does not mean much because any probability distribution can be thought of as energy-based. So the question is whether the combination is innovative or not. 

Although I appreciate the authors' efforts to develop an industry-application-ready image processing system, I do not think the technical novelty meets the standard that ICLR papers are supposed to have. 


Eq.(2)
The sign of ln Z?

Eq.(3) 
y is not defined in the generative model in Eq. (1). 

p.3
 The problem setting should be clearly defined. What are the input and output? The anomaly score must be clearly defined in the main text as part of the problem setting.  - Good industrial use-case paper
- Limited technical novelty -- less innovative combination of known methods. 

The author proposes a fast adaptive anomaly detection method using an adaptive sparse coding layer in the few-shot setting. The proposed method outperforms baselines. Strength:

+ The experimental result on two benchmark datasets outperforms most of the baselines.
+ An ablation study is presented to demonstrate the robustness of the proposed method.

However, there are also some places where this paper needs further clarifications:
- It is not clearly shown in the experiment as to how the technique in Section 3.1 improves the robustness of the network on different types of objects.
- A small typo in Section 3.1 - visualizations of the hard shrinkage function and SigShrink with different values of $\tau$ are presented in Fig. 3 rather than Fig. 7.
- The caption of Table I suggests that “Col 2-5 are fully supervised methods trained with massive normal samples” and are considered as the “upper-bounds”. However, it seems like the first column with AE (SSIM) is able to deliver better results in many categories than these upper bounds? (such as Carpet, Grid, and Bottle).
- It seems from Table 2 that the proposed method without leveraging any temporal information can achieve even better performance in the CUHK Avenue and the SH-Tech dataset. More discussions are encouraged on the benefit of incorporating such temporal information versus image frames randomly sampled from the target scenes.
 To summarize, this paper proposed a fast adaptive anomaly detection framework for anomaly detection in images. The proposed method is technically sound. However, the novelty is limited on the adaptive space coding layer with receptive field, while the rest of the network structure seems to be a big melting pot incorporating the Energy Bassed Model and episodic training in the context of meta-learning based few-shot learning. 

The paper presents an anomaly detection algorithm that uses an EBM (Energy Based Model) to distinguish between 'normal' and 'anomaly'. This model generates pseudo-anomaly instances on-the-fly for each 'normal' instance and then learns to assign low energy to normal instances and high energy to pseudo-anomaly instances. The model is further designed to be able to adapt quickly (few 'normal' labeled examples) to new tasks. Pros:
1. Presents a well-reasoned set of design choices
2. Good ablation studies

Cons:
1. Assumes all training data is 'normal'
2. Lacks experiments with contaminated training data


Main Comments

1. Expects all training data to be 'normal' -- effectively makes the algorithm fully supervised since we must manually make sure that all input train data is 'normal'.

2. Should show performance of algorithm when training data is contaminated with anomalies (varying contamination fraction, say, from 0.0 ... 0.10).

3. Section 3.1: "The sparsity regularization to ... formulated as the mean squared error (MSE) between the original" -- This is a nice explanation of the design choice.
 The paper mixes existing techniques in a nice way to make the overall solution very effective.

Meta-review: This work tries to tackle the problem of anomaly detection across different tasks. To do so, authors employ energy-based models (EBMs) and define an outlier score in terms of the EBM energies, having a shared sparse code for different tasks. This pipeline is tested on some image and video anomaly datasets for industrial inspection.

The reviewers highlighted some concerns that need to be addressed before the paper is ready for publication. 

First, a revision could benefit from a rewriting the clearly formalizes the learning problem from page 2 and then discusses about the possible modeling options given i) the task at hand and ii) some efficiency requirements.

Second, concerning the modeling choices of the proposed pipeline, the motivation behind the choice of EBMs should be strengthened. For example, it is not clear why the proposed sparse coding could be used for any other latent variable probabilistic model. As observed by one reviewer, the pros of having energies instead of probabilities (or just reconstructions from a deterministic autoencoder) is not discussed sufficiently. Additionally, the heuristics of running Langevin dynamics for only 5 steps should be backed up by stronger empirical evidence as it lacks theory, and it should be discussed how much you should run the Markov chain to obtain sensible negative samples.

Third, conclusions over the experiments on the provided benchmarks seem preliminary. For instance, a new revision could benefit from adding a statistical significance analysis to the reported accuracies. I appreciate that authors added further ablation studies including experiments on contaminated data in the latest revision. I suggest them to extend the experimental suite to more benchmarks including the commonly used for anomaly detection.

Decision: Reject