Fast Adaptive Anomaly Detection
Authors: Ze Wang
Yipin Zhou
Rui Wang
Tsung-Yu Lin
Ashish Shah
Ser-Nam Lim
Venue: ICLR 2022 Submitted (2022)

Abstract: The ability to detect anomaly has long been recognized as an inherent human ability, yet to date, practical AI solutions to mimic such capability have been lacking.This lack of progress can be attributed to several factors.  To begin with, the distribution of “abnormalities” is intractable.   Anything outside of a given normal population is by definition an anomaly. This explains why a large volume of workin this area has been dedicated to modeling the normal distribution of a given task followed by detecting deviations from it. This direction is however unsatisfying as it would require modeling the normal distribution of every task that comes along, which includes tedious data collection.  In this paper, we report our work aiming to handle these issues. To deal with the intractability of abnormal distribution, we leverage Energy Based Model (EBM). EBMs learn to associates low energies to correct values and higher energies to incorrect values.  As its core, the EBM em-ploys Langevin Dynamics (LD) in generating these incorrect samples based on an iterative optimization procedure, alleviating the intractable problem of modeling the world of anomalies.  Then, in order to avoid training an anomaly detector for every task, we utilize an adaptive sparse coding layer. Our intention is to design a plug and play feature that can be used to quickly update what is normal during inference time. Lastly, to avoid tedious data collection, this mentioned update of the sparse coding layer needs to be achievable with just a few shots. Here, we employ a meta learning scheme that simulates such a few shot setting during training. We support our findings with strong empirical evidence.

Under review as a conference paper at ICLR 2022
FAST ADAPTIVE ANOMALY DETECTION
Anonymous authors
Paper under double-blind review
ABSTRACT
The ability to detect anomaly has long been recognized as an inherent human abil- ity, yet to date, practical Al solutions to mimic such capability have been lacking. This lack of progress can be attributed to several factors. To begin with, the dis- tribution of "abnormalities" is intractable. Anything outside of a given normal population by definition an anomaly. This explains why a large volume of work in this area has been dedicated to modeling the normal distribution of a given task followed by detecting deviations from it. This direction is however unsatisfying as it would require modeling the normal distribution of every task that comes along, which includes tedious data collection. In this paper, we report our work aiming to handle these issues. To deal with the intractability of abnormal distribution, we leverage Energy Based Model (EBM). EBMs learn to associates low energies correct values and higher energies to incorrect values. As its core, the EBM em- ploys Langevin Dynamies (LD) in generating these incorrect samples based iterative optimization procedure, alleviating the intractable problem of modeling the world of anomalies. Then, in order to avoid training anomaly detector for every task, we utilize an adaptive sparse coding layer. Our intention is to design a plug and play feature that can be used to quickly update what is normal during in- ference time. Lastly, to avoid tedious data collection, this mentioned update of the sparse coding layer needs to be achievable with just few shots. Here, we employ a meta learning scheme that simulates such a few shot setting during training. We support our findings with strong empirical evidence.
INTRODUCTION
Anomaly detection is an important area of study in the field of artificial intelligence. It has found utility in computer vision applications such as industrial inspection (Bergmann et al., 2019) and video surveillance (Liu et al. 2018, Zhao et al. 2011 Nguyen & Meunier 2019), the context of abuse prevention such as misinformation, fraud and network intrusion detection (Zhang et al.) 2019 Bolton & Hand 2002) Mukherjee al.) 1994). and others such as system health monitoring and fault detection (Bao et al. 2019 surarjomandlangrud et al.] 2014). In this paper, we propose an approach for detecting anomaly in images, where we have carefully designed steps handle some of the bigger issues that have prevented the deployment of image anomaly detection in the
real-world.
Image anomaly detection can generally be defined as the identification of abnormalitics in a given image. An exact definition of abnormality in this case is clusive because abnormality can be derived from any unknown distribution outside of a normal population. Many studies have hence focused on modeling the normal population instead of learning irregularities, where the goal is to capture the shared concept among all of the normal data as one or several reference models. This process usually will require investing significant efforts in curating a large enough set of normal samples for each task, after which anomaly detected as deviations from the reference model(s) (An & Cho. 2013] Xu et al.] 2018). Recent work from (Sheynin et al., (022)) provides algorithms that utilize only a few normal samples to train models from scratch. However, the models still have to be provisioned for each new task, which requires considerable human efforts and expertise. and thus lack the fast deployment criterion that is often time critical for real-world applications. In view of these challenges, our goals for this work are threefold. We are interested in designing an anomaly detection system that is capable of: (GI) modeling the normal population while at the same time has a principled approach towards modeling the abnormalities; (G2) quickly adapting to a new task at inference time; and (G3) requiring only few normal shots update itself to the new task hand.

---

Under review as a conference paper at ICLR 2022
For (G1), we introduce the class of Energy Based Model (EBM), which is important family of generative models (Zhao et al., 2016 Du & Mordatch, 2019 Xie et al., 2016). EBMs have been shown to demonstrate superior capability on modeling data density and localizing anomaly (Genc) al., 202T). For our purpose, the EBM we adopted learn to assign low energy to normal samples but high energy to abnormal samples. More importantly, the abnormal samples are generated with a procedure known as Langevin Dynamics (I.D) (Welling Teh 201T), which, in its original form, starts with a noise image (see App. Fig C and gradually samples from the distribution along the direction of lower energy. This lends itsell gracefully to utilizing the generated intermediate samples as negative/abnormal. The LD procedure is then coupled with a contrastive divergence loss (Hinton 2002) that aims maximize the energy differences between the normal and abnormal samples.
To achieve (G2), we propose an adaptive sparse coding layer that is attached to the deep feature ex- tractor in the EBM. The deep features are projected into set of feature vectors along the spatial axes, after which each vector is forwarded to the sparse coding layer, where the dictionary is constructed with the features of few normal samples of the given task. In essence, the input representation has been decomposed into a linear combination of normal features with the sparsity constraint imposed. The final energy score is measured by the distance between the original and the reconstructed fea- tures (after the sparse coding layer). Under this scheme, the dictionary for a particular task is not obtained by learning, but instead is constructed by the feature representations of a few normal sam- ples during inference. As a result, this simple "plug-and-plaa" to novel tasks promptly without re- re-training. Further, we expect that the dictionary, which is formed by normal features, will not be able to explain the abnormal samples well, causing relatively high reconstruction error that lends itself for subsequent detection. As a bonus, backward pass from the reconstruction error to the image also additionally useful for localizing the abnormal regions.
trick allows the model be adapted
Towards (G3), we utilize meta learning (Vilalta & Drissi 2002 Finn et al., 2017) to simulate the scenario of being given a new task with few normal shots to update the dictionary, followed by training the EBM. This is accomplished by episodic training, where in each episode the model is adapted to held back task that is given few normal samples. To accelerate the EBM training, introduce "learning from inpainting", a simple yet effective strategy for synthesizing hard abnormal samples quicker by starting the LD procedure with synthesized image that is simply normal sample with a noise patch injected as opposed to a noise image that is traditionally what is used.
We show the proposed framework is able to efficiently adapt to novel task (e.g.. new object category or scenes from a new camera) with few normal samples on both industrial inspection and video surveillance tasks. We provide both qualitative and quantitative results to demonstrate that our method outperforms other adaptive frameworks and is comparable to methods that rely on large amount of normal samples.
2 BACKGROUNDS
We briefly introduce two key ingredients of the proposed method: EBMs and sparse coding.
Energy-based Model In EBMs, the goal is to learn an energy function Eg (x) Rd - R which parametrizes the data density px(x) as:
(x)))
(x()) Jx Coxp(-Eo(x)
(1)
where 0 is the parameter of the energy function and Zg = Jx exp(-xxp (x)) is the partition function. Approximating the true data distribution Passu (x) is equivalent to minimizing the expected negative log-likelihood function over the data distribution, defined by the loss function:
LML Ex-- log po
Ex- [E((x) + log Zo).
(2)
As the computation of LM.. involves intractable term Zg, the common practice is to represent the gradient of LML as,
VoLMI. = Exttrpu (x)Vo. - Ex Voo
This objective is commonly referred to as the contrastive divergence (Hinton) 2002). which decreases the energy of positive data samples x from the true distribution (normal samples in our use case) and increases the energy of negative samples x from the model po (synthesized abnormal samples). In practice the synthesized negative samples achieved through Langevin dynamics (Welling &
2

---

Under review as a conference paper at ICLR 2022
Teh 2011), which samples along the direction of decreasing energy score, typically starting from given noise image:
3 Vx wh, wh NO, 81),
x
(4)
where / is the step size. The synthesizing ability of EBMs enables generating abnormal samples help learning a more accurate data density, and is often touted as the one of the advantages of using an EBM.
Sparse coding. Approximating signal z E 18d with the sparse linear combination over a dictionary D E Baxk can be expressed as:
min
2 12 Dox 1 + A||e||1,
(5)
where ar is the sparse coefficients, with its sparsity (4 norm) and is the weight of the sparsity constraint. Da is sparse approximation to the original signal z. In practice, finding the dictionary atoms and the sparse coefficients is usually formulated as an optimization problem.
In this paper, adopt Iterative Soft Thresholding Continuation (ISTC) (Jiao et al. 2017 to convert this optimization problem into linear operations with a non-linear shrinkage function, which allows sparse coding to be seamlessly integrated into the deep neural networks. To compute sparse coef- ficient a, ISTC performs iterations of gradient steps on reconstruction ||z Da 2 and a proximal projection step to increase coefficient sparsity.
Formally, initializing the coefficients the first step og with all zeros, each step of ISTC refines the sparse code with descending values of from Amax to d each step of ISTC expressed as:
Amax
-n/N
aa+1 = =(a,, + +D (z Da,), An), with Ar
Amax
(6)
A.
where ot. -) here a shrinkage function that truncates small values (lower than A) of the coefficients to 0 to enforce sparsity, and can be easily implemented by customized ReLU activation function:
(z, A) sga(2)(max((2) >, 0))
A)-
3 PROPOSED METHOD
The objective of our proposed method, when given an input image or video frame, is to output an anomaly score indicating how likely this input deviates from the normal, and additionally, a pixel map grounding abnormal regions. In this section, we describe the proposed fast adaptive anomaly detection framework in details. In Section 3.I we introduce the adaptive EBM which consists of deep feature extractor followed by an adaptive sparse coding layer. From there, we further show that utilizing larger receptive field in the sparse coding could improve training robustness (Section 3.1.1. and applying smoothed shrinkage functions could help speed up convergence (Section 3.1.2.. In Sec- tion 3.2 we describe the episodic training regime on various anomaly detection tasks that mimics few-shot adaptation in the meta-testing stage while learning common knowledge across tasks. Fi- nally, Instead of synthesizing negative samples (anomaly) directly from noise, introduce simple but effective "learning from inpainting" operation to accelerate the training in Section 3.3
3.1 ADAPTIVE ENERGY-BASEN BASED MODEL
An EBM is form of generative model and it is widely used for modeling data density and sampling. While there has been recent work (Genc et al.] 2021) applying EBM to anomaly detection, still requires re- re-training for each new task. To efficiently adapt the EBM to novel tasks, we introduce an adaptive sparse coding layer which is conditioned on the dictionary constructed by the features of normal samples. Specifically, obtain the corresponding feature z e dgxh'' xw' from the deep feature extractor V with parameters 0, so that = (x; 0). All feature vectors along spatial axes of z are then sparsely decomposed through the sparse coding layer over a task-specific dictionary D E IRdx w' which contains the features of K normal samples of the current task shown in the Fig aa). Each feature vector
first illustrated in Fig 1 given input image, x 33xhx
3

---

Under review as a conference paper at ICLR 2022
Spurse coding (b)
4x XW
dxx NKW
Reshige
1- DDD
D
11
J- D7D
normal samples from the new task
Dictionury "plug and play" (a)
D'D
500
S
Eg (x: D) = MSE(z x')
Da
Testing normal samples
Testing sumples
Gradiont maps
Backward pass
Figure 1: Overview the inference stage on new task. (a) Adapting the task-specific dictionary with K normal samples. (b) Sparse coding with three iterations as Equ shows. We also show backward pass from the reconstruction error localize the abnormal regions
of the normal sample feature is then directly used posed coefficients are = Szz; D), where a E IR jgk'''' wf "h'x xw and S denotes the iterative sparse decomposition process of (6). By multiplying the coefficient a with the dictionary D, we obtain the reconstructed features = Da. The sparsity regularization to is important, as it encourages input features be reconstructed by simple combinations of dictionary atoms (normal features), so that it would be difficult for features of abnormal samples to be well-approximated, therefore producing higher reconstruction errors that make it conducive for detecting anomalies. From here, the final energy score is formulated as the mean squared error (MSE) between the original and the reconstructed features:
atom in the task dictionary. The decom-
(x D) MSE(2, z') (x; 0) DS ( (x; D)11
(8)
effect, Eqn. (enn.SJeppn depicts a conditional EBM, which is conditioned on the task-specific D formed by normal features. In the following sections, will discuss how make the training of this adaptive structure more robust.
3.1.1 SPARSE CODING WITH RECEPTIVE FIELD.
As discussed in Section 3.1 the input feature is represented h' w/ of d-dim feature vectors and they are treated independently while passing through the sparse coding layer. The region input image that affects one feature vector is determined by the receptive field of the feature extractor. The trade-off that small receptive field may not capture enough contextual information, while applying large receptive field would make feature maps spatially coarse and make it hard to spot small anomaly regions. To solve this dilemma, instead of carefully tuning the receptive field each layer of the feature extractor, we introduce simple yet effective technique of applying the receptive field on the sparse coding layer. Specifically, as Fig 2 shows, rather than performing sparse coding to each individual d-dim feature vectors, we apply it on d x X I volumes centered around each feature vector, where the receptive field. This equivalent to applying a ( sliding window on spatial axes of the feature map and can be easily implemented by image to column (Im2Col) operation. Then we flatten the feature volumes into d/ vectors and adjust the shape of the dictionary accordingly. In this way, we are able to capture contextual information without needing to carefully tune the architecture of the feature extractor and show in the later experiments that this technique improves the robustness of the network on different types of objects.
3.1.2 SHRINKAGE FUNCTION
The effectiveness of training the EBM for localizing anomaly regions heavily depends the gra- dient propagation from later to earlier layers. is shown in (Du et al. 2021) that smooth activation functions like Swish Ramachandran al. (017) could be beneficial here. Notably, the gradients of
4

---

Under review as a conference paper at ICLR 2022
Shrinknge fanctions (A- -0.25)
180
Hat Strkag
Leee
Sygki
939
033
3.9
-359
435
-.00
-10
0.00
0.25
6.55
100
Figure 2: Illustration sparse coding layer with / recep- tive filed.
X
Figure 3: Shrinkage functions.
the dictionary D are determined by the sparse coding coefficients as shown in Eqn. 6 However, the sparsity constraint of a would turn off the gradient computation of many elements D and this could be detrimental during the early stage of the training. To alleviate the sparse gradient issue, we replace the RELU-like shrinkage function in Eqn. with its smoothed counterparts by introduc- ing the Sigmoid based shrinkage functions (SigShrink). The SigShink is originally proposed for non con-paramttric signal estimation in Atto et al. (2008), and can be defined as:
0, 0z A) + exp(-r(Iz) - A))*
2
where is the hyperparameter of smoothness. We present visualizations of the hard shrinkage function Eqn. / and SigShrink with different values of r in Fig Comparing to the hard shrinkage function which truncates small values into zeros, the SigShrink with large can sharply force small values near-zeros. Therefore, the SigShrink will guarantee non-zero gradients everywhere.
3.2 EPISODIC TRAINING
To train the proposed adaptive EBM, we perform episodic training that is widely adopted by meta- learning based few-shot learning tasks Finn et al. (2017); Snell et al.] (2017). Following the termi- nology of few-shot learning, in each training episode, the model is adapted and tested with task sampled from the underlying task distribution. Specifically, the model adapted a support set of the given task, then a query set with ground truth labels is applied to evaluate the adaptation, which used update the model parameters accordingly. As shown Fig the support set contains K 1K normal samples {xx jk of the current task, where K is usually small number. The feature repre-
(0nl)
sentations 24 L (xx: 0) of these normal samples are plugged into the dictionary D, E d Kh'w' corresponding to the i-th task during the i-th episode to adapt the dictionary to the normal samples of the task. After that, the adapted model is measured by a query set consisting of M normal samples {xm ) and abnormal samples (&/m J. 1-M Note that there is no actual abnormal samples given during training. instead, they are iteratively sampled from the EBM and we will discuss the initial- ization and sampling of these synthetic samples details in Section 3.3 Recall that the training of EBM with contrastive divergence described in Eqn. 33 requires the estimation of energy scores of both positive samples from the true data distribution and negative samples sampled from the model distribution. The positive energy can be estimated empirically with normal samples from the query set. The negative energy can be estimated by performing the MCMC-based (Markov Chain Monte Carlo) sampling technique (Neal et al., 2011] Welling & Teh 2011). typically Langevin Dynamics as described in Eqn. 4 Denoting the output of Langevin dynamics (sampled abnormal samples) with the initialization & as LD(x)), we now have the empirical estimation of the contrastive divergence of the i-th episode
M Σ
1
Led
Ex(xm; D.) - (LD ( (.)):
(10)
With the energy score equivalent to the feature reconstruction error in Eqn. minimizing Les en- courages normal features be well-reconstruucted by a sparse linear combination of dictionary atoms while the features from abnormal samples tend to produce relatively higher reconstruction errors so that they can be easily spotted.
5

---

Under review as a conference paper at ICLR 2022
Support set
(Adaptation)
D
episode
Dictionury "plug and play'
Query set
(Evaluation)
V
Led
episode n-1
Normal samples
episode n
Symthesieed
Step
Step
Step
Step
Step
shnomaal samples
(a)
Process of Langevin dynamics
Figure 4: Illustration of episodic training and (a) "learning by inpainting".
3.3 SYNTHESIZINGG NEGATIVE SAMPLES
Typical EBM training with contrastive divergence conducts negative sampling from the modeled density using techniques such as Langevin Dynamics, which applies gradient descent to a noise ini- tialization (App. C) with small step size and large number of steps & Mordatch (2019). Such negative sampling steps can be costly and we argue that it is unnecessary in our case. Instead, we introduce new strategy of "learning by inpainting". Starting from a positive query sample Xm. synthesize the corresponding negative sample & by randomly placing small uniform noise patch on the image. The Langevin Dynamics procedure is then initialized with the resulting image instead of a noise image. As the Langevin Dynamics proceeds, synthesized abnormal samples LD( (x,,) are inpainted along the direction of "normal", Xm, and introduce the following reconstruction loss:
Sie MSE(LD (x,)), &.
(11)
The gradient map from Lie reveals anomaly regions, which helps with localization. We show in Fa) that, starting from synthesized abnormal sample, only steps of Langevin dynamic would be sufficient to make it visually close to the corresponding normal sample during training, serving as "hard negatives" that further facilitates the learning. The final loss of the episodic training is simply:
L = Cre + 7/ Lods
(12)
where and are hyperparameters balancing two loss terms.
RELATED WORK
4.1 ANOMALY DETECTION
Sparse coding. Early efforts adopting sparse coding in anomaly detection based opti-
mization (with LI penalty) (Lu et al. 2013 Zhao et al. 201T). Recent advances on iterative sparse thresholding algorithms (Daubechies et al. 2004 Jiao et al. 2017) allow scamless integration of online sparse coding with deep neural networks, and (Luo et al. (017) formulates the sparse coding stack RNNs for video anomaly detection.
Generative models. Generative models are widely utilized in anomaly detection due the capa- bility in modeling the density of desired data distribution. Early efforts variational autoencoders (VAE) based methods (An Cho 2013) Xu al.
2018) are arguably having hard time calibrat-
ing uncertainties in novel samples Nalisnick et al.
2018). accurately localizing abnormal regions
through reconstruction errors (Dehaene al. 2020). Recent efforts have explored variant generative architectures like energy-based models (EBM) (Genc et al. 202T), GANs (Sheynin et al.) (202T). and combining VAE with EBM (Dehaene et al. (2020). Various methods also exploit intra-image struc- tures (Cohen & Hoshen 2020 Bergmann et al. 2018), cross-frame consistency (Lu et al.) 2019), and motion-appearance consistency videos (Nguyen & Meunier 2019) while detecting anomaly.

---

Under review as a conference paper at ICLR 2022
Category
AE (SSIM)
(MSE)
AncGAN
VE-YAB
MAML-AE
Ours
Carpet
0.69
0.38
0.34
D.1
0.20
0.28
0.87
0.59
0.54
0.78
0.68
0.83
Grid
0.94
0.90
0.58
0.73
0.53
0.01
0.81
0.71
0.67
0.34
0.74
0.12
0.42
0.78
0.75
0.87
0.77
0.98
Tile
0.04
0.23
0.08
0.14
0.14
0.28
Wood
0.73
0.36
0.73
0.29
0.14
0.62
0.47
0.91
0.68
0.11
0.78
0.23
Bottle
0.15
0.22
0.05
0.07
0.02
0.23
0.93
0.86
0.86
0.78
0.56
0.82
Cable
0.01
0.05
0.01
0.18
0.04
0.24
0.82
0.86
0.78
0.90
0.74
01.7
Capsule
0.94
0.88
0.84
0.74
0.68
0.90
Hazelnut
0.00
0.41
0.02
0.44
0.11
0.40
0.97
0.95
0.87
0.98
0.72
0.94
Metal
0.01
0.26
0.00
0.49
0.10
0.39
Pill
0.07
0.91
0.85
025
0.17
0.87
0.18
0.83
0.62
0.10
0.88
0.22
0.03
0.34
0.01
0.17
0.02
0.17
Screw
0.96
0,96
0.80
0.97
0.55
0.83
Toothbrush
0.51
0.07
0.14
0.06
0.23
0.92
0.93
0.90
0.94
0.80
0.82
Transistor
0.90
0.86
0.80
0.93
0.76
0.85
zipper
0.10
0.13
0.01
0.06
0.04
0.12
Table 1: Numerical evaluation of anomaly localization on MVTec-AD. We report both mloU (top rows) and AUC-ROC (bottom rows) values. Col are fully supervised methods trained with massive normal samples.
Input
Prediction
Figure 5: Visualizations of local- ized anomaly by our method.
4.2 FEW-SHOT LEARNING
Few-shot learning is extensively explored in classification tasks. Proposed methods are based on optimization (Finn al. 2017 Rusu al., [019 Finn et al.] (2018; Yoon al.] [0188] Ravi & Larochelle| 2016), learning metric (Snell et al. 2017 Vinyals et al. 2016) and parameter prediction (Gordon et al.] 20188 Qiao et al. 2018 Gidaris & Komodakis 2019). These technologies are further applicd in other tasks like image generation (Clouâtre & Demers, 2019 Liu et al. 2019) and out-of-
distribution detection (Sehwag et al. 2021).
EXPERIMENTS
In this section, we conduct evaluation the industrial inspection task with the benchmark MVTee-- AD dataset Bergmann et al. (2019) (Section 5.T). Even though our proposed framework image- based, further demonstrate it's efficacy the video anomaly detection task in Section 5.2 In Section 5.3 we show ablations and insights relating to the adaptive sparse coding components.
also show additional ablations on robustness pose variations (e.g. rotation) while applying few- shot adaptation (App. B.T) and comparing with naive background subtraction to detect anomaly (App. B.2 We provide implementation details in App.
5.1 INDUSTRIAL INSPECTION
The goal of this anomaly detection task is to predict whether a manufactured component contains any defects. The MVTec-AD dataset includes 15 categories of object. To demonstrate the fast adaptation capability of the proposed method, we adopt a leave-one-oul training strategy. Specifically, samples of each target category reserved for testing only, and the episodic training is performed remaining categories. During the training stage, the model will not see any samples from the target category. During testing, first adapt the model to the target category with 10 randomly selected
the

---

Under review as a conference paper at ICLR 2022
Target datascts
Methods
I-shot
5-shot
10-shot
I-GAN Pre-train
73.10
73,10
73.10
F-GAN Fine-tune
76.99
77.85
78.23
UCSD Ped
T-GAN MAML
80.60
81.42
82.38
MAML-AE
64.12
66.88
67.34
Ours
77.42
78.12
78.65
1-GAN Pre-train
81.95
81.95
81.95
r-GAN Fine-tune
85.64
89.66
91.11
UCSD Ped
1-GAN MAML
91.19
91.80
92.80
MAML-AE
78.24
82.04
83.30
Ours
91.22
92.00
92.45
r-GAN Pre-train
71.43
71.43
71.43
F-GAN Fine-tune
75.43
76.52
77.77
CUHK Avenue
T-GAN MAML
76.58
77.10
78.79
MAML-AE
68.72
69.67
70.01
Ours
80.68
83,41
84,46
F-GAN Pre-train
70.11
70.11
70.11
r-GAN Fine-tune
71.61
70.47
71.59
Sh-Tech
I-GAN MAML
74.51
75.28
77.36
MAML-AE
66.62
67.12
68.00
Ours
75.32
79.64
81.28
Input
GT
Prediction
Table 2: Frame-level AUC-ROC for the video anomaly detection tasks.
Figure 6: Visualizations of anomaly local- ization with video anomaly detection.
normal samples, then measure the performance with the entire testing set. We run the test 5 times, each time the model adapted to random sets of 10 normal samples from the target category. The final result is the average of the 5 runs. Following common practice (Bergmann et al.] 2019 Liu al.) (2020). we report the performance of pixel-wise anomaly localization with AUC-ROC and mloU (mean intersection over union). IoU is the area of overlap between the predicted map and the ground truth divided by the area of union between these two.
To the best of our knowledge, no existing methods feature fast adaptation like ours. We first show the performance of fully supervised methods, which train each category with normal samples from scratch, as the "upper-bounds". Specifically, (Bergmann et al.) 2018 2019) trains auto-encoders (AE) massive number of normal samples and measure the reconstruction errors during the in- ference; AnoGAN (Schlegl et al. (017) adopts a generative adversarial network (GAN) to learn manifold of normal; VE- VAE (Liu al. 2020) presents visually explainable variational auto- encode through gradient- based attention. Furthermore, we create strong baseline by applying AE (denoted as MAML-AE, detailed in App. Sec. A.3). Numerical results pixel-wise anomaly localization are Table Note that results of our methods are obtained without any data augmentation. Our proposed method outperforms MAML-AE by large margin and is competitive with the "upper-bounds We show the localized anomaly regions from our method in Fig Additional visualizations are in the App. Fig A
model-agnostic meta-learning (Finn et al. 2017)
5.2 VIDEO SURVEILLANCE
In video anomaly detection, a common goal is to detect abnormal events captured by surveillance cameras (e.g.. motorcycle on the sidewalk). A model trained on videos from one camera might not generalize well on other cameras due to different locations / mounting heights lightning conditions, and it is not feasible to train model for every new camera practice. The ability to quickly adapt to new scenes is a significant contribution to the task video surveillance. We are only aware the work in (Lu et al. 2020) (r-GAN) that has such adaptation capability. Specifically, the model adapts new scene using gradient descent with several beginning frames of a query video, after which GAN is applied to generate future frames. Anomaly is then detected via the discrepancy between predicted future frames and the original frames.
We follow the same evaluation regime as r-GAN by training with normal samples in all 13 scenes from SH-Tech Liu et al. (2018) and testing UCSD Pedestrian UCSD Pedestrian 2 (Mahadevan ett al. 2010), and CUHK Avenue Lu et al. (2013). Note that since our method is image-based, predicts the video frames independently without leveraging any temporal information as in r-GAN.
8

---

Under review as a conference paper at ICLR 2022
Calegory
Leather
Grid
Hazelnut
Cable
Pill
0.12
1
0.41
0.98
0.11
0.80
0.36
0.91
0.21
0.85
0.10
0.85
0.10
0.42
0.98
0.12
0.40
0.94
0.24
0.87
0.22
0.88
Table 3: Comparison of different sparse coding receptive fields.
We report both mloU (left) and AUC-ROC (right) values.
0.0
0.2
W Marhhaah
Category
Leather
Hazelnut
Cable
Ours
0.42
0.98
1.6e4
0.40
0.94
2.4e-4
0.24
0.87
2.0c-
Irensions
10000
No sparsity
0.32
0.90
0.9c-4
0.24
0.80
1.70-4
0.12
0.68
1.5044
Figure Loss curves with smooth (SigShrink) and non-smooth (hard- shrink RELU-like) shrinkage func- tions.
Table 4: Performance w/ and w/o sparsity constraint. From left right: mloU; AUC-ROC; the difference of averaged reconstruc- tion errors between abnormal/normal samples.
In each episode, we adapt our model with a support set containing a few normal frames randomly sampled from the target scenes. In Table we compare our method against r-GAN pre-trained on SH-Tech only (r-GAN Pre-train), fine-tuned on target datasets (r-GAN Fine-tune), and with one step gradient descent with meta-learning (r-GAN MAML). We also show the performance of MAML-AE as baseline for image-based meta-learning method. In the last section of Table we present intra- dataset results as well by training with 6 scenes of SH-Tech and testing on remaining 7. We follow common evaluation protocol and measure the frame-level AUC-ROC, Without leveraging temporal information and performing re-training (gradient descent), our method achieves comparable results r-GAN MAML and outperforms image-based meta-learning method by large margin.
5.3 ABLATION STUDIES
Sparse coding receptive fields. To evaluate the effectiveness of using large receptive fields in the sparse coding layer, we conduct additional experiments on the MVTec-AD dataset, and select representative categories with different levels of difficulties to present the comparisons with = and / = (Sec. 3.1.I in Table 3 Sparse coding with large receptive field clearly benefits more complex structural objects (hazelnut, cable, and capsule), while the improvements are limited for the texture objects (leather and grid), where contextual regularization is intuitively less important.
Shrinkage funetions. To show the benefits of smooth shrinkage function, we plot the loss curves of models trained with smooth SigShrink (Eqn. (9] and non-smooth RELU-like shrinkage (Eqn. 7 functions in Fig The model with smooth shrinkage function converges notably faster in the early training stage and achieves lower loss.
Sparsity constraint. As discussed in Section 3.1 we impose sparsity constraint to the feature de- composition in the adaptive sparse coding layer, order to prevent abnormal features from being well-approximated by the linear combinations of normal features, so that the reconstruction errors are effective for detecting anomaly. To validate this, we conduct experiments by removing the shrinkage function a the sparse coding stage (Eqn. (). We show comparison in Table 4 with mloU, AUC-ROC, and the difference of averaged reconstruction errors between abnormal and nor- mal samples. Without sparsity, the performance drops dramatically, and reconstruction errors of normal and abnormal samples become closer.
6 CONCLUSION
In this paper, we introduced a novel framework for anomaly detection and localization that allows fast adaptation to new tasks. We formulated our model tive sparse coding layer, of which the dictionary is directly formed by normal features of a target task. We adopted episodic meta-learning to extract common knowledge across tasks, which has the effect of enabling few shots adaptation. We further introduced smooth shrinkage functions, sparse coding with large receptive fields, and learning by inpainting to improve and accelerate the EBM training. It's worthy to note that when evaluating our method's performance on industrial inspection and video anomaly detection, our method is comparable and even boasts better performance than methods trained with a large amount of normal samples. Through this work, we hope to have made a significant contribution to the important problem of anomaly detection by shedding light on our findings that anomaly detection can indeed be generalized to new tasks.
energy based model with adap-
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruc- tion probability. Special Lecture on IE, 2(1):1-18, 2015.
Abdourrahmane Atto, Dominique Pastor, and Gregoire Mercier. Smooth sigmoid wavelet shrink- age for non-parametric estimation. In 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 3265-326666 1268..26...0 IEFE, 2008
Yuequan Bao, Zhiyi Tang, Hui Li, and Yufeng Zhang. Computer vision and deep learning-based data anomaly detection method for structural health monitoring. Structural Health Monitoring, 18(2)401-421, 2019.
Paul Bergmann, Sindy Löwe, Michael Fauser, David Sattlegger, and Carsten Steger. Improving un- supervised defect segmentation by applying structural similarity to autoencoders. arXiv preprint arXiv: 2018.
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad-a comprehen- sive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9592-96000 2019.
Richard J Bolton and David Hand. Statistical fraud detection: review. Statistical science, 17(3): 235-255, 2002.
Louis Clouâtre and Mare Demers. Figr: Few-shot image generation with reptile. arXiv preprint arXiv:1901.02 199, 2019.
Niv Cohen and Yedid Hoshen. Sub-image anomaly detection with deep pyramid correspondences. arXiv preprint (xXi::0005002357, 2020.
Ingrid Daubechies, Michel Defrise, and Christine De Mol. An iterative thresholding algorithm for linear inverse problems with sparsity constraint. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 57(11):141) 1457,2004
David Dehaene. Oriel Frigo, Sébastien Combrexelle, and Pierre Eline. Iterative energy- pro- jection on a normal data manifold for anomaly localization. ICLR, 2020.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy- based models. NeurlPS, 2019.
Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. International Conference on Machine Learning. 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, pp. 1126-1135. PMLR, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. arXiv preprint arXiv:1806.024 02817, 2018.
Ergin Utku Gene, Nilesh Ahuja, Ibrahima J Ndiour, and Omesh Tickoo. Energy-based anomaly detection and localization. arXiv preprint arXiv:2105.03270, 2021.
Spyros Gidaris and Nikos Komodakis. Generating classification weights with gnn denoising autoen- coders for few-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 21-30, 2019.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E Turner. Meta- learning probabilistic inference for prediction. arXiv preprint arXiv: (.0ivv/8855099)) 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp- 770-778, 2016.
10

---

Under review as a conference paper at ICLR 2022
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800) 2002.
Yuling Jiao, Bangti Jin, and Xiliang Lu. Iterative softhard thresholding with homotopy continuation for sparse recovery. IEEE Signal Processing Letters, 24(6):784-7888 2017.
Ming- Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen. and Jan Kautz. Few-shot unsupervised image-to-image translation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 10551-10560, 2019.
W. Liu, D. Lian W. Luo, and S. Gao. Future frame prediction for anomaly detection new baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
Wengian Liu, Runze Li, Meng Zheng, Srikrishma Karanam, Ziyan Wu, Bir Bhanu, Richard J Radke, and Octavia Camps. Towards visually explaining variational autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8642-8655 1651, 2020.
Cewu Lu, Jianping Shi, and Jiaya Jia. Abnormal event detection at 150 fps in matlab. In Proceedings of the IEEE international conference computer vision, pp. 2720-2727, 2013.
Jia.
Yiwei Lu, K Mahesh Kumar, Seyed shahabeddin Nabavi, and Yang Wang. Future frame prediction using convolutional vmn for anomaly detection. In 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1-8. IEEE, 2019.
Yiwei Lu, Frank Yu, Mahesh Kumar Krishna Reddy, and Yang Wang. Few-shot scene-adaptive anomaly detection. In European Conference on Computer Vision, pp. 125-141. Springer, 2020.
Weixin Luo, Wen Liu, and Shenghua Gao. A revisit of sparse coding based anomaly detection in stacked mn framework. In Proceedings of the IEEE International Conference on Computer Vision, pp. 341-349) 2017.
Vijay Mahadevan, Weixin Li, Viral Bhalodia, and Nuno Vasconcelos. Anomaly detection in crowded scenes. In 20/0 JEEE Computer Sociery Conference on Computer Vision and Pattern Recognition, pp. 1975-1981. IEEE, 2010.
Biswanath Mukherjee, I. Todd Heberlein, and Karl N Levitt. Network intrusion detection. IEEE network, 8(3):26-41 1994.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanaan Do deep generative models know what they don't 't know? arXiv preprint arXiv:1810.09113, 2018.
Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011.
Trong-Nguyen Nguyen and Jean Meunier. Anomaly detection in video sequence with appearance- motion correspondence. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 1273-1283. 2019.
Afrooz Purarjomandlangrudi, Amir Hossein Ghapanchi, and Mohammad Esmalifalak. A data min- ing approach for fault diagnosis: An application of anomaly detection algorithm. Measurement, 55:343-352, 2014.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L Yuille. Few-shot image recognition by predicting parameters from activations. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 7229-222222 2018
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint 17X1v:1110.05941, 2017.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning.
International
Conference on Learning Representations, 2016.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. International Conference Learning Representations 2019.
11

---

Under review as a conference paper at ICLR 2022
Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In International conference information processing in medical imaging, pp. 146- 157. Springer, 2017.
Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. arXiv preprint arXiv:2103//2051, 2021.
Shelly Sheynin, Sagie Benaim, and Lior Wolf. A hierarchical transformatton-disciinnnntt gener- ative model for few shot anomaly detection. arXiv preprint arXiv:2104.14553. 2021.
Jake Snell, Kevin Swersky, and Richard S Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems, 2017.
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-leaming. Artificial intelligence review, 18(2):77-95, 2002.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. Advances in Neural Information Processing Systems, 2016.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-I1), pp. 681-688. Citeseer, 2011.
Jianwen Xie, Yang Lu, Song-Chun Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning, pp. 6635-2644 PMLR, 2016.
Haowen Xu, Wenxiao Chen, Nengwen Zhao, Zeyan Li, Jiahao Bu, Zhihan Li, Ying Liu, Youjian Zhao, Dan Pei, Yang Feng, et al. Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications. Proceedings of the 2018 World Wide Web Conference, pp. 187-196, 2018
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 7343-7353. 2018.
Qiang Zhang. Aldo Lipani, Shangsong Liang, and Emine Yilmaz. Reply-aided detection of misin- formation via bayesian deep learning. In The world wide web conference, pp. 2333-2343, 2019.
Bin Zhao, Li Fei-Fei, and Eric Xing. Online detection of unusual events in videos via dynamic sparse coding. In CVPR 2, pp. 3313-3320. IEEE, 2011.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. arXiv preprint 1roX.v:6009.33111 2016.
Yang Zhao, Jianwen Xie, and Ping Li. Learning energy-based generative models via coarse-to-fine expanding and sampling. In International Conference Learning Representations, 2020.
12

---

Under review as a conference paper at ICLR 2022
APPENDIX
A IMPLEMENTATION DETAILS
We adopt a residual network (ResNet) He et al. (2016) based feature extractor, with ELU as the activation function. Following (Du et al. 2021), ). adopt Group Normalization for better stability
of the network. The detailed network construction is shown in Table A
ResNet-10
Output size
Layers
224x
Input images
56x36
Conv(7 x7, stride=2), GroupNorm, ELU, Average stride=2)
[Conv (3x3), GroupNorm, ELU] x 2,
56x56x32
Skip connect Conv (1x1, stride=2), GroupNorm
Conv (3x3, stride=2), GroupNorm, ELU,
28x28x64
Conv (3)3), BatchNorm, ELU Skip connect Conv (1x11, 1, stride=2), GroupNorm
Conv (3x3, stride=2), GroupNorm, ELU,
14x14x128
Conv (3x3), GroupNorrm ELU Skip connect Conv (1x1, 1, stride=2), GroupNorm
Conv (3x3, stride=2), GroupNorm, ELU,
7x7x256
Conv (3x3), GroupNorm, ELU Skip connect Conv (1 x 1, stride=2), GroupNorm
7x7x256
Conv (1 11, stride=2), GroupNorm, Tanh
Table A: The architectures of feature extractor.
A.1 INDUSTRIAL INSPECTION
training strategy for obtaining the results each of the categories of MVTec-AD. All experiments are performed with the same settings and hyperparameters. We resize all images to 128 x 128, and do not perform any data augmentation. We adopt simple reduced- sized ResNet as the feature extractor as shown in Table A Following (Du et al. 202T), we adopt group normalization (denoted as GroupNorm) instead of batch normalization, and use Exponential Linear Unit (ELU) as the activation function. We empirically observed that using Tanh as the final activation function can remarkably improve the numerical stability of the sparse coding stage as the magnitude of the feature values is effectively bounded by the final activation function.
We adopt the "leave-one-out"
We adopt Adam as the optimizer, with consistent learning rate of lc-4. We do not apply any net regularization methods like dropout or weight decay in training. The weights of reconstruction no and contrastive divergence 7/1 are set to 1.0 and 0.25, respectively. Each training batch contains 4 randomly sampled training tasks with 10 query (M 10) for each task. All training can be conducted on a single NVIDIA Tesla 1100 GPU.
We perform 8 steps of sparse coding in the adaptive sparse coding layer, with an initial Amux 0.3
and a final X, = 0.05. We perform 5 steps of Langevin dynamics with a step size of 3 1.0 to synthesize negative samples, which we show in the examples of Figure 4 and Figure C to be sufficient for producing hard negative samples.
A.2 VIDEO ANOMALY DETECTION
We resize each frame to 240 x 320. No data augmentation is performed. All other hyperparameters equal those applied in industrial inspection experiments.
A.3
MAML-AE
We adopt a full-convolutional auto-encoder network to construct the MAML-AE. A 10 layer ResNet (He al. 2016) the encoder, and consecutive transpose-convollttional layers with batch normal-
13

---

Under review as a conference paper at ICLR 2022
ization and RELU activation function the decoder to recover the feature resolution. The hyperpa- rameters of the episodic training of MAML AE are exactly the same with those our methods. We perform 5 steps of gradient descent as the inner-loop adaption.
We directly use the energy score in (Eqn. g as the anomaly score. When evaluating AUC-ROC and mloU, we obtain the normalized energy score for each sample by directly performing an uniformed normalization the scores of all samples in test set.
A.4 INITIALIZING NEGATIVE SAMPLE
In the synthesis of negative samples, we randomly place at most three random patches to each normal image. Each patch is created by one of the following:
Random uniform: A random patch with the values of each pixel sampled from uniform distribution. The minimum and maximum values of the uniform distribution equal to those of the pixel values of the images.
Random consistent: random patch with consistent pixel values sampled from uniform distribution. The minimum and maximum values of the uniform distribution is equal to those of the pixel values of the images.
Random copy-paste: random patch randomly cropped from the same image.
Random blurring: Applying Gaussian blurring to a random patch of the normal image.
Denoting (a, b) as an uniform distribution with minimum and maximum values of and b, re- spectively, the relative size of the random patch w.r.. the original image randomly sampled from 4(0.0025, 0.025). and the aspect ratio is randomly sampled from (0.001, 100.0). See Fig. B for il- lustrations of synthesized negative samples and the corresponding generative sequences of Langevin dynamics.
B ADDITIONAL ABLATIONS
B. ROBUSTNESS TO POSE VARIATIONS
To validate the robustness of our method to pose variations, we perform additional experiments with pose variations and provide the quantitative comparisons in Table We report results by performing random +:90 rotation to samples during testing stage when adapting the trained model to novel categories. We observe that the textures categories (leather and grid) are robust to any pose variations in both the query and support samples. For the hazelnut category, where there intrinsically significant pose variations across samples, performing any new rotations does not influence the results. On the other hand, for the pill category, where all samples are aligned horizontally (see examples in Appendix, Figure A), performing rotation to only query samples results in performance drop. Performing rotation to both query and support samples help to recover the performance. The above results suggest the robustness of our methods under the condition of sufficient sample diversity in the support set.
Category
Leather
Grid
Hazelnut
Cable
Pill
Original
0.42
0.98
0.12
0.81
0.40
0.94
0.24
0.87
0.22
0.88
Rotate query
0.41
0.98
0.12
0.80
0.40
0.94
0.21
0.82
0.10
0.71
Rotate query and support
0.42
0.98
0.11
0.81
0.40
0.94
0.23
0.85
0.19
0.86
Table B: Robustness to pose variations. We report both mloU (left) and AUC-ROC (right) values.
B.2 COMPARISON TO IMAGE DIFFERENCING
Image differencing and its variances are an intuitive approach for detecting image anomaly. We per- form image differencing and background subtraction the industrial inspection and video anomaly detection task, respectively, and report performance in Table C Our method demonstrates clear advantages.
14

---

Under review as a conference paper at ICLR 2022
Category
Leather
Hazelnut
Cable
UCSD Ped1
UCSD Ped2
CUHK
Image differencing
0.02
0.57
0.12
0.77
0.07
0.65
65.34 77.42
59.03
57.12
Ours
0.42
0.98
0.40
0.94
0.24
0.87
91.22
75.32
Table C: Comparing with image differencing. We report mloU (left) and AUC-ROC (right) for industrial inspection task and AUC-ROC for video anomaly detection task.
B.3 ROBUSTNESS AGAINST CONTAMINATED TRAINING DATA
While it is common practice among machine learning practitioners to assume clean training data, this may not be true in real world applications. In this section, we evaluate the robustness of our method against contaminated training data by inserting certain amount of abnormal samples. As shown in Table D we progressively contaminate the normal training data by increasing the amount of abnormal data from 1% to 10%. Our proposed method is in general robust to data contamination, where contamination under 5% only decreases the performance slightly. The network is still able to perform decently in the extreme case of 10% contamination.
Contamination
Leather
Grid
Hazelnut
Cable
Pill
0%
0.42
0.98
0.12
0.81
0.40
0.94
0.24
0.87
0.22
0.88
1%
0.42
0.98
0.11
0.80
0.40
0.94
0.23
0.86
0.21
0.88
2%
0.41
0.98
0.11
0.80
0.40
0.93
0.23
0.84
0.21
0.87
5%
0.78
0.84
0.87
10%
0.38
0.95
0.10
0.76
0.37
0.90
0.21
0.82
0.19
0.85
Table D: Performance evaluation against data contamination. We contaminate the normal training data by inserting increasing percentages of abnormal data.
B.4 LEVERAGING TEMPORAL INFORMATION
EBM is agnostic the underlying backbone network architecture, therefore it is straightforward to incorporate temporal information into the image-based anomaly detection framework described in Section 5.2. We replace the 2D convolutions in the first four layers with 3D convolutions, and have the model accepts a 5-frame input instead. Without heavy tuning, we report 5-shot performance on both I frame' (original, without temporal) and frames' (with temporal) in Table E The results show that when temporal information is added, it leads to further improvements.
Category UCSD Ped1
UCSD Ped2
CUHK
Sh-Tech
frame
78.12 79.06
92.00
83.41 84.27
79.64
5 frames
91.94
80.80
Table E: Performance comparisons with temporal information incorporated.
C QUALITATIVE RESULTS
We present Figure T addiiionaal anomaly localization results categories with different anomalies in the MVTec-AD dataset.
D ADDITIONAL FIGURES
Common practice of sampling from EBMs is computationally costly, requiring many 50 steps of Langevin Dynamies when initialized with full noise, instead of synthesized negative sample as shown in Fig. B Some examples are provided in Fig. C.
15

---

Under review as a conference paper at ICLR 2022
Ingut
GT
Prodiction
Input
Prediction
Input
GT
Prediction
Figure A: Visualizations of anomaly localization industrial inspection data. All results are ob- tained by adapting the model using 10 normal samples only.
16

---

Under review as a conference paper at ICLR 2022
sample
Synlhssize negaive
Step
Step
Step
Slep
Figure B: Sampling outputs after few steps of Langevin Dynamics starting from each synthesized negative sample. 5 steps of Langevin dynamics are sufficient to quickly generate hard negative samples with minor artifacts.
Figure C: Figure from (hhao et al. 2020 Generated sequences from the process of Langevin Dynamics. Initialization from noise usually requires around 50 steps to synthesize the images with desired quality (images are shown every 5 steps).
17


Reviews:
[{"review_id": "vTre9Owal6f", "reviewer": "Reviewer_rnYM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The method adapts to new tasks autonomously;\n- Ablation study and comparison with supervised models;\n- Exhaustive implementation details in the Appendix section.\n\nWeaknesses: \n\n- Experimental analysis is limited to image and video datasets;\n- The mIoU metric is never defined in the paper;\n- Fairly limited scope of competitor methods (which could depend on the limited suitability of other methods in the adopted setting).", "summary_of_the_paper": "The paper proposes a framework for anomaly detection and localization that allows fast adaptation to new tasks. \nSpecifically, the authors propose an energy-based model (EBM) with an adaptive sparse coding layer directly trained with normal features of a target task.\n\nA meta-learning process is followed to extract common knowledge across tasks enabling few shots adaptation. \nShrinkage functions, sparse coding with large receptive fields, and learning by inpainting are introduced to improve and accelerate the EBM\ntraining. ", "summary_of_the_review": "The paper proposes an interesting anomaly detection approach and an experimental evaluation that also involves fully supervised models.\n\nResults are competitive, and some of them appear close to the upper bound performances provided by supervised alternatives.\n\nThe authors also provide an exhaustive description of the architecture and hyperparameters in the Appendix section.\n\nI think the main merit of the proposed method is the ability to achieve satisfactory anomaly detection performance without large availability of normal samples while adapting to new tasks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bTGe3YpARgd", "reviewer": "Reviewer_egaq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n+ The experimental result on two benchmark datasets outperforms most of the baselines.\n+ An ablation study is presented to demonstrate the robustness of the proposed method.\n\nHowever, there are also some places where this paper needs further clarifications:\n- It is not clearly shown in the experiment as to how the technique in Section 3.1 improves the robustness of the network on different types of objects.\n- A small typo in Section 3.1 - visualizations of the hard shrinkage function and SigShrink with different values of $\\tau$ are presented in Fig. 3 rather than Fig. 7.\n- The caption of Table I suggests that “Col 2-5 are fully supervised methods trained with massive normal samples” and are considered as the “upper-bounds”. However, it seems like the first column with AE (SSIM) is able to deliver better results in many categories than these upper bounds? (such as Carpet, Grid, and Bottle).\n- It seems from Table 2 that the proposed method without leveraging any temporal information can achieve even better performance in the CUHK Avenue and the SH-Tech dataset. More discussions are encouraged on the benefit of incorporating such temporal information versus image frames randomly sampled from the target scenes.\n", "summary_of_the_paper": "The author proposes a fast adaptive anomaly detection method using an adaptive sparse coding layer in the few-shot setting. The proposed method outperforms baselines.", "summary_of_the_review": "To summarize, this paper proposed a fast adaptive anomaly detection framework for anomaly detection in images. The proposed method is technically sound. However, the novelty is limited on the adaptive space coding layer with receptive field, while the rest of the network structure seems to be a big melting pot incorporating the Energy Bassed Model and episodic training in the context of meta-learning based few-shot learning. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_gCmFqMziZD", "reviewer": "Reviewer_4fuZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. Presents a well-reasoned set of design choices\n2. Good ablation studies\n\nCons:\n1. Assumes all training data is 'normal'\n2. Lacks experiments with contaminated training data\n\n\nMain Comments\n\n1. Expects all training data to be 'normal' -- effectively makes the algorithm fully supervised since we must manually make sure that all input train data is 'normal'.\n\n2. Should show performance of algorithm when training data is contaminated with anomalies (varying contamination fraction, say, from 0.0 ... 0.10).\n\n3. Section 3.1: \"The sparsity regularization to ... formulated as the mean squared error (MSE) between the original\" -- This is a nice explanation of the design choice.\n", "summary_of_the_paper": "The paper presents an anomaly detection algorithm that uses an EBM (Energy Based Model) to distinguish between 'normal' and 'anomaly'. This model generates pseudo-anomaly instances on-the-fly for each 'normal' instance and then learns to assign low energy to normal instances and high energy to pseudo-anomaly instances. The model is further designed to be able to adapt quickly (few 'normal' labeled examples) to new tasks.", "summary_of_the_review": "The paper mixes existing techniques in a nice way to make the overall solution very effective.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IfVjLa5G-W", "reviewer": "Reviewer_Mzn6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper can be viewed as a proposal of a new image anomaly detection system that combines existing methods. The proposed \"adaptive\" dictionary-based approach is not new. The use of a deep encoder for image analysis is, of course, not new. The term \"energy-based\" does not mean much because any probability distribution can be thought of as energy-based. So the question is whether the combination is innovative or not. \n\nAlthough I appreciate the authors' efforts to develop an industry-application-ready image processing system, I do not think the technical novelty meets the standard that ICLR papers are supposed to have. \n\n\nEq.(2)\nThe sign of ln Z?\n\nEq.(3) \ny is not defined in the generative model in Eq. (1). \n\np.3\n The problem setting should be clearly defined. What are the input and output? The anomaly score must be clearly defined in the main text as part of the problem setting. ", "summary_of_the_paper": "This paper proposes an image classification system, where a set of normal patterns are stored as a \"dictionary,\" and the degree of deviation is used as the anomaly score. \n \nThe overall architecture seems to feature a conventional deep encoder and a pattern matching module that compares the encoded latent vector against the pattern dictionary. The pattern matching is done by solving the lasso regression problem. \n\nThe authors propose a certain online learning approach, following existing few-shot learning methods, and also a synthetic sample generation approach using random perturbation combined with a gradient method. ", "summary_of_the_review": "- Good industrial use-case paper\n- Limited technical novelty -- less innovative combination of known methods. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7JKIsZpFTL-", "reviewer": "Reviewer_d5Wi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper claims a \"principled\" approach to detecting rare samples compared to normal ones.  One would expect that abnormality implies improbability.  The \"principle\" here is that something abnormal is something whose probability is small.  The paper never avails itself of any discussion of  what it means to be \"anomalous\" in terms of probability.  Similarly neither does the evaluation section mention anything about false positive or false negative detection rates. \n\nAs an example of this confusion, the use of the term \"normal\" in the introduction could refer to a normal, that is Gaussian distribution of the non-anomalous population. Fitting a normal distribution to samples, then looking for samples that fall on the tails of the distribution is a conventional method to detect anomalies.  Since this is not what the paper proposes, the authors may want to clarify this at the outset.  However it raises the speculation whether the mean squared error expressed in Equation 8 is just implicitly an appeal to a normal distribution. A quadratic error term is consistent with the use of a Gaussian (normal) distribution. \n\nHonestly I find it hard to tell if the evaluation presented in the paper makes the case for improvement over other deep learning methods, or more conventional image analysis methods.  It appears the main take-away from the evaluation section is the acceptable detection ability on limited number of samples.   How does this method stack up with previous methods when given comparably large numbers of training samples?  ", "summary_of_the_paper": "This paper presents a model for detecting irregularities in images, such as would occur in manufacturing acceptance testing.  As a form of anomaly detection, the claims made are for a principled modeling approach and an adaptive algorithm quick to learn from just a few samples.   Based on \"energy based models\" for modeling data densities, the paper claims improvements to avoid the need for retraining for new tasks, such as, instead of synthesizing negative samples rom noise, using a more targeted method of “learning from\ninpainting” operation to learn anomalies. \n\n", "summary_of_the_review": "In short, the term 'anomaly detection' has been applied to a wide range of unsupervised techniques, with little in common among them.  This literature, using deep learning methods that have no connection to the existing literature that is more than a decade old, calls into question whether the proposed methods are competitive with previous methods that do not originate by applying deep learning techniques.  One really has to look at the history of the field to claim to have made an advance. \n\nMy apologies for a shallow review due to a lack of familiarity with this branch of the field, however there are some basic questions that need to be raised about the lack of connection of the field with the vast work that precedes the current trend in deep learning to really understand the significance of the work. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work tries to tackle the problem of anomaly detection across different tasks. To do so, authors employ energy-based models (EBMs) and define an outlier score in terms of the EBM energies, having a shared sparse code for different tasks. This pipeline is tested on some image and video anomaly datasets for industrial inspection.

The reviewers highlighted some concerns that need to be addressed before the paper is ready for publication. 

First, a revision could benefit from a rewriting the clearly formalizes the learning problem from page 2 and then discusses about the possible modeling options given i) the task at hand and ii) some efficiency requirements.

Second, concerning the modeling choices of the proposed pipeline, the motivation behind the choice of EBMs should be strengthened. For example, it is not clear why the proposed sparse coding could be used for any other latent variable probabilistic model. As observed by one reviewer, the pros of having energies instead of probabilities (or just reconstructions from a deterministic autoencoder) is not discussed sufficiently. Additionally, the heuristics of running Langevin dynamics for only 5 steps should be backed up by stronger empirical evidence as it lacks theory, and it should be discussed how much you should run the Markov chain to obtain sensible negative samples.

Third, conclusions over the experiments on the provided benchmarks seem preliminary. For instance, a new revision could benefit from adding a statistical significance analysis to the reported accuracies. I appreciate that authors added further ablation studies including experiments on contaminated data in the latest revision. I suggest them to extend the experimental suite to more benchmarks including the commonly used for anomaly detection.
Decision: Reject