Few-Shot Fast-Adaptive Anomaly Detection
Authors: Ze Wang
Yipin Zhou
Rui Wang
Tsung-Yu Lin
Ashish Shah
Ser-Nam Lim
Venue: NeurIPS 2022 Accept (2022)

Abstract: The ability to detect anomaly has long been recognized as an inherent human ability, yet to date, practical AI solutions to mimic such capability have been lacking. This lack of progress can be attributed to several factors. To begin with, the distribution of ``abnormalities'' is intractable. Anything outside of a given normal population is by definition an anomaly. This explains why a large volume of work in this area has been dedicated to modeling the normal distribution of a given task followed by detecting deviations from it. This direction is however unsatisfying as it would require modeling the normal distribution of every task that comes along, which includes tedious data collection. In this paper, we report our work aiming to handle these issues. To deal with the intractability of abnormal distribution, we leverage Energy Based Model (EBM). EBMs learn to associates low energies to correct values and higher energies to incorrect values. At its core, the EBM employs Langevin Dynamics (LD) in generating these incorrect samples based on an iterative optimization procedure, alleviating the intractable problem of modeling the world of anomalies. Then, in order to avoid training an anomaly detector for every task, we utilize an adaptive sparse coding layer. Our intention is to design a plug and play feature that can be used to quickly update what is normal during inference time. Lastly, to avoid tedious data collection, this mentioned update of the sparse coding layer needs to be achievable with just a few shots. Here, we employ a meta learning scheme that simulates such a few shot setting during training. We support our findings with strong empirical evidence.

Few-Shot Fast-Adaptive Anomaly Detection
Ze Wang -t, Yipin Zhou', Rui Wang', Tsung-Yu Yu Lin', Ashish Shah', and Ser-Nam Lim !Purdue University
Meta Al
Abstract
The ability to detect anomaly has long been recognized as an inherent human ability, yet to date, practical Al solutions to mimic such capability have been lacking. This lack of progress can be attributed to several factors. To begin with, the distribution of "abnormalities" is intractable. Anything outside of a given normal population is by definition an anomaly. This explains why large volume of work in this area has been dedicated to modeling the normal distribution of a given task followed by detecting deviations from it. This direction is however unsatisfying would require modeling the normal distribution of every task that comes along, which includes tedious data collection. In this paper, we report our work aiming to handle these issues. To deal with the intractability of abnormal distribution, we leverage Energy Based Model (EBM). EBMs learn to associate low energies to correct values and higher energies to incorrect values. At its core, the EBM employs Langevin Dynamies (LD) in generating these incorrect samples based on an iterative optimization procedure, alleviating the intractable problem of modeling the world of anomalies. Then, in order to avoid training anomaly detector for every task, we utilize an adaptive sparse coding layer. Our intention is design a plug and play feature that can be used quickly update what is normal during inference time. Lastly, to avoid tedious data collection, this mentioned update of the sparse coding layer needs to be achievable with just few shots. Here, we employ a meta learning scheme that simulates such a few shot setting during training. We support our findings with strong empirical evidence.
Introduction
Anomaly detection is an important area of study in the field of artificial intelligence. It has found utility in computer vision applications such as industrial inspection [6] and video surveillance [28] 61 39), in the context of abuse prevention such misinformatton, fraud and network intrusion detection
60 X 35). and others such as system health monitoring and fault detection 4 42). In this paper, we propose approach for detecting anomaly in images, where have carefully designed steps to handle some of the bigger issues that have prevented the deployment of image anomaly detection in the real-world.
Image anomaly detection can generally be defined as the identification of abnormalities in given image. An exact definition of abnormality in this case is elusive because abnormality can be derived from any unknown distribution outside of a normal population. Many studies have hence focused on modeling the normal population instead of lcarning irregularities, where the goal is to capture the shared concept among all of the normal data as one or several reference models. This process usually requires investing significant efforts in curating large of normal samples for each task, after which anomaly is detected as deviations from the reference model(s) 58 Recent work from (50) provides algorithms that utilize only few normal samples to train models from scratch. However, the models still have to be provisioned for each new task, which requires considerable human efforts
Work done as intern at Meta Al. Contact: zewang@purdue..ed
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

and expertise, and thus lack the fast deployment criterion that is often time critical for real-world applications. In view of these challenges, our goals for this work are threefold. We are interested in designing an anomaly detection system that is capable of: (GI) modeling the normal population while the same time has a principled approach towards modeling the abnormalities; (G2) quickly adapting to new task at inference time; and (G3) requiring only a few normal shots to update itself to the new task at hand.
For (GI), we introduce the class of Energy Based Model (EBM), which is an important family EBMs have been shown demonstrate superior capability on modeling data density and localizing anomaly [20] For our purpose, the EBM we adopted learns to assign low energy to normal samples but high energy to abnormal samples. More importantly, the abnormal samples are generated with procedure known as Langevin Dynamics (LD) [54], which, in its original form, starts with a noise image and gradually samples from the distribution along the direction of lower energy. This lends itself gracefully to utilizing the generated intermediate samples as negative/abnormal. The LD procedure then coupled with maximum likelihood loss 24 that aims to maximize the energy differences between the normal and abnormal samples.
of generative models [62 17 57
To achieve (G2). we propose an adaptive sparse coding layer that is attached to the deep feature extractor in the EBM as Figure shows. The extracted deep feature is forwarded to the sparse coding layer, where the dictionary is constructed with the features of few normal samples of the given task. In essence, the input representation has been decomposed into a linear combination of normal features with the sparsity constraint imposed. The final energy score is measured by the distance between the original and the reconstructed features (after the sparse coding layer). Under this scheme, the dictionary for a particular task is not obtained by learning, but instead is constructed by the feature representations of few normal samples during inference. As result, this simple "plug-and-plaa trick allows the model to be adapted to novel tasks promptly without re-training. Further, we expect that the dictionary, which is formed by normal features, will not be able to explain the abnormal samples well, causing relatively high reconstruction error that lends itself for subsequent detection. As a bonus, a backward pass of energy score minimization can be used for localizing abnormal regions. We show that using gradient to localize anomalies yields superior robustness.
Towards (G3), we utilize meta learning 22 18 to simulate the scenario of being given new task with few normal shots to update the dictionary, followed by training the EBM. This is accomplished by episodic training, where in each episode the model is adapted to a held back task that is given a few normal samples. To accelerate the EBM training, we introduce "learning from inpainting". simple yet effective strategy for synthesizing hard abnormal samples quicker by starting the LD procedure with a synthesized image that is simply a normal sample with a noise patch injected as opposed to a noise image that is traditionally what is used.
We show the proposed few-shot aast-adaptive anomaly detection and localization framework is able to efficiently adapt to novel task (c.g., a new object category or scenes from new camera) with a few normal samples without training on both industrial inspection and video surveillance. Compared with previous methods that adapts to new task through either from scratch training in few shots 50 (55) or few-shot with few steps of gradient descent k the proposed framework is the first that
performs task adaptation with a single forward pass and without any gradient descent. Despite the fast adaptation, we provide both qualitative and quantitative results to demonstrate that our method outperforms other adaptive frameworks and is comparable to methods that rely on large amount of normal samples.
2 Backgrounds
In this section, we briefly introduce two key ingredients of the proposed method: EBMs and sparse coding.
Energy- -baed Model. In EBMs, the goal is to learn an energy function Ex(x): : Rd + R which parametrizes the data density (x( as:
(x))) Ix exep(_FFo Fig (x)))
where 0 is the parameter of the energy function and Zo ffx (xp(-EE(x)) is the partition function.
Approximating the true data distribution Pdata (x) is equivalent to minimizing the expected negative
2

---

log-likelihood function over the data distribution, defined by the loss function:
MLL =xxx-ppnxx log pe [x] [x--pan (xx) [x( + log ZØ].
(2)
As the computation of LML involves an intractable term Zø. the common practice is to represent the gradient of LML as,
VeLML- = (x)
- B
(3)
This objective decreases the energy of positive data samples x from the true distribution (normal samples in our use case) and increases the energy of negative samples x from the model po (synthesized abnormal samples). In practice, the synthesized negative samples achieved through Langevin dynamics [54], which a J-steps sampling along the direction of energy minimization is given by:
x = x
V FoFo wt, wh ~ (T), j j= (1......, J}
(4)
where is the step size, and the initialization x2 is sampled from a predefined prior distribution. The synthesizing ability of EBMs enables generating abnormal samples to help in learning more accurate data density, and often touted as the one the advantages of using an EBM.
Sparse coding. Approximating signal E IRd with the sparse linear combination over dictionary D gdxk can be expressed as:
min 2 11 Do + A||a||1,
(5)
where a is the sparse coefficients, with sparsity (4 norm) and the weight of the sparsity constraint. Da is sparse approximation to the original signal z. In practice, finding the dictionary atoms and the sparse coefficients usually formulated as an optimization problem.
In this paper, we adopt Iterative Soft Thresholding Continuation (ISTC) 25 to convert this optimiza- tion problem into linear operations with non-linear shrinkage function, which allows sparse coding to be seamlessly integrated into the deep neural networks. To compute sparse coefficient a, ISTC performs iterations of gradient steps on reconstruction z Da and proximal projection step to increase coefficient sparsity.
Formally, initializing the coefficients the first step on with all zeros, each step of ISTC refines the sparse code with descending values of from Amax to d.: each step ISTC expressed as:
-n/N
an+1 = aa + +D Dn),,An), with An
Amaa
Ama
(6)
X
where (-. here is a shrinkage function that truncates small values (lower than A) of the coefficients enforce sparsity, and can easily implemented by customized ReLU activation function:
(z, A) sgn(z)(max((z) >. 0)) sgn(z) X)-
Proposed Method
In this section, describe the proposed fast adaptive anomaly detection framework in details. In Section 3.I. we introduce the adaptive EBM which consists of a deep feature extractor followed by an adaptive sparse coding layer. From there, we further show that utilizing larger receptive field in the sparse coding could improve training robustness (Section 3.1.I. and applying smoothed shrinkage functions could help speed up convergence (Section 3.1.2 In Section 3.2] we describe the episodic training regime on various anomaly detection tasks that mimics few-shot adaptation the meta-testing stage while learning common knowledge across tasks. Instead of synthesizing negative samples (anomaly) directly from noise, we introduce simple but effective "learning from inpainting" operation to accelerate the training Section 3.3 Finally, we summarize training steps of the proposed method in Algorithm | and inference steps in Algorithm 2

---

d
d KhWww'
Sparse coding (b)
Reshupe
KX
K normal samples from the new task
Dictionary plug and play' (a)
Testing samples
Testing
Du
abnornal samples
Energy estimation Eg (x: D) MSB(x
Gradient maps
Backward pass
Figure 1: Overview the inference stage new task. (a) Adapting the task-specific dictionary with K normal samples. (b) Three iterations of sparse coding based on Eqn. [ We also show
backward pass from the reconstruction error localize the abnormal regions
3.1 Adaptive Energy-based Model
An EBM form of generative model and is widely used for modeling data density and sampling. While there has been recent work [20] applying EBM to anomaly detection, it still requires re- training for each new task. To efficiently adapt the EBM to novel tasks, we introduce an adaptive sparse coding layer which is conditioned on the dictionary constructed by the features of normal samples. Specifically, as illustrated in Fig given an input image, x g33hxx we first obtain the
corresponding feature z E Raxk' xw" from the deep feature extractor with parameters 8, so that then sparsely decomposed through the sparse coding layer over a task-specific dictionary D E pdxKkk' whiic contains the features of K normal samples of the current task as shown in the Fig [ a). Each feature vector of the normal sample features then directly used as an atom the task dictionary. The decomposed coefficients
(x; All feature vectors along spatial axes of
are S(z; D), where a RRK w ' xw' and denotes the iterative sparse decomposition process of (6. By multiplying the coefficient a with the dictionary D, we obtain the reconstructed encourages input features to be reconstructed by simple combinations of dictionary atoms (normal features), so that it would be difficult for features of abnormal samples to vell-approximated, therefore producing higher reconstruction errors that make it conducive for detecting anomalies. From here, the final energy score formulated as the mean squared error (MSE) between the original and the reconstructed features:
features z' Do. The sparsity regularization to a is important,
Ex( D) = MSE(2,, z')
L
0) DS (x; 8): D)
(8)
In effect, Eqn. depicct conditional EBM, which is conditioned the task-specific D formed by normal features. With the energy score, we can obtain pixel-wise anomaly localization maps through Vx (x; D), i.e., the gradients of pixels along the direction of minimization. High gradient magnitudes indicate regions that cannot be well explained by the dictionary D. Modifications to these regions can potentially remove the anomaly and reduce the energy as in Eqn. 4 In Section 4.] and Appendix Section B.5] we show that using the gradient (as a natural ingredient of EBMs with LD) more robust compared with auto-encoder and reconstruction based methods to generalize well to unseen tasks (Appendix Figure C.. In the following sections, we will discuss how to make the training of this adaptive structure more robust.
3.1.1 Sparse Coding with Receptive Field.
As discussed in Section 3.1 the input feature is represented as h' w' of d-dim feature vectors and they are treated independently while passing through the sparse coding layer. The region of the input image that affects one feature vector is determined by the receptive field of the feature extractor. The trade-off is that a small receptive field may not capture enough contextual information, while applying large receptive field would make feature maps spatially coarse and make it hard to

---

Support set
Adaptation
episode
Dictionary "plag and play'
Query set
(Evaluation)
V
episode n-1
Normal samples
episode
Symthesized
Step
abeormal samples
Process of Langevin dynamics
Figure 2: Illustration of episodic training and (a) "learning by inpainting". In each episode, a support is constructed with normal samples. The features of the normal samples are plugged into the adaptive sparse coding layer as the dictionary. Synthesized abnormal samples are corrected by performing gradient corrections with the gradient obtained by the direction of energy minimization.
spot small anomaly regions. To solve this dilemma, instead of carefully tuning the receptive field of each layer the feature extractor, we introduce simple yet effective technique of applying the receptive field on the sparse coding layer. Specifically, as illustrated in Appendix Fig A rather than performing sparse coding to each individual d-dim feature vectors, we apply on d x volumes centered around each feature vector, where / is the receptive field. This is equivalent applying I sliding window on spatial axes of the feature map and can be casily implemented by image column (Im2Col) operation. Then we flatten the feature volumes into dl--dim vectors and adjust the shape of the dictionary accordingly. In this way, we are able to capture contextual information without needing carefully tune the architecture of the feature extractor and we show in the later experiments that this technique improves the robustness of the network on different types of objects.
3.1.2 Shrinkage Function
The effectiveness of training the EBM for localizing anomaly regions heavily depends on the gradient propagation from later to earlier layers.
shown in [13] that smooth activation functions like Swish 45 could be beneficial here. Notably, the gradients of the dictionary D are determined by the sparse coding coefficients a as shown in Eqn. 6 However, the sparsity constraint of a would turn off the gradient computation of many elements D and this could be detrimental during the early stage of the training. To alleviate the sparse gradient issue, we replace the RELU-like shrinkage function in Eqn. 7 with its smoothed counterparts by introducing the Sigmoid based shrinkage functions (SigShrink). The SigShrink function is originally proposed for non-parametric signal estimation in and be defined as:
(z A) 1 exp(-T(pz) X))
where 7 the hyperparameter of smoothness. We present visualizations of the hard shrinkage function Eqn. and SigShrink with different values of r in Fig B Comparing the hard shrinkage function which truncates small values into zeros, the SigShrink with a large can sharply force small values near-zeros. Therefore, the SigShrink will guarantee non-zero gradients everywhere.
3.2 Episodic Training
To train the proposed adaptive EBM, we perform episodic training that is widely adopted by meta- learning few-shot learners |8 55 Following the terminology of few-shot learning, in each training episode, the model is adapted and tested with a task sampled from the underlying task distribution. Specifically, the model is adapted to support set of the given task, then query with ground truth labels is applied to evaluate the adaptation, which used to update the model parameters. As
shown Fig 2 the support of the i-th episode task contains small number of K normal samples { 1K Jmm1 The features z 4 (x 0) of these normal samples are plugged into the dictionary D Kh'w" corresponding to the i-th task to adapt the dictionary. After that, the adapted model
5

---

is measured by a query set consisting of M normal samples {q Jmm and M abnormal samples {á /m=1. Note that there no actual abnormal samples given during training. instead, they are iteratively sampled from the EBM and the sampling will be discussed in details in Section 3.3 Recall that the training of EBM with contrastive divergence as Eqn. L requires the estimation energy scores of both positive samples from the true data distribution and negative samples from the modeled distribution. The positive energy can be estimated empirically with normal query set samples. The negative energy can be estimated by performing the MCMC-based sampling technique [37 (54). typically Langevin Dynamics as described Eqn. Denoting the output of Langevin dynamics (sampled abnormal samples) initialized with qm LD(qm)) we have the empirical estimation of the contrastive divergence of the i-th episode as:
Ced
Bold'n - (LDD(L)::D))
(10)
With the energy score equivalent to the feature reconstruction error in Eqn. g minimizing Lat encourages normal features to be well-reconstructed by sparse linear combination of dictionary atoms while the features from abnormal samples tend to produce relatively higher reconstruction errors that they can be casily spotted.
3.3 Synthesizing Negative Samples
Typical EBM training with contrastive divergence conducts negative sampling from the modeled density using techniques such Langevin Dynamies, which applies gradient descent to noise initialization with small step size and large number steps [T7 Such negative sampling steps can be costly and argue that is unnecessary by inpainting". Starting from a positive query sample I we synthesize the corresponding negative sample q.. by randomly placing small uniform noise patch the image. The Langevin Dynamics procedure is then initialized with the resulting image instead of a noise image. As the Langevin Dynamics proceeds, synthesized abnormal samples LD(q')) are inpainted along the direction of "normal", In. and we introduce the following reconstruction loss:
case. Instead, we introduce
strategy of "learning
L100
(11)
We show Fig ((a) that, starting from synthesized abnormal sample, only 5 steps of Langevin dynamic would be sufficient to make it visually close to the corresponding normal sample during training, serving as "hard negatives" that further facilitates the learning. The final loss of the episodic training is simply:
== YoCree + To
(12)
where Tlu and Th are balance two loss terms. We summarize the overall training and inference procedure in Alg. and Alg. 2 Prespectively.
Experiments
In this section, we conduct evaluation on the industrial inspection task with the MVTec-AD dataset
[5, 6 (Section 4.1 Even though our proposed framework is image-based, we further demonstrate
it's efficacy on the video anomaly detection task in Section 4.2] In Section [
we show ablations and
insights relating to the adaptive sparse coding components. We show additional ablations including the superiority of using gradient of EBMs over pixel-wise reconstruction localize anomalies in App. B and we provide implementation details in App.
4.1 Industrial Inspection
The goal of this anomaly detection task is to predict whether manufactured component contains any defects. The MVTec-AD dataset includes 15 categories of object. To demonstrate the fast adaptation capability of the proposed method, we adopt a leave-one-oul training strategy. Specifically, samples of each target category are reserved for testing only, and the episodic training is performed on the

---

Algorithm Training procedure.
1: Given: feature extractor with parameter 6; training dataset of multiple tasks with positive (normal)
samples only.
2: Given: Number of shots K; number of query samples Q: step size / of Langevin dynamics; total training
episodes I; and learning
Initialize the feature extractor y..
4: for Episode i = 1: do
5: Sample the i-th task from the dataset, and randomly pick K + M samples to form the support set {st Jk and the query set (q
6: Generate corrupted query samples {q Jm=1 by placing random patches to {qin h I'm=l.
7: Extract the support and query sample features with v and update the adaptive sparse coding layer with i-th task dictionary D which is constructed by support sample features. The energy function of the i-th task is now parametrized by Eo(-, D").
Obtain synthesized negative samples (LD(G()) (M))) with the updated energy function using Langevin dynamic in
9:
Obtain the final loss & with Led 10 and free
10: Update parameters 0 +- 0 vVo..
11:
end for
Return with parameter 0.
Algorithm 2 Inference procedure on task indexed by i.
1: Given: Feature extractor V with trained parameter O.
2: Given: The support set (st, Tk=1 and query samples {q'n JM be tested.
3: Extract normal feature tensor Z E RR RRxxxaa xxax'''x''
4: Reshape the normal feature tensor into matrix, and it the task-specific dictionary D' E RdxKk'''
5: Estimate the anomaly score a test sample qin by its energy score using Equ g with (qi: D =
D' D'S( (qm: 0); 1' 2 where higher energy score indicates that q'n is more likely to be an abnormal sample.
|| (q'm
6: The pixel-wise anomaly map a test sample q can obtain by visualizing Vat.. EE D'), where abnormal regions show higher magnitude.
remaining categories. During the training stage, the model will not see any samples from the target category. During testing, we first adapt the model to the target category with 10 randomly selected normal samples, then measure the performance with the entire testing set. We run the test 5 times, each time the model is adapted to random sets of 10 normal samples from the target category.
In Table we first show performance of "upper- bound" methods, which train each category from scratch with massive normal samples. Specifically, 7 G train auto-encoders (AF) with normal samples and measure the reconstruction errors during the inference; AnoGAN 48 adopts a generative
adversarial network (GAN) to learn the manifold of normal; VE-VAE
20 presents a visually
explainable variational auto-encoder through gradient-basee attention. For apple-to-apple comparison, we create strong baseline by applying model-agnostic meta-learning [88
AE (denoted as
MAML-AE, more details App. Sec. A.3). To the best of our knowledge, the proposed method the first that is capable of producing strong performance pass and no further training. This strongly suggests that the learned parameters are effectively shared across tasks, greatly helping to accelerate the model deployment process that is typically cumbersome otherwise. All results from our methods are obtained with flipping and 90" rotation augmentations. The proposed method outperforms MAML-AF by large margin. Our results are even competitive with the "upper-bounds" some categories. We show the localized anomaly regions from our method in Fig Additional visualizations are in the App. Fig D
new tasks, using just single forward
4.2 Video Surveillance
In video anomaly detection, common goal is to detect abnormal events captured by surveillance cameras (e.g., a motorcycle on the sidewalk). A model trained on videos from one camera might not generalize well on other cameras due to different locations / mounting heights / lightning conditions, and not feasible to train one model for every new camera in practice. The ability to quickly adapt to new scenes is significant contribution to the task of video surveillance. We are only aware of the
7

---

Category
AB (SSIM)
AE (MSE)
ABoGAN
VE-VAE
MAML-AE
Ous
Curpet
0.69
0,38
0.34
0.1
0.20
0.26
0.87
0.59
0.54
0.78
0.68
0.84
Grid
0.88
0.83
0.04
0.02
0.01
0.12
0.94
0.90
0.58
0.73
0.53
0.82
Leatber
0.71
0.78
0.67
0.75
0.34
0.64
0.74
0.87
0.77
0.12
0.95
0.40
Tile
0.04
0.23
0.08
0.14
0.14
0.26
0.59
0.51
0.50
0.93
0.52
0.76
Wood
0.36
0.29
0.14
0.47
0.11
0.23
0.73
0.73
0.62
0.91
0.68
0.78
Boule
0.15
0.22
0.05
0.07
0.02
0.23
0.93
0.86
0.86
0.78
0.56
0.82
Cahle
0.01
0.05
0.01
0.18
0.04
0.18
0.82
0.86
0.78
0.90
0.74
0.80
Capsule
0.09
0.11
0.04
0.11
0.03
0.10
0.94
0.88
0.84
0.74
0.68
0.90
Hazelnut
0.00
0.41
0.02
0.44
0.11
0.40
0.97
0.95
0.87
0.98
0.72
0.94
Metal nut
0.01
0.26
0.00
0.49
0.10
0.28
0.89
0.86
0,76
0.94
0.78
0.78
Pill
0.07
0.25
0.17
0.18
0.10
0.11
0.91
0.85
0.87
0.83
0.62
0.88
Screw
0.03
0.34
0.01
0.17
0.02
0.08
0.96
0.96
0.80
0.97
0.55
0.86
Toothbrush
0.08
0.51
0.07
0.14
0.06
0.18
0.92
0,93
0.90
0.94
0.80
0.85
Transistor
0.01
0.22
0.08
0.30
0.02
0.18
0.90
0.86
0.80
0.93
0.76
0.80
zipper
0.10
0.13
0.01
0.06
0.04
0.15
0.88
0.77
0.78
0.78
0.68
0.86
Input
GT
Prediction
Table I: Numerical evaluation of anomaly localization on MVTec-AD. We report both mloU (top rows) and AUC-ROC (bottom rows) values. Col 2-5 are upper-bound methods trained with massive normal samples.
Visualizations of localized anomaly by our method.
Figure 3:
work in 32 (r-GAN) that has such adaptation capability. Specifically, the model adapts to new scene using gradient descent with several beginning frames of query video, after which a GAN is applied to generate future frames. Anomaly is then detected via the discrepancy between predicted future frames and the original frames. Note that the MAML-AE baseline we conducted Section can be seen as an ablation of r-GAN on the single-frame frame without temporal information.
We follow the same evaluation regime r-GAN by training with normal samples in all 13 scenes from SH-Tech [28] and testing on UCSD Pedestrian 1, UCSD Pedestrian 2 44 and CUHK Avenue Note that since our method is image-based, it predicts the video frames independently without leveraging any temporal information as in r-GAN. In each episode, we adapt our model with support set containing few normal frames randomly sampled from the target scenes. In Table A we compare our method against r-GAN tre-trained on SH-Tech only (r-GAN Pre-train), fine-tuned on target datasets (r-GAN Fine-tune), and with one step gradient descent with meta-learning (r-GAN MAML). We also show the performance of MAML-AE as a baseline for image-based meta-learning method. In the last section of Table 2 we present
3C
results as well by training with scenes of SH-Tech and testing on remaining 7. We follow common evaluation protocol and measure the frame- AUC-ROC. Without leveraging temporal information and re-training (gradient descent), our method achieves comparable results to r-GAN MAML and outperforms image-based meta-learning method by a large margin. In App B.4 we show that incorporating simple temporal information can further improve the performance.
4.3 Ablation Studies
Sparse coding receptive fields. evaluate the effectiveness of using large receptive fields the sparse coding layer, we conduct additional experiments on the MVTec-AD dataset, and select representative categories with different levels of difficulties to present the comparisons with and I = (Sec. B.I.I. in Table Sparse coding with large receptive field clearly benefits more complex

---

Targel datasets
Methods
5-shot
10-shot
GAN Pre train
73.10
73.10
73.10
Fine-tune
UCSD Ped
G-N MAML
80.60
81.42
82.38
MAML-AE
64.12
66.88
67.34
Ours
77.42
78.12
78.65
=GNN Pre-train
81.95
81.95
81.95
r-GAN Fine-tune
85.64
89.66
91.11
UCSD
I-GAN MAML
91.80
92.80
MAML-AE
78.24
82.04
83.30
Ours
91.22
92.00
92.45
=-GAN Pre-train
71.43
71.43
71.43
rGN Fine tune
75.43
76.52
77.77
CUHK Avenue
I-GAN MAML
76.58
77.10
78.79
MAML-AE
70.01
Ours
80.68
83.41
84,46
r-AN Pre-train
70.11
70.11
70.11
r-GAN Fine-tune
71.61
70.47
71.59
MAMI.
77.36
MAML-AE
66.62
67.12
68.04 68.04
Ours
75.32
79.64
81.28
Input
Prediction
Table 2: Frame-level AUC-ROC for the video anomaly detec- tion tasks.
Figure 4: Visualizations of anomaly localization with video anomaly detection.
0.12
stinkoge
Category
Leather
Grid
Hazelnut
Cable
Zipper
0.95
0.80
0.36
0.91
0.76
0.12
0.84 0.86
0.40
0.95
0.12
0.81
0.40
0.94
0.18
0.80
0.15
Table Comparison of different sparse coding receptive fields. We report both mloU (left) and AUC-ROC (right) values.
0.06
6.04
2 Mn
0.02
Category
Leather
Hazelnot
Cable
10505
15000
Ours
0.40
0.95
1 1.6e-4
0.40
0.94
24c-4
0.18
0.80
2.0c-4
Terations
sparsityy
0.32
0.90
0.9c.4
0.24
0.80
1.7e.4
0.12
0.68
1.5e
Figure Loss curves with smooth (SigShrink) and non-smooth (hard- shrink RELU-like) shrinkage func- tions.
Table Performance w/ and w/o sparsity constraint. From left to right: mloU; AUC-ROC; the difference of averaged reconstruction errors between abnormal/nornal samples.
structural objects (hazelnut, cable, and capsule), while the improvements are limited for the texture objects (leather and grid), where contextual regularization is intuitively less important.
Shrinkage functions. To show the benefits of smooth shrinkage function, we plot the loss curves of models trained with smooth SigShrink (Eqn. 9 and non-smooth RELU-like shrinkage (Eqn. (7) functions in Fig 5 The model with smooth shrinkage function converges notably faster in the early training stage and achieves lower loss.
we impose sparsity constraint to the feature decomposition in the adaptive sparse coding layer, in order to prevent abnormal features from being well-approximated by the linear combinations of normal features, so that the reconstruction errors effective for detecting anomaly. To validate this. conduct experiments by removing the shrinkage function o the sparse coding stage (Eqn. 6). We show comparison in Table with mloU, AUC-ROC, and the difference of averaged reconstruction errors between abnormal and normal samples. Without sparsity, the performance drops dramatically, and reconstruction errors of normal and abnormal samples become closer.
Sparsity constraint. As discussed in Section 3.1

---

Related Work
Anomaly detection with sparse coding. Early efforts on adopting sparse coding in anomaly detection are based on optimization (with L1 penalty) RD [I]. Recent advances on iterative sparse thresholding algorithms [| 25 allow seamless integration of online sparse coding with deep neural
networks, and 33 formulates the sparse coding as stack RNNs for video anomaly detection.
Anomaly detection with generative models. One of the core challenges anomaly detection is that abnormal samples are usually unavailable in the training stage. Generative models are widely utilized in anomaly detection due the capability in modeling the density of desired data distribution. Early efforts on variational autoencoders ( (VAE) based methods [1] [58] are arguably having hard time calibrating uncertainties in novel samples [6. accurately localizing abnormal regions through reconstruction errors 12]. Recent efforts have explored variant generative architectures like energy-
based models (EBM) 220, GANs
[50, and combining VAE with EBM [2]. Various methods
also exploit intra-image structures [TO 7 cross-frame consistency
B1 and motion-appearam
consistency in videos
39 while detecting anomaly.
Few-shot learning. Few-shot learning is extensively explored in classification tasks. It leverages common knowledge extracted from distribution of tasks, and induces an adaptive model that fits to new classification task with as few optimization 18 47
sample per class. Proposed methods are based on
19 [59 46. learning metric [51] 53) and parameter prediction [22 + 21 These
technologies further applied in other tasks like image generation detection
27 and out-of- -ifriitiinttion
Energy-based models. As family of generative models, studies EBMs 26
mainly focused
on probabilistic modeling and sampling over data, either unconditionally B8
40
1
[S]
[3
[57]
56 2), or conditionally [13] [4] This has been recently followed by extensions other applications
that include reasoning [6] latent space modeling of generative models [1]] and anomaly detection
12
6
Conclusion
In this paper, we introduced few-shot fast-adaptive anomaly detection. We formulated our model as an energy-based model with an adaptive sparse coding layer, which the dictionary directly formed by normal features of target task. We adopted episodic meta-learning to learn common knowledge across tasks, which enables few shots adaptation. We further introduced smooth shrinkage functions, sparse coding with large receptive fields, and learning by inpainting to improve and accelerate the training. Notably, when evaluating our method's performance on industrial inspection and video anomaly detection, our method is comparable and even boasts better performance than methods trained with a large amount of normal samples. Through this work, we hope to have made a significant contribution the important problem of anomaly detection by shedding light our findings that anomaly detection can indeed be generalized to new tasks with a few normal samples only.
Social Impact and Ethics. As general framework for few-shot anomaly detection, the proposed method does not suffer from particular ethical concerns or negative social impacts. All datasets used public, and we have blurred all human faces in the qualitative visualizations.
10

---

References
[1] Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruction probability. Special Lecture on IE, 2()::1118, 2015.
[2] Michael Arbel, Liang Zhou, and Arthur Gretton. Generalized energy based models. ICLR, 2021.
[3] Abdourrahmane M Atto, Dominique Pastor, and Gregoire Mercier. Smooth sigmoid wavelet shrinkage for non-parametric estimation. In 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 3265 3268. IEEE, 2008
[4] Yuequan Bao, Zhiyi Tang, Hui Li, and Yufeng Zhang. Computer vision and deep learning learning-based data anomaly detection method for structural health monitoring. Structural Health Monitoring. 18(2)400-442, 2019.
[5] Paul Bergmann, Kilian Batzner, Michael Fauser, David Sattlegger, and Carsten Steger. The mvtec anomaly detection dataset: comprehensive real-world dataset for unsupervised anomaly detection. International Journal of Computer Vision, 129(4))0038- 10098, 2021.
[6] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad comprehensive real-world dataset unsupervised anomaly detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2019.
[7] Paul Bergmann, Sindy Löwe, Michael Fauser, David Sattlegger, and Carsten Steger. Improving unsupervised defect segmentation by applying structural similarity to autoencoders. arXiv preprint arXiv: wwXiv:180..00011 2018.
[8] Richard Bolton and David Hand. Statistical fraud detection: A review. Statistical science,, 17(3):335-255, 2002.
[9] Louis Clouâtre and Marc Demers. Figr: Few-shot image generation with reptile. arXiv preprint 1eXiv:9901.02 2019.
[10] Niv Cohen and Yedid Hoshen. Sub-image anomaly detection with deep pyramid correspondences. arXiv preprint urXiv:2005.02355 2020.
[11] Ingrid Daubechies. Michel Defrise, and Christine De Mol. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 57(11):1413-1457 2004.
[12] David Dehaene, Oriel Frigo, Sébastien Combrexelle, Pierre Eline. Iterative energy- projection on normal data manifold for anomaly localization. ICI.R, 2020.
[13] Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. NeurIPS, 33:6637-6647, 2020.
[14] Yilun Du, Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. NeurIPS, 34:156666 1611561--1620. 2021.
[15] Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. International Conference on Machine Learning, 2021.
[16] Yilun Du, Shuang Li. Joshua Tenenbaum, and Igor Mordatch. Learning iterative reasoning through energy minimization. In /CML, pages 5570-5582. PMLR, 2022.
[17] Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. NeurIPS, 2019.
[18] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, pages 1126 1135, PMLR. 2017.
[19] Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic madel-agnostic meta-le arXiv preprint arXiv:1806. 100..02880.8 2018.
[20] Ergin Utku Genc, Nilesh Ahuja, Ibrahima Ndiour, and Omesh Tickoo. Energy-based anomaly detection and localization. arXiv preprint arXiv:2105.0322 2021.
[21] Spyros Gidaris and Nikos Komodakis. Generating classification weights with gan denoising autoencoders for few-shot learning, In IEEE Conference on Computer Vision and Pattern Recognition, pages 21-30, 2019.
11

---

[22] Jonathan Gordon, John Bronskill. Matthias Bauer. Sebastian Nowozin, and Richard E Turner. Meta-learning probabilistic inference for prediction. arXiv preprini vrXiv:1805.09921, 2018.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[24] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neurul computation. 14(8):1771-1800, 2002.
[25] Yuling Jiao, Bangti Jin, and Xiliang L.. Iterative soft/hard thresholding with homotopy continuation for sparse recovery. IEEE Signal Processing Letters, 24(6):784-788, 2017.
[26] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
(27] Ming- Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few- shot unsupervised image-to-image translation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10551-10560, 2019.
[28] W. Liu, D. Lian W. Luo, and S. Gao. Future frame prediction for anomaly detection new baseline. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[29] Wenqian Liu, Runze Li, Meng Zheng. Srikrishna Karanam, Ziyan Wu, Bir Bhanu, Richard J Radke, and Octavia Camps. Towards visually explaining variational autoencoders. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8642-8651, 2020.
[30] Cewu Lu, Jianping Shi, and Jiaya Jia, Abnormal event detection at 150 fps in matlab. In Proceedings of the IEEE international conference on computer vision, pages 2720-2727, 2013.
[31] Yiwei Lu, K Mahesh Kumar, Seyed shahaheddin Nahavi, and Yang Wang. Future frame prediction using convolutional vmn for anomaly detection. In 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pages 1-8. IEFE, 2019.
[32] Yiwei Lu, Frank Yu, Mahesh Kumar Krishna Reddy, and Yang Wang. Few-shot seene- aeene-adaptvve anomaly detection. In European Conference on Computer Vision, pages 125-141. Springer, 2020.
[33] Weixin Luo, Wen Liu, and Shenghua Gao. A revisit of sparse coding based anomaly detection in stacked rnn framework, In Proceedings of the IEEE International Conference on Computer Vision, pages 341-349, 2017.
[34] Vijay Mahadevan, Weixin Li, Viral Bhalodia, and Nuno Vasconcelos. Anomaly detection in crowded scenes. In 2010 IEFE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1975-1981, IEFF, 2010.
[35] Biswanath Mukherjee, Todd Heberlein, and Karl N Levitt. Network intrusion detection. IEEE network, 8(3):26-41, 1994.
[36] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Tch, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprint
09666, 2018.
[37] Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo. 2(11):2, 2011.
[38] Jiquan Ngiam, Zhenghao Chen, Pang W Koh, and Andrew Y Ng. Learning deep energy models. In ICML, 2011.
[39] NoogeN Nguyen and Jean Meunier. Anomaly detection video sequence with appearance-motte motion correspondence. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1273-1283, 2019.
[40] Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of meme-based maximum likelihood learning of energy- based models. In AAAI, 2020.
[41] Bo Pang, Tian Han, Erik Nijkamp. Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. NewrlPS, 33:2 1994-99--2008, 2020.
[42] Afrooz Purarjomandlangrudd, Amir Hossein Ghapanchi, and Mohammad Esmalifalak. A data mining approach for fault diagnosis: An application of anomaly detection algorithm. Measurement, 55:343-351 2014.
12

---

[43] Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L Yuille. Few-shot image recognition by predicting parameters from activations. In IEEE Conference on Computer Vision and Pattern Recognition, pages 7229-7238, 2018.
[44] Yixuan Qiu, Lingsong Zhang, and Xiao Wang. Unbiased contrastive divergence algorithm for training energy-based latent variable models. In ICLR, 2019.
[45] Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint urXiv:1710.0594] 2017.
[46] Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. International Conference Learning Representations, 2016.
[47] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-leamingg with latent embedding optimization. International Conference on Learning Representations, 2019.
[48] Thomas Schlegl, Philipp Seebück, Sebastian M Waldstein, Ursula Erfuuthh,,, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In International conference information processing medical imaging, pages 146-157. Springer, 2017.
[49] Vikash Sehwag, Mung Chiang. and Prateek Mittll Ssd: A unified framework for self-supervised outlier detection. arXiv preprint arXiv:2103. /2051, 2021.
[50] Shelly Sheynin, Sagie Benaim, and Lior Wolf. hierarchical transfomatiion-diseriiiin generative model for few shot anomaly detection. arXiv preprint M..v:22004.15333 2021.
[51] Jake Snell, Kevin Swersky, and Richard S Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems, 2017.
[52] Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial intelligence review, 18(2):77-9) 95, 2002.
[53] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning, Advances in Neural Information Processing Systems, 2016.
[54] Max Welling and Yee Teh. Bayesian learning via stochastic gradient langevin dynamics. Proceedings of the 28th international conference on machine learning (ICML-11), pages 681 1-888. Citeseer, 2011.
[55] Chih-Ciang Wu, Ding-Jie Chen, Chiou-Shann Fuh, and Tyng-Luh Liu. Learning unsupervised metaformer for anomaly detection. In International Conference on Computer Vision, pages 4369-4378, 4378, 2021.
[56] Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: A symbiosis between variational autoencoders and energy- based models. ICLR, 2020.
[57] Jianwen Xie, Yang Lu, Song- Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning, pages 2635 2644. PMLR, 2016.
[58] Haowen Xu, Wenxiao Chen, Nengwen Zhao, Zeyan Li, Jiahao Bu, Zhihan Li, Ying Liu, Youjian Zhao, Dan Pei, Yang Feng, et al. Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications. In Proceedings of the 2018 World Wide Web Conference, pages 196, 2018.
[59] Jaesik Yoon, Taesup Kim. Ousmane Dia, Sungwoong Kim. Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning, In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 7343-7353, 2018
2018.
[60] Qiang Zhang, Aldo Lipani. Shangsong Liang, and Emine Yilmaz. Reply-aided detection of misinformation via bayesian deep lcarning. In The world wide web conference, pages 2333 2343, 2019.
[61] Bin Zhao, Li Fei-Fei, and Eric P Xing. Online detection of unusual events in videos via dynamic sparse coding. In CVPR 2011, pages 3313 33200. IEEE, 2011.
[62] Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. arXiv preprint arXiv: /609.00:60000000 00090019933333 2016.
[63] Yang Zhao, Jianwen Xie, and Ping Li. Learning energy energy-based generative models via coarse-to-fin expanding and sampling. In International Conference on Learning Representations. 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report bars (c.g.. with respect the random seed after running experiments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) your work uses existing assets, you cite the creators? [Yes]
(h) Did you mention the license of the assets? [N/A]
(c) Did you include any
either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you're 're
ing/curating? [No]
(c) Did you discuss whether the data you using/curating contains personally identifiable informa- tion offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "p0hp-dbwXyb", "reviewer": "Reviewer_ezfd", "summary": "The paper proposes a framework for building an anomaly detector which can be adapted to new tasks at inference time with a few examples from the new task. This is in-contrast to standard anomaly detection methods which had to be rebuild for each new task. For example, a standard autoencoder can be trained for normal tasks(like video surveillance)  and anything with a large reconstruction error can be labeled as anomalous. Now if the scene or camera changes then the autoencoder will have to be re-trained. This paper proposes a three tier framework where the trained anomaly detector can be adapted at inference time.", "questions": "How well does the method do with so called \"co-variate shift\". The video surveillance example might be an example of co-variate shift ?", "limitations": "No potential negative societal impact.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(i) Paper clearly defines the three tasks G1, G2 and G3 which makes it easy to comprehend.\n\n(ii) The idea of using intermediate examples during the LD process as anomalies and using them to train an EBM is novel. Similarly the idea of borrowing ideas from sparse autoencoders and \"re-training\" it with a few examples makes the detector adaptive.\n\n(iii) The results are quite impressive. For example, in Table 1, the proposed method is even competitive with \"upper-bound\" where the autoencoders (or other methods) are trained with many samples from each category.\n\nWeakness:\n\n(i) My slight concern is that the whole framework might be too complicated to be actually deployed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VqQvqzq9LC9", "reviewer": "Reviewer_hTet", "summary": "The paper presents an anomaly detection technique based on EBM (energy based model). The proposed technique is supposed to help adapt to new tasks faster with just a few rounds of training.", "questions": "Could the authors elaborate more clearly on how the task-specific dictionary is created?", "limitations": "N/A", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed algorithm is a combination of several existing techniques. A fundamental contribution is hard to identify.\n\n\nThe paper is very hard to read and also has numerous typos such as:\nLine 11: associates => associate\nLine 47: learn => learns\nLine 146: pixe => pixel\nLine 178: SigShink => SigShrink\nLine 245: auto-encode => auto-encoder\nLine 249, 251: adapt => adapting\n\n\nFigure 1 does not really have labeled parts a, b. It also is not very intuitive or informative in explaining parts of the algorithm, e.g., line 135 refers to Fig (a) in relation to the dictionary D. But no additional clarity about D is offered by reference to this figure.\n\n\nThe construction of task-specific dictionary D has not been explained very well. In general it would help to present the content in Sections 3.1 and 3.2 in a more intuitive manner.\n\n\nIt certainly helps that the paper does an ablation study (Sec 4.3) and shows the benefits of the various parts of the algorithm.\n\n=============\nUpdate after author rebuttal:\n\nThanks to the authors for addressing my concerns. I am revising my scores.\n--------------------", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "P4rUcBhf0Mg", "reviewer": "Reviewer_3BqJ", "summary": "This work presents a new algorithm for image-based anomaly detection. The algorithm relies upon the contrastive-divergence method for training energy based models and it presents a differentiable \"sparse-coding layer\" which seems to be novel. Beyond the algorithmic innovations the paper also presents practical techniques for improving the speed of generating samples from an EBM for images. This work also presents experiments for few-shot learning their proposed algorithm across new domains. \n\nThe basic idea is to use sparse-coding as a way of approximating whether a novel sample is similar to in-distribution samples or not. The innovation lies in implementing sparse-coding in a differentiable manner that lets them train the entire architecture in an end-to-end manner.", "questions": "Please see the questions mentioned in the \"Strengths and Weaknesses\" section.", "limitations": "Yes the authors adequately discusses the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper presents a new algorithm for image anomaly detection with good empirical performance and some innovative ideas. I believe the results/techniques could be quite useful in other contexts where researchers want to perform nearest-neighbor decomposition in a differentiable manner. \n\nHowever the techniques in the paper sometimes seem to be put together a little haphazardly and the paper does not do a good job of clearly explaining the contribution of many of the important parts. For example, a lot of space is devoted in the paper on episodic training as a way of enabling few-shot learning. However it isn't clear from the experiments that how important is the episodic meta-learning for few-shot learning. The biigest help for few-shot learning comes from the sparse-coding layer which doesn't have any learnable parameters.  For example if instead of breaking up the training into $I$ separate episodes where each episode was one task, if we simply trained in one episode on all the data what will be the performance loss ?\n\nAlso the paper sometimes makes claims that are a hard to understand/verify. For example, the paper claims that proposed method is the \"first to allow [adaptation] to new tasks with a single forward pass without any training\". However even a simple process such as finding the distance to the 1-nearest neighbor allows some level of adaptation in a single forward pass.\n\n--- After the author response -- \n\nthe authors replied back with more evidence about the importance of meta-learning which seems sufficient, updating my rating now\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept