Patchwise Sparse Dictionary Learning from pre-trained Neural Network Activation Maps for Anomaly Detection in Images

1 INTRODUCTION . Anomaly detection is the problem of identifying , given a set of data , those examples that deviate from the expected or standard behavior . In computer vision , anomaly detection tasks recognize those images showing objects or patterns with corrupted components or irregularities . Its applications range from industrial quality control to medical analysis . The typical approach to anomaly detection tries to model data distribution without anomalies and then estimate irregularities as examples with lower probability outcomes . The task is particularly demanding because it is difficult to understand and model anomaly occurrences due to three main challenges : data imbalance , concept drift , and selection bias . The first relates to the lack of anomalous examples ; the second represents the variation of anomalies over time ; the third reveals that the partial knowledge of the phenomenon introduces a bias in acquired data . These reasons pushed researchers to focus on unsupervised or weekly supervised machine learning techniques , leveraging part of the available data , usually composed of defect-free examples . Indeed especially in the context of anomaly detection over images for industrial quality control , we face data imbalance and concept drift problems , but we usually can collect sets of high-confidence level normal images . The variety of methods for unsupervised anomaly detection is wide and diversified ; the following works give a comprehensive review of existing solutions : Pimentel et al . ( 2014 ) , and Ehret et al . ( 2019 ) . In this section , we restrict ourselves to a brief overview of a specific subset of them , more recent and closer to our work . In recent years , at the 2019 International Conference on Pattern Recognition ( ICPR ) , a new dataset containing few thousand images of objects and textures with and without anomalies has been released by MVTec Software GmbH company ( Bergmann et al. , 2019 ) . The purpose has been to provide the scientific community with a dataset to address real-world industrial scenarios , compare recent techniques , and understand the overall scientific progress over the topic . At the same time , deep learning and artificial neural networks based models have had a significant impact on computer vision , leading to improvements in image classification , segmentation , and object detection , e.g . Krizhevsky et al . ( 2012 ) , Simonyan & Zisserman ( 2014 ) , Xie et al . ( 2017 ) , He et al . ( 2016 ) , Szegedy et al . ( 2015 ) . Many of these techniques have also been proposed for anomaly detection . In the article cited above ( Bergmann et al. , 2019 ) , authors have also proposed a summary and an evaluation of these techniques . They start analyzing reconstruction-based models such as Convolutional Autoencoders ( Goodfellow et al. , 2016 ) . This generative neural network based on convolutional layers compresses the image and generates a reconstruction . The network is trained over non-defective images with classic L2 or with Structural Similarity Measure as loss function ( Bergmann et al. , 2018 ) . High reconstruction errors should characterize images with anomalies . Authors then investigate a model based on Generative Adversarial Networks ( Goodfellow et al. , 2014 ) . This is a two-networks model , a Generator and a Discriminator , trained simultaneously with a two-player min-max optimization process . The Generator learns a distribution on healthy samples from a latent space to a manifold trying to fool the Discriminator . The latter computes the likelihood of a given sample to be a real or fake image . The assessed GAN-based model was the one proposed in Schlegl et al . ( 2017 ) , named ANOGAN , appositely designed for the anomaly detection task . Authors of Bergmann et al . ( 2019 ) continue the analysis considering traditional feature-based methods such as Gaussian Mixture Model ( Böttger & Ulrich , 2016 ) based on hand-crafted feature descriptors from defect-free images . Another important methodology assessed is a Convolutional Neural Network self-similarity method proposed by Napoletano et al . ( 2018 ) . The interesting point of this latest work is to use clustered feature descriptions obtained from the activation maps of a ResNet-18 ( He et al. , 2016 ) pre-trained over ImageNet dataset ( Deng et al. , 2009 ) . Convolutional Neural Networks usually have various layers , and intermediate outputs are identified as features maps , activation maps , or simply activations . In Napoletano et al . ( 2018 ) , training features are extracted from patches that are cropped from the input images . First , activations are dimensionally reduced with a Principal Component Analysis ; then , a K-Means algorithm calculates k clusters and relative centroids . These are used as atoms to build a dictionary . Normal images coming from a part of the validation set are then used to estimate the mean and variance of the euclidean distance from most similar dictionary elements . Boundaries of the estimated Gaussian distribution are used at inference time as a threshold . If the Euclidean distance to the most n similar dictionary elements is higher than the threshold , they flag the patch as anomalous . Starting from the idea of Napoletano et al . ( 2018 ) , different authors have investigated the adoption of pre-trained neural network activation maps to model distributions over images without anomalies . In particular , the authors of Defard et al . ( 2020 ) ( PaDiM ) show the effectiveness of modeling each patch activation map as Multivariate Gaussian Standard distributions , considerably improving the results presented in Bergmann et al . ( 2019 ) in terms of per-pixel and per-image scores . Other papers leverage the same idea : Cohen & Hoshen ( 2021 ) , Rippel et al . ( 2020 ) , Yang et al . ( 2020 ) , Yi & Yoon ( 2020 ) , Roth et al . ( 2021 ) , Li et al . ( 2021 ) . Their methodologies differ in the pre-trained architecture they have chosen , how they have aggregated information from different layers , estimated the distribution , and calculated the anomaly score . Moreover , to achieve high-performance scores , they usually operate class by class , i.e. , in a one-class learning scenario . The work proposed in Roth et al . ( 2021 ) , named PatchCore , builds a maximally representative memory bank of nominal patchfeatures and then applies a core set-reduction method to increase efficiency . Anomaly detection is then performed at a patch level with distance from the closest memory bank element . Other works , as Pirnay & Chai ( 2021 ) , involve specific techniques , such as inpainting , to obscure certain parts of the image and teach the network to reconstruct the missing part according to what is present in anomaly-free examples . The methodology presented in Zavrtanik et al . ( 2021 ) shows a full multiscale inpainting strategy to force the reconstruction of subregions of the image . Similarly , Wang et al . ( 2021 ) uses a student-teacher framework , a technique closer to GAN . Our contribution is a continuation of the original work of Napoletano et al . ( 2018 ) , and it assesses the application of sparse learning representation methods . While in Napoletano et al . ( 2018 ) the authors build the dictionary out of a PCA and K-Means of latent representation features , we propose a more structured approach based on K-SVD . This is a well-known sparse dictionary learning algorithm effectively used to address image denoising and anomaly detection tasks ( Aharon et al. , 2006 ) . The algorithm is used to learn a dictionary describing the essential channels features for each spatial element of the activation maps . While there has been wide adoption of dictionary learning and transfer learning for classification purposes in literature ( Maurer et al. , 2013 ; Gu & Li , 2021 ; Tang et al. , 2020 ; Zarka et al. , 2019 ) , we are the first to adopt them in an unsupervised scenario for anomaly detection The proposed pipeline results to be more robust than other methods , allowing to drop the one-class learning scenario and managing a more significant variance at one time . This is a fundamental aspect of real industrial world applications : products can vary in size and shape , and data gathered might not be sufficient for all sub-product ; building a different model for each sub-product might also be expensive in terms of time and memory . Results show that performance is competitive with respect to state-of-the-art and confirm that features extracted from robust classification networks can be leveraged to address other tasks in computer vision too . This paper is organized into a total of 4 sections . We have presented an introduction to the problem and described related solutions in Section 1 . Section 2 describes in detail the methodology adopted . Section 3 briefly describes the conducted experiments in terms of ablation studies and comparison with state-of-the-art models . Section 4 discusses the results and concludes by highlighting possible future development for the described methodology . The appendix contains further experiments conducted to assess the impact of K-SVD main hyper-parameters and the best possible configuration . 2 METHOD . This section presents how to combine the features extracted from a pre-trained network and a sparse dictionary learning procedure in an end-to-end pipeline for anomaly detection over images . The goal of the learning method is to obtain a dictionary that provides a sparse representation along the channels dimension for each spatial element of the activation maps . Because this spatial element corresponds to a receptive field in the original image , the method retain the most relevant features extracted by the network of each image patch . We underline that the method does not build a single dictionary : but a collection of dictionaries , one for each receptive field ( or patch ) of the selected layer embedding . The learning step is done through a sparse coding approach . Sparse coding or sparse dictionary learning is a representation learning method which aims at finding a sparse representation X of the input data S in the form of a linear combination of basic elements of a dictionary D , namely atoms . To give a more formal description , we define S ∈ Rm×n to be a matrix representing a collection of signals , with n > > m. Let D ∈ Rm×m , X ∈ Rm×n be respectively the dictionary and the sparse representation of the signals . We then define the following problem , known as simultaneous sparse approximation : argmin D , X 1 2 ‖DX − S‖2F + λ‖X‖1 such that ‖dk‖2 = 1∀k . ( 1 ) In Equation 1 , ‖ · ‖2F is the square of the Frobenius norm , λ > 0 , ‖ · ‖1 is the L1 norm ( in some formulation we can find the L0 norm ) , and dk are the columns of D. The formulated problem aims to simultaneously solve the task of learning a representative set of elements from a set of signals ( the dictionary ) and a sparse representation of them . The latter should retain the most important features of the signal . 2.1 TRANSFER LEARNING FROM PRE-TRAINED CONVOLUTIONAL NEURAL NETWORKS . Our patchwise sparse dictionary learning algorithm leverages Transfer Learning . Pre-trained Convolutional Neural Networks activation maps have proven to be effective features descriptors in different pattern recognition tasks : Shin et al . ( 2016 ) , Kandaswamy et al . ( 2014 ) , Raina et al . ( 2006 ) , Bergman & Hoshen ( 2020 ) . In particular , as highlighted in the introduction , pre-training over the Imagenet dataset can help build descriptors for modeling normal data distributions . In our work , we have selected architectures that achieve state-of-the-art classification error rate over Imagenet ( such as ResNet He et al . ( 2016 ) , or EfficientNet Tan & Le ( 2019 ) ) . We have also chosen the output of specific layers , rescaling them with an average pooling operation to match different spatial dimensions ( Rippel et al. , 2020 ) . Each channel of this output contains spatial representations of input image patches at a specific network depth . As done in Defard et al . ( 2020 ) , we studied how to effectively combine different layers activation maps to maximize the detection capability of the pipeline . In our case , there is a constrain to the total number of layers output we can select ; indeed , solving a dictionary learning problem usually goes through solving an undetermined linear system problem , and the total number of dictionary atoms can not exceed the total number of examples . This means we can not select architectures or layers with a number of channels greater than the total number of examples . Moreover , building a comprehensive dictionary is memory and time-consuming . If dimensions grew indefinitely , also subsequent calculations done with this dictionary would request impractical times . At the start of our pipeline , we pass all the images of the training dataset into the neural network and collect the intermediate outputs . We then build a tensor with the following dimensions : number of examples , number of channels , height , and width ; the last two dimensions correspond to those of the activation maps closer to the network ’ s input .

Summary: This paper proposes an anomaly detection algorithm based on dictionary learning on the representations obtained by pre-trained deep convolutional networks. More precisely, the authors consider location-specific signals from a collection of training images, extracted via pre-trained neural networks of different depth and types. They then train a dictionary via K-SVD to obtain sparse representation for these patches, which are then modeled via multivariate Gaussian distribution. At testing time, the representation for each patch is computed, and a metric is computed over all patches to declare an image as an outlier/anomalous, or normal.