Patchwise Sparse Dictionary Learning from pre-trained Neural Network Activation Maps for Anomaly Detection in Images

1 INTRODUCTION . Anomaly detection is the problem of identifying , given a set of data , those examples that deviate from the expected or standard behavior . In computer vision , anomaly detection tasks recognize those images showing objects or patterns with corrupted components or irregularities . Its applications range from industrial quality control to medical analysis . The typical approach to anomaly detection tries to model data distribution without anomalies and then estimate irregularities as examples with lower probability outcomes . The task is particularly demanding because it is difficult to understand and model anomaly occurrences due to three main challenges : data imbalance , concept drift , and selection bias . The first relates to the lack of anomalous examples ; the second represents the variation of anomalies over time ; the third reveals that the partial knowledge of the phenomenon introduces a bias in acquired data . These reasons pushed researchers to focus on unsupervised or weekly supervised machine learning techniques , leveraging part of the available data , usually composed of defect-free examples . Indeed especially in the context of anomaly detection over images for industrial quality control , we face data imbalance and concept drift problems , but we usually can collect sets of high-confidence level normal images . The variety of methods for unsupervised anomaly detection is wide and diversified ; the following works give a comprehensive review of existing solutions : Pimentel et al . ( 2014 ) , and Ehret et al . ( 2019 ) . In this section , we restrict ourselves to a brief overview of a specific subset of them , more recent and closer to our work . In recent years , at the 2019 International Conference on Pattern Recognition ( ICPR ) , a new dataset containing few thousand images of objects and textures with and without anomalies has been released by MVTec Software GmbH company ( Bergmann et al. , 2019 ) . The purpose has been to provide the scientific community with a dataset to address real-world industrial scenarios , compare recent techniques , and understand the overall scientific progress over the topic . At the same time , deep learning and artificial neural networks based models have had a significant impact on computer vision , leading to improvements in image classification , segmentation , and object detection , e.g . Krizhevsky et al . ( 2012 ) , Simonyan & Zisserman ( 2014 ) , Xie et al . ( 2017 ) , He et al . ( 2016 ) , Szegedy et al . ( 2015 ) . Many of these techniques have also been proposed for anomaly detection . In the article cited above ( Bergmann et al. , 2019 ) , authors have also proposed a summary and an evaluation of these techniques . They start analyzing reconstruction-based models such as Convolutional Autoencoders ( Goodfellow et al. , 2016 ) . This generative neural network based on convolutional layers compresses the image and generates a reconstruction . The network is trained over non-defective images with classic L2 or with Structural Similarity Measure as loss function ( Bergmann et al. , 2018 ) . High reconstruction errors should characterize images with anomalies . Authors then investigate a model based on Generative Adversarial Networks ( Goodfellow et al. , 2014 ) . This is a two-networks model , a Generator and a Discriminator , trained simultaneously with a two-player min-max optimization process . The Generator learns a distribution on healthy samples from a latent space to a manifold trying to fool the Discriminator . The latter computes the likelihood of a given sample to be a real or fake image . The assessed GAN-based model was the one proposed in Schlegl et al . ( 2017 ) , named ANOGAN , appositely designed for the anomaly detection task . Authors of Bergmann et al . ( 2019 ) continue the analysis considering traditional feature-based methods such as Gaussian Mixture Model ( Böttger & Ulrich , 2016 ) based on hand-crafted feature descriptors from defect-free images . Another important methodology assessed is a Convolutional Neural Network self-similarity method proposed by Napoletano et al . ( 2018 ) . The interesting point of this latest work is to use clustered feature descriptions obtained from the activation maps of a ResNet-18 ( He et al. , 2016 ) pre-trained over ImageNet dataset ( Deng et al. , 2009 ) . Convolutional Neural Networks usually have various layers , and intermediate outputs are identified as features maps , activation maps , or simply activations . In Napoletano et al . ( 2018 ) , training features are extracted from patches that are cropped from the input images . First , activations are dimensionally reduced with a Principal Component Analysis ; then , a K-Means algorithm calculates k clusters and relative centroids . These are used as atoms to build a dictionary . Normal images coming from a part of the validation set are then used to estimate the mean and variance of the euclidean distance from most similar dictionary elements . Boundaries of the estimated Gaussian distribution are used at inference time as a threshold . If the Euclidean distance to the most n similar dictionary elements is higher than the threshold , they flag the patch as anomalous . Starting from the idea of Napoletano et al . ( 2018 ) , different authors have investigated the adoption of pre-trained neural network activation maps to model distributions over images without anomalies . In particular , the authors of Defard et al . ( 2020 ) ( PaDiM ) show the effectiveness of modeling each patch activation map as Multivariate Gaussian Standard distributions , considerably improving the results presented in Bergmann et al . ( 2019 ) in terms of per-pixel and per-image scores . Other papers leverage the same idea : Cohen & Hoshen ( 2021 ) , Rippel et al . ( 2020 ) , Yang et al . ( 2020 ) , Yi & Yoon ( 2020 ) , Roth et al . ( 2021 ) , Li et al . ( 2021 ) . Their methodologies differ in the pre-trained architecture they have chosen , how they have aggregated information from different layers , estimated the distribution , and calculated the anomaly score . Moreover , to achieve high-performance scores , they usually operate class by class , i.e. , in a one-class learning scenario . The work proposed in Roth et al . ( 2021 ) , named PatchCore , builds a maximally representative memory bank of nominal patchfeatures and then applies a core set-reduction method to increase efficiency . Anomaly detection is then performed at a patch level with distance from the closest memory bank element . Other works , as Pirnay & Chai ( 2021 ) , involve specific techniques , such as inpainting , to obscure certain parts of the image and teach the network to reconstruct the missing part according to what is present in anomaly-free examples . The methodology presented in Zavrtanik et al . ( 2021 ) shows a full multiscale inpainting strategy to force the reconstruction of subregions of the image . Similarly , Wang et al . ( 2021 ) uses a student-teacher framework , a technique closer to GAN . Our contribution is a continuation of the original work of Napoletano et al . ( 2018 ) , and it assesses the application of sparse learning representation methods . While in Napoletano et al . ( 2018 ) the authors build the dictionary out of a PCA and K-Means of latent representation features , we propose a more structured approach based on K-SVD . This is a well-known sparse dictionary learning algorithm effectively used to address image denoising and anomaly detection tasks ( Aharon et al. , 2006 ) . The algorithm is used to learn a dictionary describing the essential channels features for each spatial element of the activation maps . While there has been wide adoption of dictionary learning and transfer learning for classification purposes in literature ( Maurer et al. , 2013 ; Gu & Li , 2021 ; Tang et al. , 2020 ; Zarka et al. , 2019 ) , we are the first to adopt them in an unsupervised scenario for anomaly detection The proposed pipeline results to be more robust than other methods , allowing to drop the one-class learning scenario and managing a more significant variance at one time . This is a fundamental aspect of real industrial world applications : products can vary in size and shape , and data gathered might not be sufficient for all sub-product ; building a different model for each sub-product might also be expensive in terms of time and memory . Results show that performance is competitive with respect to state-of-the-art and confirm that features extracted from robust classification networks can be leveraged to address other tasks in computer vision too . This paper is organized into a total of 4 sections . We have presented an introduction to the problem and described related solutions in Section 1 . Section 2 describes in detail the methodology adopted . Section 3 briefly describes the conducted experiments in terms of ablation studies and comparison with state-of-the-art models . Section 4 discusses the results and concludes by highlighting possible future development for the described methodology . The appendix contains further experiments conducted to assess the impact of K-SVD main hyper-parameters and the best possible configuration . 2 METHOD . This section presents how to combine the features extracted from a pre-trained network and a sparse dictionary learning procedure in an end-to-end pipeline for anomaly detection over images . The goal of the learning method is to obtain a dictionary that provides a sparse representation along the channels dimension for each spatial element of the activation maps . Because this spatial element corresponds to a receptive field in the original image , the method retain the most relevant features extracted by the network of each image patch . We underline that the method does not build a single dictionary : but a collection of dictionaries , one for each receptive field ( or patch ) of the selected layer embedding . The learning step is done through a sparse coding approach . Sparse coding or sparse dictionary learning is a representation learning method which aims at finding a sparse representation X of the input data S in the form of a linear combination of basic elements of a dictionary D , namely atoms . To give a more formal description , we define S ∈ Rm×n to be a matrix representing a collection of signals , with n > > m. Let D ∈ Rm×m , X ∈ Rm×n be respectively the dictionary and the sparse representation of the signals . We then define the following problem , known as simultaneous sparse approximation : argmin D , X 1 2 ‖DX − S‖2F + λ‖X‖1 such that ‖dk‖2 = 1∀k . ( 1 ) In Equation 1 , ‖ · ‖2F is the square of the Frobenius norm , λ > 0 , ‖ · ‖1 is the L1 norm ( in some formulation we can find the L0 norm ) , and dk are the columns of D. The formulated problem aims to simultaneously solve the task of learning a representative set of elements from a set of signals ( the dictionary ) and a sparse representation of them . The latter should retain the most important features of the signal . 2.1 TRANSFER LEARNING FROM PRE-TRAINED CONVOLUTIONAL NEURAL NETWORKS . Our patchwise sparse dictionary learning algorithm leverages Transfer Learning . Pre-trained Convolutional Neural Networks activation maps have proven to be effective features descriptors in different pattern recognition tasks : Shin et al . ( 2016 ) , Kandaswamy et al . ( 2014 ) , Raina et al . ( 2006 ) , Bergman & Hoshen ( 2020 ) . In particular , as highlighted in the introduction , pre-training over the Imagenet dataset can help build descriptors for modeling normal data distributions . In our work , we have selected architectures that achieve state-of-the-art classification error rate over Imagenet ( such as ResNet He et al . ( 2016 ) , or EfficientNet Tan & Le ( 2019 ) ) . We have also chosen the output of specific layers , rescaling them with an average pooling operation to match different spatial dimensions ( Rippel et al. , 2020 ) . Each channel of this output contains spatial representations of input image patches at a specific network depth . As done in Defard et al . ( 2020 ) , we studied how to effectively combine different layers activation maps to maximize the detection capability of the pipeline . In our case , there is a constrain to the total number of layers output we can select ; indeed , solving a dictionary learning problem usually goes through solving an undetermined linear system problem , and the total number of dictionary atoms can not exceed the total number of examples . This means we can not select architectures or layers with a number of channels greater than the total number of examples . Moreover , building a comprehensive dictionary is memory and time-consuming . If dimensions grew indefinitely , also subsequent calculations done with this dictionary would request impractical times . At the start of our pipeline , we pass all the images of the training dataset into the neural network and collect the intermediate outputs . We then build a tensor with the following dimensions : number of examples , number of channels , height , and width ; the last two dimensions correspond to those of the activation maps closer to the network ’ s input .

Summary: This paper tackles image anomaly detection by CNNs as feature extraction and sparse dictionary learning as its classifier solved by KSVD. In reference, the sparse coding can be effectively solved by OMP and its reconstruction errors denote the anomaly scores of images. As this paper states,  it is the first work to use a sparse dictionary learning algorithm to handle deep network features. Also, the proposed method achieves the sota performance.