Classification and Uncertainty Quantification of Corrupted Data using Semi-Supervised Autoencoders

1 INTRODUCTION AND MOTIVATION . Many real-world applications of data-driven classifiers , e.g. , neural networks , involve corruptions that pose significant challenges to the pretrained classifiers . Often , the corruption must previously be included , and thus already be known , during the process of training . For instance , noise ( e.g. , due to sensor imperfections ) and convolutions ( e.g. , due to lens flares or unfocused images ) are inevitable in image processing systems and may occur spontaneously and irregularly . The same holds for masking , which may occur when a foreign object occludes the actual object of interest ( e.g. , water droplets or dirt or scratches on the camera lens ) . Hence , we aim to answer the following question in this paper : How can we classify corrupted data with a parametric classifier trained exclusively on uncorrupted data ? As classifying corrupted data naturally demands a measure of uncertainty for validation ( corruption may , in the worst case , lead to a total loss of information ) , we include both model uncertainty δm and reconstruction uncertainty δr in the classification . We refer to δm as the model ’ s confidence on the classification itself . In contrast , we refer to δr as the confidence of the process on reconstructing the latent space activations given some corrupted datum . An overview of the proposed method is illustrated in Figure 1 . 2 CLASSIFICATION AND UNCERTAINTY QUANTIFICATION OF CORRUPTED DATA . 2.1 METHODOLOGY OVERVIEW AND RELATED WORK . To address the challenge of classification and uncertainty quantification of corrupted data , we propose the following core approach , illustrated in Figure 2 . 1 In the first step , we train a semi-supervised autoencoder ( Le et al. , 2018 ) that is : ( a ) capable of classifying the input data with its latent space activations h , and ( b ) capable of decoding the ( supervised ) latent space activations to generate higher-dimensional data , targeting it to be identical to the input . Except for these two constraints ( a ) and ( b ) , we do not impose any further restrictions on the autoencoder and train it as a standard feedforward neural network . 2 In the second step , we decouple the decoder g from the autoencoder and treat the decoder as a fixed generative function g. Neither retraining nor further modifying of g is done in the following steps . 3 In the third step , we include g in an ADDITIVE WHITE GAUSSIAN NOISE ( AWGN ) channelmodel d = mCg ( h ) + n. This AWGN channel model additionally involves heavy corruption like convolution C and maskingm . 4 In the final step , we approximate the posterior probability distribution P ( h|d ) in the latent space , and derive the mean and standard deviation , corresponding optimally to some uncorrupted datum g ( h ) , given the corrupted datum d. Due to supervision at the latent space , this reconstruction enables a direct classification of d including model and reconstruction uncertainty quantification , even though the decoding function was trained on uncorrupted data . We use a set of samplesH from the approximate posterior probability distribution to determine the sampling mean mean ( H ) = H as well as the set ’ s reconstruction uncertainty δr with the sampling standard deviation std ( H ) . Samples are statistically inferred by METRIC GAUSSIAN VARIATIONAL INFERENCE ( MGVI ) ( Knollmüller & Enßlin , 2020 ) . In addition to reconstruction uncertainty δr , we determine the model uncertainty by calculating the MAHALANOBIS-distance ( M-distance ) ( De Maesschalck et al. , 2000 ) in the latent space representation , slightly different to Lee et al . ( 2018 ) . For details of our implementation , see Algorithm 2 in the appendix . We here distinguish between reconstruction uncertainty δr and model uncertainty δm to evaluate the confidence of the process of inferring h and to evaluate the confidence of the classification given by the supervised latent space , respectively . Similarly to our approach , Böhm et al . ( 2019 ) and Böhm & Seljak ( 2020 ) have shown that the reconstruction of the latent space by posterior inference and by using generative models ( ( Adler & Öktem , 2018 ) , ( Seljak & Yu , 2019 ) , ( Wu et al. , 2018 ) ) for a corrupted datum can lead to an optimal image restoration with uncertainty quantification . These methods do , however , not focus on classifying the corrupted datum in the latent space , nor using supervised autoencoder structures . In the field of quantifying uncertainties of classifications there exist several methods . Predominantly BAYESIAN NEURAL NETWORKS ( BNNs ) including neural network ensembling ( ( Depeweg et al. , 2017 ) , ( Neal , 1995 ) , ( Pearce et al. , 2020 ) ) and MONTE CARLO-dropout ( MC dropout ) ( Gal & Ghahramani , 2016 ) have lately shown success . More recently , EVIDENTIAL DEEP LEARNING ( Sensoy et al. , 2018 ) was introduced as yet another probabilistic method to quantify classification uncertainty . The latter two methods will be compared to our method in Section 3 . Finally , various methods to perform image restoration exist in the literature ( see ( Dong et al. , 2012 ) , ( Lehtinen et al. , 2018 ) , ( Mao et al. , 2016a ) , ( Mao et al. , 2016b ) , ( Zoran & Weiss , 2011 ) ) . Similar to the well-known denoising autoencoder ( Vincent et al. , 2008 ) , almost all methods require prior knowledge of the corruption to be included in the training data . We argue that these methods lack flexibility , as they deal with one specific type of corruption . Once the model is trained , one can not include other corruption types without retraining . Moreover , many methods focus on either classification or eliminating corruption , but none of the named approaches combine both . 2.2 GENERATIVE MODEL AND BAYESIAN INFERENCE WITH NEURAL NETWORKS . The first step of our method is to train a semi-supervised autoencoder . The autoencoder involves the encoding function f ( mapping data x ∈ Rp to the latent space representation with activations h ∈ Rz , z ∈ N ) as well as the decoding function g ( mapping h to the data space representation x̂ ∈ Rp , p ∈ N , p z ) . Parameters of f : Rp → Rz and g : Rz → Rp are optimized via a combination of two loss terms Lgf ( representing reconstruction loss in the data space ) and Lf ( representing classification loss in the latent space ) : LSAE = Lgf ( g ( f ( x ) ) , x ) + Lf ( f ( x ) j , y ) = Lgf ( x̂ , x ) + Lf ( hj , y ) . ( 1 ) where j denotes the number of dimensions of h that are supervised , i.e. , hj = [ h1 , . . . , hj ] . The number of classes to be classified equals j . After normalizing all data samples ( i.e. , pixel values range in between [ 0 , 1 ] ) , we use the corresponding cross-entropy for each respective loss term to penalize false classifications in the latent space and inaccurate reconstructions in the data space . The binary cross-entropy represents the reconstruction loss , while we use the sparse categorical cross-entropy on integer labels y = [ 0 , 1 , . . . , 9 ] to represent the classification loss . Note that for training , we process the latent space activations h through the softmax-function before feeding them to the sparse categorical cross-entropy . However , the softmax-function is not included as an activation function in our neural network . We minimize the general loss function Equation ( 1 ) using Adam optimizer ( Kingma & Ba , 2015 ) 1 . Once the training procedure has converged , we decouple the decoding function g from the autoencoder and extend it to model different types of corruption ( this is necessary as the decoder is trained on uncorrupted data ) . Without loss of generality , we use an AWGN model including the nonlinearity g ( h ) , which additionally involves maskingm and convolution C on g : d = mCg ( h ) + n. ( 2 ) In the data space , additive white Gaussian noise , n ∈ Rp ∼ N ( 0 , Σn ) , is applied to the decoded latent space signal g ( h ) , which yields the corrupted data d ∈ Rp . Note for the implementation of h = Aξ + µh , the reparametrization trick Kingma & Welling ( 2014 ) is applied.2 In addition to AWGN , we include corruptions of masking m and convolutions C , which are both linear operations . See the Appendix A , for details on the implementation of n , m and C. Since we are interested in reconstructing the latent space activation h from d alongside uncertainty quantification , the goal is to determine the posterior distribution P ( h|d ) ∝ P ( d|h ) P ( h ) . The log-probability distribution reads − lnP ( h|d ) = 1 2 ( ( d−mCg ( h ) ) T Σ−1n ( d−mCg ( h ) ) + ( hTΣ−1h h ) ) + const. , ( 3 ) where ( · ) T denotes the matrix transpose . Since we are finally interested in the analytically intractable mean of h , 〈h〉P ( h|d ) = ∫ hP ( h|d ) dh , we approximately determine mean and variance of P ( h|d ) by applying MGVI . Similar to other variational inference methods ( Kingma & Welling ( 2014 ) , Kucukelbir et al . ( 2017 ) ) , MGVI approximates the distribution by a simper , but tractable distribution from within a variational family , Q ( h ) . The parameters of Q ( h ) , i.e. , mean η and covariance ∆ , are obtained by the minimization of the variational lower bound . The size of a full variational covariance scales quadratically with the number of latent variables . Taking these limitations into account , we employ MGVI , which locally approximates the target distribution using the inverse Fisher metric as an uncertainty estimate around the variational mean η , for which we optimize . The approximation is represented by an ensemble of samplesH = { h̃1 , h̃2 , . . . , h̃n } with h̃n ∈ Rz , which we use for our analysis . ( ̃· ) refers to the inferred sample . We here call H the posterior mean and δr the posterior standard deviation , or , the reconstruction uncertainty . 2.3 CLASSIFICATION AND UNCERTAINTY QUANTIFICATION . The supervision of the latent space allows us to classify the input d in a straightforward manner by evaluating the sampling mean and sampling standard deviation of the set H. While the sampling mean of the set mean ( H ) = H gives the class of the most likely classification , the sampling standard deviation reflects the reconstruction uncertainty δr of the latent space posterior distribution . δr depends on the type and magnitude of the corruption as well as the prior probability distribution we include in the channel model ( Equation ( 3 ) ) . We visualize this dependency with various experiments , see Figure 4 and Figure 10 . The straightforward classification does not yet provide information about the model uncertainty on the classification . Since we are additionally interested in the uncertainty of the model , δm , we evaluate the M-distance of all samples in H to all class conditional distributions in the latent space . The closest class conditional distribution to a single sample h̃n corresponds to the most likely class . The absolute value of the M-distance to the closest class conditional distribution serves as a measure of the 1Test accuracy of [ 98 , 6 % ; 89 , 4 % ] on the encoding function f with [ MNIST ; Fashion-MNIST ] . 2Σh = cov ( f ( XVal ) ) , Σh = AA T , Σh ∈ Rz×z , ξ ∼ N ( 0 , I ) , ξ ∈ Rz , µh = mean ( XVal ) , µh ∈ Rz model uncertainty δm . In this work , all class conditional distributions in the latent space are assumed to follow multivariate Gaussian distributions with covariance Σi and mean µi . We determine the parameters of these class conditional distributions by passing the uncorrupted data samples from an independent ( i.e. , independent of training and testing ) datasetXVal ( see Section 3 ) through the encoder f . This method is an implementation slightly different to Lee et al . ( 2018 ) , where it was shown that the Mahalanobis metric is not only an accurate classifier in this context but also a reliable out-of-distribution detector reflecting the model uncertainty . Lee et al . ( 2018 ) uses tied covariance matrices instead of individual covariance matrices for each class conditional distribution , as done in our method . Concretely , we calculate the M-distance of all samples inH to all class-conditional clusters Ck within the latent space , each characterized by µCk and ΣCk for K-classes . We then determine sampling mean δm and sampling standard deviation δr of the M-distances of all samples . This way , we can represent reconstruction uncertainty by the sampling standard deviation ( resulting directly from the shape of the inferred posterior distribution ) and model uncertainty by the absolute value of the M-distance . For a graphical illustration , refer to Figure 3 . The pseudo-code is given in Algorithm 2 in the appendix .

Summary: This paper proposes a false classification detection method. The proposed method first trains an autoencoder using uncorrupted data to obtain a decoder to define a generative model. The generative model is then exploited for quantifying the uncertainty of the model. Using MGVI, the generative model infers the posterior distribution of the corresponding latent vector for each data. The latent space can explain the model uncertainty, which can be measured by the Mahalanobis distance. From experiments, it is shown that the proposed method can detect corrupted data more accurately compared to MC dropout and EDL for MNIST dataset.