Classification and Uncertainty Quantification of Corrupted Data using Semi-Supervised Autoencoders

1 INTRODUCTION AND MOTIVATION . Many real-world applications of data-driven classifiers , e.g. , neural networks , involve corruptions that pose significant challenges to the pretrained classifiers . Often , the corruption must previously be included , and thus already be known , during the process of training . For instance , noise ( e.g. , due to sensor imperfections ) and convolutions ( e.g. , due to lens flares or unfocused images ) are inevitable in image processing systems and may occur spontaneously and irregularly . The same holds for masking , which may occur when a foreign object occludes the actual object of interest ( e.g. , water droplets or dirt or scratches on the camera lens ) . Hence , we aim to answer the following question in this paper : How can we classify corrupted data with a parametric classifier trained exclusively on uncorrupted data ? As classifying corrupted data naturally demands a measure of uncertainty for validation ( corruption may , in the worst case , lead to a total loss of information ) , we include both model uncertainty δm and reconstruction uncertainty δr in the classification . We refer to δm as the model ’ s confidence on the classification itself . In contrast , we refer to δr as the confidence of the process on reconstructing the latent space activations given some corrupted datum . An overview of the proposed method is illustrated in Figure 1 . 2 CLASSIFICATION AND UNCERTAINTY QUANTIFICATION OF CORRUPTED DATA . 2.1 METHODOLOGY OVERVIEW AND RELATED WORK . To address the challenge of classification and uncertainty quantification of corrupted data , we propose the following core approach , illustrated in Figure 2 . 1 In the first step , we train a semi-supervised autoencoder ( Le et al. , 2018 ) that is : ( a ) capable of classifying the input data with its latent space activations h , and ( b ) capable of decoding the ( supervised ) latent space activations to generate higher-dimensional data , targeting it to be identical to the input . Except for these two constraints ( a ) and ( b ) , we do not impose any further restrictions on the autoencoder and train it as a standard feedforward neural network . 2 In the second step , we decouple the decoder g from the autoencoder and treat the decoder as a fixed generative function g. Neither retraining nor further modifying of g is done in the following steps . 3 In the third step , we include g in an ADDITIVE WHITE GAUSSIAN NOISE ( AWGN ) channelmodel d = mCg ( h ) + n. This AWGN channel model additionally involves heavy corruption like convolution C and maskingm . 4 In the final step , we approximate the posterior probability distribution P ( h|d ) in the latent space , and derive the mean and standard deviation , corresponding optimally to some uncorrupted datum g ( h ) , given the corrupted datum d. Due to supervision at the latent space , this reconstruction enables a direct classification of d including model and reconstruction uncertainty quantification , even though the decoding function was trained on uncorrupted data . We use a set of samplesH from the approximate posterior probability distribution to determine the sampling mean mean ( H ) = H as well as the set ’ s reconstruction uncertainty δr with the sampling standard deviation std ( H ) . Samples are statistically inferred by METRIC GAUSSIAN VARIATIONAL INFERENCE ( MGVI ) ( Knollmüller & Enßlin , 2020 ) . In addition to reconstruction uncertainty δr , we determine the model uncertainty by calculating the MAHALANOBIS-distance ( M-distance ) ( De Maesschalck et al. , 2000 ) in the latent space representation , slightly different to Lee et al . ( 2018 ) . For details of our implementation , see Algorithm 2 in the appendix . We here distinguish between reconstruction uncertainty δr and model uncertainty δm to evaluate the confidence of the process of inferring h and to evaluate the confidence of the classification given by the supervised latent space , respectively . Similarly to our approach , Böhm et al . ( 2019 ) and Böhm & Seljak ( 2020 ) have shown that the reconstruction of the latent space by posterior inference and by using generative models ( ( Adler & Öktem , 2018 ) , ( Seljak & Yu , 2019 ) , ( Wu et al. , 2018 ) ) for a corrupted datum can lead to an optimal image restoration with uncertainty quantification . These methods do , however , not focus on classifying the corrupted datum in the latent space , nor using supervised autoencoder structures . In the field of quantifying uncertainties of classifications there exist several methods . Predominantly BAYESIAN NEURAL NETWORKS ( BNNs ) including neural network ensembling ( ( Depeweg et al. , 2017 ) , ( Neal , 1995 ) , ( Pearce et al. , 2020 ) ) and MONTE CARLO-dropout ( MC dropout ) ( Gal & Ghahramani , 2016 ) have lately shown success . More recently , EVIDENTIAL DEEP LEARNING ( Sensoy et al. , 2018 ) was introduced as yet another probabilistic method to quantify classification uncertainty . The latter two methods will be compared to our method in Section 3 . Finally , various methods to perform image restoration exist in the literature ( see ( Dong et al. , 2012 ) , ( Lehtinen et al. , 2018 ) , ( Mao et al. , 2016a ) , ( Mao et al. , 2016b ) , ( Zoran & Weiss , 2011 ) ) . Similar to the well-known denoising autoencoder ( Vincent et al. , 2008 ) , almost all methods require prior knowledge of the corruption to be included in the training data . We argue that these methods lack flexibility , as they deal with one specific type of corruption . Once the model is trained , one can not include other corruption types without retraining . Moreover , many methods focus on either classification or eliminating corruption , but none of the named approaches combine both . 2.2 GENERATIVE MODEL AND BAYESIAN INFERENCE WITH NEURAL NETWORKS . The first step of our method is to train a semi-supervised autoencoder . The autoencoder involves the encoding function f ( mapping data x ∈ Rp to the latent space representation with activations h ∈ Rz , z ∈ N ) as well as the decoding function g ( mapping h to the data space representation x̂ ∈ Rp , p ∈ N , p z ) . Parameters of f : Rp → Rz and g : Rz → Rp are optimized via a combination of two loss terms Lgf ( representing reconstruction loss in the data space ) and Lf ( representing classification loss in the latent space ) : LSAE = Lgf ( g ( f ( x ) ) , x ) + Lf ( f ( x ) j , y ) = Lgf ( x̂ , x ) + Lf ( hj , y ) . ( 1 ) where j denotes the number of dimensions of h that are supervised , i.e. , hj = [ h1 , . . . , hj ] . The number of classes to be classified equals j . After normalizing all data samples ( i.e. , pixel values range in between [ 0 , 1 ] ) , we use the corresponding cross-entropy for each respective loss term to penalize false classifications in the latent space and inaccurate reconstructions in the data space . The binary cross-entropy represents the reconstruction loss , while we use the sparse categorical cross-entropy on integer labels y = [ 0 , 1 , . . . , 9 ] to represent the classification loss . Note that for training , we process the latent space activations h through the softmax-function before feeding them to the sparse categorical cross-entropy . However , the softmax-function is not included as an activation function in our neural network . We minimize the general loss function Equation ( 1 ) using Adam optimizer ( Kingma & Ba , 2015 ) 1 . Once the training procedure has converged , we decouple the decoding function g from the autoencoder and extend it to model different types of corruption ( this is necessary as the decoder is trained on uncorrupted data ) . Without loss of generality , we use an AWGN model including the nonlinearity g ( h ) , which additionally involves maskingm and convolution C on g : d = mCg ( h ) + n. ( 2 ) In the data space , additive white Gaussian noise , n ∈ Rp ∼ N ( 0 , Σn ) , is applied to the decoded latent space signal g ( h ) , which yields the corrupted data d ∈ Rp . Note for the implementation of h = Aξ + µh , the reparametrization trick Kingma & Welling ( 2014 ) is applied.2 In addition to AWGN , we include corruptions of masking m and convolutions C , which are both linear operations . See the Appendix A , for details on the implementation of n , m and C. Since we are interested in reconstructing the latent space activation h from d alongside uncertainty quantification , the goal is to determine the posterior distribution P ( h|d ) ∝ P ( d|h ) P ( h ) . The log-probability distribution reads − lnP ( h|d ) = 1 2 ( ( d−mCg ( h ) ) T Σ−1n ( d−mCg ( h ) ) + ( hTΣ−1h h ) ) + const. , ( 3 ) where ( · ) T denotes the matrix transpose . Since we are finally interested in the analytically intractable mean of h , 〈h〉P ( h|d ) = ∫ hP ( h|d ) dh , we approximately determine mean and variance of P ( h|d ) by applying MGVI . Similar to other variational inference methods ( Kingma & Welling ( 2014 ) , Kucukelbir et al . ( 2017 ) ) , MGVI approximates the distribution by a simper , but tractable distribution from within a variational family , Q ( h ) . The parameters of Q ( h ) , i.e. , mean η and covariance ∆ , are obtained by the minimization of the variational lower bound . The size of a full variational covariance scales quadratically with the number of latent variables . Taking these limitations into account , we employ MGVI , which locally approximates the target distribution using the inverse Fisher metric as an uncertainty estimate around the variational mean η , for which we optimize . The approximation is represented by an ensemble of samplesH = { h̃1 , h̃2 , . . . , h̃n } with h̃n ∈ Rz , which we use for our analysis . ( ̃· ) refers to the inferred sample . We here call H the posterior mean and δr the posterior standard deviation , or , the reconstruction uncertainty . 2.3 CLASSIFICATION AND UNCERTAINTY QUANTIFICATION . The supervision of the latent space allows us to classify the input d in a straightforward manner by evaluating the sampling mean and sampling standard deviation of the set H. While the sampling mean of the set mean ( H ) = H gives the class of the most likely classification , the sampling standard deviation reflects the reconstruction uncertainty δr of the latent space posterior distribution . δr depends on the type and magnitude of the corruption as well as the prior probability distribution we include in the channel model ( Equation ( 3 ) ) . We visualize this dependency with various experiments , see Figure 4 and Figure 10 . The straightforward classification does not yet provide information about the model uncertainty on the classification . Since we are additionally interested in the uncertainty of the model , δm , we evaluate the M-distance of all samples in H to all class conditional distributions in the latent space . The closest class conditional distribution to a single sample h̃n corresponds to the most likely class . The absolute value of the M-distance to the closest class conditional distribution serves as a measure of the 1Test accuracy of [ 98 , 6 % ; 89 , 4 % ] on the encoding function f with [ MNIST ; Fashion-MNIST ] . 2Σh = cov ( f ( XVal ) ) , Σh = AA T , Σh ∈ Rz×z , ξ ∼ N ( 0 , I ) , ξ ∈ Rz , µh = mean ( XVal ) , µh ∈ Rz model uncertainty δm . In this work , all class conditional distributions in the latent space are assumed to follow multivariate Gaussian distributions with covariance Σi and mean µi . We determine the parameters of these class conditional distributions by passing the uncorrupted data samples from an independent ( i.e. , independent of training and testing ) datasetXVal ( see Section 3 ) through the encoder f . This method is an implementation slightly different to Lee et al . ( 2018 ) , where it was shown that the Mahalanobis metric is not only an accurate classifier in this context but also a reliable out-of-distribution detector reflecting the model uncertainty . Lee et al . ( 2018 ) uses tied covariance matrices instead of individual covariance matrices for each class conditional distribution , as done in our method . Concretely , we calculate the M-distance of all samples inH to all class-conditional clusters Ck within the latent space , each characterized by µCk and ΣCk for K-classes . We then determine sampling mean δm and sampling standard deviation δr of the M-distances of all samples . This way , we can represent reconstruction uncertainty by the sampling standard deviation ( resulting directly from the shape of the inferred posterior distribution ) and model uncertainty by the absolute value of the M-distance . For a graphical illustration , refer to Figure 3 . The pseudo-code is given in Algorithm 2 in the appendix .

Summary: This paper presents a method that simultaneously classifies corrupted data and quantifies uncertainty, despite the model being fitted only on uncorrupted data. The idea is to fit a semi-supervised autoencoder. Then, the encoded representation $h$ is fed into a decoder $g$, and the output $g(h)$ is augmented with various perturbations (additive noise, blur, and masking), resulting in a corrupted image $d$. Lastly, the method of MGVI is used to estimate $P(h \mid d)$, which is then used both to classify $d$ and to estimate the uncertainty. In experiments, the authors show the advantage of this method over baseline methods.