Characterizing Missing Information in Deep Networks Using Backpropagated Gradients

1 INTRODUCTION . The generalizable representation of data from deep network has largely contributed to the success of deep learning in diverse applications ( Bengio et al. , 2013 ) . The representation from deep networks is often obtained in the form of activation . The activation is constructed by the weights which contain specific knowledge learned from training samples . Recent studies reveal that deep networks still face robustness issues when input that can not be properly represented by learned knowledge is given to the networks ( Goodfellow et al. , 2014 ; Hendrycks & Dietterich , 2018 ; Liang et al. , 2017 ) . One of the reasons for the vulnerability of deep networks is the limitation in the activation-based representation , which inherently focused on the learned knowledge . However , the part of the input that causes problems in deep networks is mainly from the information that deep networks were not able to learn from the training data . Therefore , it is more appropriate to complement the representation of input data from the perspective of information that has not been learned for enhancing the robustness of machine learning algorithms . The gradient is another fundamental element in deep networks that is utilized to learn new information from given inputs by updating model weights ( Goodfellow et al. , 2016 ) . It is generated through backpropagation to train deep networks by minimizing designed loss functions ( Rumelhart et al. , 1986 ) . During the training of network , the gradient with respect to the weights provides directional information to update the deep network and learn a better representation for the inputs . In other words , gradients guide the network to learn new information that was not learned from data that it has seen so far but is presented in the current input . Considering this role during training , gradients can provide a complementary perspective with respect to activation and characterize missing information that the network has not learned for each unseen image . We demonstrate the role of gradients with an example in Fig . 1 . Assume that a deep network has only learned curved edge features from training images of the digit ‘ 0 ’ . During testing , the digit ‘ 6 ’ is given to the network . The digit ‘ 6 ’ consists of both learned information ( curved edges ) and missing information ( straight edges on top ) . Since the activation-based representation is constructed based on the information that the network has already learned , the curved part of the digit ‘ 6 ’ will be characterized effectively by the activation . However , the network still has to learn the straight edge features to perform successfully on the digit ‘ 6 ’ . Therefore , the gradients which guide updates in the deep network can characterize straight edge information that has not been learned . We propose analyzing the representation capability of gradients in characterizing missing information for deep networks . Gradients have been utilized in diverse applications such as adversarial attack generation and visualization ( Zeiler & Fergus , 2014 ; Goodfellow et al. , 2014 ) . However , using gradients with respect to weight as the representation of data has not been actively explored yet . Through the comprehensive analysis with activation-based representations , we show the effectiveness of gradient representation in characterizing the information that has not been learned for deep network . Furthermore , we show that gradient representation can achieve state-of-the-art performance in detecting potentially invalid data for the network . The main contributions of this paper are three folds : i We propose utilizing gradients as a representation to characterize information that has not been learned from the training data but is currently presented in the input data . ii We analyze the representation capability of gradient compared to activation for detecting samples which possess features that have not been learned for the network . iii We propose a gradient-based anomaly detection algorithm that outperforms state-of-the-art algorithms based on activation representations . 2 RELATED WORKS . Existing works have focused on achieving reliable activation-based representations to properly handle inputs that can cause significant performance degradation to deep networks . One of the most intuitive ways is to enhance the representation capability of activation by finetuning trained models and learning more from augmented data . Goodfellow et al . ( 2014 ) ; Vasiljevic et al . ( 2016 ) ; Temel et al . ( 2017 ) utilize adversarial images , blurred images , and distorted virtual images , respectively , to finetune networks to improve the classification performance based on the activation features . Also , several pre-processing techniques have been developed to make the representation of data similar to that of training data to achieve the robustness of algorithms . Discrete cosine transform ( DCT ) has been explored as a simple pre-processing technique to eliminate the effect of distortion and adversarial attacks ( Hossain et al. , 2018 ; Das et al. , 2018 ) . On the other hand , methods developed for detecting and filtering out problematic samples have focused on making the representation of such samples as distinguishable as possible from that of training images in the activation space . Liang et al . ( 2017 ) propose a gradient-based pre-processing technique to make the activation-based representation of adversarial images and out-of-distribution images dissimilar to that of training images . Furthermore , constrained activation representations have been actively utilized to make normal and abnormal images statistically separable in the activation space for abnormal data detection ( Pidhorskyi et al. , 2018 ; Perera et al. , 2019 ; Abati et al. , 2018 ) . Aforementioned works exclusively focus on activation-based representations which are based on learned information to handle the information that has not been learned . We complement these by proposing backpropagated gradients-based representation of data which particularly focuses on what has not been learned in the trained network . The backpropagated gradients have been utilized in diverse applications including but not limited to visualization , adversarial attacks , and image classification . The backpropagated gradients have been widely used for the visualization of deep networks . In Zeiler & Fergus ( 2014 ) ; Springenberg et al . ( 2014 ) , information that networks have learned for a specific target class is mapped back to the pixel space through the backpropagation and visualized . Selvaraju et al . ( 2017 ) utilize the gradient with respect to activation to weight the activation and visualize the reasoning for prediction that deep networks have made . Adversarial attack is another application of gradients . Goodfellow et al . ( 2014 ) ; Kurakin et al . ( 2016 ) show that adversarial attacks can be generated by adding an imperceptibly small vector which is the signum of input gradients . In Kwon et al . ( 2019 ) , the backpropagated gradients are utilized to classify and objectively estimate the quality of distorted images . Several works have incorporated gradients with respect to input in the form of regularization during the training of deep networks to improve the robustness ( Drucker & Le Cun , 1991 ; Ross & Doshi-Velez , 2018 ; Sokolić et al. , 2017 ) . Although existing works have shown that gradients with respect to input or activation can be useful for diverse applications , gradients with respect to weight remain almost unexplored aside from its role in training deep networks . In the following section , we demonstrate the interpretation of weight gradients to further elaborate the role of weight gradients as data representation . 3 GEOMETRIC INTERPRETATION OF GRADIENTS . We use an autoencoder which is an unsupervised representation learning framework to explain the geometric interpretation of gradients . An autoencoder consists of an encoder , fθ , and a decoder , gφ as shown in Fig . 2 ( a ) . From an input image , x , the latent variable , z , is generated as z = fθ ( x ) and the reconstructed image is obtained by feeding the latent variable into the decoder , gφ ( fθ ( x ) ) . The training is performed by minimizing the loss function , J ( x ; θ , φ ) , defined as follows : J ( x ; θ , φ ) = L ( x , gφ ( fθ ( x ) ) ) + Ω ( z ; θ , φ ) , ( 1 ) where L is a reconstruction error which measures the dissimilarity between the input and the reconstructed image and Ω is a regularization term for the latent variable . We visualize the geometric interpretation of backpropagated gradients in Fig . 2 ( b ) . The autoencoder is trained to accurately reconstruct training images and the reconstructed training images form a manifold . We assume that the structure of the manifold is a linear plane which is spanned by the weights of the encoder and the decoder as shown in the figure for the simplicity of explanation . Also , we explains the generalization of this concept in the following section . During test time , any given input to the autoencoder is projected onto the reconstructed image manifold through the projection , gφ ( fθ ( · ) ) . The closer test images are to the reconstructed image manifold , the more accurately reconstructed they are . Let us assume that test data distribution is outside of the reconstructed image manifold . When the test image , xout , sampled from the test data distribution is given to the autoencoder , it will be reconstructed as x̂out by the projection , gφ ( fθ ( xout ) ) . Since test images from the test data distribution have not been utilized for training , they will be poorly reconstructed . Given that the reconstructed image manifold contains data reconstructed by learned knowledge , the orthogonal gap between xout and x̂out measures the missing knowledge that the autoencoder has not learned and can not reconstruct . Using the error defined in ( 1 ) , the gradient of weights , ∂J∂θ , ∂J ∂φ , can be calculated through the backpropagation . These gradients represent required changes in the reconstructed image manifold to incorporate the test images and reconstruct them accurately . In other words , these gradients characterize orthogonal variations of the test distribution with respect to the reconstructed image manifold , which is missing information in the trained networks . The gradients complement activation-based representation in characterizing missing information . In this setup of the autoencoder , missing information is characterized by the reconstruction error using activation . It provides distance information between test images and reconstructed images and is utilized as a primal loss to generate gradients . However , it does not provide any directional informa- tion about the deviation of test data distribution . The gradients focus on the directional information to characterize information that has not be learned . Considering that weights contain learned knowledge from training samples , the weight gradients indicate the directional changes required to obtain new knowledge with respect to current knowledge of networks . Therefore , the gradients can provide a comprehensive perspective to represent missing knowledge in trained networks by complementing distance information from activation-based representations with the directional information . 4 GENERALIZATION OF THE GEOMETRIC INTERPRETATION FOR GRADIENTS . The gradients require a discriminant capability in high dimensional space to be an effective representation for missing information in practical scenarios of deep networks . As a representation for the information that has not been learned , the representation is expected to clearly differentiate information that has been learned and has not been learned from the input . Therefore , gradients from images with learned information should be distinguishable in terms of direction from those generated by images with new information . For instance , the gradient generated from xout in Fig . 2 is orthogonal to the reconstructed image manifold . However , test images with learned information do not result in high reconstruction errors and do not require significant changes in the reconstructed image manifold . In this case , gradients from these test images will be more tangential to the manifold and distinguishable from the gradient of xout . This separation between gradients from learned and not learned information is required to effectively characterize what is missing and not in deep networks . In addition , gradients from the learned information should be constrained to make the direction of gradients for missing information more distinctive . In Fig . 2 , the reconstructed image manifold is a two dimensional plane in three dimensional space . The gradients generated by test images with learned information are tangential to the manifold and constrained to the two dimensional space . These constrained gradients for test images with learned information allow gradients from xout to be more distinctive and characterize missing information in the network . We design two experiments to analyze the separation between gradients from learned and missing information in the deep network . We train a convolutional autoencoder ( CAE ) which consists of 4-layer encoder and decoder , respectively . We use training images from Airplane class in CIFAR-10 dataset ( Krizhevsky & Hinton , 2009 ) to train the CAE . A test set contains 1000 images from Airplane class and the same number of images randomly sampled from all other 9 classes in test split . The test Airplane class images contain learned knowledge and images from other classes contain information that has not been learned . In the first experiment , we extract the backpropagated gradients for all test images and measure the average alignment of gradients using cosine similarity ( cosSIM ) . In Fig . 3 ( a ) , we visualize the distribution of the average cosine similarity calculated between the gradient from each Airplane class image and that from all other remaining Airplane class images ( Learned ) , and between each Airplane class image and test images from all other classes ( Missing ) . We calculate the average cosine similarity over all 4 layers in decoder . Ideally for gradients to be an effective representation , cosine similarity between images with learned information should be clearly separable to that calculated between images with learned and missing information . We also calculated the number of samples in the overlapped region of histograms . A large overlapped region ( 793 samples out of 1000 samples ) between cosine similarity from ‘ Learned ’ and ’ Missing ’ indicates that gradients in general can not be distinguishable in high dimensional space . However , with our proposed method which will be described in the following section , the number of overlapped samples significantly decreases from 793 to 296 as shown in Fig . 3 ( b ) . In addition , gradients become more aligned for learned information by achieving high cosine similarity and become orthogonal for missing information . This shows that gradients become more separable for learned and missing information with the proposed approach . In the second experiment , we analyze the constraint on gradients by measuring the alignment of them generated while the training of CAE . We train the same architecture of the CAE described in the first experiment using Airplane class images . To measure the alignment of training gradients , we calculate the cosine similarity between the gradients of a certain layer i at the k th iteration of training , ∂L∂Wi k , and the average of training gradients of layer i obtained until the k − 1 th iteration , ∂L ∂Wi k−1 avg , defined as follows : cosSIM ( ∂L ∂Wi k−1 avg , ∂L ∂Wi k ) where ∂L ∂Wi k−1 avg = 1 ( k − 1 ) k−1∑ t=1 ∂L ∂Wi t , ( 2 ) W is the weight of autoencoder , i is the layer index , and superscripts are used to indicate the iteration number . We visualize the progress of the cosine similarity while training over 1000 epochs in Fig . 3 ( c ) . We particularly focus on the cosine similarity in the 4 layers of decoder . To enhance the distinction between gradients from learned information and missing information , training gradients are expected to be constrained and aligned at the end of training by achieving high cosine similarity . However , we observe that the cosine similarity converges to 0 at the end of training . This indicates that even when most of information in training images is learned , the gradients is orthogonal to the average training gradients and remain unconstrained . On the other hand , we show our proposed method effectively constrains the gradients while training and obtain around 0.55 ∼ 0.85 cosine similarity in Fig . 3 ( d ) . This constraint leads to the clear separability observed in Fig . 3 ( b ) . In following section , we explain the details of our proposed method .

Summary: The authors consider gradients of some loss as a feature representation. The intro starts describing a form of classification loss, but this is switched to a GAE unsupervised loss in Eq. 1. In that section the authors somewhat show that a naive gradient representation does not work well, but at the same time show that it /may/ work with their method, introduced right after, and which consists in an "orthogonality" regularizer, introduced with any arbitrary loss later in Eq.3, and back to GAE in Eq.4. The authors evaluate whether this regularizer hurst reconstruction error