Self-Interpretable Model with Transformation Equivariant Interpretation

1 Introduction . Deep learning ( DL ) models have been a great success in various domains of applications , including object detection , image classification , etc . However , many applications suffer from the overfitting problem , which is usually due to the lack of various training data . For scenarios with limited data access , data augmentation is usually applied to alleviate the overfitting problem . As one of the most simple but effective data augmentation methods , geometric transformation plays an important role in exploring the intrinsic visual structures of image data [ 44 , 34 ] . Transformation equivariance refers to the property that data representations learned from the model capture the intrinsic coordinates of the entities [ 15 ] , i.e. , transformations on the data will result in the same transformations to the model representations . Building transformation-equivariant DL models is desired in many kinds of applications , such as medical image analysis [ 11 ] , reinforcement learning [ 27 ] , etc . Although DL models can exert excellent performance in various tasks , DL models are usually expressed as black boxes . Therefore , DL models can have great performance in complex tasks but lack an explanation of the results [ 12 ] . In low-risk tasks such as adaptive email filtering , the direct deployment of black-box models without reasoning might be acceptable . However , for high-risk decision-making tasks such as disease diagnosis and autonomous vehicles [ 18 ] , the applied model needs to be more convincing than a black box . On the one hand , by faithfully explaining the model behavior , it can ensure the end user intuitively understands and trusts the DL model . On the other hand , the explanation of black-box models can provide insights into the relationship between input ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and output , thereby improving model design . However , with the rapid growth in computational power , DL models are designed to be more and more complex to meet the performance [ 35 ] , and the most advanced DL models can have billions of trainable parameters [ 36 ] . High complexity leads to the complete black box for human beings , which results in a lack of trust in the model . The demand for building more reliable and easy-to-understand DL models is growing rapidly . Depending on the stages where predictions and interpretations are conducted , the methods can be divided into two opposing categories : self-interpretable models and post-hoc models [ 30 ] . Unlike posthoc models , which generate interpretations to pre-trained black-box models , self-interpretable models aim to build models that are intrinsically interpretable themselves . The main difference between these two categories is that for post-hoc models , the interpretation and prediction are obtained in two different stages . The interpretation is separately obtained after the black-box models are trained . Therefore , interpretations obtained from post-hoc models are considered to be more fragile , sensitive , and less faithful to the predictive mechanism [ 1 , 19 , 50 ] . In contrast , self-interpretable models make interpretations at the same time as predictions , thus revealing the intrinsic mechanism of the models , and are thereby preferred by users in high-stakes tasks [ 40 ] . Besides , considering how powerful and common transformation can be in data augmentations , it is reasonable to take the robustness of interpretation to transformations into consideration when designing and evaluating the interpretations . Robust interpretation towards transformation implies two requirements : 1 ) the predictive mechanism indicated by the interpretation should remain the same after transformation ( e.g. , the highlighted region should remain the same despite the transformation ) ; 2 ) the location of interpretation should change according to the transformation . These two requirements naturally lead to transformation equivariance on interpretation . The transformation-equivariance property will enhance robust and faithful interpretation , where the interpretation is aware of the transformations and preserves the predictive mechanism . This correspondence between transformation equivariance and faithfulness suggests that self-interpretable models may perform better than post-hoc models in transformation awareness given their higher faithfulness . And the experiments also demonstrate this . Although self-interpretable models surpass post-hoc models in faithfulness and stability , there are non-negligible challenges in building self-interpretable models . First , the interpretations may need additional regularization to be in forms that are rational to humans . This process usually involves prior domain knowledge provided by human experts [ 21 , 39 ] . Besides , since the interpretability is intrinsic , specific constraints are required in the models to ensure the interpretability . The prediction power of such models will be damaged since it is essentially adding constraints to optimization problems . It is acknowledged that the increase of the interpretation quality is likely to decrease the performance of prediction results [ 13 , 31 ] . As a consequence , self-interpretable models are usually less expressive compared with black-box models , which can be interpreted by post-hoc models . Our Model : In this paper , we develop a transformation-equivariant self-interpretable model for classification tasks . As a self-interpretable model , our method makes predictions and generates interpretations of the predictions at the same stage . In other words , the interpretations are directly involved in the feed-forward prediction process , and are therefore faithful to the final results . We name our method as SITE ( Self-Interpretable model with Transformation Equivariant Interpretation ) . In SITE , we generate data-dependent prototypes for each class and formulate the prediction as the inner product between each prototype and the extracted features . The interpretations can be easily visualized by upsampling from the prototype space to the input data space . Besides , we introduce transformation regularization and reconstruction regularization to the prototypes . The reconstruction regularizer regularizes the interpretations to be meaningful and comprehensible for humans , while the transformation regularizer constrains the interpretations to be transformation equivariant . We validate that SITE presents understandable and faithful interpretations without requiring additional domain knowledge , and preserves high expressive power in prediction . We summarize the main contributions through this work as : • To our best knowledge , we are the first to learn transformation equivariant interpretations . • We build a self-interpretable model SITE with high-quality faithful and robust interpretation . • SITE preserves the high expressive power with comparable or better accuracy than related black-box models . • We propose self-consistency score , a new quantitative metric for interpretation methods . It quantifies the robustness of interpretation by measuring the consistency of interpretations to geometric transformations . 2 Related Work . Machine learning interpretation can have different goals , such as attribution [ 24 ] , interpretable clustering [ 28 ] , interpretable reinforcement learning [ 29 ] , disentanglement [ 43 ] , etc . Our method lies in the attribution category , thus we mainly review the related interpretation methods for attribution . Attribution methods target at identifying the contribution of different elements in the prediction . Based on if the prediction and interpretation are obtained in the same stage , the methods can be divided into post-hoc interpretation and self-interpretable methods . For post-hoc interpretation , the prediction results are obtained by a black-box model while the interpretation is obtained separately to explain the predictive mechanism of the black box . Among the different post-hoc interpretation techniques , backpropagation methods [ 60 , 41 , 52 , 45 , 37 , 4 ] , trace from the output back to the input to determine how the different elements in the input contribute to the prediction result . Class Activation Mapping ( CAM ) [ 60 ] visualizes the feature importance in convolutional neural networks by mapping the weights in the last fully connected layer to the input layer via upsampling . Score-CAM learns weighting scores for the activation maps by integrating the increase in confidence for an improved CAM visualization [ 52 ] . While for approximation methods [ 38 ] , the interpretation is obtained by fitting an interpretable model to the black-box prediction around the target sample . Deconvolution methods [ 58 ] interpret a convolutional neural network via image deconvolution . For perturbation-based interpretation [ 32 , 14 ] , the methods interpret the feature importance by imposing perturbation to certain feature and checking the changes in the output . Moreover , Shapley values [ 24 ] have been used to calculate the feature importance due to the nice properties preserved by Shapley values . For post-hoc interpretation , since the prediction and interpretation are separated , the prediction can be obtained by a highly expressive black-box model to handle complex tasks . However , the post-hoc interpretation may not capture the true predictive mechanism of the black box and is less reliable [ 1 , 19 ] . Different from post-hoc interpretation , self-interpretable models target at building white boxes that are intrinsically interpretable , which are able to conduct prediction and interpretation at the same time . A self-interpretable model preserves faithful interpretation since the model itself is a white box . However , the self-interpretation constraints can limit the expressive power , thus sacrificing prediction performance . For example , in order to build an interpretable decision set [ 22 ] , there is a constraint on the number of rules for the sake of interpretation , which restricts its application to complex tasks . Recent models propose to build self-interpretable models with neural network [ 2 , 3 , 6 , 17 , 20 , 53 , 54 ] and kernel methods [ 7 ] . FRESH [ 17 ] focuses on the interpretability for natural language processing tasks . SENN [ 3 ] , Concept Bottleneck Models [ 20 ] generate interpretations in high-level spaces instead of the raw pixel space . ProtoPNet [ 6 ] provides interpretations in the pixel space , but it focuses more on the local patches corresponding to the local areas of the image instead of the global interpretation . NAM [ 2 ] provides the same kind of interpretations as SITE . It combines neural networks with additive models to facilitate the self-interpretation via component function . But it decouples all pixels , which results in low expressiveness . Moreover , attention models have been widely used to build interpretable predictions [ 26 ] . However , recent works find that the interpretation via attention weights can fail to identify the important representations [ 42 ] . Different from the related works , our goal is to build a self-interpretable model that learns faithful interpretation and has high expressive power . Previously the transformation equivariance property has been studied in prediction via deep neural networks . Many recent studies integrate the transformation equivariance in object detection , with the goal of building convolutional neural networks that are equivariant to image translations . The models learn features equivariant to translation and rotation [ 56 , 55 , 10 ] , 3D symmetries [ 51 ] , and build sets with symmetric elements for general equivariant tasks [ 25 ] . Despite transformation equivariance in the prediction , these methods may not guarantee the transformation equivariance in the interpretation , i.e. , the prediction mechanisms of transformed and untransformed inputs may be inconsistent . We thus introduce the interpretation equivariance to complement the prediction equivariance . To the best of our knowledge , we are the first to learn transformation-equivariant interpretations to ensure faithful and robust interpretation . 3 Building Transformation Equivariant Interpretation in SITE In this section , we introduce the structure of SITE . For notations , all normal lowercase letters stand for numbers ; all bold lowercase letters stand for tensors ; all normal uppercase letters stand for operations ( including functions , networks , etc ) ; and all curly uppercase letters denote sets and families . Additionally , all Greek letters will be explained when they are introduced in context .

Summary: The paper proposes SITE Self-Interpretable model with Transformation Equivariant Interpretation.  Features are first extracted by feature extraction network such as CNN/Resnet F(x) such that F(x)=z SITE uses generative model G that maps z to c prototypes each corresponds to a specific class. y_hat = softmax(G(z)^T x) where y_hat_i = softmax(w_i^Tz) so  w_i is the interpretation of the i-th prediction. The paper then proposes to to regularize the prototypes for an input image x, they enforce each generated prototype to be similar to its corresponding class’s latent representation. They also propose to regularize on the transformation equivariance property of interpretation produced by the model.  The paper also proposed self-consistency score to compare different interpretation methods. For an input x and transformation T and interpretation I(x) the self-consistency score is defined as the cosine similarity between the transformation of the interpretation to x and the interpretation of the transformed images cosine similarity(T(I(x)),I(T(x)) . Higher indicates a more robust interpretation.  The paper compared SITE interpretation with post hoc interpretability methods and the accuracy maintained by SITE with that of inherently interpretable architecture that exhibit a decrease in accuracy due to their interpretable nature. Comparison was done on MNIST and CIFAR10.