A Framework to Learn with Interpretation

1 Introduction . Interpretability in machine learning systems [ 16 , 37 , 42 ] has recently attracted a large amount of attention . This is due to the increasing adoption of these tools in every area of automated decisionmaking , including critical domains such as law [ 25 ] , healthcare [ 48 ] or defence . Besides robustness , fairness and safety , it is considered as an essential component to ensure trustworthiness in predictive models that exhibit a growing complexity . Explainability and interpretability are often used as synonyms in the literature , referring to the ability to provide human-understandable insights on the decision process . Throughout this paper , we opt for interpretability as in [ 15 ] and leave the term explainability for the ability to provide logical explanations or causal reasoning , both requiring more sophisticated frameworks [ 17 , 19 , 44 ] . To address the long-standing challenge of interpreting models such as deep neural networks [ 43 , 10 , 9 ] , two main approaches have been developed in literature : post-hoc approaches and “ by design methods ” . Post-hoc approaches [ 7 , 41 , 38 , 45 ] generally analyze a pre-trained system locally and attempt to interpret its decisions . “ Interpretable by design ” [ 3 , 1 ] methods aim at integrating the interpretability objective into the learning process . They generally modify the structure of predictor function itself or add to the loss function regularizing penalties to enforce interpretability . Both approaches offer different types of advantages and drawbacks . Post-hoc approaches guarantee not affecting the performance of the pre-trained system but are however criticized for computational costs , robustness and faithfulness of interpretations [ 54 , 28 , 5 ] . Interpretable systems by-design on the other hand , although preferred for interpretability , face the challenge of not losing out on performance . Here , we adopt another angle to learning interpretable models . As a starting point , we consider that prediction ( computing ŷ the model ’ s output for a given input ) and interpretation ( giving a human-understandable description of properties of the input that lead to ŷ ) are two distinct but strongly related tasks . On one hand , they do not involve the same criteria for the assessment of their 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . quality and might not be implemented using the same hypothesis space . On the other hand , we wish that an interpretable model relies on the components of a predictive model to remain faithful to it . These remarks yield to a novel generic task in machine learning called Supervised Learning with Interpretation ( SLI ) . SLI is the problem of jointly learning a pair of dedicated models , a predictive model and an interpreter model , to provide both interpretability and prediction accuracy . In this work , we present FLINT ( Framework to Learn With INTerpretation ) as a solution to SLI when the model to interpret is a deep neural network classifier . The interpreter in FLINT implements the idea that a prediction to be understandable by a human should be linearly decomposed in terms of attribute functions that encode high-level concepts as other approaches [ 4 , 18 ] . However , it enjoys two original key features . First the high-level attribute functions leverage the outputs of chosen hidden layers of the neural network . Second , together with expansion coefficients they are jointly learnt with the neural network to enable local and global interpretations . By local interpretation , we mean a subset of attribute functions whose simultaneous activation leads to the model ’ s prediction , while by global interpretation , we refer to the description of each class in terms of a subset of attribute functions whose activation leads to the class prediction . Learning the pair of models involves the minimization of dedicated losses and penalty terms . In particular , local and global interpretability are enforced by imposing a limited number of attribute functions as well as conciseness and diversity among the activation of these attributes for a given input . Additionally we show that FLINT can be specialized to post-hoc interpretability if a pre-trained deep neural network is available . Key contributions : • We present FLINT devoted to Supervised Learning with Interpretation with an original interpreter network architecture based on some hidden layers of the network . The role of the interpreter is to provide local and global interpretability that we express using a novel notion of relevance of concepts . • We propose a novel entropy and sparsity based criterion for promoting conciseness and diversity in the learnt attribute functions and develop a simple pipeline to visualize the encoded concepts based on previously proposed tools . • We present extensive experiments on 4 image classification datasets , MNIST , FashionMNIST , CIFAR10 , QuickDraw , with a comparison with state-of-the-art approaches and a subjective evaluation study . • Eventually , a specialization of FLINT to post-hoc interpretability is presented while corresponding numerical results are deferred to supplements . 2 Related Works . We emphasize here more on the methods relying upon a dictionary of high level attributes/concepts , a key feature of our framework . A synthetic view of this review is presented in the supplements to effectively view the connections and differences w.r.t wider literature regarding interpretability . Post-hoc interpretations . Most works in literature focus on producing a posteriori interpretations for pre-trained models via input attribution . They often consider the model as a black-box [ 41 , 38 , 8 , 31 , 14 ] or in the case of deep neural networks , work with gradients to generate saliency maps for a given input [ 46 , 47 , 45 , 40 ] . Very few post-hoc approaches rely on high level concepts or other means of interpretations [ 22 ] . Methods utilizing high level concepts come under the subclass of concept activation vector ( CAV ) -based approaches . TCAV [ 27 ] proposed to utilize human-annotated examples to represent concepts in terms of activations of a pre-trained neural network . The sensitivity of prediction to these concepts is estimated to offer an explanation . ACE [ 18 ] attempts to automate the human-annotation process by super-pixel segmentation and clustering these segments based on their perceptual similarity where each cluster represents a concept . ConceptSHAP [ 51 ] introduces the idea of “ completeness ” in ACE ’ s framework . The CAV-based approaches already strongly differ from us in context of problem as they only consider post-hoc interpretations . TCAV generates candidate concepts using human supervision and not from the network itself . While ACE automates concept discovery , the concepts are less dynamic as by design they are associated to a single class and rely on being represented via spatially connected regions . Moreover , since ACE depends on using a CNN as perceptual similarity metric for image segments ( regardless of aspect ratio , scale ) , it is limited in applicability ( experimentally supported in supplement Sec . S.3 ) . Interpretable neural networks by design . Most works from this class learn a single model by either modifying the architecture [ 3 ] , the loss functions [ 55 , 13 ] , or both [ 6 , 36 , 4 , 12 ] . Methods like FRESH [ 24 ] and INVASE [ 53 ] perform selection over raw input tokens/features . The selected input features then are used by the final prediction model . GAME [ 35 ] shapes the learning problem as a co-operative game between predictor and interpreter . However , it learns a separate local interpreter for each sample rather than a single model . The above methods do not utilize high-level concepts for interpretation and offer local interpretations , with the exception of neural additive models [ 2 ] , which are currently only suitable for tabular data . Self Explaining Neural Networks ( SENN ) [ 4 ] presented a generalized linear model wherein coefficients are also modelled as a function of input . The linear structure is to emphasize interpretability . SENN imposes a gradient-based penalty to learn coefficients stably and other constraints to learn human understandable features . Unlike SENN , to avoid trade-off between accuracy and interpretability in FLINT , we allow the predictor to be an unrestrained neural network and jointly learn the interpreter . Interpretations are generated at a local and global level using a novel notion of relevance of attributes . Moreover , FLINT can be specialized for generating post-hoc interpretations of pre-trained networks . Known dictionary of concepts . Some recent works have focused on different ways of utilizing a known dictionary of concepts for interpretability [ 26 ] , by transforming the latent space to align with the concepts [ 13 ] or by adding user intervention as an additional feature to improve interactivity [ 29 ] . It should be noted that these methods are not comparable to FLINT or other interpretable networks by design as they assume availability of a ground truth dictionary of concepts for training . 3 Learning a classifier and its interpreter with FLINT . We introduce a novel generic task called Supervised Learning with Interpretation ( SLI ) . Denoting X the input space , and Y the output space , we assume that the training set S = { ( xi , yi ) Ni=1 } is composed of n independent realizations of a pair of random variables ( X , Y ) defined over X × Y . SLI refers to the idea that the interpretation task differs from the prediction task and must be taken over by a dedicated model that depends on the predictive model to be interpreted . Let us call F the space of predictive models from X to Y . For a given model f ∈ F , we denote Gf the family of models gf : X → Y , that depend on f and are devoted to its interpretation . For sake of simplicity , an interpreter gf ∈ Gf is denoted g , omitting the dependency on f . With these assumptions , the empirical loss of supervised learning is revisited to include explicitly an interpretability objective besides the prediction loss yielding to the following definition . Supervised Learning with Interpretation ( SLI ) : Problem 1 : arg min f∈F , g∈Gf Lpred ( f , S ) + Lint ( f , g , S ) , where Lpred ( f , S ) denotes a loss term related to prediction error and Lint ( f , g , S ) measures the ability of g to provide interpretations of predictions by f . The goal of this paper is to address Supervised Learning with Interpretation when the hypothesis space F is instantiated to deep neural networks and the task at hand is multi-class classification . We present a novel and general framework , called Framework to Learn with INTerpretation ( FLINT ) that relies on ( i ) a specific architecture for the interpreter model which leverages some hidden layers of the neural network network to be interpreted , ( ii ) notions of local and global interpretation and ( iii ) corresponding penalties in the loss function . 3.1 Design of FLINT . All along the paper , we take X = Rd and Y = { y ∈ { 0 , 1 } C , ∑C j=1 y j = 1 } , the set of C one-hot encoding vectors of dimension C. We set F to the class of deep neural networks with l hidden layers of respective dimension d1 , . . . , dl . Each element f : X → Y of F satisfies : f = fl+1 ◦ fl ◦ ... ◦ f1 where fk : Rdk−1 → Rdk , d0 = d , dl+1 = C , k = 1 , ... , l + 1 is the function implemented by layer k. A network f in F is completely identified by its generic parameter θf . As for the interpreter model g ∈ Gf , we propose the following original architecture which exploits the outputs of chosen hidden layers of f . Denote I = { i1 , i2 , ... , iT } ⊂ { 1 , . . . , l } the set of indices specifying the intermediate layers of network f to be accessed and chosen for the representation of input . We define D = ∑T t=1 dit . Typically these layers are selected from the latter layers of the network f . The concatenated vector of all intermediate outputs for an input sample x is denoted as fI ( x ) ∈ RD . Given f a network to be interpreted and a positive integer J ∈ N∗ , an interpreter network g computes the composition of a dictionary of attribute functions Φ : X → RJ and an interpretable function h : RJ → Y . ∀x ∈ X , g ( x ) = h ◦ Φ ( x ) , ( 1 ) In this work , we take : h ( Φ ( x ) ) : = softmax ( WTΦ ( x ) ) but other models like decision trees could be eligible . The attribute dictionary is composed of functions φj : X → R+ , j = 1 , . . . J whose non-negative images φj ( x ) can be interpreted as the activation of some high level attribute , i.e . a `` concept '' over X . A key originality of the model lies in the fact that the attribute functions φj ( referred to as attribute for simplicity ) leverage the outputs of hidden layers of f specified by I : ∀j ∈ { 1 , . . . , J } , φj ( x ) = ψj ◦ fI ( x ) ( 2 ) where each ψj : RD → R+ operates on the accessed hidden layers . Here , the set of functions ψj , j = 1 , . . . J is defined to form a shallow network Ψ ( around 3 layers ) whose output is Ψ ( fI ( x ) ) = Φ ( x ) ( example architecture in Fig . 1 ) . Interestingly , φj are defined over X and as a consequence can be interpreted in the input space which is the most meaningful for the user ( see Sec . 4 ) . For sake of simplicity , we denote Θg = ( θΨ , θh ) the specific parameters of this model , while the parameters devoted to the computation of fI ( x ) are shared with f .

Summary: The paper introduces FLINT, a framework that jointly trains a classifier with an “interpreter”, an interpretable version of the model. The interpreter is a linear classifier over a set of attributes (each associated with a “concept”), which are derived from hidden states of the neural network. The interpreter is trained with three objectives: (1) matching the output of the original classifier, (2) a sparsity regularizer, and (3) an anto-encoding objective to reconstruct the input. Each attribute in the interpreter can be visualized by finding input that maximizes the attribute (with some regularization), which allows the interpreter to produce global interpretation for a class and local interpretation for an example. The method is tested on four image classification tasks. It has higher accuracy than two previous interpretable models (SENN and PrototypeDNN) and higher fidelity than two post hoc interpretation methods (LIME and VIBI). According to a human survey, the visualizations are human interpretable in most cases.