A Framework to Learn with Interpretation

1 Introduction . Interpretability in machine learning systems [ 16 , 37 , 42 ] has recently attracted a large amount of attention . This is due to the increasing adoption of these tools in every area of automated decisionmaking , including critical domains such as law [ 25 ] , healthcare [ 48 ] or defence . Besides robustness , fairness and safety , it is considered as an essential component to ensure trustworthiness in predictive models that exhibit a growing complexity . Explainability and interpretability are often used as synonyms in the literature , referring to the ability to provide human-understandable insights on the decision process . Throughout this paper , we opt for interpretability as in [ 15 ] and leave the term explainability for the ability to provide logical explanations or causal reasoning , both requiring more sophisticated frameworks [ 17 , 19 , 44 ] . To address the long-standing challenge of interpreting models such as deep neural networks [ 43 , 10 , 9 ] , two main approaches have been developed in literature : post-hoc approaches and “ by design methods ” . Post-hoc approaches [ 7 , 41 , 38 , 45 ] generally analyze a pre-trained system locally and attempt to interpret its decisions . “ Interpretable by design ” [ 3 , 1 ] methods aim at integrating the interpretability objective into the learning process . They generally modify the structure of predictor function itself or add to the loss function regularizing penalties to enforce interpretability . Both approaches offer different types of advantages and drawbacks . Post-hoc approaches guarantee not affecting the performance of the pre-trained system but are however criticized for computational costs , robustness and faithfulness of interpretations [ 54 , 28 , 5 ] . Interpretable systems by-design on the other hand , although preferred for interpretability , face the challenge of not losing out on performance . Here , we adopt another angle to learning interpretable models . As a starting point , we consider that prediction ( computing ŷ the model ’ s output for a given input ) and interpretation ( giving a human-understandable description of properties of the input that lead to ŷ ) are two distinct but strongly related tasks . On one hand , they do not involve the same criteria for the assessment of their 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . quality and might not be implemented using the same hypothesis space . On the other hand , we wish that an interpretable model relies on the components of a predictive model to remain faithful to it . These remarks yield to a novel generic task in machine learning called Supervised Learning with Interpretation ( SLI ) . SLI is the problem of jointly learning a pair of dedicated models , a predictive model and an interpreter model , to provide both interpretability and prediction accuracy . In this work , we present FLINT ( Framework to Learn With INTerpretation ) as a solution to SLI when the model to interpret is a deep neural network classifier . The interpreter in FLINT implements the idea that a prediction to be understandable by a human should be linearly decomposed in terms of attribute functions that encode high-level concepts as other approaches [ 4 , 18 ] . However , it enjoys two original key features . First the high-level attribute functions leverage the outputs of chosen hidden layers of the neural network . Second , together with expansion coefficients they are jointly learnt with the neural network to enable local and global interpretations . By local interpretation , we mean a subset of attribute functions whose simultaneous activation leads to the model ’ s prediction , while by global interpretation , we refer to the description of each class in terms of a subset of attribute functions whose activation leads to the class prediction . Learning the pair of models involves the minimization of dedicated losses and penalty terms . In particular , local and global interpretability are enforced by imposing a limited number of attribute functions as well as conciseness and diversity among the activation of these attributes for a given input . Additionally we show that FLINT can be specialized to post-hoc interpretability if a pre-trained deep neural network is available . Key contributions : • We present FLINT devoted to Supervised Learning with Interpretation with an original interpreter network architecture based on some hidden layers of the network . The role of the interpreter is to provide local and global interpretability that we express using a novel notion of relevance of concepts . • We propose a novel entropy and sparsity based criterion for promoting conciseness and diversity in the learnt attribute functions and develop a simple pipeline to visualize the encoded concepts based on previously proposed tools . • We present extensive experiments on 4 image classification datasets , MNIST , FashionMNIST , CIFAR10 , QuickDraw , with a comparison with state-of-the-art approaches and a subjective evaluation study . • Eventually , a specialization of FLINT to post-hoc interpretability is presented while corresponding numerical results are deferred to supplements . 2 Related Works . We emphasize here more on the methods relying upon a dictionary of high level attributes/concepts , a key feature of our framework . A synthetic view of this review is presented in the supplements to effectively view the connections and differences w.r.t wider literature regarding interpretability . Post-hoc interpretations . Most works in literature focus on producing a posteriori interpretations for pre-trained models via input attribution . They often consider the model as a black-box [ 41 , 38 , 8 , 31 , 14 ] or in the case of deep neural networks , work with gradients to generate saliency maps for a given input [ 46 , 47 , 45 , 40 ] . Very few post-hoc approaches rely on high level concepts or other means of interpretations [ 22 ] . Methods utilizing high level concepts come under the subclass of concept activation vector ( CAV ) -based approaches . TCAV [ 27 ] proposed to utilize human-annotated examples to represent concepts in terms of activations of a pre-trained neural network . The sensitivity of prediction to these concepts is estimated to offer an explanation . ACE [ 18 ] attempts to automate the human-annotation process by super-pixel segmentation and clustering these segments based on their perceptual similarity where each cluster represents a concept . ConceptSHAP [ 51 ] introduces the idea of “ completeness ” in ACE ’ s framework . The CAV-based approaches already strongly differ from us in context of problem as they only consider post-hoc interpretations . TCAV generates candidate concepts using human supervision and not from the network itself . While ACE automates concept discovery , the concepts are less dynamic as by design they are associated to a single class and rely on being represented via spatially connected regions . Moreover , since ACE depends on using a CNN as perceptual similarity metric for image segments ( regardless of aspect ratio , scale ) , it is limited in applicability ( experimentally supported in supplement Sec . S.3 ) . Interpretable neural networks by design . Most works from this class learn a single model by either modifying the architecture [ 3 ] , the loss functions [ 55 , 13 ] , or both [ 6 , 36 , 4 , 12 ] . Methods like FRESH [ 24 ] and INVASE [ 53 ] perform selection over raw input tokens/features . The selected input features then are used by the final prediction model . GAME [ 35 ] shapes the learning problem as a co-operative game between predictor and interpreter . However , it learns a separate local interpreter for each sample rather than a single model . The above methods do not utilize high-level concepts for interpretation and offer local interpretations , with the exception of neural additive models [ 2 ] , which are currently only suitable for tabular data . Self Explaining Neural Networks ( SENN ) [ 4 ] presented a generalized linear model wherein coefficients are also modelled as a function of input . The linear structure is to emphasize interpretability . SENN imposes a gradient-based penalty to learn coefficients stably and other constraints to learn human understandable features . Unlike SENN , to avoid trade-off between accuracy and interpretability in FLINT , we allow the predictor to be an unrestrained neural network and jointly learn the interpreter . Interpretations are generated at a local and global level using a novel notion of relevance of attributes . Moreover , FLINT can be specialized for generating post-hoc interpretations of pre-trained networks . Known dictionary of concepts . Some recent works have focused on different ways of utilizing a known dictionary of concepts for interpretability [ 26 ] , by transforming the latent space to align with the concepts [ 13 ] or by adding user intervention as an additional feature to improve interactivity [ 29 ] . It should be noted that these methods are not comparable to FLINT or other interpretable networks by design as they assume availability of a ground truth dictionary of concepts for training . 3 Learning a classifier and its interpreter with FLINT . We introduce a novel generic task called Supervised Learning with Interpretation ( SLI ) . Denoting X the input space , and Y the output space , we assume that the training set S = { ( xi , yi ) Ni=1 } is composed of n independent realizations of a pair of random variables ( X , Y ) defined over X × Y . SLI refers to the idea that the interpretation task differs from the prediction task and must be taken over by a dedicated model that depends on the predictive model to be interpreted . Let us call F the space of predictive models from X to Y . For a given model f ∈ F , we denote Gf the family of models gf : X → Y , that depend on f and are devoted to its interpretation . For sake of simplicity , an interpreter gf ∈ Gf is denoted g , omitting the dependency on f . With these assumptions , the empirical loss of supervised learning is revisited to include explicitly an interpretability objective besides the prediction loss yielding to the following definition . Supervised Learning with Interpretation ( SLI ) : Problem 1 : arg min f∈F , g∈Gf Lpred ( f , S ) + Lint ( f , g , S ) , where Lpred ( f , S ) denotes a loss term related to prediction error and Lint ( f , g , S ) measures the ability of g to provide interpretations of predictions by f . The goal of this paper is to address Supervised Learning with Interpretation when the hypothesis space F is instantiated to deep neural networks and the task at hand is multi-class classification . We present a novel and general framework , called Framework to Learn with INTerpretation ( FLINT ) that relies on ( i ) a specific architecture for the interpreter model which leverages some hidden layers of the neural network network to be interpreted , ( ii ) notions of local and global interpretation and ( iii ) corresponding penalties in the loss function . 3.1 Design of FLINT . All along the paper , we take X = Rd and Y = { y ∈ { 0 , 1 } C , ∑C j=1 y j = 1 } , the set of C one-hot encoding vectors of dimension C. We set F to the class of deep neural networks with l hidden layers of respective dimension d1 , . . . , dl . Each element f : X → Y of F satisfies : f = fl+1 ◦ fl ◦ ... ◦ f1 where fk : Rdk−1 → Rdk , d0 = d , dl+1 = C , k = 1 , ... , l + 1 is the function implemented by layer k. A network f in F is completely identified by its generic parameter θf . As for the interpreter model g ∈ Gf , we propose the following original architecture which exploits the outputs of chosen hidden layers of f . Denote I = { i1 , i2 , ... , iT } ⊂ { 1 , . . . , l } the set of indices specifying the intermediate layers of network f to be accessed and chosen for the representation of input . We define D = ∑T t=1 dit . Typically these layers are selected from the latter layers of the network f . The concatenated vector of all intermediate outputs for an input sample x is denoted as fI ( x ) ∈ RD . Given f a network to be interpreted and a positive integer J ∈ N∗ , an interpreter network g computes the composition of a dictionary of attribute functions Φ : X → RJ and an interpretable function h : RJ → Y . ∀x ∈ X , g ( x ) = h ◦ Φ ( x ) , ( 1 ) In this work , we take : h ( Φ ( x ) ) : = softmax ( WTΦ ( x ) ) but other models like decision trees could be eligible . The attribute dictionary is composed of functions φj : X → R+ , j = 1 , . . . J whose non-negative images φj ( x ) can be interpreted as the activation of some high level attribute , i.e . a `` concept '' over X . A key originality of the model lies in the fact that the attribute functions φj ( referred to as attribute for simplicity ) leverage the outputs of hidden layers of f specified by I : ∀j ∈ { 1 , . . . , J } , φj ( x ) = ψj ◦ fI ( x ) ( 2 ) where each ψj : RD → R+ operates on the accessed hidden layers . Here , the set of functions ψj , j = 1 , . . . J is defined to form a shallow network Ψ ( around 3 layers ) whose output is Ψ ( fI ( x ) ) = Φ ( x ) ( example architecture in Fig . 1 ) . Interestingly , φj are defined over X and as a consequence can be interpreted in the input space which is the most meaningful for the user ( see Sec . 4 ) . For sake of simplicity , we denote Θg = ( θΨ , θh ) the specific parameters of this model , while the parameters devoted to the computation of fI ( x ) are shared with f .

Summary: The paper proposes FLINT, a new learning schema in which a neural net and a gray-box model are learned jointly by minimizing a composite loss: a loss on the labels + a set of losses controlling the concepts acquired by the gray-box models + a distillation loss that encourages the gray-box models to behave similarly to the neural net.  The concepts in the gray-box model are built from intermediate neurons of the neural net to encourage faithfulness.  The authors also describe a visualization pipeline and a post-hoc explanation protocol.