A Framework to Learn with Interpretation
Year: 2022

Abstract:
To tackle interpretability in deep learning, we present a novel framework to jointly learn a predictive model and its associated interpretation model. The interpreter provides both local and global interpretability about the predictive model in terms of human-understandable high level attribute functions, with minimal loss of accuracy. This is achieved by a dedicated architecture and well chosen regularization penalties. We seek for a small-size dictionary of high level attribute functions that take as inputs the outputs of selected hidden layers and whose outputs feed a linear classifier. We impose strong conciseness on the activation of attributes with an entropy-based criterion while enforcing fidelity to both inputs and outputs of the predictive model. A detailed pipeline to visualize the learnt features is also developed. Moreover, besides generating interpretable models by design, our approach can be specialized to provide post-hoc interpretations for a pre-trained neural network. We validate our approach against several state-of-the-art methods on multiple datasets and show its efficacy on both kinds of tasks.

heading: 1 Introduction
text: Interpretability in machine learning systems [16, 37, 42] has recently attracted a large amount of attention. This is due to the increasing adoption of these tools in every area of automated decisionmaking, including critical domains such as law [25], healthcare [48] or defence. Besides robustness, fairness and safety, it is considered as an essential component to ensure trustworthiness in predictive models that exhibit a growing complexity. Explainability and interpretability are often used as synonyms in the literature, referring to the ability to provide human-understandable insights on the decision process. Throughout this paper, we opt for interpretability as in [15] and leave the term explainability for the ability to provide logical explanations or causal reasoning, both requiring more sophisticated frameworks [17, 19, 44]. To address the long-standing challenge of interpreting models such as deep neural networks [43, 10, 9], two main approaches have been developed in literature: post-hoc approaches and “by design methods”.
Post-hoc approaches [7, 41, 38, 45] generally analyze a pre-trained system locally and attempt to interpret its decisions. “Interpretable by design” [3, 1] methods aim at integrating the interpretability objective into the learning process. They generally modify the structure of predictor function itself or add to the loss function regularizing penalties to enforce interpretability. Both approaches offer different types of advantages and drawbacks. Post-hoc approaches guarantee not affecting the performance of the pre-trained system but are however criticized for computational costs, robustness and faithfulness of interpretations [54, 28, 5]. Interpretable systems by-design on the other hand, although preferred for interpretability, face the challenge of not losing out on performance.
Here, we adopt another angle to learning interpretable models. As a starting point, we consider that prediction (computing ŷ the model’s output for a given input) and interpretation (giving a human-understandable description of properties of the input that lead to ŷ) are two distinct but strongly related tasks. On one hand, they do not involve the same criteria for the assessment of their
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
quality and might not be implemented using the same hypothesis space. On the other hand, we wish that an interpretable model relies on the components of a predictive model to remain faithful to it. These remarks yield to a novel generic task in machine learning called Supervised Learning with Interpretation (SLI). SLI is the problem of jointly learning a pair of dedicated models, a predictive model and an interpreter model, to provide both interpretability and prediction accuracy. In this work, we present FLINT (Framework to Learn With INTerpretation) as a solution to SLI when the model to interpret is a deep neural network classifier. The interpreter in FLINT implements the idea that a prediction to be understandable by a human should be linearly decomposed in terms of attribute functions that encode high-level concepts as other approaches [4, 18]. However, it enjoys two original key features. First the high-level attribute functions leverage the outputs of chosen hidden layers of the neural network. Second, together with expansion coefficients they are jointly learnt with the neural network to enable local and global interpretations. By local interpretation, we mean a subset of attribute functions whose simultaneous activation leads to the model’s prediction, while by global interpretation, we refer to the description of each class in terms of a subset of attribute functions whose activation leads to the class prediction. Learning the pair of models involves the minimization of dedicated losses and penalty terms. In particular, local and global interpretability are enforced by imposing a limited number of attribute functions as well as conciseness and diversity among the activation of these attributes for a given input. Additionally we show that FLINT can be specialized to post-hoc interpretability if a pre-trained deep neural network is available.
Key contributions: • We present FLINT devoted to Supervised Learning with Interpretation with an original
interpreter network architecture based on some hidden layers of the network. The role of the interpreter is to provide local and global interpretability that we express using a novel notion of relevance of concepts.
• We propose a novel entropy and sparsity based criterion for promoting conciseness and diversity in the learnt attribute functions and develop a simple pipeline to visualize the encoded concepts based on previously proposed tools.
• We present extensive experiments on 4 image classification datasets, MNIST, FashionMNIST, CIFAR10, QuickDraw, with a comparison with state-of-the-art approaches and a subjective evaluation study.
• Eventually, a specialization of FLINT to post-hoc interpretability is presented while corresponding numerical results are deferred to supplements.

heading: 2 Related Works
text: We emphasize here more on the methods relying upon a dictionary of high level attributes/concepts, a key feature of our framework. A synthetic view of this review is presented in the supplements to effectively view the connections and differences w.r.t wider literature regarding interpretability.
Post-hoc interpretations. Most works in literature focus on producing a posteriori interpretations for pre-trained models via input attribution. They often consider the model as a black-box [41, 38, 8, 31, 14] or in the case of deep neural networks, work with gradients to generate saliency maps for a given input [46, 47, 45, 40]. Very few post-hoc approaches rely on high level concepts or other means of interpretations [22]. Methods utilizing high level concepts come under the subclass of concept activation vector (CAV)-based approaches. TCAV [27] proposed to utilize human-annotated examples to represent concepts in terms of activations of a pre-trained neural network. The sensitivity of prediction to these concepts is estimated to offer an explanation. ACE [18] attempts to automate the human-annotation process by super-pixel segmentation and clustering these segments based on their perceptual similarity where each cluster represents a concept. ConceptSHAP [51] introduces the idea of “completeness” in ACE’s framework. The CAV-based approaches already strongly differ from us in context of problem as they only consider post-hoc interpretations. TCAV generates candidate concepts using human supervision and not from the network itself. While ACE automates concept discovery, the concepts are less dynamic as by design they are associated to a single class and rely on being represented via spatially connected regions. Moreover, since ACE depends on using a CNN as perceptual similarity metric for image segments (regardless of aspect ratio, scale), it is limited in applicability (experimentally supported in supplement Sec. S.3).
Interpretable neural networks by design. Most works from this class learn a single model by either modifying the architecture [3], the loss functions [55, 13], or both [6, 36, 4, 12]. Methods like FRESH [24] and INVASE [53] perform selection over raw input tokens/features. The selected input features then are used by the final prediction model. GAME [35] shapes the learning problem as a co-operative game between predictor and interpreter. However, it learns a separate local interpreter for each sample rather than a single model. The above methods do not utilize high-level concepts for interpretation and offer local interpretations, with the exception of neural additive models [2], which are currently only suitable for tabular data. Self Explaining Neural Networks (SENN) [4] presented a generalized linear model wherein coefficients are also modelled as a function of input. The linear structure is to emphasize interpretability. SENN imposes a gradient-based penalty to learn coefficients stably and other constraints to learn human understandable features. Unlike SENN, to avoid trade-off between accuracy and interpretability in FLINT, we allow the predictor to be an unrestrained neural network and jointly learn the interpreter. Interpretations are generated at a local and global level using a novel notion of relevance of attributes. Moreover, FLINT can be specialized for generating post-hoc interpretations of pre-trained networks. Known dictionary of concepts. Some recent works have focused on different ways of utilizing a known dictionary of concepts for interpretability [26], by transforming the latent space to align with the concepts [13] or by adding user intervention as an additional feature to improve interactivity [29]. It should be noted that these methods are not comparable to FLINT or other interpretable networks by design as they assume availability of a ground truth dictionary of concepts for training.

heading: 3 Learning a classifier and its interpreter with FLINT
text: We introduce a novel generic task called Supervised Learning with Interpretation (SLI). Denoting X the input space, and Y the output space, we assume that the training set S = {(xi, yi)Ni=1} is composed of n independent realizations of a pair of random variables (X,Y ) defined over X × Y . SLI refers to the idea that the interpretation task differs from the prediction task and must be taken over by a dedicated model that depends on the predictive model to be interpreted. Let us call F the space of predictive models from X to Y . For a given model f ∈ F , we denote Gf the family of models gf : X → Y , that depend on f and are devoted to its interpretation. For sake of simplicity, an interpreter gf ∈ Gf is denoted g, omitting the dependency on f . With these assumptions, the empirical loss of supervised learning is revisited to include explicitly an interpretability objective besides the prediction loss yielding to the following definition.
Supervised Learning with Interpretation (SLI):
Problem 1: arg min f∈F,g∈Gf Lpred(f,S) + Lint(f, g,S),
where Lpred(f,S) denotes a loss term related to prediction error and Lint(f, g,S) measures the ability of g to provide interpretations of predictions by f .
The goal of this paper is to address Supervised Learning with Interpretation when the hypothesis space F is instantiated to deep neural networks and the task at hand is multi-class classification. We present a novel and general framework, called Framework to Learn with INTerpretation (FLINT) that relies on (i) a specific architecture for the interpreter model which leverages some hidden layers of the neural network network to be interpreted, (ii) notions of local and global interpretation and (iii) corresponding penalties in the loss function.

heading: 3.1 Design of FLINT
text: All along the paper, we take X = Rd and Y = {y ∈ {0, 1}C , ∑C j=1 y
j = 1}, the set of C one-hot encoding vectors of dimension C. We set F to the class of deep neural networks with l hidden layers of respective dimension d1, . . . , dl. Each element f : X → Y of F satisfies: f = fl+1 ◦ fl ◦ ... ◦ f1 where fk : Rdk−1 → Rdk , d0 = d, dl+1 = C, k = 1, ..., l + 1 is the function implemented by layer k. A network f in F is completely identified by its generic parameter θf . As for the interpreter model g ∈ Gf , we propose the following original architecture which exploits the outputs of chosen hidden layers of f . Denote I = {i1, i2, ..., iT } ⊂ {1, . . . , l} the set of indices specifying the intermediate layers of network f to be accessed and chosen for the representation of input. We define D = ∑T t=1 dit . Typically these layers are selected from the latter layers of the network f . The
concatenated vector of all intermediate outputs for an input sample x is denoted as fI(x) ∈ RD. Given f a network to be interpreted and a positive integer J ∈ N∗, an interpreter network g computes the composition of a dictionary of attribute functions Φ : X → RJ and an interpretable function h : RJ → Y . ∀x ∈ X , g(x) = h ◦ Φ(x), (1) In this work, we take: h(Φ(x)) := softmax(WTΦ(x)) but other models like decision trees could be eligible. The attribute dictionary is composed of functions φj : X → R+, j = 1, . . . J whose non-negative images φj(x) can be interpreted as the activation of some high level attribute, i.e. a "concept" over X . A key originality of the model lies in the fact that the attribute functions φj (referred to as attribute for simplicity) leverage the outputs of hidden layers of f specified by I:
∀j ∈ {1, . . . , J}, φj(x) = ψj ◦ fI(x) (2)
where each ψj : RD → R+ operates on the accessed hidden layers. Here, the set of functions ψj , j = 1, . . . J is defined to form a shallow network Ψ (around 3 layers) whose output is Ψ(fI(x)) = Φ(x) (example architecture in Fig. 1). Interestingly, φj are defined over X and as a consequence can be interpreted in the input space which is the most meaningful for the user (see Sec. 4). For sake of simplicity, we denote Θg = (θΨ, θh) the specific parameters of this model, while the parameters devoted to the computation of fI(x) are shared with f .

heading: 3.2 Interpretation in FLINT
text: The interpreter being defined, we need to specify its expected role and corresponding interpretability objective. In FLINT, interpretation is seen as an additional task besides prediction. We are interested by two kinds of interpretation, one at the global level that helps to understand which attribute functions are useful to predict a class and the other at the local level, that indicates which attribute functions are involved in prediction of a specific sample. As a preamble, note that, to interpret a local prediction f(x), we require that the interpreter output g(x) matches f(x). When the two models disagree, we provide a way to analyze the conflictual data and possibly raise an issue about the confidence on the prediction f(x) (see Supplementary Sec. S.2). To define local and global interpretation, we rely on the notion of relevance of an attribute.
Given an interpreter with parameter Θg = (θΨ, θh) and some input x, the relevance score of an attribute φj is defined regarding the prediction g(x) = f(x) = ŷ. Denoting ŷ ∈ Y the index of the predicted class and wj,ŷ ∈ W the coefficient associated to this class, the contribution of attribute φj to unnormalized score of class ŷ is αj,ŷ,x = φj(x).wj,ŷ. The relevance score is computed by normalizing contribution α as rj,x =
αj,ŷ,x maxi |αi,ŷ,x| . An attribute φj is considered as relevant for a
local prediction if it is both activated and effectively used in the linear (logistic) model. The notion of relevance of an attribute for a sample is extended to its "overall" importance in the prediction of any class c. This can be done by simply averaging relevance scores from local interpretations over a random subset or whole of the training set S, where predicted class is c. Thus, we have: rj,c =
1 |Sc| ∑ x∈Sc rj,x, Sc = {x ∈ S|ŷ = c}. Now, we can introduce the notions of local and global
interpretations that the interpreter will provide.
Definition 1 (Global and Local Interpretation) For a prediction network f , the global interpretation G(g, f) provided by an interpreter g, is the set of class-attribute pairs (c, φj) such that their global relevance rj,c is greater than some threshold 1/τ, τ > 1. A local interpretation for a sample x provided by an interpreter g of f denoted L(x, g, f) is the set of attribute functions φj with local relevance score rj,x greater than some threshold 1/τ, τ > 1.
It is important to note that these definitions do not prejudge the quality of local and global interpretations. Next, we convert desirable properties of the interpreter into specific loss functions.

heading: 3.3 Learning by imposing interpretability properties
text: Although converting desirable interpretability properties into losses is shared by several by-design approaches [11, 38], there is no consensus on these properties. We propose below a minimal set of penalties which are suitable for the proposed architecture and sufficient to provide relevant interpretations.
Fidelity to Output. The output of the interpreter g(x) should be "close" to f(x) for any x. This can be imposed through a cross-entropy loss:
Lof (f, g,S) = − ∑ x∈S h(Ψ(fI(x))) T log(f(x))
Conciseness and Diversity of Interpretations. For any given sample x, we wish to get a small number of attributes in its associated local interpretation. This property of conciseness should make the interpretation easier to understand due to fewer attributes to be analyzed and promote the "highlevel" character in the encoded concepts. However, to encourage better use of available attributes we also expect activation of multiple attributes across many randomly selected samples. We refer to this property as diversity. This is also important to avoid the case of attribute functions being learnt as class exclusive (for eg. reshuffled version of class logits). To enforce these conditions we utilize notion of entropy defined for real vectors proposed by Jain et al [23] to solve problem of efficient image search. For a real-valued vector v, the entropy is defined as E(v) = − ∑ i pi log(pi),
pi = exp(vi)/( ∑ i exp(vi)).
Conciseness is promoted by minimizing E(Ψ(fI(x))) and diversity is promoted by maximizing entropy of average Ψ(fI(x)) over a mini-batch. Note that this can be seen as encouraging the interpreter to find a sparse and diverse coding of fI(x) using the function Ψ. Since entropy-based losses have inherent normalization, they do not constrain the magnitude of the attribute activation. This often leads to poor optimization. Thus, we also minimize the `1 norm ‖Ψ(fI(x))‖1 (with hyperparameter η) to avoid it. Note that `1-regularization is a common tool to encourage sparsity and thus conciseness, however we show in the experiments that entropy provides a more effective way.
Lcd(f, g,S) = −E(Φ̄S)+ ∑ x∈S E(Ψ(fI(x)))+ ∑ x∈S η‖Ψ(fI(x))‖1 with Φ̄S = 1 |S| ∑ x∈S Ψ(fI(x))
Fidelity to Input. To encourage encoding high-level patterns related to input in Φ(x), we use a decoder network d : RJ → X that takes as input the dictionary of attributes Ψ(fI(x)) and reconstructs x. A similar penalty has previously been applied by [4].
Lif (f, g, d,S) = ∑ x∈S (d(Ψ(fI(x)))− x)2
Note that one can modify Lif with other reconstruction losses as well (such as `1-reconstruction). Given the proposed loss terms, the loss for interpretability writes as follows:
Lint(f, g, d,S) =βLof (f, g,S) + γLif (f, g, d,S) + δLcd(f, g,S) where β, γ, δ are non-negative hyperparameters. The total loss to be minimized L = Lpred + Lint, where the prediction loss, Lpred, is the well-known cross-entropy loss. Let us denote Θ = (θf , θd, θΨ, θh) the parameters of these networks. Learning the models f , Ψ, h and d boils down to learning Θ. In practice, introducing all the losses at once often leads to very poor optimization. Thus, we follow the procedure described in Alg. 1. We train the networks with Lpred,Lif for the first two epochs and gain a reasonable level of accuracy. From the third epoch we introduce Lof and from the fourth epoch we introduce Lcd loss.

heading: 4 Understanding encoded concepts in FLINT
text: Once the predictor and interpreter are jointly learnt, interpretation can be given at the global and local levels as in Def. 1. A key component to grasp the interpretations is to understand the concept
Algorithm 1 Learning algorithm for FLINT 1: Input: S & parameters Θ = (θf , θd, θΨ, θh) & hyperparameters: β0, γ0, δ0, η0 & number of
batches B & number of training epochs Nepoch. 2: Random initialization of parameter Θ0 3: Θ1← Train (S,Θ0, β = 0, γ0, δ = 0, η = 0, B, 2) {% Trains 2 epochs with Lpred,Lif} 4: Θ2← Train (S,Θ1, β = β0, γ0, δ = 0, η = 0, B, 1) {% Trains 1 epoch with Lpred,Lif ,Lof} 5: Θ̂← Train (S,Θ2, β0, γ0, δ0, η0, B,Nepoch − 3) {% Trains with all losses} 6: Output: Θ̂ = (θ̂f , θ̂d, θ̂Ψ, θ̂h)
encoded by each individual attribute function φj , previously defined in Eq. 2. In this work, we focus on image classification and propose to represent an encoded concept as a set of visual patterns in the input space which highly activate φj . We present a pipeline to generate visualizations for global and local interpretation by adapting various previously proposed tools [4, 39].
Algorithm 2 Visualization of global interpretation 1: Input: (class,attribute):(c, φj) & subset size:l & training set:Sn & AM+PI params:(λφ, λtv, λbo) 2: Sc = {x|(x, c) ∈ Sn} 3: MAS(c, φj , l)← arg maxM⊂Sc,|M|=l ∑ xi∈M φj(x)
4: FOR xk ∈ MAS(c, φj , l) 5: xkvis ← AM+PI(xk, λφ, λtv, λbo) 6: ENDFOR 7: Output:{x1vis, . . . , xlvis}, MAS(c, φj , l)
Visualization of global interpretation. Given any class-attribute pair (c, φj) in the global interpretation G(g, f), we first select a small subset of training samples from class c that maximally activate φj . This set of samples is referred to as maximum activating samples and denoted MAS(c, φj , l) where l is the size of the subset (chosen as 3 in the experiments). Although, MAS reveal some information about the encoded concept, it might not be apparent what aspect of these samples causes activation of φj . We thus propose further analyzing each element in MAS through tools that enhance the detected concept. This results in a much better understanding. The primary tool we employ is a modified version of activation maximization [39], which we refer to as activation maximization with partial initialization (AM+PI).
Given a maximum activating sample x′ ∈ MAS(c, φj , l), the key idea behind AM+PI is to synthesize appropriate input via optimization, that maximally activates φj . We thus optimize a common activation maximization objective [39]: arg maxx λφφj(x)− λtvTV(x)− λboBo(x) , where TV(.),Bo(.) are regularization terms. However, we initialize the procedure by low-intensity version of sample x′. This makes the optimization easier with the detected concept weakly present in the input. This also allows the optimization to “fill” the input to enhance the encoded concept. As an output, we obtain a map adapted to x′, that strongly activates φj . Complete details of the AM+PI procedure are given in supplementary (Sec. S.2). Visualization of a class-attribute pair is summarized in Alg. 2. Alternative useful tools are discussed in the supplementary (Sec. S.2).
Local analysis. Given any test sample x0, one can determine its local interpretation L(x0, f, g), the set of relevant attribute functions accordingly to Def. 1. To visualize a relevant attribute φj ∈ L(x0, f, g), we can repeat the AM+PI procedure with initialization using low-intensity version of x0 to enhance concept detected by φj in x0. Note that the understanding built about any attribute function φj via global analysis, although not essential, can still be helpful to understand the generated AM+PI maps during local analysis, as these maps are generally similar.

heading: 5 Numerical Experiments for FLINT
text: Datasets and Networks. We consider 4 datasets for experiments, MNIST [34], FashionMNIST [50], CIFAR-10 [30], and a subset of QuickDraw dataset [20]. Additional results on CIFAR100 [30] (large number of classes) and Caltech-UCSD Birds-200-2011 [49] (large-scale images and large number of
classes) are covered in supplementary (Sec. S.2.2). Our experiments include 2 kinds of architectures for predictor f : (i) LeNet-based [33] network for MNIST, FashionMNIST, and (ii) ResNet18-based [21] network for QuickDraw, CIFAR. We select one intermediate layer for LeNet based network and two for ResNet based networks, from the last few convolutional layers as they are expected to capture higher-level features. We set the number of attributes J = 25 for MNIST, FashionMNIST, J = 24 QuickDraw and J = 36 for CIFAR. Further details about the QuickDraw subset, precise architecture, ablation studies about choice of hyperparameters (hidden layers, size of attribute dictionary, loss scheduling) and optimization details are available in supplementary (Sec. S.2).

heading: 5.1 Quantitative evaluation of FLINT
text: We evaluate and compare our model with other state-of-the-art systems regarding accuracy and interpretability. The evaluation metrics for interpretability [15] are defined to measure the effectiveness of the losses proposed in Sec. 3.3. Our primary method for comparison, wherever applicable, is SENN, as it is an interpretable network by design with same units for interpretation as FLINT. Other baselines include PrototypeDNN [36] for predictive performance, LIME [41] and VIBI [8] for fidelity of interpretations. Implementation of our method is available on Github 1. Details for implementation of baselines are in supplementary (Sec. S.2).
Predictive performance of FLINT. There are two goals to validate related to predictor trained with FLINT (denoted FLINT-f ), (i) Jointly training f with g and backpropagating loss term Lint does not negatively impact performance, and (ii) The achieved performance is comparable with other similar interpretable by-design models. For the former we compare the accuracy of FLINT-f with same predictor architecture trained just with Lpred (denoted by BASE-f ). For the latter goal we compare accuracy of FLINT-f with accuracy of SENN and another interpretable network by design PrototypeDNN [36] that does not use input attribution for interpretations. Note that PrototypeDNN requires non-trivial changes to the model for running on more complex datasets, CIFAR10 and QuickDraw. To avoid any unfair comparison we skip these results. The accuracies are reported in Tab. 1. They indicate that training f within FLINT does not result in any significant accuracy loss on any dataset. Also, FLINT is competitive with other interpretable by-design models.
Fidelity of Interpreter. The fraction of samples where prediction of a model and its interpreter agree, i.e predict the same class, is referred to as fidelity. It is a commonly used metric to measure how well an interpreter approximates a model [8, 32]. Note that, typically, for interpretable by design models, fidelity cannot be measured as they only consider a single model. However, to validate that the interpreter trained with FLINT (denoted as FLINT-g) achieves a reasonable level of agreement with FLINT-f , we benchmark its fidelity against a state-of-the-art black-box explainer VIBI [8] and a traditional method LIME [41]. The results for this are provided in Tab. 1 (last three columns). FLINT-g consistently achieves higher fidelity. Even though it is not a fair comparison as other systems are black-box explainers and FLINT-g accesses intermediate layers, they clearly show that FLINT-g demonstrates high fidelity to FLINT-f .
Conciseness of interpretations. We evaluate conciseness by measuring the average number of important attributes in generated interpretations. For a given sample x, it can be computed as number of attributes φj with rj,x greater than a threshold 1/τ, τ > 1, i.e. CNSg,x = |{j : |rj,x| > 1/τ}|. For different thresholds 1/τ , we compute the mean of CNSg,x over test data to estimate conciseness of g, CNSg. Lower conciseness indicates need to analyze a lower number of attributes on an
1https://github.com/jayneelparekh/FLINT
average. SENN is the only other system for which this curve can be computed. We thus compare the conciseness of SENN with FLINT on all four datasets. Fig. 2a depicts the same. It can be easily observed that FLINT produces lot more concise interpretations compared to SENN. Moreover, SENN even ends up with majority of concepts being considered relevant for lower thresholds (higher τ ).
Entropy vs `1 regularization. We validate the effectiveness of entropy losses by computing conciseness curve at various levels of `1 regularization strength, with and without entropy, for ResNet with QuickDraw. This is reported in Fig. 2b. The figure confirms that using the entropy-based loss is more effective in inducing conciseness of explanations compared to using just `1-regularization, with the difference being close to use of 1 attribute less when entropy losses are employed.
Importance of attributes. Additional experiments evaluating meaningfulness of attributes by shuffling them and observing the effect (for FLINT and SENN) are given in supplementary (Sec. S.2).

heading: 5.2 Qualitative analysis
text: Global interpretation. Fig. 2c depicts the generated global relevances rj,c for all class-attribute pairs on QuickDraw and CIFAR. Each class-attribute pair with ‘high’ relevance needs to be analyzed as part of global analysis. Some example class-attribute pairs, with high relevance, are visualized in Fig. 3. For each pair we select MAS of size 3 and also show their AM+PI outputs. As mentioned before, simply analyzing MAS reveals useful information about the encoded concept. For instance, based on MAS, φ15, φ19 on MNIST, relevant for class ‘One’, clearly seem to activate for vertical and diagonal strokes respectively. However, AM+PI outputs give deeper insights about the concept by revealing more clearly what parts of input activate an attribute function. For eg., while MAS indicate that φ5 on FashionMNIST activates for heels (one type of ‘Sandal’), φ2 on CIFAR10 activates for white dogs, it is not clear what part the attribute focuses on. AM+PI outputs indicate that φ2 focuses on the area around eyes and nose (the most enhanced regions), φ5 primarily detects a thin diagonal stroke of the heel surrounded by empty space. AM+PI outputs generally become even more important for attributes relevant for multiple classes. One such example is the function φ5 on QuickDraw, relevant for both ‘Dog’ and ‘Lion’. It activates for very similar set of strokes for all samples, as indicated by AM+PI maps. For ‘Dog’ this corresponds to ears and mouth and for ‘Lion’ it corresponds to the mane. Other such attribute functions in the figure include φ24 on FashionMNIST, relevant for ‘Pullover’, ‘Coat’ and ‘Shirt’ which detects long sleeves and φ29 on CIFAR10, relevant for ‘Trucks’, ‘Cars’ and primarily detects wheels and parts of upper body. Further visualizations including those of other relevant classes for φ24, φ29 and global relevances are available in supplementary (Sec. S.2).
Local interpretation. Fig. 4 (left) displays the local interpretation visualizations for test samples. f and g both predict the true class in all the cases. We show the top 3 relevant attributes to the prediction with their relevances and their corresponding AM+PI outputs. Based on the AM+PI outputs it can be observed that the attribute functions generally activate for patterns corresponding to the same concept as inferred during global analysis. This can be easily seen for attribute functions present in both Fig. 3, 4 (left). This is further illustrated by Fig. 4 (right) where we illustrate AM+PI outputs for two attributes from Fig. 3. These functions are relevant for more than one class and detect the same concept across various test samples, namely long sleeves for φ24 and primarily wheels for φ29.

heading: 5.3 Subjective evaluation
text: We conducted a survey based subjective evaluation with QuickDraw dataset for FLINT with 20 respondents. We selected 10 attributes, covering 17 class-attribute pairs from the QuickDraw dataset. For each attribute we present the respondent with our visualizations (3 MAS and AM+PI outputs) for each of its relevant classes along with a textual description. We ask them if the description meaningfully associates to patterns in the AM+PI outputs. They indicate level of agreement with choices: Strongly Agree (SA), Agree (A), Disagree (D), Strongly Disagree (SD), Don’t Know (DK). Descriptions were manually generated by our understanding of encoded concept for each attribute. 40% incorrect descriptions were carefully included to ensure informed responses. These were forcefully related to the classes shown to make them harder to identify. Results – for correct descriptions: 77.5% – SA/A, 10.0% – DK, 12.5% – D/SD. For incorrect descriptions: 83.7% – D/SD, 7.5% – DK, 8.8% – SA/A. These results clearly indicate that concepts encoded in FLINT’s learnt attributes are understandable to humans. Survey details are given in supplementary (Sec. S.2).

heading: 6 Specialization of FLINT to post-hoc interpretability
text: While interpretability by design is the primary goal of FLINT, it can be specialized to provide a posthoc interpretation when a classifier f̂ is already available. The Post-hoc interpretation learning (see for instance [41]) comes as a special case of SLI and is defined as follows. Given a classifier f̂ ∈ F and a training set S, the goal is to build an interpreter of f̂ by solving:
Problem 2: arg min g∈Gf̂ Lint(f̂ , g,S).
With FLINT, we have g(x) = h ◦ Φ(x) and Φ(x) = Ψ ◦ f̂I(x) for a given set of accessible hidden layers I and a attribute dictionary size J . Learning can be performed by specializing Alg. 1 with slight modification of replacing Θ as Θ = (θΨ, θh, θd) while θf̂ is fixed and eliminating Lpred from training loss L.
Experimental results for post-hoc FLINT: We validate this ability of our framework by interpreting fixed models trained only for accuracy, i.e, BASE-f models from section 5.1. Even after not tuning the internal layers of f , the system is still able to generate high-fidelity and meaningful interpretations. Fidelity comparisons against VIBI, class-attribute pair visualizations and experimental details are available in supplementary (Sec. S.3).

heading: 7 Discussion and Perspectives
text: FLINT is a novel framework for learning a predictor network and its interpreter network with dedicated losses. It provides local and global interpretations in terms of high-level learnt attributes/concepts by relying on (some) hidden layers of the prediction network. This however leaves some under-explored questions about faithfulness of interpretations to the predictor. Defining faithfulness of an interpretation regarding a decision process [4] has not yet reached a consensus particularly in the case of post-hoc interpretability or when the two models, predictor and interpreter, differ [52]. Even though generating interpretations based on hidden layers of predictor ensures high level of faithfulness of the interpreter to the predictor, a complete faithfulness cannot be guaranteed since predictor and interpreter differ in their last part. However if ensuring faithfulness by design is regarded as the primary objective, nothing stops the use of interpreter FLINT-g as the final decisionmaking network. In this case, there is only one network and the so-called prediction network has only played the useful role of providing relevant hidden layers.
Retaining only the interpreter model additionally provides a compression of the predictor and can be relevant when frugality is at stake. Further works will investigate this direction and the enforcement of additional constraints on attribute functions to encourage invariance under various transformations. Eventually FLINT can be extended to other tasks or modalities other than images in particular by adapting the design of attributes and the pipeline to understand them.
Acknowledgments and Disclosure of Funding
This work has been funded by the research chair Data Science & Artificial Intelligence for Digitalized Industry and Services of Télécom Paris. This work also benefited from the support of French National Research Agency (ANR) under reference ANR-20-CE23-0028 (LIMPID project). The authors would like to thank Sanjeel Parekh for fruitful discussions and anonymous reviewers for their valuable comments and suggestions.

Review summaries:
This paper proposes an architecture which jointly learns to classify as well as provide an explanation behind the classification. The explanation is generated from a separate part of the model which doesn’t affect the model performance much as the classification performance is achieved by a backbone network. The explanation generation part takes inputs from the intermediate layers of the backbone network which helps to produce better and more meaningful explanations. The results are promising on experiments shown on MNIST, FMNIST, CIFAR10 and QuickDraw datasets.

The paper introduces FLINT, a framework that jointly trains a classifier with an “interpreter”, an interpretable version of the model. The interpreter is a linear classifier over a set of attributes (each associated with a “concept”), which are derived from hidden states of the neural network. The interpreter is trained with three objectives: (1) matching the output of the original classifier, (2) a sparsity regularizer, and (3) an anto-encoding objective to reconstruct the input. Each attribute in the interpreter can be visualized by finding input that maximizes the attribute (with some regularization), which allows the interpreter to produce global interpretation for a class and local interpretation for an example. The method is tested on four image classification tasks. It has higher accuracy than two previous interpretable models (SENN and PrototypeDNN) and higher fidelity than two post hoc interpretation methods (LIME and VIBI). According to a human survey, the visualizations are human interpretable in most cases.

The paper proposes FLINT, a new learning schema in which a neural net and a gray-box model are learned jointly by minimizing a composite loss: a loss on the labels + a set of losses controlling the concepts acquired by the gray-box models + a distillation loss that encourages the gray-box models to behave similarly to the neural net.  The concepts in the gray-box model are built from intermediate neurons of the neural net to encourage faithfulness.  The authors also describe a visualization pipeline and a post-hoc explanation protocol.

The paper proposes FLINT a Framework to Learn with Interpretation to solve Supervised Learning with Interpretation task. FLINT involves training 3 models on a preditor model an interpreter and a decoder. This framework provides both global and local interpretations.  The interpreter network takes as an input the output of different hidden layers from the predictor network then computes an attribute dictionary of functions phi g(x)= softmax(Wphi(x)). Different phi corresponds to different concepts  phi_j(x)= Psi_j (f_I(x)) where Psi_j is shallow network and f_I(x) is the output of hidden state I.  So The model parameters of g depend on the parameters of Psi and Weights. Given interpreter with parameter and input x the contribution of phi_j = phi_j(x)*w_j,y_hat/max|ph_i*w_i_yhat| where y_hat the class output.  Global Interpretation is a set of class-attribute pairs (c, ph_j) such that their global relevance is greater than some threshold. Local interpretation for a sample x provided is the set of attribute functions ph_j  with a local relevance score greater than some threshold.  FLINT has multiple loss functions, first to ensure that the interpreter output is similar to that of the model output cross-entropy is used. An additional loss is used to ensure diversity of ph_j and ensure input fidelity the input is reconstructed from phi using a decoder network which gives us the final loss.  FlINT was evaluated on 4 different image datasets, it was compared with other self explain networks SENN and PrototypeDNN to terms of accuracy and saliency methods LIME and VIBI in terms of fidelity.