A Framework to Learn with Interpretation
Authors: Jayneel Parekh
Pavlo Mozharovskyi
Florence d'Alché-Buc
Venue: NeurIPS 2021 Poster (2021)
Keywords: Interpretability
Explainability
Deep Neural Networks

Abstract: To tackle interpretability in deep learning, we present a novel framework to jointly learn a predictive model and its associated interpretation model. The interpreter provides both local and global interpretability about the predictive model in terms of human-understandable high level attribute functions, with minimal loss of accuracy. This is achieved by a dedicated architecture and well chosen regularization penalties. We seek for a small-size dictionary of high level attribute functions that take as inputs the outputs of selected hidden layers and whose outputs feed a linear classifier. We impose strong conciseness on the activation of attributes with an entropy-based criterion while enforcing fidelity to both inputs and outputs of the predictive model. A detailed pipeline to visualize the learnt features is also developed. Moreover, besides generating interpretable models by design, our approach can be specialized to provide post-hoc interpretations for a pre-trained neural network. We validate our approach against several state-of-the-art methods on multiple datasets and show its efficacy on both kinds of tasks.

A Framework to Learn with Interpretation
Jayneel Parekh Pavlo Mozharovskyi 1, Florence d' A'Alcéé-Buc
1 LTCI, Télécom Paris,
Institut Polytechnique de Paris, France
jjayneel.parkkh.pavlooo.
www.:ree....lloooooooorrrrsroao floreneeaaceeeeeecchce (eeeeaaleeeeeeeeeeeeeeee @telecom-paris.h
Abstract
To tackle interpretability in deep learning, we present novel framework jointly learn a predictive model and its associated interpretation model. The interpreter provides both local and global interpretability about the predictive model terms of human-understene understandable high level attribute functions, with minimal loss of accuracy. This is achieved by a dedicated architecture and well chosen regularization penalties. We seek for small-size dictionary of high level attribute functions that take as inputs the outputs of selected hidden layers and whose outputs feed linear classifier. We impose strong conciseness on the activation of attributes with an entropy-based criterion while enforcing fidelity to both inputs and outputs the predictive model. A detailed pipeline to visualize the learnt features is also developed. Moreover, besides generating interpretable models by design, our approach can be specialized to provide post-hoc interpretations for pre-trained neural network. We validate our approach against several state-of-ther -fe-----thht-- art methods multiple datasets and show its efficacy on both kinds of tasks.
Introduction
Interpretability in machine learning systems [16 37 42 has recently attracted a large amount of attention. This is due to the increasing adoption of these tools in every area of automated decision- making, including critical domains such as law (25]. healthcare [48] or defence. Besides robustness, fairness and safety, is considered as an essential component to ensure trustworthiness in predictive models that exhibit a growing complexity. Explainability and interpretability are often used as synonyms the literature, referring to the ability to provide numan-understandable insights on the decision process. Throughout this paper, we opt for interpretability as in [15] and leave the term explainability the ability to provide logical explanations or causal reasoning, both requiring more sophisticated frameworks [17 19 44 To address the long-standing challenge of interpreting models two main approaches have been developed in literature: post-hoc approaches and "by design methods".
such as deep neural networks [43 9
generally analyze a pre-trained system locally and attempt to interpret its decisions. "Interpretable by design" [ methods aim at integrating the interpretability objective into the learning process. They generally modify the structure of predictor function itself or add to the loss function regularizing penalties to enforce interpretability. Both approaches offer different types of advantages and drawbacks. Post-hoc approaches guarantee not affecting the performance of the pre-trained system but are however criticized for computational costs, robustness and faithfulness of interpretations [54 28 5). Interpretable systems by-design on the other hand, although preferred for interpretability, face the challenge of not losing out performance.
Post-hoe approaches 7
45
Here, we adopt another angle to learning interpretable models. As starting point, consider that prediction (computing ÿ the model's output for given input) and interpretation (giving human- human-understanme understaddalle description of properties of the input that lead to ÿ) are two distinet but strongly related tasks. On one hand, they do not involve the same criteria for the assessment of their
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

quality and might not be implemented using the same hypothesis space. On the other hand, we wish that an interpretable model relies on the components of predictive model to remain faithful to These remarks yield to a novel generic task in machine learning called Supervised Learning with Interpretation (SI.). SLI the problem of jointly learning a pair of dedicated models, a predictive model and an interpreter model, to provide both interpretability and prediction accuracy. In this work, we present FLINT (Framework to Learn With INTerpretation) as a solution to SLI when the model to interpret is deep neural network classifier. The interpreter in FLINT implements the idea that a prediction to be understandable by human should be linearly decomposed in terms of attribute functions that encode high-level concepts other approaches [ [8]. However, it enjoys two original key features. First the high-level attribute functions leverage the outputs of chosen hidden layers of the neural network. Second, together with expansion coefficients they jointly learnt with the neural network to enable local and global interpretations. By local interpretation, we mean a subset of attribute functions whose simultaneous activation leads to the model's prediction, while by global interpretation, we refer to the description of each class in terms of a subset of attribute functions whose activation leads to the class prediction. Learning the pair of models involves the minimization of dedicated losses and penalty terms. In particular, local and global interpretability are enforced by imposing a limited number of attribute functions as well as conciseness and diversity among the activation of these attributes for given input. Additionally we show that FLINT can be specialized to post-hoc interpretability if a pre- pre-trained deep neural network is available.
Key contributions:
We present FLINT devoted to Supervised Learning with Interpretation with an original interpreter network architecture based on some hidden layers of the network. The role of the interpreter is to provide local and global interpretability that we express using a novel notion of relevance of concepts.
We propose novel entropy and sparsity based criterion for promoting conciseness and diversity in the learnt attribute functions and develop a simple pipeline to visualize the encoded concepts based on previously proposed tools.
We present extensive experiments on 4 image classification datasets, MNIST, FashionM- NIST, CIFARIO, QuickDraw, with a comparison with state-of-the-art approaches and subjective evaluation study.
Eventually, a specialization of FLINT to post-hoc interpretability is presented while corre- sponding numerical results are deferred to supplements.
2
Related Works
We emphasize here more on the methods relying upon a dictionary of high level attributes/concepss, key feature of our framework. A synthetic view of this review is presented in the supplements to effectively view the connections and differences w.r.t wider literature regarding interpretability.
Post-hoc interpretations. Most works in literature focus on producing posteriori interpretations for pre-trained models via input attribution. They often consider the model as a black- box or in the case of deep neural networks, work with gradients to generate saliency maps for a given input [46] 47 44 44].. Very few post-hoc approaches rely on high level concepts or other means of interpretations [2 Methods utilizing high level concepts come under the subelass of concept activation vector (CAV)-based approaches. TCAV (27 proposed to utilize human-annotated examples represent concepts in terms of activations of a pre-trained neural network. The sensitivity of prediction to these concepts is estimated to offer an explanation. ACE (18) attempts to automate the human-annotee annotation process by super-pixel segmentation and clustering these segments based on their perceptual similarity where each cluster represents a concept. ConcepiSHAP (51] introduces the idea of "completeness" in ACE's framework. The CAV-based approaches already strongly differ from us in context of problem as they only consider post-hoe interpretations. TCAV generates candidate concepts using human supervision and not from the network itself. While ACE automates concept discovery, the concepts are less dynamic as by design they are associated to a single class and rely on being represented via spatially connected regions. Moreover, since ACE depends on using a CNN as perceptual similarity metric for image segments (regardless of aspect ratio, scale), it limited in applicability (experimentally supported supplement See. S.3).
41 38 g
31
14
2

---

Interpretable neural networks by design. Most works from this class learn single model by
either modifying the architecture [3) the loss functions [55] [3] or both 6 36 [2]. Methods like FRESH [24] and INVASE [53] perform selection over raw input tokens/features. The selected input
features then are used by the final prediction model. GAME 35 shapes the learning problem as a co-operative game between predictor and interpreter. However, it learns a separate local interpreter for each sample rather than a single model. The above methods do not utilize high-level concepts for interpretation and offer local interpretations, with the exception of neural additive models (2), which are currently only suitable for tabular data.
Self Explaining Neural Networks (SENN) 4 presented generalized linear model wherein coeffi- cients are also modelled as a function of input. The linear structure is to emphasize interpretability. SENN imposes gradient- based penalty to learn coefficients stably and other constraints learn hu- man understandable features. Unlike SENN, to avoid trade-off between accuracy and interpretability in FLINT, we allow the predictor to be an unrestrained neural network and jointly learn the interpreter. Interpretations are generated at a local and global level using a novel notion of relevance of attributes. Moreover, FLINT can be specialized for generating post-hoc interpretations of pre-trained networks. Known dictionary of concepts. Some recent works have focused on different ways of utilizing known dictionary of concepts for interpretability [26]. by transforming the latent space align with the concepts 3 or by adding user intervention as an additional feature to improve interactivity [29]. should be noted that these methods are not comparable to FLINT or other interpretable networks by design as they assume availability of a ground truth dictionary of concepts for training.
3 Learning classifier and its interpreter with FLINT
We introduce novel generic task called Supervised Learning with Interpretation (SLI). Denoting X' the input space, and y the output space, we assume that the training set S {I., y1 )) is composed of n independent realizations of a pair of random variables (X, Y) defined over X x V. SLI refers to the idea that the interpretation task differs from the prediction task and must be taken over by dedicated model that depends on the predictive model to be interpreted. Let us call F the space of predictive models from to y. For given model f F, we denote gy the family of models 9f X' + ), that depend on ʃ and are devoted to its interpretation. For sake of simplicity, an interpreter 9 E G is denoted g, omitting the dependency on f. With these assumptions, the empirical loss of supervised learning is revisited to include explicitly an interpretability objective besides the prediction loss yielding to the following definition.
Supervised Learning with Interpretation (SLI):
Problem 1: arg min feF,gE0, (prea(S, + Line(J,g,,)
where [f S) denotes loss term related to prediction error and Line (J,g, S) measures the ability of g to provide interpretations of predictions by f.
The goal of this paper is to address Supervised Learning with Interpretation when the hypothesis space F is instantiated deep neural networks and the task at hand is multi-class classification. We present a novel and general framework, called Framework to Learn with INTerpretation (FLINT) that relies on (i) a specific architecture for the interpreter model which leverages some hidden layers of the neural network network to be interpreted, (ii) notions of local and global interpretation and (iii) corresponding penalties in the loss function.
3.1 Design of FLINT
the set of C' one-hot encoding vectors of dimension C. We set F to the class of deep neural networks with / hidden layers of respective dimension d1, -i..... ........ dh. Fach element f X y of F satisfies: f= fi+1 fi ... o fi where f& d+ + Rdd,, do = d. d141 = C. 1, ..., + 1 is the function implemented by layer k. A network in J is completely identified by its generic parameter 0y. As for the interpreter model g E Gy, we propose the following original architecture which exploits the outputs of chosen hidden layers of f. Denote I {ir,
All along the paper, we take X' = RRd and = = E (0,
=1. 1
C [I....,(} the set of indices specifying the intermediate layers of network f to be accessed and chosen for the representation of input. We define
D IT_ di. Typically these layers are selected from the latter layers of the network f. The
3

---

Predictor
+5
*/
+/x
f(a)
x/ (x(x)
== (5,0,8)
Let
a(x) h ((s)
V
9
Interpreter
Eet
(x)
Figure 1: (Left) General view FLINT. (Right) Instantiation of FLINT on a deep architecture.
concatenated vector of all intermediate outputs for an input sample x is denoted as fr(x) E R...
Given network to be interpreted and a positive integer J E No, an interpreter network computes the composition of a dictionary of attribute functions X + IR and an interpretable function h RR- + y
Vx E X, ((x) = ho (2),
(1)
In this work, we take: h(((x)) := softmax (WT ((x)) but other models like decision trees could be eligible. The attribute dictionary is composed of functions Φj : X R-+ .j 1.... ... whose non-negative images (x) can be interpreted as the activation of some high level attribute, i.e. "concept" over X. A key originality of the model lies the fact that the attribute functions oj (referee to as attribute for simplicity) leverage the outputs of hidden layers of specified by I:
Vj E {I....., (x) bhy fI())
(2)
where each v/, RDD - RR+ operates on the accessed hidden layers. Here, the of functions 1/j. ..... is defined to form shallow network V (around 3 layers) whose output is V ((z((z)) (x)) (example architecture in Fig. [T. Interestingly, Φ, are defined over X and as a consequence can be interpreted in the input space which is the most meaningful for the user (see Sec. 4). For sake of simplicity, we denote θ = (0v, On.) the specific parameters of this model, while the parameters devoted to the computation of fz(x)) are shared with f.
3.2 Interpretation in FLINT
The interpreter being defined, we need to specify its expected role and corresponding interpretability objective. In FLINT, interpretation seen as an additional task besides prediction. We are interested by two kinds of interpretation, one at the global level that helps to understand which attribute functions are useful to predict a class and the other at the local level, that indicates which attribute functions are involved in prediction of a specific sample. As preamble, note that, to interpret local prediction fz)), we require that the interpreter output g2x) matches f(z). When the two models disagree, we provide way to analyze the conflictual data and possibly raise an issue about the confidence on the prediction f(x) (see Supplementary Sec. S.2). To define local and global interpretation, we rely on the notion of relevance of an attribute.
Given an interpreter with parameter θ, (0v, 0%) and some input x, the relevance score of an attribute Φj is defined regarding the prediction g(x) = (x) ÿ. Denoting ý E y the index of the predicted class and Wj.s W the coefficient associated to this class, the contribution of attribute é, to unnormalized score of class ÿ is ajgg.æ = 6z (x)),,,, The relevance score is computed by f.. [....=]' An attribute Φú is considered as relevant for a local prediction if it is both activated and effectively used in the linear (logistic) model. The notion of relevance of an attribute for a sample is extended to its "overall" importance in the prediction of any class c. This can be done by simply averaging relevance scores from local interpretations over a random subset or whole of the training set S, where predicted class is c. Thus, we have: introduce the notions of local and global
normalizing contribution a as Tj,x =
Tj.e [8.T Exes, Tizz S. {x Sly c}. Now, interpretations that the interpreter will provide.
Definition (Global and Local Interpretation) For prediction network f. the global interpre- tation G(g, provided by an interpreter g, is the set of class-atribute puirs (c, Φ5) such that their globul relevance Tje is greater than some threshold 1 1/T,T >> 1. A local interpretation for sample x provided by an interpreter g of I denoted M(x, 9: is the set of attribute functions 63 with local relevance score z greater than some threshold 1/T,5 1.

---

It is important to note that these definitions do not prejudge the quality of local and global interpreta- tions, Next, we convert desirable properties of the interpreter into specific loss functions.
3.3 Learning by imposing interpretability properties
Although converting desirable interpretability properties into losses is shared by several by-design approaches [T] B8), there is consensus these properties. We propose below minimal set of penalties which are suitable for the proposed architecture and sufficient to provide relevant interpretations.
be imposed through cross-entropy loss: Fidelity to Output. The output of the interpreter g(z) should "close" to ((x) for any æ. This can
Lof(1g. -- Σ MV(T(()))""og((((
#ES
Conciseness and Diversity of Interpretations. For any given sample æ, we wish to get small number of attributes in its associated local interpretation. This property of conciseness should make the interpretation easier to understand due to fewer attributes to analyzed and promote the "high- level" character in the encoded concepts. However, to encourage better use of available attributes we also expect activation of multiple attributes across many randomly selected samples. We refer to this property diversity. This is also important avoid the case of attribute functions being learnt as class exclusive (for eg. reshuffled version class logits). To enforce these conditions we utilize notion of entropy defined for real vectors proposed by Jain al [23] solve problem of
efficient image search. For a real-valued vector v, the entropy is defined as {(v) = >; log(p:).
P: (xpp(.)) (>E; exp(v:)).
Conciseness is promoted by minimizing ((f((())) and diversity is promoted by maximizing entropy of average T (f)(()) over mini-batch. Note that this be seen encouraging the interpreter to find a sparse and diverse coding of fz(x) using the function W. Since satropy-based losses have inherent normalization, they do not constrain the magnitude of the attribute activation. This often leads to poor optimization. Thus, we also minimize the f norm ([(((( (with hyperparameter 11) to avoid it. Note that é é-rreuularization is common tool encourage sparsity and thus conciseness, however we show in the experiments that entropy provides more effective way.
Led(f.g S) -E Σ (()()())))
Σ (f2)))l #ES
with $
(fz(z))
S
#ES
RES
Fidelity to Input. To encourage encoding high-level patterns related to input in ((x),, we use decoder network d R + X' that takes as input the dictionary of attributes Vi (ff((z)) and reconstructs x. similar penalty has previously been applied by 4
Lisls.g,
Σ (d((((2(<)))
#ES
Note that one can modify Lis with other reconstruction losses as well (such as -frrconsttruction) Given the proposed loss terms, the loss for interpretability writes as follows:
Live (f, d, S) = =3C,
+ + g, d, S) 8Cealf,g,
where B,7; are non-negative hyperparameters. The total loss be minimized L Lpred + Lints where the prediction loss, Lpred, is the well-known cross-entropy loss.
Let us denote θ (0,, Os. 8v. 0%)) the parameters of these networks. Learning the models f. V, h and d boils down to learning θ. In practice, introducing all the losses at once often leads to very poor optimization. Thus, Lepred, Lif for the first two epochs and gain a reasonable level of accuracy. From the third epoch we introduce fof and from the fourth epoch introduce Led loss.
follow the procedure described in Alg. We train the networks with
4 Understanding encoded concepts in FLINT
Once the predictor and interpreter are jointly learnt, interpretation can be given the global and local levels as in Def. [ A key component to grasp the interpretations is to understand the concept
5

---

Algorithm Learning algorithm for FLINT
1: Input: S & parameters θ (0,, Oa, Ov, 0n) & hyperparameters: 3o, 70, do no & number of
batches B & number of training epochs Nopoch.
2: Random initialization of parameter O
3: 1 < Train (S, Og. 8 = 0, 7o. 0, B, 2) (% Trains epochs with Lpred: Lis
4: O₂ f Train (S, 1 3 So, 70. 8 0, 0, B, 1) 1% Trains epoch with Cpreds Lis, Los!
5: é +- Train -,,, , Bo. yo. dū, nu. B, Nepoch - 1% Trains with all losses)
6: Output: ô = (0z, dd. fv 8%)
encoded by each individual attribute function Φÿ. previously defined in Eq. 2 In this work, we focus on image classification and propose to represent an encoded concept set of visual patterns in the input space which highly activate Ø5. We present a pipeline to generate visualizations for global and local interpretation by adapting various previously proposed tools 4 39].
Algorithm 2 Visualization of global interpretation
I: Input: (clbss,atrributt)),,,,, 05)) & subset size:l & training set:S,, & AM+PI params: (Ns, dev Abo)
2: S. {2|(2, c) E 5
3: MAS(c, Øj. + arg maxmmaxmmaa M... EzEMM
4: FOR MAS(c, 11)
5: z &- AM+PI (x,, Ao, Ato Abo)
6: ENDFOR
7: Output: Jiih MAS(C, 1)
Visualization of global interpretation. Given any class-attribute pair (c, Φÿ) in the global interpre- tation Gigg f), we first select small subset of training samples from class e that maximally activate Φj. This set of samples is referred to as maximum activating samples and denoted MAS(c, oj. where I the size of the subset (chosen as 3 in the experiments). Although, MAS reveal some information about the encoded concept, it might not be apparent what aspect of these samples causes activation of dy. We thus propose further analyzing each element in MAS through tools that enhance the detected concept. This results in much better understanding. The primary tool we employ is modified version of activation maximization [39, which we refer to as activation maximization with partial initialization (AM+PI).
Given maximum activating sample x' E MAS(c, 1),4), the key idea behind AM+PI is synthesize appropriate input via optimization, that maximally activates Φj.. We thus optimize a common activa- tion maximization objective [9 arg max, λgo; (z) TV(x)- Bo(x)) where TV(.), Bo(.) are regularization terms. However, we initialize the procedure by low-intensity version of sample x². This makes the optimization easier with the detected concept weakly present in the input. This also allows the optimization to "fill" the input to enhance the encoded concept. As an output, we obtain map adapted to x'. that strongly activates dbj. Complete details of the AM+PI procedure are given in supplementary (Sec. S.2). Visualization of class-attribute pair is summarized in Alg. Alternative useful tools are discussed in the supplementary (Sec. S.2).
Local analysis. Given any test sample Io. one can determine its local interpretation L(zo: f.g). the of relevant attribute functions accordingly to Def. [ To visualize relevant attribute Φj E L(zo. f.g), we can repeat the AM+PI procedure with initialization using www-intensity version of Zo to enhance concept detected by Φ in Zg- Note that the understanding built about any attribute function é, via global analysis, although not essential, can still be helpful to understand the generated AM+PI maps during local analysis. as these maps are generally similar.
Numerical Experiments for FLINT
Datasets and Networks. We consider 4 datasets for experiments, MNIST 34
FashionMNIST [50]
CI (30), and subset of QuickDraw dataset (0). Additional results on CIFARTO0 (30 (large
number of classes) and Caltech-UCSSD Birds-200-2011
[9 (large-scale images and large number of

---

Accuracy (in %6
Fidelity (in %)
BASE f
SENN
PNNNotypeDNN
FLINT
FLINT-5
LIME
VIBI
FLINT-g
MNIST
98.9:10.11
98.41:0.1
98.9:00222
98.31:0.2
95.6:1:0.4
96.6:20.7
98.7:10..1
PashionMNIST
90.4+-011
84.2+00.3
90.0
90.5:200.2
86,8+00,4
67.3++13
88,4=0.3
91.5+-0.1
CIFARIO
84.7:10.3
77.8+00.7
84.5:10.2
84.0:20.4
31.5:0099
65.5=0.3
93.2:1:0.2
QuickDraw 85.3.10.2
85.51.0.4
85.7.10.3
85.410.11
76.3100.1
78.610..4
90.810.44
Table I: Results for accuracy (in %6 and fidelity to FLINT- ff on different datasets. BASE- is system trained with just accuracy loss, FLINT-f, FLINT-g denote the predictor and interpreter trained in our framework. Mean and standard deviation of 4 runs for each system are reported
classes) are covered in supplementary (Sec. S.2.2). Our experiments include kinds of architectures for predictor f: (i) LeNet-based [33] network for MNIST, FashionMNNIST, and (ii) ResNet1 8-based [21] network for QuickDraw, CIFAR. We select one intermediate layer for LeNet based network and two for ResNet based networks, from the last few convolutional layers as they are expected to capture higher-level features. We set the number of attributes J = 25 for MNIST, FashionMNIST, J 24 QuickDraw and 36 for CIFAR. Further details about the QuickDraw subset, precise architecture. ablation studies about choice of hyperparameters (hidden layers, size of attribute dictionary, loss scheduling) and optimization details are available in supplementary (Sec. S.2).
5.1 Quantitative evaluation of FLINT
We evaluate and compare our model with other state-of-the -hrrtrr systems regarding accuracy and inter- pretability. The evaluation metrics for interpretability (15) are defined to measure the effectiveness of the losses proposed in Sec. [.3 Our primary method for comparison, wherever applicable, is SENN, it is interpretable network by design with same units for interpretation FLINT. Other baselines include Prototype/DNN (6 for predictive performance, LIME [1 and VIBI g for fidelity of interpretations. Implementation of our method is available on Github Details for implementation of baselines are in supplementary (Sec. S.2).
Predictive performance of FLINT. There are two goals to validate related to predictor trained with FLINT (denoted FLINT- f), (i) Jointly training f with g and backpropagating loss term Cine does not negatively impact performance, and (ii) The achieved performance is comparable with other similar interpretable by-design models. For the former we compare the accuracy of FLINT- with same predictor architecture trained just with Lpred (denoted by BASE-/)- For the latter goal compare accuracy of FLINT-f with accuracy of SENN and another interpretable network by design input attribution for interpretations. Note that PNototypeDNNN requires non-trivial changes to the model for running on more complex datasets, CIFAR 10 and QuickDraw. To avoid any unfair comparison we skip these results. The accuracies are reported in Tab. They indicate that training f within FLINT does not result in any significant accuracy loss on any dataset. Also, FLINT is competitive with other interpretable by-design models.
PrototypeDNN (36) that does
Fidelity of Interpreter. The fraction of samples where prediction of a model and its interpreter agree, ie predict the same class, is referred to fidelity. It is commonly used metric to measure how well an interpreter approximates a model g 32). Note that, typically, for interpretable by design models, fidelity cannot be measured as they only consider single model. However, validate that the interpreter trained with FLINT' (denoted as FLINT-9)) achieves a reasonable level of agreement with FLINT-f. we benchmark its fidelity against a state-of-the-art black-box explainer VIBI (8) and a traditional method LIME [41] The results for this are provided in Tab. (last three columns).
FLINT-g consistently achieves higher fidelity. Even though it is not a fair comparison as other systems are black-box explainers and FLINT-g accesses intermediate layers, they clearly show that FLINT-9 demonstrates high fidelity to FLINT-f.
Conciseness of interpretations. We evaluate conciseness by measuring the average number of important attributes in generated interpretations. For given sample it can be computed numher of attributes 6j with Fr, greater than a threshold 1/T, 7 > 1, i.e. CNS,,- = Ki |ry.al > 1/T)-- For different thresholds 1 //r, we compute the mean of CNSy.a over test data to estimate conciseness of CNS,- Lower conciseness indicates need to analyze a lower number of attributes on an
7

---

25
31
(a)
(b)
(c)
Figure 2: (a) Conciseness comparison of FLINT and SENN. (b) Effect of entropy losses on con-
ciseness of ResNet for QuickDraw for various & é-regularization levels. (c) Global class- relevances rj. for Draw (Left) and CIFARIO (Right). 24 class-attribute pairs for Quick and 32 pairs for CIFARIE 10 have relevance Tj.c 0.2.
MAS1
AlNeI MAS
One -91s
Sandal de
MNIST
FashionMNIST
One -du
Putlover- -upht
Apple-gg -d
Airplane
CIFAR10
de
QuickDraw
Truck
Lion-fo
Figure 3: Example class- dats-attribbtt pair analysis on all datasets, with global relevance Tie > 0.2. Each row contains MAS with corresponding AM+PI outputs
IRPUT
AM--I OUTTUTS
FULLOVER
COAT
DONT
SWERT
SHRT
AMH
CIFA100
TRUCK
CAR
CAR
TRUCH
sample
42
Figure 4: (Left) Local interpretations for test samples. Top 3 attributes with corresponding AM+PI output are shown. True labels for inputs are: Pullover, Airplane, Apple, Dog. (Right) Examples of attribute functions detecting same part across various test samples. For each sample, their relevance greater than 0.8. True labels of samples indicated above them.
average. SENN is the only other system for which this curve be computed. We thus compare the conciseness SENN with FLINT on all four datasets. Fig. 2a depicts the same. It can be easily observed that FLINT produces lot more concise interpretations compared SENN. Moreover, SENN even ends with majority of concepts being considered relevant for lower thresholds (higher r).
Entropy vs l regularization. We validate the effectiveness of entropy losses by computing concise- ness curve at various levels & regularization strength, with and without entropy, for ResNet with QuickDraw. This is reported in Fig, 2b The figure confirms that using the entropy based loss is more effective inducing conciseness of explanations compared to using just E (-rrgullarization, with the difference being close use of attribute less when entropy losses are employed.
Importance of attributes. Additional experiments evaluating meaningfulness attributes by shuf- fling them and observing the effect (for FLINT and SENN) are given in supplementary (Sec. S.2).

---

5.2 Qualitative analysis
Global interpretation. Fig. 2c depicts the generated global relevances Tie for all class-attribute pairs on QuickDraw and CIFAR. Each class-atrribute pair with 'high" relevance needs to be analyzed as part of global analysis. Some example class-attribute pairs, with high relevance, are visualized in Fig. 3 For each pair we select MAS of size 3 and also show their AM+PI outputs. As mentioned before, simply analyzing MAS reveals useful information about the encoded concept. For instance, based on MAS, Φ15. 19 on MNIST, relevant for class 'One', clearly seem to activate for vertical and diagonal strokes respectively. However, AM+PI outputs give deeper insights about the concept by revealing more clearly what parts of input activate attribute function. For eg.. while MAS indicate that Φ on FashionMNIST activates for heels (one type of 'Sandal'), ) 22 on CIFARIO 10 activates for white dogs, it is not clear what part the attribute focuses AM+PI outputs indicate that Φ2 focuses on the area around eyes and nose (the most enhanced regions), Os primarily detects a thin diagonal stroke of the heel surrounded by empty space. AM+PI outputs generally become for attributes relevant for multiple classes. One such example is the function Φs on QuickDraw, relevant for both 'Dog' and 'Lion'. It activates for very similar set of strokes for all samples, as indicated by AM+PI maps. For 'Dog' this corresponds to ears and mouth and for 'Lion' it corresponds to the mane. Other such attribute functions in the figure include 224 on MNIST, relevant for 'Pullover', "Coat' and 'Shirt' which detects long sleeves and Φ2 on CIFAR 10, relevant for 'Trucks', 'Cars' and primarily detects wheels and parts of upper body. Further visualizations including those of other relevant classes for Φ24. 02 and global relevances are available in supplementary (Sec. S.2).
more important
Local interpretation. Fig. 4 (left) displays the local interpretation visualizations for test samples. f and both predict the true class in all the cases. We show the top 3 relevant attributes to the prediction with their relevances and their corresponding AM+PI outputs. Based on the AM+PI outputs it can be observed that the attribute functions generally activate for patterns corresponding to the same concept as inferred during global analysis. This can be easily seen for attribute functions present in both Fig. 3 4 (left). This is further illustrated by Fig. 4 (right) where we illustrate AM+PI outputs for two attributes from Fig. 3 These functions relevant for more than one class and detect the same concept across various test samples, namely long sleeves for Φ2 and primarily wheels for $29-
5.3 Subjective evaluation
We conducted a survey based subjective evaluation with QuickDraw dataset for FLINT with 20 respondents. We selected 10 attributes, covering 17 class-attribute pairs from the QuickDraw dataset. For each attribute we present the respondent with our visualizations (3 MAS and AM+PI outputs) for each of its relevant classes along with a textual description. We ask them if the description meaningfully associates to patterns in the AM+PI outputs. They indicate level of agreement with choices: Strongly Agree (SA), Agree (A), Disagree (D), Strongly Disagree (SD), Don't 't Know (DK). Descriptions were manually generated by our understanding of encoded concept for each attribute. 40% incorrect descriptions were carefully included to ensure informed responses. These were forcefully related to the classes shown to make them harder identify. Results for correct descriptions: 77.5 7.5% SA/A, 10.0% DK, 12.5% D/SD. For incorrect descriptions: 83.7% - DISD, 7.5% - DK, 8.8% SA/A. These results clearly indicate that concepts encoded in FLINT's learnt attributes are understandable to humans. Survey details are given in supplementary (Sec. S.2).
6 Specialization of FLINT to post-hoc interpretability
While interpretability by design is the primary goal of FLINT, it can be specialized to provide a post- hoc interpretation when a classifier f is already available. The Post-hoc interpretation learning (see for instance (41)) comes as a special case of SLI and is defined as follows. Given a classifier F and a training set S, the goal to build an interpreter of ff by solving:
Problem 2: arg min Line (f. S)) 9E9,
With FLINT, we have g(x) = h (xx) and ((x)= V (z) for a given set of accessible hidden layers I and a attribute dictionary size J. Learning can be performed by specializing Alg, with slight modification of replacing θ as θ = (0v, On, 0z) while 02 is fixed and eliminating Lpred from training loss L.

---

Experimental results for post-hoe FLINT:
We validate this ability of our framework by interpreting fixed models trained only for accuracy, i.e, BASE-f models from vection[5.1] Even after not tuning the internal layers of f. the system is still able to generate high-fidelity and meaningful interpretations. Fidelity comparisons against VIBI, class-attribute pair visualizations and experimental details are available in supplementary (Sec. S.3).
7 Discussion and Perspectives
FLINT is novel framework for learning predictor network and its interpreter network with dedicated losses. It provides local and global interpretations in terms of high-level learnt at- unbutes/concepts by relying on (some) hidden layers of the prediction network. This however leaves some under-explored questions about faithfulness of interpretations to the predictor. Defining faithfulness of an interpretation regarding a decision process 4 has not yet reached consensus particularly in the case of post-hoc interpretability or when the two models, predictor and interpreter, differ [52]. Even though generating interpretations based on hidden layers of predictor ensures high level of faithfulness of the interpreter to the predictor, a complete faithfulness cannot be guaranteed since predictor and interpreter differ in their last part. However ensuring faithfulness by design regarded as the primary objective, nothing stops the use of interpreter FLINTT- as the final decision- making network. In this case, there is only one network and the so-called prediction network has only played the useful role of providing relevant hidden layers.
Retaining only the interpreter model additionally provides a compression of the predictor and can be relevant when frugality is at stake. Further works will investigate this direction and the enforcement of additional constraints on attribute functions to encourage invariance under various transformations. Eventually FLINT can be extended to other tasks or modalities other than images in particular by adapting the design of attributes and the pipeline to understand them.
Acknowledgments and Disclosure of Funding
This work has been funded by the research chair Data Science & Artificial Intelligence for Digitalized Industry and Services of Télécom Paris. This work also benefited from the support of French National Research Agency (ANR) under reference ANR-20-CE23-0028 (LIMPID project). The authors would like thank Sanjeel Parekh for fruitful discussions and anonymous reviewers for their valuable comments and suggestions.
References
[1] Tameem Adel, Zoubin Ghahramani, and Adrian Weller. Discovering interpretable represen- tations for both deep generative and discriminative models. In International Conference on Machine Learning, pages 50-59. PMLR, 2018
[2] Rishabh Agarwal, Nicholas Frosst, Xuezhou Zhang, Rich Caruana, and Geoffrey Hinton. Neural additive models: Interpretable machine learning with neural nets. arXiv preprint arXiv:2004.1 13912, 2020.
[3] Maruan Al-Shedivat, Avinava Dubey, and Eric Xing. Contextual explanation networks. arXiv preprint arXiv: 1705. 10301, 2017.
[4] David Alvarez-Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 7775-7784, 2018.
[5] David Alvarez-Mehh and Tommi S Jaakkola. On the robustness of interpretability methods. arXiv preprint arXiv: 08049, 2018.
[6] Plamen Angelov and Eduardo Soares. Towards explainable deep neural networks (xdnn). Neural Networks, 130: 130.185-194, 2020.
[7] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Robert Müller. How to explain individual classification decisions. Mach. Learn. Res., 11:1803-1831, August 2010. ISSN 1532-4435.
10

---

[8] Seojin Bang, Pengtao Xie, Wei Wu, and Eric Xing. Explaining black- box using deep variational information bottleneck approach. arXiv preprint 4TXv:1902.06918. 2019.
[9] Alejandro Barredo Arrieta, Natalia Díaz- Rodríguz,, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez. Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable artificial intelligence (xai): Concepts, tax- onomies, opportunities and challenges toward responsible ai. Information Fusion, 58:82 115, 2020.
[10] Valérie Beaudouin, Isabelle Bloch, David Bounie, Stéphan Clémençon, Florence d' Alché- Buc, James Eagan, Winston Maxwell, Pavlo Mozharovskyi, and Jayneel Parekh. Flexible and context-specific Al explainability: A multidisciplinary approach. CoRR, abs/2003.07703, 2020.
[11] Diogo V. Carvalho, Eduardo M. Pereira, and Jaime S. Cardoso. Machine learning interpretability: survey on methods and metrics. Electronics, 8(8), 2019.
[12] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. In Advances in Neural Information Processing Systems (NeurIPS), pages 8928-8939, 2019.
[13] Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782, 2020.
[14] Jonathan Crabbe, Yao Zhang. William Zame, and Mihacla van der Schaar. Learning outside the black-box: The pursuit of interpretable models. In Advances in Neural Information Processing Systems, pages 17838-17449, 2020.
[15] Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprini 1wXXiv:/02..000
2017.
[16] Finale Doshi- Velez and Been Kim. Interpretable machine learning. In Proceedings of the Fourty-fourth International Conference on Machine Learning, 2017.
[17] D. Dubois and H. Prade. Possibilistic logic-an overview. Computational logic, 9, 2014.
[18] Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept- based explanations. In Advances in Neural Information Processing Systems (NeurIPS), pages 9277-9286 2019.
[19] Riccardo Guidotti, Anna Monreale, Fosca Giannotti, Dino Pedreschi, Salvatore Ruggieri, and Franco Turini. Factual and counterfactual explanations for black box decision making. IEEE Intelligent Systems, 2019.
[20] David Ha and Douglas Eck. A neural representation of sketch drawings. In International Conference on Learning Representations, 2018.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. European conference on computer vision, pages
Springer, 2016.
[22] Lisa Anne Hendricks, Zeynep Akata, Mareus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. Generating visual explanations. In European Conference on Computer Vision, pages 3-19. Springer, 2016.
[23] Himalaya Jain, Joaquin Zepeda, Patrick Pérez, and Rémi Gribonval. Subie: A supervised, structured binary code for image search. In Proceedings of the IEEE International Conference on Computer Vision, pages 833-842, 2017.
[24] Sarthak Jain, Sarah Wiegreffe. Yuval Pinter, and Byron C Wallace. Learning to faithfully rationalize by construction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4459-4473, 2020.
[25] Berk Ustun Jyaming Zeng and Cynthia Rudin. Interpretable classification models for recidivism prediction, Journal of the Royal Statistical Society, Series A, Statistics in Society, 180, part 689-722, 2017.

---

[26] Dmitry Kazhdan, Botty Dimanov, Mateja Jamnik, Pictro Lio, and Adrian Weller. Now you see me (cme): Concept-based model extraction. arXiv preprint arXiv:2010.13233, 2020.
[27] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fermanda Viegas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). arXiv preprint arXiv:/711.11299 2017.
[28] Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, pages 267-280. Springer, 2019.
[29] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pages 3338-5348. PMLR, 2020.
[30] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[31] Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. Faithful and customizable explanations of black box models. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 131-138, 2019.
[32] Himabindu Lakkaraju, Nino Arsov, and Osbert Bastani. Robust and stable black box explana- tions. In International Conference on Machine Learning, pages 5628-5638. PMLR, 2020.
[33] Yann LeCun. Lenet-5, convolutional neural networks. URL.: http://yann. lecun. confexdb//enet, 2015.
[34] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 36(11))2278-233 1998.
[35] Guang-He Lee, Wengong Jin, David Alvarez-Melis, and Tommi S Jaakkola. Functional transparency for structured data: game-theoretic approach. arXiv preprint 102.07790..09737, 2019.
[36] Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[37] Zachary C. Lipton. The mythos of model interpretabilityy Commun. ACM, 61(10):36 2018.
[38] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pages 4765-4774, 2017.
[39] Aravindh Mahendran and Andrea Vedaldi. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision, 120(3):233 255, 2016.
[40] Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Müller. Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 73:1-15, 2018.
[41] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144. ACM, 2016.
[42] Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. Interpretable machine learning: Fundamental principles and 10 grand challenges. CORR, abs/2103. 12251, 2021.
[43] Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robers Müller, editors. Explainable Al: Interpreting, Explaining and Visualizing Deep Learning. volume 11700 of Lecture Notes in Computer Science. Springer, 2019.
[44] Sascha Saralajew, Lars Holdijk, Maike Rees, Ebubekir Asan, and Thomas Villmann. Probabilistic modeling of reasoning over a set of components. In Advances in Neural Information Processing Systems (NeurIPS), pages 2788-2799 2019.
12

---

[45] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, pages 618-628 626, 2017.
[46] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.600 2013.
[47] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smooth- grad: removing noise by adding noise. arXiv preprint arXiv:1706.0382. 2017.
[48] Gregor Stiglic, Primoz Kocbek, Nino Fijacko, Marinka Zitnik, Katrien Verbert, and Leona Cilar. Interpretabilityy of machine learning- based prediction models in healthcare, Wiley Interdiscip. Rev. Data Min. Knowl. Discov., 10(5), 2020.
[49] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
[50] Han Xiao. Kashif Rasul, and Roland Vollgraf. Fashion-mnis novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 2017.
[51] Chih-Kuan Yeh, Been Kim, Sercan O Arik, Chun-Liang Li, Pradeep Ravikumar, and Tomas Pfis- ter. On concept-based explanations in deep neural networks. arXiv preprint arXiv: 1910.07969, 2019.
[52] Fan Yin, Zhouxing Shi, Cho-Jui Hsieh, and Kai- Lei Chang. On the faithfulness measurements for model interpretations. arXiv preprint
08782, 2021.
[53] Jinsung Yoon, James Jordon, and Mihaela van der Schaar. Invase: Instance-wise variable selection using neural networks. In International Conference Learning Representations, 2018.
[54] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arXiv preprint 15X1v:/500.06677,, 2015.
[55] Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. Interpretable convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8827-8836, 2018.
13


Reviews:
[{"review_id": "v6sLd60vO70", "reviewer": "Reviewer_gNz2", "summary": "The paper proposes FLINT a Framework to Learn with Interpretation to solve Supervised Learning with Interpretation task.\nFLINT involves training 3 models on a preditor model an interpreter and a decoder. This framework provides both global and local interpretations.\n\nThe interpreter network takes as an input the output of different hidden layers from the predictor network then computes an attribute dictionary of functions phi g(x)= softmax(Wphi(x)). Different phi corresponds to different concepts  phi_j(x)= Psi_j (f_I(x)) where Psi_j is shallow network and f_I(x) is the output of hidden state I.  So The model parameters of g depend on the parameters of Psi and Weights. Given interpreter with parameter and input x the contribution of phi_j = phi_j(x)*w_j,y_hat/max|ph_i*w_i_yhat| where y_hat the class output.\n\nGlobal Interpretation is a set of class-attribute pairs (c, ph_j) such that their global relevance is greater than some threshold.\nLocal interpretation for a sample x provided is the set of attribute functions ph_j  with a local relevance score greater than some threshold.\n\nFLINT has multiple loss functions, first to ensure that the interpreter output is similar to that of the model output cross-entropy is used. An additional loss is used to ensure diversity of ph_j and ensure input fidelity the input is reconstructed from phi using a decoder network which gives us the final loss.\n\nFlINT was evaluated on 4 different image datasets, it was compared with other self explain networks SENN and PrototypeDNN to terms of accuracy and saliency methods LIME and VIBI in terms of fidelity.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n- Although self-explaining networks have been previously proposed FLINT framework itself is original and novel.\n- The paper is technically sound and the claims are well supported.\n- The paper organized and is well written.\n- The relation to previous work is clearly outlined.\n- The empirical results are quite impressive epically that FLINT_g  produces accuracy similar to that of FLINT_f.\n- The paper included a subject evaluation showing that the concepts learned by FLINT are understandable by humans.\n- The proposed method is significant, the constraints used to impose Fidelity to Output, Conciseness and Diversity of Interpretations, and Fidelity to Input can be adapted to many methods.\n\n\n\nWeakness:\n\n- It is unclear how one can choose the hidden layer to give input to the interpreter.\n- Comparison was done on very simple datasets, it is unclear if this will generalize to larger more complicated datasets.\n- Baseline models used have very low accuracy compared to state-of-the-art accuracy.\n- Empirical evaluation is somewhat limited.\n\nQuestions and Comments:\n\n- Accuracy reported on CIFAR10 as an example has baseline accuracy 84.7 and FLINT was able to maintain this however state of the art accuracy is around 98% if that model was used as a baseline or other high performing network will FLINT be able to maintain such accuracy?\n- The paper only used very simple datasets which are generally easy to train, how will this approach work on larger datasets like image net will the proposed optimization still work.\n- It would be interesting to see how  FLINT compares to other self-explaining networks such as INVASE [1] or FRESH[2].\n\n\n\n[1] Yoon, Jinsung, James Jordon, and Mihaela van der Schaar. \"INVASE: Instance-wise variable selection using neural networks.\" International Conference on Learning Representations. 2018.\n[2] Jain, Sarthak, et al. \"Learning to faithfully rationalize by construction.\" arXiv preprint arXiv:2005.00115 (2020).\n\n\nOverall, I believe the proposed method is great and I really enjoyed reviewing the paper. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, in the supplementary.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5.5"}, "extra_text": {}}, {"review_id": "g0-5xTkc80U", "reviewer": "Reviewer_WJT2", "summary": "This paper proposes an architecture which jointly learns to classify as well as provide an explanation behind the classification. The explanation is generated from a separate part of the model which doesn’t affect the model performance much as the classification performance is achieved by a backbone network. The explanation generation part takes inputs from the intermediate layers of the backbone network which helps to produce better and more meaningful explanations. The results are promising on experiments shown on MNIST, FMNIST, CIFAR10 and QuickDraw datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives:\n\n+ The overall idea of integrating explanations while learning to classify is important and timely, especially considering the spate of post-hoc explanation methods available today.\n+ The use of an exclusive backbone network for classification helps maintain good classification performance.\n+ Post-hoc explanation generation techniques can be applied in the framework.\n+ The results show that the method performs better in terms of classification performance and explanation generation as well compared to other baseline methods.\n\nConcerns:\n\n- One of the primary concerns is whether the method is applicable to complex large-scale datasets, on which post-hoc attribution methods show their results (e.g. ImageNet). The paper does not show results on large-scale datasets, limiting the understanding of its applicability. As much as the proposed idea is interesting and new, to a certain extent, it attempts to improve over a couple of earlier methods in this direction such as SENN (Ref [4] in the paper). Showing it works on large-scale datasets would be one important improvement to show, especially considering the pieces used in the method are known before.\n\n- The contribution of every attribute is given by the coefficients w which are given by the matrix W (considering all the classes) and trained as a simple fully connected layer. Once the model is trained, these contributions are fixed and are used as it is during inference. This is different for SENN, as the contributions are given by a relevance function there (realized by a neural network) and hence produces different contributions for different inputs. How do you view this difference? Do you think the same contributions for every input across all data points is justified?\n\n- The interpreter selects outputs from one or more intermediate layers from the backbone network specified by the function f. The layers chosen for different architectures are shown in Fig.1 and Fig.2 of supplementary section. Are these choices arbitrary? If not, how were the layers chosen? I’m curious to know if this choice affects the quality of the interpretation. Also, if this is non-trivial and empirical, this makes the framework difficult to scale to newer datasets/models.\n\n- In L56, it is said that ‘FLINT can be specialized to post-hoc interpretability if a pre-trained deep neural network is available.’ As far as I understood, the post-hoc interpretability is generated from the backbone network given by f, which is part of the whole network. Then, what is meant by availability of a pre-trained deep neural network here, and how is this achieved?\n\n- The final loss function is an aggregation of many different loss functions which makes this network highly dynamical in nature. One needs to take proper care while selecting the coefficients and the number of epochs where every loss function was introduced. This was pointed out in Algorithm 1, but how were these epoch numbers arrived at? How does one choose these on a dataset?\n\n- Table 1 compares the accuracy numbers with SENN and the proposed method uses a similar architecture as SENN for MNIST. However, for CIFAR10, SENN used smaller architectures for their results, which were not considered why. Was there any reason for this? Would this be fair comparison? I’m convinced with the explanations provided for the training time differences with SENN though.\n\n- In L106-108, the paper talks about other methods which use a known dictionary of concepts (such as concept bottleneck models). Was there a particular reason not to compare with such methods? While it may be understandable, it may be nice for the paper to explicitly state why the baselines used in the experiments were chosen (and other such models were not).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not in the main paper, as suggested in paper submission guidelines (provided in the supplementary)", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "LsnnA85KfFp", "reviewer": "Reviewer_eCgH", "summary": "The paper introduces FLINT, a framework that jointly trains a classifier with an “interpreter”, an interpretable version of the model. The interpreter is a linear classifier over a set of attributes (each associated with a “concept”), which are derived from hidden states of the neural network. The interpreter is trained with three objectives: (1) matching the output of the original classifier, (2) a sparsity regularizer, and (3) an anto-encoding objective to reconstruct the input. Each attribute in the interpreter can be visualized by finding input that maximizes the attribute (with some regularization), which allows the interpreter to produce global interpretation for a class and local interpretation for an example. The method is tested on four image classification tasks. It has higher accuracy than two previous interpretable models (SENN and PrototypeDNN) and higher fidelity than two post hoc interpretation methods (LIME and VIBI). According to a human survey, the visualizations are human interpretable in most cases.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThe framework is novel, intuitive, and flexible. It allows us to provide an interpretable version of any neural classifier. The experiments are extensive, and the results seem good. The model can retain high accuracy while producing high-fidelity interpretation. \n\nWeakness:\nMy biggest concern is that the framework introduces extra complexity to model training. For example, the paper acknowledges that optimizing the full objective is hard, so they need a schedule to gradually increase the complexity of the objective (Section 3.1). This schedule may need to be changed for a new model. The framework also introduces several hyperparameters: the weight of different objectives and the number of attributes. These hyperparameters need to be tuned separately for different tasks/datasets. \n\nIn summary, I am leaning towards accepting the paper. However, I am not very familiar with the latest interpretation methods and how to evaluate them, so I might miss something.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not have any suggestions for this part.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "IAt4g_3qCv", "reviewer": "Reviewer_ZpKd", "summary": "The paper proposes FLINT, a new learning schema in which a neural net and a gray-box model are learned jointly by minimizing a composite loss: a loss on the labels + a set of losses controlling the concepts acquired by the gray-box models +\na distillation loss that encourages the gray-box models to behave similarly to the neural net.  The concepts in the gray-box model are built from intermediate neurons of the neural net to encourage faithfulness.  The authors also describe a visualization pipeline and a post-hoc explanation protocol.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is generally well written;  all concepts are explained clearly.  The text is structured just fine.  I do not have any major gripes with the presentation itself.\n\nOne issue is that the message is unfocused: it is not exactly clear what the authors want to say with this paper.  Apparently it is only about \"going beyond SENNs\", but I think that there are some interesting and deeper issues implicit in this work that the authors do not address:\n\n    1.  There is an unstated benefit of gray-box models like SENNs that the authors overlook:  SENNs are not only interpretable by design but also *faithful* by design.  This is no longer the case in FLINT: there is no guarantee that the g network is 100% faithful to the f network.  This also explains why SENNs underperform in Table 1: they trade off performance for interpretability AND faithfulness.  This actually highlights a limitation of FLINT and that this should be explicitly discussed in the paper.\n\n    2.  This also means that, implicitly at lesat, the authors suggest that it is fine to give up on faithfulness -- that faithfulenss is not always fundamental.  I don't think that it is reasonable to do so in high-stakes applications.  In order to defend this point, the authors neglect to clarify which applications their method is best suited for.\n\nThe core idea -- namely that of training two twin networks based on (roughly speaking) the same base intermediate layers but with different aims -- is sensible.  Re-using the intermediate layer to tie the two models is an interesting choice and is reminiscent of multi-task learning solutions.  Again, this introduces a performance-faithfulness trade-off that should be addressed.  Some modeling choices (like combining the L1 norm with entropy) are quite empirical, but on the bright side they seem to perform well in practice.\n\nThe experimental validation is a bit limited, but the choice of competitors and research questions is reasonable.  The results seem to favor the proposed method.\n\n\nAs for the score, I am a bit on the fence.  All in all, the contribution is sensible and the experiments are reasonable, but the paper is somewhat lacking in terms of focus, significance and depth.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are barely discussed in the introduction; there is a section on limitations and negative impact in the supplementary.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers agree that the paper has some nice and interesting ideas and it is indeed quite promising that the proposed method gives minimal accuracy loss while providing interpretability.

Having said that, a majority of them felt that a more thorough analysis of the method (e.g., ablation studies) and evaluation on at least one dataset on the scale of ImageNet is necessary to properly evaluate the utility of the proposed method. This is particularly so because of the large amount of existing work in this area and the limited novelty of the proposed method. I am sharing some of the key concerns and points that arose during the discussion period.

- As a problem on which there is earlier work (SENN, PrototypeDNN), restricting to small-scale datasets seemed limiting. The new results on CIFAR100 were nice to see, but results on one larger complex dataset (TinyImagenet, CUB, ImageNet) would have made it better to see the usefulness of the work (considering the method by itself has no new components, it's an aggregation of existing pieces).

- The paper seems to be missing analysis (ablation studies in the case of this work) where the results are shown with choice of different layers, or choice of different epochs where loss functions are introduced. These are important decisions in the framework, and how they impact the final result would have been nice to see (even if some of the choices did not yield strong results, and the method worked only for certain choices).

[Update] The authors' response dated 2nd September indeed satisfactorily addresses the two main issues of (i) evaluation on a larger dataset and (ii) analysis of the effect of different choices in the proposed framework. Since the reviewer/AC discussion had concluded by 30th August, this last authors' response was not noticed at the time of the original metareview. Given that all the reviewers agreed that the paper has interesting results, and that the two main criticisms have been satisfactorily addressed, I recommend acceptance of the paper.
Decision: Accept (Poster)