Explaining A Black-box By Using A Deep Variational Information Bottleneck Approach
Authors: Seojin Bang
Pengtao Xie
Heewook Lee
Wei Wu
Eric Xing
Venue:  (2020)
Keywords: interpretable machine learning
information bottleneck principle
black-box

Abstract: Interpretable machine learning has gained much attention recently. Briefness and comprehensiveness are necessary in order to provide a large amount of information concisely  when explaining a black-box decision system. However, existing interpretable machine learning methods fail to consider briefness and comprehensiveness simultaneously, leading to redundant explanations. We propose the variational information bottleneck for interpretation, VIBI, a system-agnostic interpretable method that provides a brief but comprehensive explanation. VIBI adopts an information theoretic principle, information bottleneck principle, as a criterion for finding such explanations. For each instance, VIBI selects key features that are maximally compressed about an input (briefness), and informative about a decision made by a black-box system on that input (comprehensive). We evaluate VIBI on three datasets and compare with state-of-the-art interpretable machine learning methods in terms of both interpretability and fidelity evaluated by human and quantitative metrics.

Under review as a conference paper at ICLR 2020
EXPLAINING A BLACK-BOX BY USING A DEEP VARI- ATIONAL INFORMATION BOTTLENECK APPROACH
Anonymous authors Paper under double-hlind review
ABSTRACT
Interpretable machine learning has gained much attention recently. Briefness and comprehensiveness are necessary in order to provide large amount of informa- tion concisely when explaining black-box decision system. However, existing interpretable machine learning methods fail to consider briefness and compre- hensiveness simultaneouslyy, leading to redundant explanations. We propose the variational information bottleneck for interpretation, VIBI, a system-agnostic inter- pretable method that provides brief but comprehensive explanation. VIBI adopts an information thcoretic principle, information bottleneck principle, as a criterion for finding such explanations. For each instance, VIBI selects key features that are maximally compressed about an input (briefness), and informative about a decision made by a black-box system that input (comprehensive). We evaluate VIBI on three datasets and compare with state-of-the-art interpretable machine learning methods in terms of both interpretability and fidelity evaluated by human and quantitative metrics.
INTRODUCTION
Interpretability is crucial in building and deploying black-box decision systems such as deep learning models. Interpretation of a black-box system helps decide whether or not to follow its decisions, or understand the logic behind the system. In recent years, the extensive use of deep learning black-box systems has given rise to interpretable machine learning approaches (Lipton) 2016 Doshi-Vos Velez & Kim 2017). which aim to explain how black-box systems work or why they reach certain decisions. In order to provide sufficient information while avoiding redundancy when explaining a black-box decision, we need to consider both briefness and comprehensiveness. However, existing approaches lack in-depth consideration for and fail to find both brief but comprehensive explanation.
In order to obtain brief but comprehensive explanation, we adopt the information bottleneck princi- ple (Tishby et al. 2000). This principle provides appealing information theoretic perspective for learning supervised models by defining what we mean by a 'good' representation. The principle says that the optimal model transmits much information as possible from its input to its output through a compressed representation called the information bottleneck. Then, the information bottleneck will maximally compress the mutual information (MI) with an input while preserving as much as possible MI with the output. Recently, it has been shown that the principle also applies to deep neural networks and each layer of deep neural network can work an information bottleneck (Tishby & Zaslavsky 2015, Shwartz- Ziv & Tishby 2017). Using this idea of information bottleneck principle, we define brief but comprehensive explanation as maximally informative about the black-box decision while compressive about a given input.
In this paper, we introduce the variational information bottleneck for interpretation (VIBI), system- agnostic information bottleneck model that provides brief but comprehensive explanation for every single decision made by a black-box model. VIBI composed of two parts: explainer and approximator, each of which is modeled by deep neural network. The explainer returns probability whether a chunk of features such as a word, phrase, sentence or a group of pixels will be selected as an explanation or not for each instance, and an approximator mimics behaviour of black-box model. Using the information bottleneck principle, we learn an explainer that favors brief explanations while enforcing that the explanations alone suffice for accurate approximations to a black-box model.

---

Under review as a conference paper at ICLR 2020
1.1 CONTRIBUTION
Our main contribution is to provide a new framework that systematically defines and generates a 'good' (i.e. brief but comprehensive) explanation using the information bottleneck principle. Based on this principle, we develop VIBI that favors a brief but comprehensive explanation. In order to make the objective function of VIBI tractable, we derive a variational approximation to the objective.
The benefits of our method are as follows. 1) System-agnostic VIBI can be applied to explain any black-box system. 2) Post-hoe learning: VIBI is learned in post-hoc manner, hence there is no trade- off between task accuracy of a black-box system and interpretability of an explainer. 3) Cognitive chunk: Cognitive chunk defined as group of raw features whose identity is understandable to human. VIBI groups non-cognitive raw features such pixel and letter into a cognitive chunk (e.g. group of pixels, a word, a phrase, a sentence) and selects each unit as an explanation. 4) Separate explainer and approximator: The explainer and approximator are designed for separated tasks so that we do not need to limit the approximator to have a simple structure, which may reduce the fidelity (the ability to imitate the behaviour of black-box) of approximator.
2 RELATED WORK
Most prior interpretable machine learning methods have been focusing on local interpretation, which implies knowing the reasons why a black-box system makes certain decision at a very local point of interest, and our work is also situated in this line. Existing methods can be categorized into system-specite and gystem-agnoste black-box decision systems (e.g. using backpropagation algorithm, or having CNN structure), while system-agnostic methods explain any black-box decision systems.
method. System-specitn methods only explain certain
System-specific methods. To measure change of output with respect to changes of input is an intuitive way of obtaining feature attribution for the output. Using this idea, Zeiler & Fergus (2014), and Zintgraf et al. (2017) observe how outputs change when they make perturbations to each instance. Baehrens et al. (2010); Simonyan et al. (0013), and Smilkov et al. (2017) use computationally more efficient approaches; they measure change of output by propagating contributions through layers of a deep neural network towards an input than the perturbation. However, these approaches fail to detect the changes of output when the prediction function is flattened at the instance (Shrikumar et al.) 2017). which leads to interpretations focusing on irrelevant features. In order to solve this problem, the layer-wise relevance propagation (Bach et al. 2015 Binder et al. 2016), DeepLIFT (Shrikumar iet al. 2017)) and Integrated Gradients (Sundararajan et al. 2017) compare the changes of output to its reference output. Ross et al. (2017) learn a more generalizable model as well as desirable explanations by constraining its explanations (i.e. input gradient) to match domain knowledge.
System-agnostic methods. The great advantage of system-agnosee interpretable machine learning methods over system-specific methods is that their usage is not restricted to specific black-box system. One of the most well-known system-agnostie methods LIME (Ribeiro et al.) 2016).
explains the decision of an instance by locally approximating the black-box decision boundary around the instance with an inherently interpretable model such as sparse lincar or decision trees. The approximator is learned by samples generated by perturbing given instance. Lundberg & Lee (2017) proposed SHAP, a unified measure defined over the additive feature attribution scores in order to achieve local accuracy, missingness, and consistency. L2X (Chen et al.) 2018 learns stochastic map that selects instance-wise features that are most informative for black-box decisions. Unlike LIME and SHAP, which approximate local behaviors of black-box system with a simple (linear) model, L.2X does not put a limit on the structure of the approximator: hence it avoids losing fidelity of the approximator. As SHAP does, Dabkowski & Gal (2017): FFng & Vedaldi (2017). and Petsiuk et al. (2018) use sample perturbation but they rather learn or estimate desired perturbation masks than using perturbed samples to learn an approximator, DDbkowski Gal (2017), and Fong & Vedaldi[(201) learn smallest perturbation mask that alters black-box outputs as much as possible. Petsiuk et al. (2018) empirically estimate feature attribution as a sum of random masks weighted by class scores corresponding to masked inputs. Our method VIBI is similar with L2X in that both learn a stochastic explainer that returns a distribution over the subset of features given the input and performs instance wise feature selection based that. However, given the same number of explanations, our explainer favors both briefness and comprehensiveness while the L2X explainer favors comprehensiveness of the explanation and does not account for briefness.
ii

---

Under review as a conference paper at ICLR 2020
3
METHOD
A
chunkc
Task-performing System
X
Black Box
ICE
right
Explainer p(z|x)
T(X
@ r x=
X
Explainer T(X) Approx.
cough
right
Approximator
g g
Not
Figure I: Illustration of VIBI. (A) VIBI is composed of two parts; the explainer and approximator The explainer selects group of k key cognitive chunks given an instance while the approximator mimics the behaviour of the black bbx system using the selected keys as the input. (B) We set each
word as cognitive chunk and k 2. ! The explainer takes an input x and returns a stochastic k-hot random vector which indicates whether each cognitive chunk will be selected as explanation or not. ② t(x) provides instance- specific explanation. The approximator takes t(x) as an input and approximates the black-box output.
3.1 PERSPECTIVE FROM INFORMATION BOTTLENECK PRINCIPLE
The information bottleneck principle (Tishby et al. 2000) provides an appealing information theoretic view for learning supervised model by defining what mean by 'good" representation. The principle says that the optimal model transmits as much information as possible from the input x to the output through compressed representation (called the information bottleneck). The representation is stochastically defined and the optimal stochastic mapping p(t[x) is obtained by optimizing the following problem with a Markov chain assumption y + t:
p(t[x) =
arg max p(t)))).n(y(b).p(t))
I(t.y) 3 I(x,t)
(1)
where I(-.-) the MI and [ is Lagrange multiplier representing the trade-off between the compres- siveness - -I(x, and informativeness [(t., y) of the representation t.
We adopt the information bottleneck principle as criterion for finding brief but comprehensive explanations. Our aim is to learn an explainer generating explanations that are maximally informative about the black-box decision while compressive about a given input.
3.2 PROPOSED APPROACH
We introduce VIBI, a system-agnostic interpretation approach that provides brief but comprehensive explanations for decisions made by black-box decision system. In order to achieve this, we optimize the following information bottleneck objective.
xzz
arg max p(y(x).r(ylt)
I(t,y) - B t)
where I(t,y) represents the sufficiency of information retained for explaining the black- box output
y. -I(x, represents the briefness of the explanation f, and 33 is a Lagrange multiplier representing
a trade-off between the two. The primary difference between our information bottleneck objective (2)
and the one in Tishby et (2000) is follows: the latter aims to identify stochastic map of the representation t that itself works as an information bottleneck, whereas our objective aims to identify
stochastic map of performing instance- wise selection of cognitive chunks and define information
bottleneck as a function of z and the input x.
As illustrated in Figure TA VIBI is composed of two parts: the explainer and the approximator, each of which is modeled by deep neural network. The explainer selects a group of k key cognitive chunks given instance while the approximator mimics the behaviour of the black-box system using
iii

---

Under review as a conference paper at ICLR 2020
the selected keys the imput. i controls the level of sparsity in In detail, the explainer p(z]x: 0e) a map from an imput x to its attribution scores (x) = p(z, (x) where j is for the j-th cognitive chunk and zj is binary indicator whether the chunk will be selected or not. The attribution score indicates the probability that each cognitive chunk to be selected.
order to select top &o cognitive chunks explanation, k-hot vector is sampled from categorical distribution with class probabilities Pj (x) = p(z,|x x) and the j-th cognitive chunk is selected if Z, = 1. More specifically, the explanation defined follows:
1 (x= ② X ?,
where j indicates cognitive chunk, each of which corresponds to multiple row features i. The approximator is modeled by another deep neural network pty 0z), which mimics the black-box decision system. It takes
an input and returns output approximating the black-box output for the instance x. 0₂ and Oe represent the weight parameters of neural networks. The explainer and approximator are trained jointly by minimizing cost function that favors concise explanations while enforcing that the explanations alone suffice for accurate prediction.
To achieve compressiveness, addition to encouraging small MI between explanations and inputs, we also encourage the number of selected cognitive chunks to be small, i.e., encouraging z to be sparse. Note that MI and sparsity are two complementary approaches for achieving compression. MI aims at reducing semantic redundancy i explanations. Sparsity cannot achieve such goal. For example, consider a movie review where "great" occurs a lot and two explanations in judging the sentiment of the review: "great, great" and "great, shought-provoking sparsity (k = 2), but the former has semantic redundancy. In this case, MI helps to choose better explanation. The first explanation has larger MI with the input document. The second explanation has smaller MI and hence more brief and preferable.
They have the same level of
3.2.1 THE VARIATIONAL BOUND
The current form of information bottleneck objective is intractable due to the MIs y(, and I(x, t). We address this problem by using a variational approximation of our information bottleneck objective. In this section, we summarize the results and refer to Supplementary Material A for details.
Variational bound for 1(x, t): We first show that 1(x, t) <. 1(x, z) + C where C is constant and use the lower bound for -I(x, z) C' as lower bound for - -[xx, t). As result, we obtain:
p(z[x) r(z)
1(x, t) < 1(x, z) + + ((xx))xp(xxx)
log
+C DRL
+ (3)
Note that with proper choices rzz) and p(z|x), we can assume that the Kullback-Leibler divergence DKL (p(z)x), r(z)) has an analytical form.
Variational bound for I(t, y): We obtain the lower bound for I(t, y) by using g [t) approximate Plytt t), which works as an approximator to the black- box system. As a result, we obtain:
l(t, y) 2 E (t.y)))p(t.y) [log q = Ex-p(x) x-p(y xc) l x>p(()x)) [log ((y]t))) [t]]
(4)
where p(t[x,, y) p(t|x) by the Markov chain assumption y <+ t.
Combining Equations and 44, we obtain the following variational bound:
T(t, y) - -3 I(xx t)
2 Exxplx) By Eyxx-plylx Bx [log < [)] Ex (es. + C", (5)
where C* = -C8 can be ignored since it is independent of the optimization procedure. We use the empirical data distribution to approximate p(x, y) = p(x)p(y) x) and p(x).
3.2.2 CONTINUOUS RELAXATION AND REPARAMETERIZATIONN
Current form of the bound is still intractable because we need to sum over the (c) combinations of feature subsets. This is because sample top & out of cognitive chunks where each chunk is
assumed drawn from a categorical distribution with class probabilities p, (x)x) = p(z, (x). In order to
avoid this, we use the generalized Gumbel-softmax trick (Jang et al. 2017 Chen et al. 2018). This is
iv

---

Under review as a conference paper at ICLR 2020
well- technique that are used approximate cin-differentiable categorical subset sampling with differentiable Gumbel-softmax samples. The steps are as follows.
First, independently sample a cognitive chunk for l times. For each time, random perturbation ej is added to the log probability of each cognitive chunk log pj (x). From this, Concrete random vector (01,""' Cd) working continuous, differentiable approximation argmax is defined:
8; log (- loge;) where ej (0,
exp ((E, + (x) /r)
Cj Σラ= exp ((S, + log (x)) /T)
where is tuning parameter for the temperature of Gumbel-Softmax distribution. Next, we define a continuous-relaxed random vector z" = izi.-. 2il as the element-wise maximum the independently sampled Concrete vectors c!!) where = 1,...,.
zj max d for I 1,...,.
With this sampling scheme, we approximate the k-hot random vector and have the continuous approximation to the variational bound (5). This trick allows using standard backpropagation to compute the gradients of the parameters via reparameterization.
By putting everything together, obtain:
N
L
ΣΣ
log (x)) @ x(:)) - x
(x))
NL
where N is the number of samples, n indicate the n-th sample. ((e X(()) = in ((M()) @ (nn) is the approximator to the black-box system and -Dx1 (p(z* x(m)), r(="(())) represents the
(1)
compactness of the explanation. Once we learn the model, the attribution score p3 (x) for each cognitive chunk is used to select top l key cognitive chunks that are maximally compressive about the input x and informative about the black-box decision y on that input.
4 EXPERIMENTS
We evaluated VIBI on three datasets and compared with state-of-the- interpretable machine learning methods. The evaluation is performed from two perspectives: interpretability and fidelity. The interpretability indicates the ability to explain black-box model with human understandable terms. The fidelity implies how accurately approximator approximates the black-box model. Based on these criteria, we compared VIBI with three state-of-the-art system-agnosee (LIME (Ribeiro al. 2016), SHAP (Lundberg Lee 2017) and 1.2X (Chen et al., 2018)), and commonly used Map, we used the smooth gradient technique (Smilkov al. 2017) to get visually sharp gradient- based sensitivity maps over the basic gradient saliency map. See Supplementary Material B for further experimental details.
methods
method called Saliency Map (Simonyan et al. 2013). For Saliency
We examined how VIBI performs across different experimental settings varying the number of selected chunks k (amount or number of explanation), size of chunk (unit explanation). and trade- off parameter 3 (trade-off between the compressiveness of explanation and information preserved about the output). The settings of hyperparameter tuning include (bold indicate the choice for our final model): the temperature for Gumbel-soffmne
approximation 7 10.1, 0.2, 0.5, 0.7, 1}, learning rate 5 5 x 10-3, 10-3,-555 ,5 x 10 4 10 4 x 10-5) and 3 - {0, 0.001, 0.01, 0. 1, 10, 100). We use Adam algorithm (Kingma & Ba 2014) with batch size 100 for MNIST and 50 for IMDB, the coefficients used for computing running averages of gradient and its square (31. 32) (0.5, 0.999).
and 18 We tuned the hyperparameters via grid search and picked up the hyperparameters that yield the best fidelity score on the validation set. The code is publicly available on GitHub https: / wtwp.coo/pb.......oco/o/pooo

---

Under review as a conference paper at ICLR 2020
Negative Sentiment if any negative words are found
Positive Sentiment if any positive words found
NOT deslaadd anyone
money
was released and being nally young and too much into oineria it was one of the most fascinating cinematic experiences I had and realy left mark inside At first I didn't quite undersland the story and probably falle He
watched movie
fresh like thes Don't get me erong - LOVE a good Western-- -- Noticn said "G00D" this just trash. The actinn is horrible Vil Kilmer musi know someori
well the opens
falls love this
something them to
deeply prohibited atfair....
Katherine
in this redculiu place of crap..
Megathee/ Negative
Box Pesiive
Negative sentiment but predicted as Positive because several positive words are found
The reality of the mafia environment is absolutaly dog-eat-dog where gangster will killed showing any sign of weakness because they become a liability. I've got problem with the human side gansters' being portrayed but Bugey steers too far in the direction of soft, comical, men. The film is enjoyable but is only light entertainment and not a biopic of man who, though exciting, was extremely dangerous and fearsome. The acting's is good and the direction very solid. The locations and
very represented and the themes very interesting....
Tou:: Regathv B B.Box Poaltie
The movie reviews and explanations provided by VIBI were randomly selected from the validation set. The selected words are colored red. Each word is used as a cognitive chunk and k words are provided for each review.
Figure 2:
4.1 LSTM MOVIE SENTIMENT PREDICTION MODEL USING IMDB
The IMDB (Mas et al. T01T is a large text dataset containing movie reviews labeled by sentiment (positive/negative) We grouped the reviews into training, validation, and test sets, which have 25,000, 12,500, and 1 12,500 reviews prediction, which has two LSTM layers where each layer encodes words and sentences respectively. It achieved 87% of test accuracy. In order explain this LSTM black-box model, applied VIBI. We parameterized the explainer using a bidirectional LSTM and approximator using 2D CNN. For the details of the black-box model and VIBI architectures, see Supplementary Material B.1
respectiveel respectively. Then, trained hierarchical LSTM for sentiment
VIBI explains why the LSTM predicts each movie review to be positive/negative and provides instance-wise key words that are the most important attributes to the sentiment prediction. As seen in the top-right and top-left of Figure 2 VIBI shows that the positive (or negative) words pass through the bottleneck and make correct prediction. The bottom of Figure[sshow that the LSTM sentiment prediction model makes wrong prediction for negative review because the review includes several positive words such as 'enjoyable' and 'exciting".
4.2
CNN DIGIT RECOGNITION MODEL USING MNIST
Same digit with different angles.
Different digits, upside down
Difference between and 1
Same digit similar features
g
6
Figure 3: The hand-written digits and explanations provided by VIBI were randomly selected from the validation set. The selected patches are colored red the pixel is activated (i.e. white) and yellow otherwise (i.c. black), A patch composed of 4 x pixels is used as cognitive chunk and A 4 patches are identified for each image.
The MNIST (LeCun et al. 1998) is a large dataset contains 28 x 28 sized images of handwritten digits (0 9). We grouped the images into training, validation, and test sets, which have 50,000, 10,000, and 10,000 images respectively, and trained simple 2D CNN for the digit recognition, which achieved 97% test accuracy. In order to explain this CNN black-box model, we applied VIBI. We parameterized each the explainer and approximator using 2D CNN. For the details of the black-box model and VIBI architectures, see Supplementary Material B.2
VIBI explains how the CNN characterizes digit and recognizes differences between digits. The first two examples in Figure 3 show that the CNN recognizes digits using both shapes and angles. In the first example, the CNN characterizes '1 's by straightly aligned patches along with the activated
Blinded due the double-blind reviewing.
vi

---

Under review as a conference paper at ICLR 2020
regions although 'I's in the left and right panels are written at different angles. Contrary the first example, the second example shows that the CNN recognizes the difference between '9' and '6' by their differences in angles. The last two examples in Figure show that the CNN catches a difference of '7's from 'I's by patches located on the activated horizontal line on '7' (see the cyan circle) and recognizes '8's by two patches the top of the digits and another two patches at the bottom circle. More qualitative examples for VIBI and the baselines are shown in Supplementary Figure 6
The briefness of explanations also depends on the sparsity k. Supplementary Figure S shows how our method works under different sparsity. When we increase k. VIBI tends to select patches that are the same with or nearby previously selected patches and additionally select patches that catch new characteristics of digits.
4.3
TCR TO EPITOPE BINDING PREDICTION MODEL USING VDJDB AND IEDB
A
The Label Bound
B
Trun Label
D
VDJdb
Banoh-- prediraton 6.a
ASSARATDTOY
gro ctor
Feeuaroy
VDJdb
Frequero
IEDB
Matched
GILGFVFT
recall 0.79
recall 0.40
Un- matahhed
Murited
The explanations
Divis Doooggece
matched
ASS ASSGRSOBTOY
LOSA TCR
0.5
3.5
All Epilpess
Chavanry
OROWn Mr
GILGFVFT
Black-box prediction
Diack-bok prediction
Proauce:
IEDB
Figure 4: Black-box prediction scores of (A) VDJdb and (B) IEDB (C) Black-box prediction scores between the matched and unmatched instances from IEDB and (D) those by six epitope sequences. (E) An example of matched explanation (The VIBI selected amino acids are shaded).
We next illustrate how VIBI can be used to get insights from a model and ensure the safety of a model in real world application. Identifying which T-cell receptor (TCR) will bind specific epitope (i.e. cancer induced peptide molecules presented by the major histocompatibility complex to T-cells) important for screening T-cells or genetically engineering T-cells that are effective in recognizing and destroying tumor cells. Therefore, there has been efforts in developing computational methods to predict binding affinity of given Tep-tpitope pairs (Jurtz et al., 2018 Dokinen et al., (0019). These approaches rely on known interacting TCR-epitope pairs available from VDJdb (Shugay et al. 2017)
and IEDB (Vita et al.) 2014), which are the largest databases of several thousand entries. However, the number of unique TCRs harbored in single individual is estimated to be 10 (Lythe et al. 2016)
and a theoretical number of epitopes of length is 20' which are much larger than the number of known interacting TCR-epitope pairs.
One of the main concerns is whether a black-box model trained on such limited dataset can accurately predict TCR- (CR-epitope bindings of out-of-samples, This concern becomes pressing in a TCR-epitope binding prediction model trained on VDJdb (For details of the data, black-box model architecture, and parameter tuning, see Supplementary Material B.3). The model accurately predicted the (in-sample) bindings from VDJdb (recall 0.79, Figure However, it achieved poor prediction performance when used to predict the cattof-sample bindings from another dataset, IEDB (recall 0.40, Figure BB. ). In an attempt to address this problem, we applied VIBI and determined whether or not to accept a decision made by the black-box model based on VIBI's explanation. As illustrated in Figure VIBI provided matched explanations the identical amino acids in same positions (S and Y in this example) are highlighted different TCR sequences when they are bound to the same epitope (GILGFVETL in this example). Moreover, we found that if two TCR sequences binding to the same epitope, each from IEDB and VDJdb, are assigned with matched explanations by VIBI, then it significantly better predicts the binding than the others with matching TCRs (Figure 4C-D. p-values are shown). Therefore, if TCR sequence from IEDB has a matched explanation to a TCR from VDIdb, then safely follow the positive decision made by the black-box model.
vii

---

Under review as a conference paper at ICLR 2020
4.4 INTERPRETABILITT EVALUATED BY HUMANS
We evaluated interpretability of the methods on the LSTM movie sentiment prediction model and the CNN digit recognition model. For the movie sentiment prediction model, we provided instances that the black-box model had correctly predicted and asked humans to infer the output of the primary sentiment of the movie review (BesttivNNegadveeNeuraaa) given five key words selected by each method. Each method was evaluated by the humans on MTurk who are awarded the Masters Qualification, nigh-performant
workers who have demonstrated excellence across a wide range of tasks). We randomly selected and evaluated 200 instances for VIBI and 100 instances for the others. Five workers were assigned per instance. For the digit recognition model, we asked humans to directly score the explanation a 0-5 scale. Each method was evaluated by 16 graduate students at EXXX-Universityy who have taken at least one graduate-level machine learning class. For each method, 100 instances were randomly selected and evaluated. Four cognitive chunks with the size 4 x 4 were provided as an explanation for each instance (3 = 0.1 for VIBI). On average, 4.26 students were assigned per instance. Further details regarding the experiments can be found in Supplementary Material C
Table I: Evaluation of interpretability
Saliency
LIME
L2X
VIBI (Ours)
IMDB 34.2%
33.8%
35.6%
44.7%
MNIST
3.448
1.369
1.936
3.526
For IMDB, the percentage indicates how well the MTuurk worker's answers match the black-box output. For MNIST, the score indicates how well the highlighted chunks catch key characteristics of handwritten digits. The average scores over all samples is shown on a 0 to 5 scale. See the survey example and detailed result in Supplementary Material Tables for the detailed result.
As shown by the Table T VIBI better explains the black-box models. When explaining the movie sentiment prediction model, humans better inferred the (correctly predicted) black-box output given the five keywords when they were provided by VIBI. Therefore, it better captures the most contributing key words to the LSTM decision and better explains why the LSTM predicted each movie review by providing five key words. For explaining the digit recognition model, VIBI also highlighted the most concise chunks for explaining key characteristics of handwritten digit. Thus, it better explains how the CNN model recognized each the handwritten digit.
4.5 FIDELITY
Table 2: Evaluation of approximator and rationale fidelity
Approximator Fidelity
Rationale Fidelity
chunk size
k
Saliency
LIME
SHAP
L2X
VIBI (Ours)
L2X
VIBI (Ours)
sentenee
1
38.7 + 0.9
72.7 0.8
49.5
+ 1.0
87.6 : 0.6
87.7 = 0.6
72.7
+ 0.8
73.1 + 0.8
word words 5 words
41.9 + 0.9
75.6 + 0.8
50.1 + 1.0
73.8 + 0.8
74.4 0.8
63.8 + 0.8
65.7 + 0.8
IMDB
42.4 0.4+00 0.9
29.0 - 0.8
49.7
1.0
75.9
0.7
76.4 = 0.7
60.1
1 0.9
63.2 - 0.8
3.
41.4 0.9
67.9
49.1 1.0
83.3 0.7
83,5 = 0.7
69.4 = 0.8
66.0 - 0.8
Fx2 2
16
91.2 0.6
77.0
94.2
0.5
93.4
94,8 = 0.4
73.5 + 0.9
77.1 + 0.8
2 2
24
93.8
0.5
80.7 + 0.8
95.4 0.4
95.1 0.4
95.3 = 0.4
77.6
85.6 - 0.7
2 2
40
85.9
95.4
0.4
96.7
0.4
96,2 + 0.4
81.1 81.1+008
91.5 + 0.5
MNIST
4x
60.9 0.7
94.8
0.4
95.3
0.4
94.8
0.9
77.5 - 0.8
4 X
6
90.6
0.6
63.7
+ 0.9
93.6
- 0.5
95.7 0.4
95.6 = 0.4
51.1 + 1.0
70.1 + 0.9
4xx4 4
10
94.9
0.4
70.5
+ 0.9
95.1
+ 0.4
96.5
0.4
96.7
0.4
83,5
+ 0.7
93.3 + 0.5
Note that the black box models achieved 87% accuracy for IMDB and 97% uccurucy for MNIST. A = 0.1 for VIBL Accuracy and 0.95
confidence interval shown. performed three runs for each method and reported the results. more evaluations using Fl Fl-score
and further results from different parameter seltings in Supplementary Material Table/o B for appeuximator lidelity and Table ( Jnnd 7 rationale fidelity.
or
We assessed fidelity of the methods in approximating the black-box output. First, we compared the ability of the approximators to imitate behaviour of the black-box, denoted as Approximator fidelity.
Amazoo Mechanical Turk, https://www.mturtt wwtp:/wwwwwwwww.o/m/// Blinded due the double-blinding reviewing
viii

---

Under review as a conference paper at ICLR 2020
(See Supplementary Material B.4 for details about how each approximator fidelity is evaluated.) As shown in Table 2 VIBI has a better approximator fidelity than Saliency, LIME and SHAP in most cases. VIBI and L2X showed similar levels of approximator fidelity, we further compared them based on Rationale fidelity. The difference between approximator and rationale fidelity is as follows. Approximator fidelity is quantified by prediction performance of the approximators that takes t', the continuous relaxation of t, as an input and the black-box output as targeted label; rationale fidelity is quantified by using instead of t". Note that only takes the top k chunks and sets the others to be zero, while t" sets the others to be small, non-zero values. Therefore, rationale fidelity allows evaluate how much information purely flows through the explanations, not through a narrow crack made during the continuous relaxation procedure. As shown in Table 2 VIBI has better rationale fidelity than L.2X most cases. Note that 1.2X can be viewed as a special case VIBI without the compressiveness term, i.e., 9 = O. The rationale fidelity empirically demonstrates that the compressiveness term can help the information to flow purely through the explanations.
5 CONCLUSION
We employ the information bottleneck principle criterion for learning 'good' explanations. Instance-wised selected cognitive chunks work as an information bottleneck, hence, provide concise but comprehensive explanations for each decision made by a black-box system. The information bottleneck framework provides a theoretical background that the bottleneck captures a minimal sufficient statistic, i.e. the most compressed representation that captures all the possible (i.e. sufficient) amount of information about output. For finite 3, the bottleneck approximates such minimal sufficient statistic.
However, the way this information is represented may have a substantial effect on interpretability. VIBI helps to address this issue some extent by always returning a certain form of output (i.e.. a k-hot vector assigned to each chunk) and having a certain form of the information bottleneck layer (i.e., a masked input) so that makes sure that the explanations are easily understandable to humans. In practice, such chunking strategy leads to deviation from the strict theory that a 'good' explanation is the most compressed one but helps to achieve better interpretability in practice.
REFERENCES
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. International Conference on Learning Representations, 2017.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):e0130 3010000000.000 2015.
David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert Müller. How to explain individual classification decisions. Journal of Machine Learning Research, 1 1000))1100-1111 2010.
Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Rober Müller, and Wojeiech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In International Conference on Artificial Neural Networks, pp. 63-71. Springer, 2016.
Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning to explain: An thoomattion-thooettie perspective on model interpretation. International Conference on Machine Learning (ICML), 2018.
Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2012.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. Advances in Neural Information Processing Systems, pp. 6967-6976, 2017.
Finale Doshi Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint arXiv: 1702.08602777 2017.
ix

---

Under review as a conference paper at ICLR 2020
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3427-3337 2017.
Steven Henikoff and Jorja G Henikoff. Amino acid substitution matrices from protein blocks. Proceedings of the National Academy of Sciences, 89(22):10115-1091 1992.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. International Conference on Learning Representations, 2017.
Emmi Jokinen, Markus Heinonen, Jani Huuhtanen, Satu Mustjoki, and Harri Lähdesmäki. Tergp: Determining epitope specificity of t cell receptors. bioRxáv, pp. 542332, 2019.
Vanessa Isabell Jurtz, Leon Eyrich Jessen, Amalie Kai Bentzen, Martin Closter Jespersen, Swapnil Mahajan, Randi Vita, Kamilla Kjærgaard Jensen, Paolo Marcatili, Sine Reker Hadrup, Bjoern Peters, al. Netter: sequence-based prediction of ter binding to peptide-mhe complexes using convolutional neural networks. bioRxiv, pp. 433706, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(11):228-22224 1998.
Zachary C Lipton. The mythos of model interpretabillity. arXiv preprint wrXiv:1606.034900 2016.
Scott M Lundberg and Su-In Lee. A unified approach interpreting model predictions. In Advances in Neural Information Processing Systems, pp. 4765-4774, 2017.
Grant Lythe, Robin E Callard, Rollo L Hoare, and Carmen Molina-Paris How many TCR clonotypes does a body maintain? Journal of Theoretical Biology, 389:214-223 2244,2224,222 2016.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. h
/ http://wwww..oooatttooooooo/////
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. The British Machine Vision Conference, 2018.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135-1144. ACM, 2016.
Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Velez. Right for the right reasons: Training differentiable models by constraining their explanations. In Proceedings of the Twenty- Sixth International Joint Conference on Artificial Intelligence, pp.
2017.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning-Voie Volume 70, pp. 3145-3153. JMLR. org, 2017.
Mikhail Shugay, Dmitriy V Bagaev, Ivan V Zvyagin, Renske M Vroomans, Jeremy Chase Crawford, Garry Dolton, Ekaterina A Komech, Anastasiya I. Sycheva, Anna E Koneva, Evgeniy S Egorov, et al. Vdjdb: a curated database of t-cell receptor sequences with known antigen specificity. Nucleic Acids Research, 46(D1):D419-4427 2017.
Ravid Shwartz-ZZi and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv: 100..00330000000 000000 2017.
X

---

Under review as a conference paper at ICLR 2020
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Vićgas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint
2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume Volume 70, pp. 3319-3328.
JMLR. org. 2017.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In Information Theory Workshop (ITW), 2015 IEEE, pp. 1 1-5. IEEE, 2015.
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/00040057, 2000.
Randi Vita, James A Overton, Jason A Greenbaum, Julia Ponomarenko, Jason D Clark, Jason R Cantrell, Daniel K Wheeler, Joseph L Gabbard, Deborah Hix, Alessandro Sette, al. The immune epitope database (iedb) 3.0. Nucleic Acids Research, (3(DI):D455-4444 2014.
Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision, pp. 818-833. Springer, 2014.
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. International Conference on Learning Representations, 2017.
A VARIATIONAL INFERENCE TO CONSTRUCT A LOWER BOUND ON THE IB OBJECTIVE
We illustrate the variational approximation of the following information bottleneck objective:
p(t|x) = arg max p(t ((t))))))()l))pp(t)
I(t, y) 33 I(x,t)
Let start with the Markov chain assumption y Therefore, the condition is written as + x ++ (Cover & Thomas 2012). Using this condition, we factorize the joint distribution ply, x, t):
tt which also implies y < t.
ply, x, t) (())p(((x)xp))pppppllpppp(p) It, x) p(x)p(()x)p(((t)
where ptylt, x) = p(y|t) by the Markov chain.
Now, will examine each of the expressions in the information bottleneck objective I(t, ,3 [(x, t) in turn.
(I) VARIATIONAL. LOWER BOUND FOR T(x,
We first show that I(x, t) < I(x, z) where C' is a constant and then use the lower bound for -I(x, 2) C as a lower bound for -T T(x,t). t).
First, we prove T(x, t.) < T(x, z) + C. From the Markov Chain x-> (xxz) z) t, we have T(x, t) <
I(x, (x, 2)). According to the chain rule for mutual information, I(x, (x. z)) = =[(x, z) I(x, x/z).
where 1(x, x x|z) = Hxx|z) + H(x|z) H(x, x|z). Further, H(x|z) < H(x). Putting these pieces together, we have
[(x, t) <
(6)
where entropy H(x) of input a constant. For simplicity, we denote as C.
We then approximate p(z) using r(z). From the fact that Kullback Leibler divergence is always positive, we have
E (xxx))xp(xa) [logp(z)]
Eznp(z) [logp(z) Ex-n(z) rz))
E E(xxx)xxpxx.) [logr(z)).
xi

---

Under review as a conference paper at ICLR 2020
From 6 and
have
1(x, t) - 1(xx z) +
log p(2|x)
(xxx)))pxxx))
+
p(z)
log p(z|x)
(xxx)
+
r(z)
Exxp(x) DKT. (p())x()))) C.
(I)) VARIATIONAL LOWER BOUND FOR y(t
Starting with yt, we have
log p(y, t) p(y)p(t)
I(t,y) (ptt))
log p(y[t)
B((y))p(te)
p(y)
-E(ty)>p(tty) [log ((y(() - Ey-rly) [lg)(yy)]
((txy))p(t.y) [[y]p(y[t)] + Const.
Note that H(y) - By-plyy [log p(y) is independent of the optimization procedure, hence can be ignored. Now, we use g(y it) to approximate p(y[t) which works as an approximator to the black-box system. Using the fact that Kullback Leibler divergence is always positive, we have
Byltsp(yit) [logp(y|t]] Eyltsp(yitt [log ((y[t)]
and
ytt, 2 ((ry)) [log ((y(t)]
=Ex=p(x) By boypppplly (x x [log <(y[t)]
Exxp(x) Eyixsplyls (xxxp(())) [log t))
where p(t(x,y) = p(t|x) by the Markov chain assumption y + x + t.
(II) VARIATIONAL LOWER BOUND FOR I(t,y) - 31((,, t)
In summary, we have the following variational bounds for each term.
I(t,y) E Ex-p(xx Byt (y]{)]
I(x, t) Dx. +C
which result in
I(t,y) - 3 ((x,t) t) Exxplx) By (xxeppttt) [log q(ylt)] - Ex-p(xx DKL. + Const.
With proper choices of r(z) and p(z)x), we assume that the Kullback-Leibler divergence (KL(p(Z)x) r(z)) is integrated analytically. We use the empirical data distribution to approxi- mate yxx = ((x) x) and p(x).
Our variational approximation is similar to the one from Alemi et al. (2017), which first developed variational lower bound on the information bottleneck for deep neural networks. However, our information bottleneck is different from theirs: their information bottleneck is the stochastic encoding z of the input as itself, whereas our information bottleneck is pairwise product of the stochastic encoding and the input x where is Boolean random vector. Due to this difference, appproximate 1(x, t) by using the lower bound [xx, z), instead directly deriving the lower bound of 1xx, t).
xii

---

Under review as a conference paper at ICLR 2020
B EXPERIMENTAL DETAILS
The settings of hyperparameter tuning include the followings (bold indicate the choice for our final model): the temperature for Gumbel-softtax approximation
{0.1, 0.2, 0.5, 0.7, 1), learning
rate 5 x 10-3 10-3 5 x 10 10 4.5 5 x 10-") and B {0, 0.001. 0.01, 0.1, 1, 10, 100). We use Adam algorithm (Kingma & Ba 2014) with batch size 100 for MNIST and 50 for IMDB, the coefficients used for computing running averages of gradient and its square (31, (2)3 (0.5, 0.999), and 10- 8 We tuned the hyperparameters via grid search and picked up the hyperparameters that yield the best fidelity score on the validation set. All implementation is performed via PyTorch an open source deep learning platform (Paszke et al. 2017).
Approximator fidelity is prediction performance of the approximator that takes relaxation to as an input and black-box output target. In detail, we get t* with the Gumble-softmax sampling and make prediction based on t'. This procedure is repeated for 12 times and the final prediction is made by averaging the 12 prediction scores. Rationale fidelity is prediction performance of the approximator that takes t as an input and black-box output as a target. (Note that a masked input that only takes the top chunks and others to zero.) The final prediction is made by its prediction score. We use the prior rz(2)) = (k/J)²; k/J)²-2, for all experiments where is the total number of chunks and r(z) r2,r) The analytical form
of the Kullback- divergence term is DKL r(zz)x),,(z)) Σÿ= (KL(p(z, (z)))
E/=1 Ezz=op(z;) (x) (logr(zylx) log (Z;)).
B.1 LSTM MOVIE SENTIMENT PREDICTION MODEL USING IMDB
Black-Box Model Structure. Each review is padded or cut to contain 15 sentences and 50 words for each sentence. The architecture consists of word-embedding layer with size 50 for each word followed by two bidirectional LSTMs, a fully connected layer with two units, and a soft-max layer. The first LSTM layer encodes the word embedding vector and generates a word-representationn vector with size 100 for each word. Within each sentence, the word representation vectors are elementwisely averaged to form size 100 sentence representation vector. The second LSTM layer encodes the sentence representation vector and generates size 60 review embedding vector.
VIBI Structure. We parameterize the explainer with bidirectional LSTM and approxima- tor with a 2D CNN. For the explainer, we use a bidirectional layer that returns multiple output vectors, each of which corresponds to recurrent unit. Each element in the output vectors are averaged over all units, and then the averaged output vector is followed by log-softmax calculation. As a result, the explainer returns vector of log-probabilities, each of which indicates whether or not each cognitive chunk will be selected as an input to the approximator. For the approximator, we use a convolutional layer followed by ReLU activation function and max-pooling layer and a fully connected layer returning a size-2 vector followed by log-softmax calculation. The final layer returns a vector of log pogpbbbbbilities for the two sentiments (positiveenegaative.
B.2 CNN DIGIT RECOGNITION MODEL USING MNIST
Black-Bor Model Structure. The architecture consists of two convolutional layers with the kernel size 5 followed by max-pooling layer with the pool size 2, two fully connected layers and soft-max layer. The two convolutional layers contain 10 and 20 filters respectively and the two fully connected layers are composed of 50 and 10 units respectively.
VIBI Structure. We parameterize each the explainer and approximator using 2D CNNs.
Structure of the explainer differs depending on the chunk size. For example, when 4 x 4 cognitive chunk is used, we use two convolutional layers with the kernel size followed by ReLU activation function and max- pooling layer with the pool size 2, and one convolutional layer with kernel size I returning 7 x 7 2D matrix followed by log-softmax calculation. The final layer returns vector
xiii

---

Under review as a conference paper at ICLR 2020
of log probabilbitiis for the 49 chunks. The three convolutional layers contains 8, 16, and I filters respectively. The output from the explainer indicates which cogmitive chunks should be taken as an input for the approximator. We parameterize the approximator using two convolutional layers with kernel size followed by ReL. activation function and max-pooling layer with pool size 2 and with 32 and 64 filters respectively, and one fully connected layer returning size-10 vector followed by log-softmax calculation so that the final layer returns a vector of log-probabilities for the ten digits.
B.3 TCR TO EPITOPE BINDING PREDICTION MODEL USING VDJDB AND IEDB
Data. We use two public datasets: VDJdb and IEDB. VDJdb (Shugay al. 2017 contains the T-cell receptor (TCR) sequences with known antigen specificity (i.e., epitope sequences). We use a VDJdb dataset preprocessed by Jokinen al. (2019) (See their paper for details in data preprocessing). The preprocessed dataset contains 5,784 samples (2,892 positively and 2,892 negatively binding pairs of TCR and epitope). This dataset consists of 4,363 unique TCR sequences and 21 unique epitope sequences. We group the pairs of TCR and epitope sequences into training, validation, and test sets, which have 4,627, 578, and 579 pairs, respectively. IEDB (Vita et al., 2014) contains TCR sequences and corresponding epitopes. We used an IEDB dataset preprocessed by Jurtz et al. (2018). The preprocessed dataset contains 9,328 samples (positively binding pairs only) and consists of 9,221 unique TCR sequences and 98 unique epitopes sequences. We used the IEDB dataset as out-ouf-samples. There are 6 epitopes contained in both VDJdb and IEDB dataset: GILGEVETL, GLCTLVAML, NLVPMVATV, LIWNGPMAV, YVLDHLIVV, CINGVCWTV.
Black-Box Model Structure. Each TCR and epitope sequence is padded or cut to contain 20 and 13 amino acids, respectively. The embedding matrix has size 24 and is initialized with BLOSUM50 matrix (Henikoff & Henikoff] 1992). The architecture consists of two sequence encoders that process TCR and epitope sequences each, and three dense layers that process the encoded sequences together. The TCR encoder consists of dropout layers with the drop-out probability 0.3, two convolutional layers with the kernel size followed by a batch normalization layer, ReLU activation function and max-pooling layer with the pool size 3. The two convolutional layers contain 32 and 16 filters respectively. Structure of the epitope encoder is the same with the one from the TCR encoder. We optimized the models with the following search space (bold indicate the choice for our final model): the batch size (25, 50, 100), learning rate 0.05, 0.01, 0.005, 0.001, 0.0005] and filter size - {116, 8), (32, 16)).
VIBI Structure. We parameterize the explainer and approximator using 2D CNNs and sev- eral dense layers. Each TCR and epitope sequence is preprocessed and embedded in the same way as for the black-box model. We have two types of explainers: one for TCR sequence and another for epitope sequence. Each explainer encodes both TCR and epitope sequences and concatenates them through three dense layers followed by a ReLU activation function. For the TCR explainer, the three dense layers contain 32, 16, 20 hidden-units, respectively. For the epitope explainer, the three dense layers contain 32, 16. 13 hidden-unine respectively. The TCR and epitope encoders have the same architectures with those of the black-box box model. Each explainer then returns a vector of explanation.
log- probaabiiiltsss that indicate which peptides in TCR or epitope should be selected The approximator has the same architecture as the black-box model. We optimize the models with the following search space (bold indicate the choice for final model): the batch size (25, 50, 100). learning rate- 0.0001, 0.001, 101,0.1..
B.4 BASELINE METHODS
Hyperparameter tuning, For L2X. we used the same hyperparameter search space of VIBI. For LIME, we tuned the segmentation filter size over 2,, 4] for MNIST and (10, 25} for IMDB, For all methods, we tuned the hyperparameters via grid search and picked up the hyperparameters that yield the best fidelity score on the validation set.
Cognitive chunk. LIME, SHAP and Saliency yield an attribution score for each feature. A score is an average of (absolute) attribution scores of features that belong to the chunk. Top & chunks that have the highest thunk-attribution scores are selected. L2X selects chunks in the same way as VIBI.
xiv

---

Under review as a conference paper at ICLR 2020
Fidelity evaluation. LIME, SHAP, Saliency, and L2X all have their approximators. LIME and SHAP have a sparse linear approximator for each black-box instance (See Section 3,4 in Ribeiro et al. 2016), and Equation (3) in Lundberg & Lee (017)). Saliency has 1st-order Taylor approximation to each black-box instance (See Equation (3) in Simonyyn et al. (2013) ). The approximator fidelity of LIME, SHAP, and Saliency is calculated using their proposed approximators that are composed of the selected features. L2X has a deep neural network that approximates black-box model (See Section in Chen al. 2018 ) The approximator and rationale fidelity of L2X is calculated in the same way as VIBI.
XV

---

Under review as a conference paper at ICLR 2020
C INTERPRETABILITY EVALUATED BY HUMANS
C.I LSTM MOVIE SENTIMENT PREDICTION MODEL USING IMDB DATASET
EVALUATION
Table 3: Evaluation of Interpretability on an LSTM movie sentiment prediction model using IMDB.
Recognized
Box Output
Saliency
LIME
L2X
VIBI (Ours)
by Mturk worker
Positive
Positive
19.8
17.4
17.6
24.3
Positive
Negative
12.6
6.8
7.2
16.9
Positive
Neutral
25.6
18.8
24.2
11.1
Negative
Positive
11.0
10.6
11.6
16.2
Negative
Negative
14.4
16.4
18.0
20.4
Negative
Neutral
16.6
30.0
21.4
11.2
The percentage of samples belongs to each combination of the black-box output and the
sentiment recognized by workers at Amazon Mechanical Turk (https: htpp:///www mturk
com/)
) are showed
We evaluated interpretability of the methods on the LSTM movie sentiment prediction model. The interpretable machine learning methods were evaluated by workers at Amazon Mechanical Turk (https: //www .tturk com/) who are awarded the Masters Qualification (i.e. high performance workers who have demonstrated excellence across wide range of task). Randomly selected instances (200 for VIBI and 100 for the others) were evaluated for each method. 5 workers are assigned per instance.
We provided instances that the black-box model had correctly predicted and asked humans to infer the output of the primary sentiment of the movie review (Posiive/Nggativ/NNetaaal) given five key words selected by each method. See the survey example below for further details. Note that this proxy for measuring how well humans infer the black-box output given explanations; we use such proxy because the workers are general public who are not familiar with the term 'black-box' or "output of the model."
In Table the percentage of samples belongs to each combination of the black-box output and the sentiment recognized by the workers are showed. VIBI has the highest percentage of samples belonging to the Positive/Positive or Negative/Negaaave and the lowest percentage of samples belonging to the Positive/Neutral or Negative/Neutral. LIME has the lowest percentage of samples belonging to the Positive/Negative Negative/Positive, but it is because LIME tends to select words such as 'that', 'the', 'is' so that most of samples are recognized as Neutral.
xvi

---

Under review as a conference paper at ICLR 2020
SURVEY EXAMPLE FOR IMDB
Title Label sentiment given few words. Description: Recognize the primary sentiment of the movie review given a few words only.
Instructions
Only few tests from a movie review are shown here, Guess the given
primary
full Intruction
Postre
Onty fou saca homa moli
Tadeedl
Figure 5: A survey example of MTurk evaluation on the LSTM movie sentiment prediction model
C.2 CNN DIGIT RECOGNITION MODEL USING MNIST
EVALUATION
Table 4: Evaluation of Interpretability a 2d CNN digit
recognition model using MNIST.
MNIST Digit
Saliency
LIME
I.2X
VIBI (Ours)
0
3.200
2.000
2.333
3.000
4.393
0.795
1.263
3,913
2
3.125
1.200
1.400
3,200
3,286
1.833
2.429
3.625
3.333
1.000
1.857
3.857
5
3.167
1.381
2,000
2.875
3,333
1.000
1.889
3.625
3.667
2.000
1.667
4.000
8
3.750
1.333
2.667
3,500
3.222
1.143
1.857
3.667
Ave. over digits
3.448
1.369
1.936
3.526
The average scores (0-5 scale) evaluated by graduate students at
XXX University (Blinded due to the double hlinding reviewing) are showed.
We evaluated interpretability of the methods on the CNN digit recognition model. The interpretable machine learning methods were evaluated by 16 graduate students at XXX University (Blinded due the double blinding reviewing) who have taken at least one graduate-level machine learning class. Randomly selected 100 instances were evaluated for each method. On average, 4.26 students are assigned per instance. See the survey example below for further details. For the digit recognition model, we asked humans to directly score the explanation on 0-5 scale.
In Table 4 the average score per digit showed. VIBI outperforms L2X and LIME and slightly outperforms Saliency in terms of the average score over digits. VIBI outperforms at digit 2, 3, 4, 6, and 9, and performs comparable to Saliency at 0, 5, 8.
xvii

---

Under review as a conference paper at ICLR 2020
Saliency
6
LIME
4
6
4
66
77
SHAP
JC
JF
G
6
6
L2X
7
VIBI
6
Saliency
4
9
LIME
9
2
2
SHAP
1
3
9
2
L2X
2
4
3
5
3
B
VIBI
Figure 6: The hand-written digits and explanations provided by VIBI and the baselines. The examples
randomly selected from the validation The selected patches are colored red if the pixel is activated (i.e. white) and yellow otherwise (i.e. black). A patch composed of 4 x 4 pixels is used as cognitive chunk and & 4 patches are identified for each image.
xviii

---

Under review as a conference paper at ICLR 2020
SURVEY EXAMPLE FOR MNIST
Instruction
2D CNN model used for digit recogrition for MNIST dalasel
Recognized
2D CNN model
9
Now; interpretable learning method explains decision made by the 2D CNN model:
Recognized Explain by highlighiin key pixels play
Importent role the 2D CNN digit recognition.
MNIST large dataset contains x sized images handwritten digits (0 to 9). Here, a 2D convolutional neural network (CNN) is used for the digit recognition for MNIST. Several interpretable machine learning methods are learned to explain the model by highlighting key pixels that play important role in the CNN digit recognition. The highlighted pixels provides an explanation a handwritten image why the CNN model recognized the handwriting it does. Your task is to evaluate the explanation each instance on scale
Please score each instance based on following criteria:
1
2
4
5
explanstion
Insufficient redundent explanation
Concise explanation
randonty
some highighted
con hee key of cranciessio each dgk.
Figure 7: A survey example of evaluation on the MNIST digit recognition model.
xix

---

Under review as a conference paper at ICLR 2020
D FIDELITY
Table 5: Evaluation of rationale fidelity on LSTM movie sentiment prediction model
using IMDB.
L2X
VIBI (Ours)
chunk size
0
0.001
0.01
0.1
1
10
100
sentence
0.727
0.693
0.711
0.731
0.729
0.734
0.734
word
5
0.638
0.657
0.666
0.657
0.648
0.640
0.654
Accuracy
words
0.601
0.630
0.624
0.632
0.628
0.623
0.628
words
0.694
0.660
0.662
0.660
0.662
0.660
0.660
sentence
0,581
0,547
0.562
0.567
0.585
0.586
0.586
word
5
0,486
0.521
0.551
0.512
0.516
0.508
0.526
F1-score
words
0.478
0.506
0.500
0.540
0.501
0.498
0,504
words
3
0.551
0.528
0.529
0.522
0.529
0.528
0.525
Rationale fidelity quantifies ability of the selected chunks to infer the black-box output. A large rationale fidelity implies that the selected chunks account for large portion of the approximator fidelity. Prediction accuracy Fl-score the approximator for the CNN model are shown. (5 0, 0.001, 0.01, 0.1, 1, 10, 100)
Table 6: Evaluation approximator fidelity on LSTM movie sentiment prediction model using IMDB.
Saliency
LIME
SHAP
L2X
VIBI (Ours)
chunk
0
0.001
0.01
0.1
10
100)
sentence
0.387
0.727
0.495
0.876
0.869
0.877
0.879
0.879
0.884
word
0.419
0.756
0.501
0.738
0.766
0.772
0.744
0.773
0.763
0.767
Accuracy
words
0.424
0.290
0.496
0.759
0.784
0.780
0.764
0.774
0.778
0.774
5 words
0.414
0.679
0.491
0.833
0.836
0.831
0.835
0.834
0.830
0.833
sentence
0.331
0.564
0.400
0.721
0.693
0.707
0.730
0.730
0.727
0.734
word
0.350
0.585
0.413
0.565
0.607
0.616
0.594
0.620
0.609
0.612
F1-score
S words
0.360
0.302
0.418
0.621
0.641
0.622
0.624
0.615
0.622
0.616
5 words
0.352
0.523
0.409
0.680
0.683
0.674
0.681
0.677
0.669
0.682
Approximator fidelity quantifies ability of the approximator imitate the behaviour of black-box. Prediction accuracy and Fi-score the approximator for the LSTM model are shown. (3 0.0000 0.01, 0.1, 1, 10, 100)
XX

---

Under review as a conference paper at ICLR 2020
Table 7: Evaluation of the rationale fidelity on CNN digit recognition model using MNIST.
L2X
VIBI (Ours)
chunk size
k
0
0.001
0.01
0.1
10
100
x
64
0.694
0.690
0.726
0.689
0.742
0.729
0.766
1
0.814
0.831
0.780
0.806
0.859
0.765
0.826
x 1
160
0.903
0.907
0.905
0.917
0.917
0.928
0.902
2 x 2
16
0.735
0.795
0.750
0.771
0.732
0.753
0.769
2
0.776
0.855
0.834
0.856
0.868
0.854
0.847
Accuracy
2
40
0,811
0.914
0.914
0.915
0.903
0.918
0.935
x 2
80
0.905
0.949
0.940
0.939
0.962
0.941
0.923
X 4
0.650
0.655
0.650
0.775
0.717
0.682
0.681
4 X 4
6
0.511
0.858
0.706
0.701
0.708
0.690
0.730
4
0.835
0.835
0.824
0.933
0.875
0.854
0.782
X 4
20
0.954
0.962
0.815
0.934
0.929
0.946
0.943
X 1
0.684
0.679
0.716
0.670
0.734
0.710
0.755
X 1
96
0.808
0.825
0.750
0.803
0.854
0.750
0.820
1
160
0.898
0.902
0.899
0.912
0.913
0.924
0.897
x 2
16
0.720
0.786
0.738
0.761
0.723
0.744
0.769
2
24
0.766
0.848
0.836
0.851
0.858
0.859
0.840
F1-score
40
0.798
0.914
0.910
0.910
0.898
0.914
0.931
2 X 2
80
0.901
0.946
0.936
0.930
0.959
0.938
0.918
4
4
0.634
0.658
0.637
0.763
0.704
0.671
0.669
X 4
0.493
0.852
0.693
0.687
0.692
0.675
0.720
4 X 4
10
0.828
0.827
0.816
0.928
0.869
0.849
0.773
4 4
20
0.950
0.959
0.806
0.931
0.926
0.942
0.940
selected chunks to infer black-box output. large rationale fidelity implies that selected chunks account for a large portion of the approximator fidelity. Prediction accuracy and F1-score of the approximator for the CNN model are shown. (8 0, 0.001, 0.01, 0.1, 1, 10, 100)
Rationale fidelity quantifies ability
Table 8: Evaluation of the approximator fidelity CNN digit recognition model using MNIST.
Saliency
LIME
SHAP
I.2X
VIBI (Ours)
chunk size
k
0.001
0.01
0.1
10
100
64
0.944
0.982
0.955
0.933
0.959
0.962
0.959
0.960
0.952
0.953
1 x
96
0.956
0.986
0.950
0.963
0.963
0.951
0.967
0.968
0.953
0.962
1 x
160
0.964
0.989
0.958
0.970
0.967
0,973
0.974
0.974
0.974
0.967
2 x
16
0.912
0.770
0.942
0.934
0.945
0.941
0.948
0.938
0.939
0.940
2 x
24
0.938
0.807
0.954
0.951
0.956
0,955
0.953
0.953
0.953
0,960
Accuracy
2 1x2
40
0.957
0.859
0.954
0.967
0.965
0.966
0.962
0.967
0.965
0.967
2 x
80
0.966
0.897
0.966
0.976
0.977
0,974
0.972
0.977
0.971
0.973
0.863
0.609
0.948
0.953
0.922
0.928
0.948
0.942
0.942
0,953
0.906
0.637
0.936
0.957
0.963
0.954
0.956
0.953
0.963
0,949
0.705
0.959
0.967
0.961
0.969
0.964
20
0.963
0.771
0.955
0.974
0.977
0.975
0.975
0.973
0.975
0.974
64
0.938
0.981
0.952
0.930
0.956
0.960
0.956
0.957
0.950
0.950
0.950
0.985
0.948
0.961
0.961
0.954
0.965
0.966
0.951
0.960
1x1
0.959
0.989
0.954
0.973
0.972
0,972
2x2
0.902
0.755
0.938
0.942
0.936
0.944
0.934
0.936
0.936
2x2
24
0,932
0.795
0.951
0.949
0.954
0,952
0.950
0.951
0.950
0.958
Fl-score
2 2
40
0.952
0.853
0.946
0.965
0.963
0,964
0.961
0.965
0.963
0.965
2
80
0,962
0.892
0.963
0.974
0.976
0.973
0.972
0.975
0.969
0.971
4
4
0.849
0.588
0.943
0.951
0.917
0.923
0.944
0.938
0.939
0.950
4
6
0.895
0.621
0,920
0.954
0.961
0.951
0.953
0.950
0.960
0.959
4
10
0.943
0.689
0,946
0.963
0.969
0,956
0.965
0.958
0.968
0.961
4 1
20
0.959
0.763
0.952
0.972
0.976
0,972
0.974
0.972
0.973
0.972
Approximator fidelity quantifies ability of the approximator imitate the behaviour of black-box. Prediction accuracy and Fi-score of the approximator for the CNN model ure shown. (6 = 0, 0,001, 0.01, 0. 10 100)
xxi

---

Under review as a conference paper at ICLR 2020
E
EXTRA EXPERIMENTS
E. CHOICE OF
The compressiveness of explanations depends on the sparsity & (i.e., the number of cognitive chunks to selected). larger allows the information bottleneck to convey more information about output, but it gives less compressive explanations than smaller k. For deciding k, we recommend choosing the minimum possible that achieves a target fidelity because an unnecessarily large l can make redundant explanations. Figure g shows how VIBI works under different sparsity k. When increase k VIBI tends to select chunks that are the same or nearby the previously selected chunks and additionally select chunks that catch another characteristics of digits. The characteristics that are caught at k = 4 tend to be caught again at a larger ks.
2
2
5
k= 4
2
21
5
k= 6
= 2
Q
5
k = 20
Figure 8: The hand-written digits and explanations provided by VIBI using various number of chunks (k). The examples are randomly selected from the validation set. The selected patches are colored red if the pixel is activated (i.c. white) and yellow otherwise (i.e. black). A patch composed of 4 x 4 pixels is used as a cognitive chunk and & 4, 6, 10, patches are identified for each image.
E.2 CHOICE OF CHUNK SIZE
The chunk size changes the way the information is represented in the information bottleneck, which can affect interpretability. For choosing the chunk size, a greater fidelity does not always mean better interpretability. For example, in the MNIST experiment, the models with the chunk size X have achieved better fidelity than those with 2 x 2 and 4 x 4 under the same number of pixels that are used for the information bottleneck, but 1 is less recognizable than 2 x 2 or 4 (Figure 9
Choosing the chunk size is not trivial and the strategy varies for different domains (except the case that each (raw) feature itself cognitive (e.g., age, gender) so that the chunk size should be 1). Obviously, the chunk size should be larger than the size of the smallest cognitive unit. For example, the IMDB experiment, the smallest cognitive unit was set to 50 because 50 raw features compose a word. In our sentiment prediction task, using word as the chunk size can be enough to explain the decision (i.e., positive/negative sentiment); to infer the sentiment, we do not need to know a whole sentence. However, we may need to such as disease diagnosis or patient need detection since decisions on such tasks are made based on contextual information. In the MNIST experiment, set the chunk size to be 4 x because the
phrase sentence as the chunk size for complex tasks
xxii

---

Under review as a conference paper at ICLR 2020
digits' stroke width is about to 4. However, may need to use different chunk sizes, for example, for automated visual inspection for identifying defects on a factory production line. In this case, the chunk size can be set to the average defect size the production line.
2
2
5
4X4
2
2
S
5
7
2X2
1X1
2
2
5
5
7
Figure 9: The hand-written digits and explanations provided by VIBI using various chunk sizes. The same examples with Figure are shown. The selected patches are colored red if the pixel is activated (i.e. white) and yellow otherwise (i.e. black). A patch composed of 4 x 4, 2 2, 1 x 1 pixels is used as a cognitive chunk.
E.3 CHOICE OF APPROXIMATOR
Different approximators affect fidelity and interpretability of the explanation, which makes sense both intuitively and theoretically. In theory, explanation is obtained via solving the problem in Equation 2 where y(t, represents the approximator. Different models of the approximator result in different (t,,y) and optimizing the information bottleneck objective amounts learning decent approximator. Intuitively, different models of the approximator indeed affect the quality as well as evaluated fidelity of the explanation. If the model of approximator has low capacity, I(t. y) and fidelity tend to be low. To remedy this, explainer has to generate less brief explanations; vice versa.
In addition to the approximator capacity, the chunk sizes, more precisely chunk types, can be an essential factor for choosing approximators. In the IMDB experiment, we investigated how the two approximators affect the fidelity and interpretability. As seen in Table the LSTM approximator has lower fidelity than the CNN approximator when use word for the chunk size, while both approximators achieve similar fidelity when we use sentence for the chunk size. When we use a sentence cognitive chunk, the LSTM achieves better interpretability than CNN, which is more substantial than when we use a word as a cognitive chunk. This may indicate that a sentence (as an explanation) should be processed with approximator that resembles the black-box model, while group of single words is relatively free from such a constraint. This may be because sentence is more similar to the original input (i.e., movie review one or two paragraphs) than group of single words.
Table 9: Fidelity and interpretability under different approximators.
Approximator Fidelity
Rationale Fidelity
Interpretability
chunk size
k
CNN
LSTM
CNN
LSTM
CNN
LSTM
sentence
87.7 + 0.6
87.8 + 1.0
73.1 + 0.8
71.7 + 2,4
45.1 49.4
65.9
word
5
74.4 + 0.8
61.3 + 3.0
65.7 + 0.8
57.1
+ 2.4
51.3
Prediction accuracy reported for all measures. Randomely selected instances (100 for each) were evaluated for each model. workers are assigned per instance.
In practice, we recommend choosing the approximator that matches the black-box model, especially if a cognitive chunk resembles the original input. We then recommend starting with an approximator with enough capacity and reducing as as the approximator achieves a target fidelity because large capacity usually makes it hard to leamn
xxiii


Reviews:
[{"review_id": "r1eN4QX8YB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Due to the rebuttal, my score was raised from a weak reject to a weak accept]\n\nSummary\nThe paper addresses the problem of interpreting predictions/decisions of a black-box classifier/regressor by masking the parts of the input that were most relevant. The proposed approach consists of, first, manually designing “cognitive chunks” of input data, e.g. individual words for sentiment classification or fixed-size image-patches for image-classification. Then, a variational IB framework is used to infer which of these chunks are relevant for the classifier’s decision. Additionally, there is a (hard) constraint, making sure that only a fixed (small) number of chunks is used. The bottleneck variable, in this case, is a sparse-chunk representation of the data. The latter is obviously a more compressed representation of the data, but importantly it is a more compressed representation that contains the largest possible amount of relevant information about the decision (because of properties of the information-bottleneck objective). Both factors together, according to the paper, constitute a “good” (i.e. brief but comprehensive) explanation which allows for interpretability and attribution of the black-box system’s decision. The method is evaluated on three tasks (sentiment prediction, image classification, TCR to epitope binding prediction) and performance is reported to be on-par or better than state-of-the-art methods.\n\nContributions\n-) Application of the IB-method for generating summaries of decision-relevant input-data, which are good candidates for interpretability. The theoretical properties of the IB objective are appealing for producing interpretable data-summaries.\n-) Adaptation of the variational IB framework, using bits and pieces reported in the literature such that the bottleneck variable is a sparse, binary vector over “cognitive chunks”.\n-) Experimental evaluation, where human judges rate the “interpretability“ of various state-of-the-art attribution methods.\n\nQuality, Clarity, Novelty, Impact\nThe paper addresses a timely and important problem, particularly the IB framework could add some solid theoretical footing (the “theory of relevant information”) to the field of interpetability methods. The paper is well written (though it needs another pass for typos, etc.), related methods and literature are discussed and compared against, and the specific variational IB objective is introduced nicely. Large parts of the method (deep variational IB, VI with categorical variables) have been published before, but these parts are combined in a novel and original way. My main issues with the current paper are (I) interpretability and comprehensiveness are not necessarily the same as maximum compression of maximally relevant information, (II) the method (in theory) depends strongly on the quality of the approximator, this is currently not mentioned and not explored, (III) the experimental section is currently not very strong, in particular the MNIST experiment. See more details for the main issues below. Overall, I personally think that the main idea of the paper is interesting, mature and fleshed out enough for a publication, however, the experimental section is somewhat lacking and (II) is missing from the current manuscript. While the method has theoretical advantages, empirically it seems to perform more or less equal to L2X (but the chunks produced seem qualitatively different which is interesting). I am therefore slightly leaning towards suggesting a major revision of the paper, but I am happy to be convinced otherwise by the other reviewers and the authors during discussion/rebuttal. (I would rate the paper as \"borderline\", but it seems that this year's review system only allows for \"weak reject\" or \"weak accept\", so I'll go for \"weak reject\" for now).\n\nImprovements / major issues\n\n(I) Good compression of highly relevant information is not (always) the same as good interpretability/comprehensiveness. In the limit, the bottleneck variable captures a minimal sufficient statistic, i.e. a maximally compressed version of all relevant information - for finite beta, the bottleneck approximates such a minimal sufficient statistic. From a theoretical point of view this is very appealing, since it is guaranteed to cover a maximal amount of information (given a certain level of compression). But the way this information is represented matters a lot for interpretability - any reversible mapping of the bottleneck variable does not change its information content but can have substantial effects on interpretability, e.g. consider encrypting or randomly perturbing elements of the explanation (i.e. the selected cognitive chunks). This is a major open problem, and some theoretical grounding in the IB framework helps by talking about this problem in very concrete terms. While I would not expect the paper to solve the problem in full generality, some discussion, and perhaps adding a “shortcomings” section would be nice.\n\n(II) The relevance of information is measured via I(t;y), which ultimately boils down to the approximation q(y|t). The quality of this approximation is crucial, which can of course be seen by how it influences the tightness of the bound. While I appreciate that the paper investigates the quality of the approximation to some degree (by inspecting the approximator fidelity), I would highly appreciate a thorough discussion of this issue (because ultimately the method will produce cognitive chunks that are relevant for q(y|t), not p(y|x) - the interpretations can be trusted only if q matches fairly well). It would be very interesting to see how quickly interpretability degrades with lower-quality q(y|t) - the latter would of course require more experiments with human “interpreters” which I would not expect to be easily feasible within the rebuttal period.\nAnother interesting experiment to test the match between q(y|t) and p(y|x) would be to “minimally intervene” on the input-chunks suggested by the method and see whether that actually affects the predictions of the black-box models. E.g. do small random perturbations to the selected cognitive chunks in the MNIST digits change the prediction of the black-box classifier? Compare this against small random perturbations in arbitrary chunks of the input.\n\n(III) Experimental section: I’m fairly happy with the IMDB experiment, and the TCR to Epitope binding is a nice non-standard application but I find the quality and significance of the results a bit hard to judge. My main concern though is the MNIST experiment: what I would have expected was the following: cognitive chunks are shown to participants and they need to guess the correct number (just like in the IMDB experiment). In the experiment reported in the paper, I’m afraid that there’s a certain bias for judges favoring explanations that lie on the digits rather than off digits. It remains unclear whether they simply prefer the chunks selected by VIBI over other methods, or whether they have actually gained more understanding of how the black-box makes decisions.\n\n(IV) Table 2 can easily be misleading because entries with highest mean-accuracy are marked in bold, regardless of whether confidence intervals overlap with other entries or not. Please fix this by either only marking entries in bold where the error bars don’t overlap with an entry in the same row, or marking all entries in bold that lie within the error bars of the best-performing entry. Particularly for “Approximate Fidelity” VIBI often does not perform significantly better than L2X but performs roughly equally well. Of course it would also help to run more repetitions to potentially shrink confidence intervals.\n\n(V) Please state the (parametric form) of the prior r(z*) used for the experiments. Also state the analytical expression for the KL-term in the final objective that this prior leads to.\n\nMinor comments\n\na) Please add some discussion on how the method depends on hand-crafting cognitive chunks, and how hard/easy this might be for different domains.\n\nb) Rather than fixing r(z), other papers have proposed to optimize the prior as well (typically in the context of VAEs / VIB) which is well justified from an IB perspective. It might be interesting to explore these possibilities for VIBI as well in the future.\n\n[1] Fixing a broken ELBO. Alemi et al. 2017\n[2] The beta-VAE’s Implicit Prior. Hoffman et al. 2017\n\nc) Instead of fixing the number of cognitive chunks in advance, it could also be interesting to infer that number as well (as a future extension of the method). This could either be achieved via a sparsity-inducing prior r(z), or perhaps by borrowing some ideas from the Deterministic IB [3], and its variational version.\n\n[3] The deterministic information bottleneck. Strouse and Schwab. 2016", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Ske-h3jRYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method to learn an explanation of black-box systems from its outputs. The method is based on the information bottleneck as the objective function is designed to measure mutual information between input x, system output y, and narrowed information of input t. t is constructed by filtering x with maintaining interpretability of y, so that it is finally assumed as the explanation of the system extracted by the proposed method.\n\nThe paper is well motivated and well written. Enough experiments were conducted to assess the advantage of the proposed method in the classification tasks. It looks a good paper.\n\nMaybe the paper is focused on only tasks that the predictor does not generate much information, such as classification. It is still unclear how the proposed method work when it is applied to the output-rich models, i.e., the model should keep as much information as inputs.\n\nThe proposed method automatically selects some important chunks from inputs, but the chunks still rely on some task-specific hand-crafted chunking strategies. The paper also conducted some experiments by changing the strategy, but it is still unclear what is the important criteria.\n\nIt is also good to show how actually the thickness of the bottleneck (controlled by k) works in actual cases, e.g., showing results for the same example with moving k.\n\nTrivial comments:\n* The example in 3.2 \"great, great\" and \"great, thought provoking\" looks still ambiguous to explain what the section want to say.\n* \"x_i \\times z_j\" in p.iv looks ambiguous.\n* z_j^* in 3rd eqn. of p.v should take l: z_j^{*(l)}\n* the max operator over l in 3rd eqn. of p.v looks to hide other values than the highest one (specifically, the L1 norm of z_j^* does not become k by this eqn. as the k-hot vector does). Summation looks intuitively better than max. Could you explain how this eqn. was constructed?\n* f(.) in the 4th eqn. of p.v may be undefined in the main text.\n* \\beta_1 of Adam looks to be set to not a standard value (0.9). Is there any reason?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeiT5Vg5r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\"an information theoretic principle, information bottleneck principle\" in the abstract is quite redundant with the use of 'principle' twice\n\n'\"great, great\" and \"great, thought provoking\". They have the same level of sparsity.' What kind of sparsity are you referring to with this example? Why can't sparsity reduce semantic redundancy? Please explain further.\n\n\"However, the first explanation has a large MI with the input document where \"great\" occurs a lot.\" What example input document are you referring to?\n\nYou should save the explanation of how your method in Equation 2 differs from the original information bottleneck of Equation 1 until after you have actually written out Equation 2. As it is now, you are referencing Equation 2 before it has been seen. \n\nI find Equation 2 confusing. Is it possible to make the dependence of the expression on z more explicit. It isn't clear from the equation itself how p(z|x) influences either quantity in Equation 2. Perhaps you should wait to introduce this equation until you have first explained how z relates to x and t. As it is, z is not clearly defined. I can gather information about it from the figure, from how you describe the difference in your method from the original information bottleneck, but the relationship is not clear enough by reading only the text of the paper before Equation 2 is presented.\n\nCan you explain briefly how your \"hierarchical LSTM\" works in the main text of the paper? Its an unusual enough term that I would want to see a citation or brief explanation right away rather than having it deferred to the Appendix. Why not use a state-of-the-art model for IMDb? Are you not using the standard splits for IMDb? the\n\nIn Appendix B.1 \"output vector is averaged and followed by log-softmax calculation. The final layer is formed to return a log-probability indicating which cognitive chunks should be taken as an input to the approximator\" The single output vector of the biLSTM is averaged and followed by log-softmax? the final layer is formed? What does this mean?\n\nI find the phrasing of \"Negative Sentiment if any negative words\" and the corresponding title for positive in Fig 2 confusing. What do you mean by \"if any\"? The phrasing makes it sound like the prediction of the model somehow depends on a logical step based on whether there are any negative/positive words found.\n\nI find the lack of a comparison to some kind of attentional method somewhat glaring in the IMDb example, since I would expect that many classifiers with attention would simply attend to the same words. What does your method give us that attention would not?\n\nThe same can be said for the MNIST example regarding an attention map.\n\n\"by the human intelligences\" sounds quite robotic\n\nCan you provide some sense of inter annotator agreement for labeling the images and sentences?\n\nIt does seem that there is key information like the definition of approximator fidelity in the appendices which is crucial to actually understanding the paper.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a system-agnostic interpretable method based on the idea of that provides a brief (=compressed) but comprehensive (=informative) explanation. Their system is build upon the idea of VIB. The authors compare against 3 state-of-the-art interpretable machine learning methods and the evaluation is terms of interpretability (=human understandable) and fidelity (=accuracy of approximating black-box model). Overall, all reviewers agreed that the topic of model interpretability is an important one and the novel connection between IB and interpretable data-summaries is a very natural one.  

This manuscript has generated a lot of discussion among the reviewers during the rebuttal and there are a number of concerns that are currently preventing me from recommending this paper for acceptance. The first concern relates to the lack of comparison against attention methods (I agree with the authors that this is a model-specific solution whereas they propose a model-agnostic one), however attention is currently the elephant in room and the first thing someone thinks of when thinking of interpretability. As such, the authors should have presented such a comparison. The second concern relates to the human evaluation protocol which could be significantly improved  (Why 100 samples from all models but 200 for VIBI? Given the small set of results, are these model differences significant? Similarly, assuming that we have multiple annotations per sample, what is the variance in the annotations?).

This paper is currently borderline and given reviewers' concerns and the limited space in the conference program I cannot recommend acceptance of this paper. 
Decision: Reject