Title: EXPLAINING A BLACK-BOX BY USING A DEEP VARI-ATIONAL INFORMATION BOTTLENECK APPROACH

Abstract: Interpretable machine learning has gained much attention recently. Briefness and comprehensiveness are necessary in order to provide a large amount of information concisely when explaining a black-box decision system. However, existing interpretable machine learning methods fail to consider briefness and comprehensiveness simultaneously, leading to redundant explanations. We propose the variational information bottleneck for interpretation, VIBI, a system-agnostic interpretable method that provides a brief but comprehensive explanation. VIBI adopts an information theoretic principle, information bottleneck principle, as a criterion for finding such explanations. For each instance, VIBI selects key features that are maximally compressed about an input (briefness), and informative about a decision made by a black-box system on that input (comprehensive). We evaluate VIBI on three datasets and compare with state-of-the-art interpretable machine learning methods in terms of both interpretability and fidelity evaluated by human and quantitative metrics.

1. INTRODUCTION:

Interpretability is crucial in building and deploying black-box decision systems such as deep learning models. Interpretation of a black-box system helps decide whether or not to follow its decisions, or understand the logic behind the system. In recent years, the extensive use of deep learning black-box systems has given rise to interpretable machine learning approaches (Lipton, 2016; Doshi-Velez & Kim, 2017) , which aim to explain how black-box systems work or why they reach certain decisions. In order to provide sufficient information while avoiding redundancy when explaining a black-box decision, we need to consider both briefness and comprehensiveness. However, existing approaches lack in-depth consideration for and fail to find both brief but comprehensive explanation.

In order to obtain brief but comprehensive explanation, we adopt the information bottleneck principle (Tishby et al., 2000) . This principle provides an appealing information theoretic perspective for learning supervised models by defining what we mean by a 'good' representation. The principle says that the optimal model transmits as much information as possible from its input to its output through a compressed representation called the information bottleneck. Then, the information bottleneck will maximally compress the mutual information (MI) with an input while preserving as much as possible MI with the output. Recently, it has been shown that the principle also applies to deep neural networks and each layer of a deep neural network can work as an information bottleneck (Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017) . Using this idea of information bottleneck principle, we define a brief but comprehensive explanation as maximally informative about the black-box decision while compressive about a given input.

In this paper, we introduce the variational information bottleneck for interpretation (VIBI), a systemagnostic information bottleneck model that provides a brief but comprehensive explanation for every single decision made by a black-box model. VIBI is composed of two parts: explainer and approximator, each of which is modeled by a deep neural network. The explainer returns a probability whether a chunk of features such as a word, phrase, sentence or a group of pixels will be selected as an explanation or not for each instance, and an approximator mimics behaviour of a black-box model. Using the information bottleneck principle, we learn an explainer that favors brief explanations while enforcing that the explanations alone suffice for accurate approximations to a black-box model. i Under review as a conference paper at ICLR 2020 1.1 CONTRIBUTION Our main contribution is to provide a new framework that systematically defines and generates a 'good' (i.e. brief but comprehensive) explanation using the information bottleneck principle. Based on this principle, we develop VIBI that favors a brief but comprehensive explanation. In order to make the objective function of VIBI tractable, we derive a variational approximation to the objective.

The benefits of our method are as follows. 1) System-agnostic: VIBI can be applied to explain any black-box system. 2) Post-hoc learning: VIBI is learned in a post-hoc manner, hence there is no tradeoff between task accuracy of a black-box system and interpretability of an explainer. 3) Cognitive chunk: Cognitive chunk is defined as a group of raw features whose identity is understandable to human. VIBI groups non-cognitive raw features such as a pixel and letter into a cognitive chunk (e.g. a group of pixels, a word, a phrase, a sentence) and selects each unit as an explanation. 4) Separate explainer and approximator: The explainer and approximator are designed for separated tasks so that we do not need to limit the approximator to have a simple structure, which may reduce the fidelity (the ability to imitate the behaviour of a black-box) of approximator.

2. RELATED WORK:

Most prior interpretable machine learning methods have been focusing on local interpretation, which implies knowing the reasons why a black-box system makes a certain decision at a very local point of interest, and our work is also situated in this line. Existing methods can be categorized into system-specific and system-agnostic method. System-specific methods only explain certain black-box decision systems (e.g. using backpropagation algorithm, or having CNN structure), while system-agnostic methods explain any black-box decision systems.

System-specific methods. To measure a change of output with respect to changes of input is an intuitive way of obtaining feature attribution for the output. Using this idea, Zeiler & Fergus (2014) , and Zintgraf et al. (2017) observe how outputs change when they make perturbations to each instance. Baehrens et al. (2010) ; Simonyan et al. (2013) , and Smilkov et al. (2017) use computationally more efficient approaches; they measure change of output by propagating contributions through layers of a deep neural network towards an input than the perturbation. However, these approaches fail to detect the changes of output when the prediction function is flattened at the instance (Shrikumar et al., 2017) , which leads to interpretations focusing on irrelevant features. In order to solve this problem, the layer-wise relevance propagation (Bach et al., 2015; Binder et al., 2016) , DeepLIFT (Shrikumar et al., 2017) , and Integrated Gradients (Sundararajan et al., 2017) compare the changes of output to its reference output. Ross et al. (2017) learn a more generalizable model as well as desirable explanations by constraining its explanations (i.e. input gradient) to match domain knowledge.

System-agnostic methods. The great advantage of system-agnostic interpretable machine learning methods over system-specific methods is that their usage is not restricted to a specific black-box system. One of the most well-known system-agnostic methods is LIME (Ribeiro et al., 2016) . It explains the decision of an instance by locally approximating the black-box decision boundary around the instance with an inherently interpretable model such as sparse linear or decision trees. The approximator is learned by samples generated by perturbing a given instance. Lundberg & Lee (2017) proposed SHAP, a unified measure defined over the additive feature attribution scores in order to achieve local accuracy, missingness, and consistency. L2X (Chen et al., 2018) learns a stochastic map that selects instance-wise features that are most informative for black-box decisions. Unlike LIME and SHAP, which approximate local behaviors of a black-box system with a simple (linear) model, L2X does not put a limit on the structure of the approximator; hence it avoids losing fidelity of the approximator. As SHAP does, Dabkowski & Gal (2017); Fong & Vedaldi (2017), and Petsiuk et al. (2018) use sample perturbation but they rather learn or estimate desired perturbation masks than using perturbed samples to learn an approximator. Dabkowski & Gal (2017) , and Fong & Vedaldi (2017) learn a smallest perturbation mask that alters black-box outputs as much as possible. Petsiuk et al. (2018) empirically estimate feature attribution as a sum of random masks weighted by class scores corresponding to masked inputs. Our method VIBI is similar with L2X in that both learn a stochastic explainer that returns a distribution over the subset of features given the input and performs instance-wise feature selection based on that. However, given the same number of explanations, our explainer favors both briefness and comprehensiveness while the L2X explainer favors comprehensiveness of the explanation and does not account for briefness. The explainer selects a group of k key cognitive chunks given an instance while the approximator mimics the behaviour of the black-box system using the selected keys as the input. (B) We set each word as a cognitive chunk and k = 2. 1 The explainer takes an input x and returns a stochastic k-hot random vector z which indicates whether each cognitive chunk will be selected as an explanation or not. 2 t(x) provides instance-specific explanation. 3 The approximator takes t(x) as an input and approximates the black-box output.

p(z | x) t = x ⊙ z q(y | t) = q(y | x ⊙ z)

3.1. PERSPECTIVE FROM INFORMATION BOTTLENECK PRINCIPLE:

The information bottleneck principle (Tishby et al., 2000) provides an appealing information theoretic view for learning a supervised model by defining what we mean by a 'good' representation. The principle says that the optimal model transmits as much information as possible from the input x to the output y through a compressed representation t (called the information bottleneck). The representation t is stochastically defined and the optimal stochastic mapping p(t|x) is obtained by optimizing the following problem with a Markov chain assumption y → x → t:

EQUATION

where I(•, •) is the MI and β is a Lagrange multiplier representing the trade-off between the compressiveness -I(x, t) and informativeness I(t, y) of the representation t.

We adopt the information bottleneck principle as a criterion for finding brief but comprehensive explanations. Our aim is to learn an explainer generating explanations that are maximally informative about the black-box decision while compressive about a given input.

3.2. PROPOSED APPROACH:

We introduce VIBI, a system-agnostic interpretation approach that provides brief but comprehensive explanations for decisions made by black-box decision system. In order to achieve this, we optimize the following information bottleneck objective.

EQUATION

where I(t, y) represents the sufficiency of information retained for explaining the black-box output y, -I(x, t) represents the briefness of the explanation t, and β is a Lagrange multiplier representing a trade-off between the two. The primary difference between our information bottleneck objective (2) and the one in Tishby et al. (2000) is as follows: the latter aims to identify a stochastic map of the representation t that itself works as an information bottleneck, whereas our objective aims to identify a stochastic map of z performing instance-wise selection of cognitive chunks and define information bottleneck as a function of z and the input x.

As illustrated in Figure 1A , VIBI is composed of two parts: the explainer and the approximator, each of which is modeled by a deep neural network. The explainer selects a group of k key cognitive chunks given an instance while the approximator mimics the behaviour of the black-box system using the selected keys as the input. k controls the level of sparsity in z. In detail, the explainer p(z|x; θ e ) is a map from an input x to its attribution scores p j (x) = p(z j |x) where j is for the j-th cognitive chunk and z j is a binary indicator whether the chunk will be selected or not. The attribution score indicates the probability that each cognitive chunk to be selected. In order to select top k cognitive chunks as an explanation, a k-hot vector z is sampled from a categorical distribution with class probabilities p j (x) = p(z j |x) and the j-th cognitive chunk is selected if z j = 1. More specifically, the explanation t is defined as follows:

t i = (x z) i = x i × z j ,

where j indicates a cognitive chunk, each of which corresponds to multiple row features i. The approximator is modeled by another deep neural network p(y|t; θ a ), which mimics the black-box decision system. It takes t as an input and returns an output approximating the black-box output for the instance x. θ a and θ e represent the weight parameters of neural networks. The explainer and approximator are trained jointly by minimizing a cost function that favors concise explanations while enforcing that the explanations alone suffice for accurate prediction.

To achieve compressiveness, in addition to encouraging small MI between explanations and inputs, we also encourage the number of selected cognitive chunks to be small, i.e., encouraging z to be sparse. Note that MI and sparsity are two complementary approaches for achieving compression. MI aims at reducing semantic redundancy in explanations. Sparsity cannot achieve such a goal. For example, consider a movie review where "great" occurs a lot and two explanations in judging the sentiment of the review: "great, great" and "great, thought-provoking". They have the same level of sparsity (k = 2), but the former has semantic redundancy. In this case, MI helps to choose a better explanation. The first explanation has a larger MI with the input document. The second explanation has smaller MI and hence is more brief and preferable.

3.2.1. THE VARIATIONAL BOUND:

The current form of information bottleneck objective is intractable due to the MIs I(t, y) and I(x, t).

We address this problem by using a variational approximation of our information bottleneck objective.

In this section, we summarize the results and refer to Supplementary Material A for details.

Variational bound for I(x, t): We first show that I(x, t) ≤ I(x, z) + C where C is constant and use the lower bound for -I(x, z) -C as a lower bound for -I(x, t). As a result, we obtain:

I(x, t) ≤ I(x, z) + C ≤ E (x,z)∼p(x,z) log p(z|x) r(z) + C = E x∼p(x) D KL (p(z|x), r(z)) + C (3)

Note that with proper choices of r(z) and p(z|x), we can assume that the Kullback-Leibler divergence D KL (p(z|x), r(z)) has an analytical form.

Variational bound for I(t, y): We obtain the lower bound for I(t, y) by using q(y|t) to approximate p(y|t), which works as an approximator to the black-box system. As a result, we obtain:

EQUATION

where p(t|x, y) = p(t|x) by the Markov chain assumption y ↔ x ↔ t.

Combining Equations ( 3) and (4), we obtain the following variational bound:

EQUATION

)

where C * = -Cβ can be ignored since it is independent of the optimization procedure. We use the empirical data distribution to approximate p(x, y) = p(x)p(y|x) and p(x).

3.2.2. CONTINUOUS RELAXATION AND REPARAMETERIZATION:

Current form of the bound ( 5) is still intractable because we need to sum over the d k combinations of feature subsets. This is because we sample top k out of d cognitive chunks where each chunk is assumed drawn from a categorical distribution with class probabilities p j (x) = p(z j |x). In order to avoid this, we use the generalized Gumbel-softmax trick (Jang et al., 2017; Chen et al., 2018) . This is a well-known technique that are used to approximate a non-differentiable categorical subset sampling with differentiable Gumbel-softmax samples. The steps are as follows.

First, we independently sample a cognitive chunk for k times. For each time, a random perturbation e j is added to the log probability of each cognitive chunk log p j (x). From this, Concrete random vector c = (c 1 , • • • , c d ) working as a continuous, differentiable approximation to argmax is defined: g j = -log (-log e j ) where e j ∼ U (0, 1)

c j = exp g j + log p j (x) /τ d j=1 exp g j + log p j (x) /τ

, where τ is a tuning parameter for the temperature of Gumbel-Softmax distribution. Next, we define a continuous-relaxed random vector

z * = [z * 1 , • • • , z * d ]

as the element-wise maximum of the independently sampled Concrete vectors c (l) where l = 1, • • • , k:

z * j = max l c (l) j for l = 1, • • • , k

With this sampling scheme, we approximate the k-hot random vector and have the continuous approximation to the variational bound ( 5). This trick allows using standard backpropagation to compute the gradients of the parameters via reparameterization.

By putting everything together, we obtain:

1 N L N n L l log q(y (n) |x (n) f (e (l) (n) , x (n) )) -β D KL (p(z * (n) |x (n) ), r(z * (n) ))

where N is the number of samples, n indicate the n-th sample, f (e

(l) (n) , x (n) ) = z * (n) , q(y (n) |x (n) z * (n) )

is the approximator to the black-box system and -D KL (p(z * |x (n) ), r(z * )) represents the compactness of the explanation. Once we learn the model, the attribution score p j (x) for each cognitive chunk is used to select top k key cognitive chunks that are maximally compressive about the input x and informative about the black-box decision y on that input.

4. EXPERIMENTS:

We evaluated VIBI on three datasets and compared with state-of-the-art interpretable machine learning methods. The evaluation is performed from two perspectives: interpretability and fidelity. The interpretability indicates the ability to explain a black-box model with human understandable terms. The fidelity implies how accurately our approximator approximates the black-box model. Based on these criteria, we compared VIBI with three state-of-the-art system-agnostic methods (LIME (Ribeiro et al., 2016) , SHAP (Lundberg & Lee, 2017) and L2X (Chen et al., 2018) ), and a commonly used model-specific method called Saliency Map (Simonyan et al., 2013) . For Saliency Map, we used the smooth gradient technique (Smilkov et al., 2017) to get visually sharp gradientbased sensitivity maps over the basic gradient saliency map. See Supplementary Material B for further experimental details.

We examined how VIBI performs across different experimental settings varying the number of selected chunks k (amount or number of explanation), size of chunk (unit of explanation), and tradeoff parameter β (trade-off between the compressiveness of explanation and information preserved about the output). The settings of hyperparameter tuning include (bold indicate the choice for our final model): the temperature for Gumbel-softmax approximation τ -{0.1, 0.2, 0.5, 0.7, 1}, learning rate -5 × 10 -3 , 10 -3 , 5 × 10 -4 , 10 -4 , 5 × 10 -5 } and β -{0, 0.001, 0.01, 0.1, 1, 10, 100}. We use Adam algorithm (Kingma & Ba, 2014) with batch size 100 for MNIST and 50 for IMDB, the coefficients used for computing running averages of gradient and its square (β 1 , β 2 ) = (0.5, 0.999), and = 10 -8 . We tuned the hyperparameters via grid search and picked up the hyperparameters that yield the best fidelity score on the validation set. The code is publicly available on GitHub https://github.com/XXX.1 v Under review as a conference paper at ICLR 2020 Negative Sentiment if any negative words are found I do NOT understand why anyone would waste their time or money on utter trash like this … Don't get me wrong --I LOVE a good Western --Notice I said "GOOD" --this is just trash. The acting is horrible --Val Kilmer must know someone or owed a favor or something for them just to use his face and name in this ridiculous piece of crap...

True: Negative / B-Box: Negative:

I watched this movie when it was released and being really young and not too much into cinema it was one of the most fascinating cinematic experiences I ever had and it really left a mark inside me. At first I didn't quite understand the story and probably failed to … He plays so well the man that falls in love slowly but so deeply with Katherine Clifton, opens up his heart and dives into this prohibited affair….

Positive Sentiment if any positive words are found:



True: Positive / B-Box: Positive:

The reality of the mafia environment is absolutely dog-eat-dog where a gangster will be killed for showing any sign of weakness because they become a liability. I've got no problem with the human side of gansters' being portrayed but Bugsy steers too far in the direction of soft, comical, men. The film is enjoyable but is only light entertainment and not a biopic of a man who, though exciting, was extremely dangerous and fearsome. The acting's all good and the direction very solid. The locations and era are very well represented and the themes very interesting….

Negative sentiment but predicted as Positive because several positive words are found 

4.1. LSTM MOVIE SENTIMENT PREDICTION MODEL USING IMDB:

The IMDB (Maas et al., 2011) is a large text dataset containing movie reviews labeled by sentiment (positive/negative). We grouped the reviews into training, validation, and test sets, which have 25,000, 12,500, and 12,500 reviews respectively. Then, we trained a hierarchical LSTM for sentiment prediction, which has two LSTM layers where each layer encodes words and sentences respectively. It achieved 87% of test accuracy. In order to explain this LSTM black-box model, we applied VIBI. We parameterized the explainer using a bidirectional LSTM and approximator using a 2D CNN. For the details of the black-box model and VIBI architectures, see Supplementary Material B.1. VIBI explains why the LSTM predicts each movie review to be positive/negative and provides instance-wise key words that are the most important attributes to the sentiment prediction. As seen in the top-right and top-left of Figure 2 , VIBI shows that the positive (or negative) words pass through the bottleneck and make a correct prediction. The bottom of Figure 2 shows that the LSTM sentiment prediction model makes a wrong prediction for a negative review because the review includes several positive words such as 'enjoyable' and 'exciting'.

4.2. CNN DIGIT RECOGNITION MODEL USING MNIST:

Same digit with different angles. Different digits, upside down Difference between 7 and 1 Same digit similar features The MNIST (LeCun et al., 1998 ) is a large dataset contains 28 × 28 sized images of handwritten digits (0 to 9). We grouped the images into training, validation, and test sets, which have 50,000, 10,000, and 10,000 images respectively, and trained a simple 2D CNN for the digit recognition, which achieved 97% of test accuracy. In order to explain this CNN black-box model, we applied VIBI. We parameterized each the explainer and approximator using a 2D CNN. For the details of the black-box model and VIBI architectures, see Supplementary Material B.2.

VIBI explains how the CNN characterizes a digit and recognizes differences between digits. The first two examples in Figure 3 show that the CNN recognizes digits using both shapes and angles. In the first example, the CNN characterizes '1's by straightly aligned patches along with the activated regions although '1's in the left and right panels are written at different angles. Contrary to the first example, the second example shows that the CNN recognizes the difference between '9' and '6' by their differences in angles. The last two examples in Figure 3 show that the CNN catches a difference of '7's from '1's by patches located on the activated horizontal line on '7' (see the cyan circle) and recognizes '8's by two patches on the top of the digits and another two patches at the bottom circle.

More qualitative examples for VIBI and the baselines are shown in Supplementary Figure 6 .

The briefness of explanations also depends on the sparsity k. Supplementary Figure 8 shows how our method works under different sparsity. When we increase k, VIBI tends to select patches that are the same with or nearby previously selected patches and additionally select patches that catch new characteristics of digits. We next illustrate how VIBI can be used to get insights from a model and ensure the safety of a model in a real world application. Identifying which T-cell receptor (TCR) will bind to a specific epitope (i.e. cancer induced peptide molecules presented by the major histocompatibility complex to T-cells) is important for screening T-cells or genetically engineering T-cells that are effective in recognizing and destroying tumor cells. Therefore, there has been efforts in developing computational methods to predict binding affinity of given TCR-epitope pairs (Jurtz et al., 2018; Jokinen et al., 2019) . These approaches rely on known interacting TCR-epitope pairs available from VDJdb (Shugay et al., 2017) and IEDB (Vita et al., 2014) , which are the largest databases of several thousand entries. However, the number of unique TCRs harbored in a single individual is estimated to be 10 10 ( Lythe et al., 2016) and a theoretical number of epitopes of length l is 20 l , which are much larger than the number of known interacting TCR-epitope pairs. One of the main concerns is whether a black-box model trained on such limited dataset can accurately predict TCR-epitope bindings of out-of-samples. This concern becomes pressing in a TCR-epitope binding prediction model trained on VDJdb (For details of the data, black-box model architecture, and parameter tuning, see Supplementary Material B.3). The model accurately predicted the (in-sample) bindings from VDJdb (recall 0.79, Figure 4A ). However, it achieved poor prediction performance when it is used to predict the (out-of-sample) bindings from another dataset, IEDB (recall 0.40, Figure 4B ). In an attempt to address this problem, we applied VIBI and determined whether or not to accept a decision made by the black-box model based on VIBI's explanation. As illustrated in Figure 4E , VIBI provided matched explanations-the identical amino acids in same positions (S and Y in this example) are highlighted in different TCR sequences when they are bound to the same epitope (GILGFVFTL in this example). Moreover, we found that if two TCR sequences binding to the same epitope, each from IEDB and VDJdb, are assigned with matched explanations by VIBI, then it significantly better predicts the binding than the others with no matching TCRs (Figure 4C-D , p-values are shown). Therefore, if a TCR sequence from IEDB has a matched explanation to a TCR from VDJdb, then we safely follow the positive decision made by the black-box model.

C IN G V C W T V 1 .9 × 1 0 -5 2 .2 × 1 0 -16 1 .2 × 1 0 -7 0.0 0.2 0.4 0.6 0.8 1.0 2 .2 × 1 0 -16 3 .1 × 1 0 -15 0 .4 1 G IL G F V F T L G LC T LV A M L LL W N G P M A V N LV P M V AT V Y V LD H LI V V All Epitopes Black-box

4.4. INTERPRETABILITY EVALUATED BY HUMANS:

We evaluated interpretability of the methods on the LSTM movie sentiment prediction model and the CNN digit recognition model. For the movie sentiment prediction model, we provided instances that the black-box model had correctly predicted and asked humans to infer the output of the primary sentiment of the movie review (Positive/Negative/Neutral) given five key words selected by each method. Each method was evaluated by the humans on MTurk2 who are awarded the Masters Qualification, high-performance workers who have demonstrated excellence across a wide range of tasks). We randomly selected and evaluated 200 instances for VIBI and 100 instances for the others. Five workers were assigned per instance. For the digit recognition model, we asked humans to directly score the explanation on a 0-5 scale. Each method was evaluated by 16 graduate students at XXX-University3 who have taken at least one graduate-level machine learning class. For each method, 100 instances were randomly selected and evaluated. Four cognitive chunks with the size 4 × 4 were provided as an explanation for each instance (β = 0.1 for VIBI). On average, 4.26 students were assigned per instance. Further details regarding the experiments can be found in Supplementary Material C. For IMDB, the percentage indicates how well the MTurk worker's answers match the black-box output. For MNIST, the score indicates how well the highlighted chunks catch key characteristics of handwritten digits.

The average scores over all samples is shown on a 0 to 5 scale. See the survey example and detailed result in Supplementary Material Tables 3 and 4 for the detailed result.

As shown by the Table 1 , VIBI better explains the black-box models. When explaining the movie sentiment prediction model, humans better inferred the (correctly predicted) black-box output given the five keywords when they were provided by VIBI. Therefore, it better captures the most contributing key words to the LSTM decision and better explains why the LSTM predicted each movie review by providing five key words. For explaining the digit recognition model, VIBI also highlighted the most concise chunks for explaining key characteristics of handwritten digit. Thus, it better explains how the CNN model recognized each the handwritten digit. 6 90.6 ± 0.6 63.7 ± 0.9 93.6 ± 0.5 95.7 ± 0.4 95.6 ± 0.4 51.1 ± 1.0 70.1 ± 0.9 4 × 4 10 94.9 ± 0.4 70.5 ± 0.9 95.1 ± 0.4 96.5 ± 0.4 96.7 ± 0.4 83.5 ± 0.7 93.3 ± 0.5

4.5. FIDELITY:

Note that the black-box models achieved 87% accuracy for IMDB and 97% accuracy for MNIST. β = 0.1 for VIBI. Accuracy and 0.95 confidence interval is shown. We performed three runs for each method and reported the best results. See more evaluations using F1-score and further results from different parameter settings in Supplementary Material Table 6 and 8 for approximator fidelity and Table 5 and 7 for rationale fidelity.

We assessed fidelity of the methods in approximating the black-box output. First, we compared the ability of the approximators to imitate behaviour of the black-box, denoted as Approximator fidelity.

(See Supplementary Material B.4 for details about how each approximator fidelity is evaluated.) As shown in Table 2 , VIBI has a better approximator fidelity than Saliency, LIME and SHAP in most cases. VIBI and L2X showed similar levels of approximator fidelity, so we further compared them based on Rationale fidelity. The difference between approximator and rationale fidelity is as follows.

Approximator fidelity is quantified by prediction performance of the approximators that takes t * , the continuous relaxation of t, as an input and the black-box output as a targeted label; rationale fidelity is quantified by using t instead of t * . Note that t only takes the top k chunks and sets the others to be zero, while t * sets the others to be small, non-zero values. Therefore, rationale fidelity allows to evaluate how much information purely flows through the explanations, not through a narrow crack made during the continuous relaxation procedure. As shown in Table 2 , VIBI has a better rationale fidelity than L2X in most cases. Note that L2X can be viewed as a special case of VIBI without the compressiveness term, i.e., β = 0. The rationale fidelity empirically demonstrates that the compressiveness term can help the information to flow purely through the explanations.

5. CONCLUSION:

We employ the information bottleneck principle as a criterion for learning 'good' explanations. Instance-wisely selected cognitive chunks work as an information bottleneck, hence, provide concise but comprehensive explanations for each decision made by a black-box system. The information bottleneck framework provides a theoretical background that the bottleneck captures a minimal sufficient statistic, i.e. the most compressed representation that captures all the possible (i.e. sufficient) amount of information about output. For finite β, the bottleneck approximates such a minimal sufficient statistic.

However, the way this information is represented may have a substantial effect on interpretability. VIBI helps to address this issue to some extent by always returning a certain form of output (i.e., a k-hot vector z assigned to each chunk) and having a certain form of the information bottleneck layer (i.e., a masked input) so that it makes sure that the explanations are easily understandable to humans. In practice, such a chunking strategy leads to a deviation from the strict theory that a 'good' explanation is the most compressed one but helps to achieve better interpretability in practice.

A VARIATIONAL INFERENCE TO CONSTRUCT A LOWER BOUND ON THE IB OBJECTIVE:

We illustrate the variational approximation of the following information bottleneck objective: p(t|x) = arg max p(t|x),p(y|t),p(t)

I(t, y) -β I(x, t)

Let us start with the Markov chain assumption y → x → t, which also implies y ← x ← t. Therefore, the condition is written as y ↔ x ↔ t (Cover & Thomas, 2012) . Using this condition, we factorize the joint distribution p(y, x, t): p(y, x, t) = p(x)p(t|x)p(y|t, x) = p(x)p(t|x)p(y|t)

where p(y|t, x) = p(y|t) by the Markov chain. Now, we will examine each of the expressions in the information bottleneck objective I(t, y)β I(x, t) in turn.

(I) VARIATIONAL LOWER BOUND FOR I(x, t)

We first show that I(x, t) ≤ I(x, z) + C where C is a constant and then use the lower bound for -I(x, z) -C as a lower bound for -I(x, t).

First, we prove I(x, t) ≤ I(x, z) + C. From the Markov Chain x → (x, z) → t, we have I(x, t) ≤ I(x, (x, z)). According to the chain rule for mutual information, I(x, (x, z)) = I(x, z) + I(x, x|z), where I(x, x|z) = H(x|z) + H(x|z) -H(x, x|z). Further, H(x|z) ≤ H(x). Putting these pieces together, we have I(x, t) ≤ I(x, z) + H(x) (6) where entropy H(x) of input is a constant. For simplicity, we denote it as C.

We then approximate p(z) using r(z). From the fact that Kullback Leibler divergence is always positive, we have

EQUATION

xi From ( 6) and ( 7), we have

I(x, t) ≤ I(x, z) + C = E (x,z)∼p(x,z) log p(z|x) p(z) + C ≤ E (x,z)∼p(x,z) log p(z|x) r(z) + C = E x∼p(x) D KL (p(z|x), r(z)) + C.

(II) VARIATIONAL LOWER BOUND FOR I(t, y)

Starting with I(t, y), we have Note that H(y) = -E y∼p(y) [log p(y)] is independent of the optimization procedure, hence can be ignored. Now, we use q(y|t) to approximate p(y|t) which works as an approximator to the black-box system. Using the fact that Kullback Leibler divergence is always positive, we have

I(t, y) = E (t,

E y|t∼p(y|t) [log p(y|t)] ≥ E y|t∼p(y|t) [log q(y|t)]

and

I(t, y) ≥ E (t,y)∼p(t,y) [log q(y|t)] = E x∼p(x) E y|x∼p(y|x) E t|x,y∼p(t|x,y) [log q(y|t)] = E x∼p(x) E y|x∼p(y|x) E t|x∼p(t|x) [log q(y|t)]

where p(t|x, y) = p(t|x) by the Markov chain assumption y ↔ x ↔ t.

(III) VARIATIONAL LOWER BOUND FOR I(t, y) -βI(x, t)

In summary, we have the following variational bounds for each term.

I(t, y) ≥ E x∼p(x) E y|x∼p(y|x) E t|x∼p(t|x) [log q(y|t)] I(x, t) ≤ E x∼p(x) D KL (p(z|x), r(z)) + C which result in I(t, y) -β I(x, t) ≥ E x∼p(x) E y|x∼p(y|x) E t|x∼p(t|x) [log q(y|t)] -β E x∼p(x) D KL (p(z|x), r(z)) + Const.

With proper choices of r(z) and p(z|x), we assume that the Kullback-Leibler divergence D KL (p(z|x), r(z)) is integrated analytically. We use the empirical data distribution to approximate p(x, y) = p(x)p(y|x) and p(x).

Our variational approximation is similar to the one from Alemi et al. (2017) , which first developed variational lower bound on the information bottleneck for deep neural networks. However, our information bottleneck is different from theirs: their information bottleneck is the stochastic encoding z of the input x as itself, whereas our information bottleneck is a pairwise product of the stochastic encoding z and the input x where z is a Boolean random vector. Due to this difference, we appproximate I(x, t) by using the lower bound of I(x, z), instead of directly deriving the lower bound of I(x, t).

B EXPERIMENTAL DETAILS:

The settings of hyperparameter tuning include the followings (bold indicate the choice for our final model): the temperature for Gumbel-softmax approximation τ -{0.1, 0.2, 0.5, 0.7, 1}, learning rate -5 × 10 -3 , 10 -3 , 5 × 10 -4 , 10 -4 , 5 × 10 -5 } and β -{0, 0.001, 0.01, 0.1, 1, 10, 100}. We use Adam algorithm (Kingma & Ba, 2014) with batch size 100 for MNIST and 50 for IMDB, the coefficients used for computing running averages of gradient and its square (β 1 , β 2 ) = (0.5, 0.999), and = 10 -8 . We tuned the hyperparameters via grid search and picked up the hyperparameters that yield the best fidelity score on the validation set. All implementation is performed via PyTorch an open source deep learning platform (Paszke et al., 2017) .

Approximator fidelity is prediction performance of the approximator that takes relaxation t * as an input and black-box output as a target. In detail, we get a t * with the Gumble-softmax sampling and make a prediction based on t * . This procedure is repeated for 12 times and the final prediction is made by averaging the 12 prediction scores. Rationale fidelity is prediction performance of the approximator that takes t as an input and black-box output as a target. (Note that t is a masked input that only takes the top k chunks and set others to zero.) The final prediction is made by its prediction score. We use the prior r j (z j ) = (k/J) zj (1 -k/J) 1-zj for all experiments where J is the total number of chunks and r(z) = j r j (z j ). The analytical form of the Kullback-Leibler divergence term is

D KL (p(z|x), r(z)) = J j=1 D KL (p(z j |x), r j (z j )) = J j=1 1 zj =0 p(z j |x) (log p(z j |x) -log r j (z j )).

B.1 LSTM MOVIE SENTIMENT PREDICTION MODEL USING IMDB:

Black-Box Model Structure. Each review is padded or cut to contain 15 sentences and 50 words for each sentence. The architecture consists of a word-embedding layer with size 50 for each word followed by two bidirectional LSTMs, a fully connected layer with two units, and a soft-max layer. The first LSTM layer encodes the word embedding vector and generates a word-representation vector with size 100 for each word. Within each sentence, the word representation vectors are elementwisely averaged to form a size 100 sentence representation vector. The second LSTM layer encodes the sentence representation vector and generates a size 60 review embedding vector. VIBI Structure. We parameterize the explainer with a bidirectional LSTM and approximator with a 2D CNN. For the explainer, we use a bidirectional layer that returns multiple output vectors, each of which corresponds to a recurrent unit. Each element in the output vectors are averaged over all units, and then the averaged output vector is followed by log-softmax calculation. As a result, the explainer returns a vector of log-probabilities, each of which indicates whether or not each cognitive chunk will be selected as an input to the approximator. For the approximator, we use a convolutional layer followed by a ReLU activation function and max-pooling layer and a fully connected layer returning a size-2 vector followed by a log-softmax calculation. The final layer returns a vector of log-probabilities for the two sentiments (positive/negative).

B.2 CNN DIGIT RECOGNITION MODEL USING MNIST:

Black-Box Model Structure. The architecture consists of two convolutional layers with the kernel size 5 followed by a max-pooling layer with the pool size 2, two fully connected layers and a soft-max layer. The two convolutional layers contain 10 and 20 filters respectively and the two fully connected layers are composed of 50 and 10 units respectively. VIBI Structure. We parameterize each the explainer and approximator using 2D CNNs. Structure of the explainer differs depending on the chunk size. For example, when 4 × 4 cognitive chunk is used, we use two convolutional layers with the kernel size 5 followed by a ReLU activation function and max-pooling layer with the pool size 2, and one convolutional layer with kernel size 1 returning a 7 × 7 2D matrix followed by a log-softmax calculation. The final layer returns a vector of log-probabilities for the 49 chunks. The three convolutional layers contains 8, 16, and 1 filters respectively. The output from the explainer indicates which cognitive chunks should be taken as an input for the approximator. We parameterize the approximator using two convolutional layers with kernel size 5 followed by a ReLU activation function and max-pooling layer with pool size 2 and with 32 and 64 filters respectively, and one fully connected layer returning a size-10 vector followed by a log-softmax calculation so that the final layer returns a vector of log-probabilities for the ten digits.

B.3 TCR TO EPITOPE BINDING PREDICTION MODEL USING VDJDB AND IEDB:

Data. We use two public datasets: VDJdb and IEDB. VDJdb (Shugay et al., 2017) contains the T-cell receptor (TCR) sequences with known antigen specificity (i.e., epitope sequences). We use a VDJdb dataset preprocessed by Jokinen et al. (2019) (See their paper for details in data preprocessing). The preprocessed dataset contains 5,784 samples (2,892 positively and 2,892 negatively binding pairs of TCR and epitope). This dataset consists of 4,363 unique TCR sequences and 21 unique epitope sequences. We group the pairs of TCR and epitope sequences into training, validation, and test sets, which have 4,627, 578, and 579 pairs, respectively. IEDB (Vita et al., 2014) contains TCR sequences and corresponding epitopes. We used an IEDB dataset preprocessed by Jurtz et al. (2018) . The preprocessed dataset contains 9,328 samples (positively binding pairs only) and consists of 9,221 unique TCR sequences and 98 unique epitopes sequences. We used the IEDB dataset as out-ouf-samples. There are 6 epitopes contained in both VDJdb and IEDB dataset: GILGFVFTL, GLCTLVAML, NLVPMVATV, LLWNGPMAV, YVLDHLIVV, CINGVCWTV.

Black-Box Model Structure. Each TCR and epitope sequence is padded or cut to contain 20 and 13 amino acids, respectively. The embedding matrix has a size 24 and is initialized with BLOSUM50 matrix (Henikoff & Henikoff, 1992) . The architecture consists of two sequence encoders that process TCR and epitope sequences each, and three dense layers that process the encoded sequences together. The TCR encoder consists of a dropout layers with the drop-out probability 0.3, two convolutional layers with the kernel size 3 followed by a batch normalization layer, a ReLU activation function and max-pooling layer with the pool size 3. The two convolutional layers contain 32 and 16 filters respectively. Structure of the epitope encoder is the same with the one from the TCR encoder. We optimized the models with the following search space (bold indicate the choice for our final model): the batch size -{25, 50, 100}, learning rate -0.05, 0.01, 0.005, 0.001, 0.0005} and filter size -{(16, 8), (32, 16)}.

VIBI Structure. We parameterize the explainer and approximator using 2D CNNs and several dense layers. Each TCR and epitope sequence is preprocessed and embedded in the same way as for the black-box model. We have two types of explainers: one for TCR sequence and another for epitope sequence. Each explainer encodes both TCR and epitope sequences and concatenates them through three dense layers followed by a ReLU activation function. For the TCR explainer, the three dense layers contain 32, 16, 20 hidden-units, respectively. For the epitope explainer, the three dense layers contain 32, 16, 13 hidden-units, respectively. The TCR and epitope encoders have the same architectures with those of the black-box model. Each explainer then returns a vector of log-probabilities that indicate which peptides in TCR or epitope should be selected as an explanation. The approximator has the same architecture as the black-box model. We optimize the models with the following search space (bold indicate the choice for our final model): the batch size -{25, 50, 100}, learning rate -0.0001, 0.001, 0.01, 0.1}.

B.4 BASELINE METHODS:

Hyperparameter tuning. For L2X, we used the same hyperparameter search space of VIBI. For LIME, we tuned the segmentation filter size over {1, 2, 4} for MNIST and {10, 25} for IMDB. For all methods, we tuned the hyperparameters via grid search and picked up the hyperparameters that yield the best fidelity score on the validation set.

Cognitive chunk. LIME, SHAP and Saliency yield an attribution score for each feature. A chunk-attribution score is an average of (absolute) attribution scores of features that belong to the chunk. Top k chunks that have the highest chunk-attribution scores are selected. L2X selects chunks in the same way as VIBI.

Fidelity evaluation. LIME, SHAP, Saliency, and L2X all have their own approximators. LIME and SHAP have a sparse linear approximator for each black-box instance (See Section 3.4 in Ribeiro et al. (2016) , and Equation (3) in Lundberg & Lee (2017)). Saliency has a 1st-order Taylor approximation to each black-box instance (See Equation (3) in Simonyan et al. (2013) ). The approximator fidelity of LIME, SHAP, and Saliency is calculated using their proposed approximators that are composed of the selected features. L2X has a deep neural network that approximates a black-box model (See Section 4 in Chen et al. (2018) ). The approximator and rationale fidelity of L2X is calculated in the same way as VIBI. The percentage of samples belongs to each combination of the black-box output and the sentiment recognized by workers at Amazon Mechanical Turk (https://www.mturk. com/) are showed

We evaluated interpretability of the methods on the LSTM movie sentiment prediction model. The interpretable machine learning methods were evaluated by workers at Amazon Mechanical Turk (https://www.mturk.com/) who are awarded the Masters Qualification (i.e. high performance workers who have demonstrated excellence across a wide range of task). Randomly selected instances (200 for VIBI and 100 for the others) were evaluated for each method. 5 workers are assigned per instance.

We provided instances that the black-box model had correctly predicted and asked humans to infer the output of the primary sentiment of the movie review (Positive/Negative/Neutral) given five key words selected by each method. See the survey example below for further details. Note that this is a proxy for measuring how well humans infer the black-box output given explanations; we use such proxy because the workers are general public who are not familiar with the term 'black-box' or 'output of the model.'

In Table 3 The average scores (0-5 scale) evaluated by graduate students at XXX University (Blinded due to the double blinding reviewing) are showed.

We evaluated interpretability of the methods on the CNN digit recognition model. The interpretable machine learning methods were evaluated by 16 graduate students at XXX University (Blinded due to the double blinding reviewing) who have taken at least one graduate-level machine learning class. Randomly selected 100 instances were evaluated for each method. On average, 4.26 students are assigned per instance. See the survey example below for further details. For the digit recognition model, we asked humans to directly score the explanation on a 0-5 scale.

In Table 4 , the average score per digit is showed. VIBI outperforms L2X and LIME and slightly outperforms Saliency in terms of the average score over digits. VIBI outperforms at digit 2, 3, 4, 6, 7, and 9, and performs comparable to Saliency at 0, 1, 5, 8. The compressiveness of explanations depends on the sparsity k (i.e., the number of cognitive chunks to be selected). A larger k allows the information bottleneck to convey more information about output, but it gives less compressive explanations than a smaller k. For deciding k, we recommend choosing the minimum possible k that achieves a target fidelity because an unnecessarily large k can make redundant explanations. Figure 8 shows how VIBI works under different sparsity k. When we increase k, VIBI tends to select chunks that are the same or nearby the previously selected chunks and additionally select new chunks that catch another characteristics of digits. The characteristics that are caught at k = 4 tend to be caught again at a larger ks. 

E.2 CHOICE OF CHUNK SIZE:

The chunk size changes the way the information is represented in the information bottleneck, which can affect interpretability. For choosing the chunk size, a greater fidelity does not always mean better interpretability. For example, in the MNIST experiment, the models with the chunk size 1 × 1 have achieved better fidelity than those with 2 × 2 and 4 × 4 under the same number of pixels that are used for the information bottleneck, but 1 × 1 is less recognizable than 2 × 2 or 4 × 4 (Figure 9 ).

Choosing the chunk size is not trivial, and the strategy varies for different domains (except the case that each (raw) feature itself is cognitive (e.g., age, gender) so that the chunk size should be 1).

Obviously, the chunk size should be larger than the size of the smallest cognitive unit. For example, in the IMDB experiment, the smallest cognitive unit was set to 50 because 50 raw features compose a word. In our sentiment prediction task, using a word as the chunk size can be enough to explain the decision (i.e., positive/negative sentiment); to infer the sentiment, we do not need to know a whole sentence. However, we may need to use a phrase or sentence as the chunk size for complex tasks such as disease diagnosis or patient need detection since decisions on such tasks are made based on contextual information. In the MNIST experiment, we set the chunk size to be 4 × 4 because the xxii Under review as a conference paper at ICLR 2020 digits' stroke width is about to 4. However, we may need to use different chunk sizes, for example, for automated visual inspection for identifying defects on a factory production line. In this case, the chunk size can be set to the average defect size on the production line. 

E.3 CHOICE OF APPROXIMATOR:

Different approximators affect fidelity and interpretability of the explanation, which makes sense both intuitively and theoretically. In theory, explanation is obtained via solving the problem in Equation ( 2) where I(t, y) represents the approximator. Different models of the approximator result in different I(t, y) and optimizing the information bottleneck objective amounts to learning a decent approximator. Intuitively, different models of the approximator indeed affect the quality as well as evaluated fidelity of the explanation. If the model of approximator has low capacity, I(t, y) and fidelity tend to be low. To remedy this, explainer has to generate less brief explanations; vice versa.

In addition to the approximator capacity, the chunk sizes, more precisely chunk types, can be an essential factor for choosing approximators. In the IMDB experiment, we investigated how the two approximators affect the fidelity and interpretability. As seen in Table 9 , the LSTM approximator has lower fidelity than the CNN approximator when we use a word for the chunk size, while both approximators achieve similar fidelity when we use a sentence for the chunk size. When we use a sentence as a cognitive chunk, the LSTM achieves better interpretability than CNN, which is more substantial than when we use a word as a cognitive chunk. This may indicate that a sentence (as an explanation) should be processed with an approximator that resembles the black-box model, while a group of single words is relatively free from such a constraint. This may be because a sentence is more similar to the original input (i.e., movie review -one or two paragraphs) than a group of single words. Prediction accuracy is reported for all measures. Randomely selected instances (100 for each) were evaluated for each model. 5 workers are assigned per instance.

In practice, we recommend choosing the approximator that matches the black-box model, especially if a cognitive chunk resembles the original input. We then recommend starting with an approximator with enough capacity and reducing it as far as the approximator achieves a target fidelity because a large capacity usually makes it hard to learn.

xxiii

:

Blinded due to the double-blind reviewing. vi

Amazon Mechanical Turk, https://www.mturk.com/

Blinded due to the double-blinding reviewing viii

Figures and Tables:

Figure 1: Illustration of VIBI. (A) VIBI is composed of two parts: the explainer and approximator.The explainer selects a group of k key cognitive chunks given an instance while the approximator mimics the behaviour of the black-box system using the selected keys as the input. (B) We set each word as a cognitive chunk and k = 2. 1 The explainer takes an input x and returns a stochastic k-hot random vector z which indicates whether each cognitive chunk will be selected as an explanation or not. 2 t(x) provides instance-specific explanation.3 The approximator takes t(x) as an input and approximates the black-box output.

Figure2: The movie reviews and explanations provided by VIBI were randomly selected from the validation set. The selected words are colored red. Each word is used as a cognitive chunk and k = 5 words are provided for each review.

Figure3: The hand-written digits and explanations provided by VIBI were randomly selected from the validation set. The selected patches are colored red if the pixel is activated (i.e. white) and yellow otherwise (i.e. black). A patch composed of 4 × 4 pixels is used as a cognitive chunk and k = 4 patches are identified for each image.

4.3 TCR TO EPITOPE BINDING PREDICTION MODEL USING VDJDB AND IEDBVDJdb

Figure 4: Black-box prediction scores of (A) VDJdb and (B) IEDB. (C) Black-box prediction scores between the matched and unmatched instances from IEDB and (D) those by six epitope sequences. (E) An example of matched explanation (The VIBI selected amino acids are shaded).

Figure 6: The hand-written digits and explanations provided by VIBI and the baselines. The examples are randomly selected from the validation set. The selected patches are colored red if the pixel is activated (i.e. white) and yellow otherwise (i.e. black). A patch composed of 4 × 4 pixels is used as a cognitive chunk and k = 4 patches are identified for each image.

Figure 8: The hand-written digits and explanations provided by VIBI using various number of chunks (k). The examples are randomly selected from the validation set. The selected patches are colored red if the pixel is activated (i.e. white) and yellow otherwise (i.e. black). A patch composed of 4 × 4 pixels is used as a cognitive chunk and k = 4, 6, 10, 20 patches are identified for each image.

Figure 9: The hand-written digits and explanations provided by VIBI using various chunk sizes. The same examples with Figure 8 are shown. The selected patches are colored red if the pixel is activated (i.e. white) and yellow otherwise (i.e. black). A patch composed of 4 × 4, 2 × 2, 1 × 1 pixels is used as a cognitive chunk.

Evaluation of interpretability

Table content: <table><tr><td/><td colspan="2">Saliency LIME</td><td>L2X</td><td>VIBI (Ours)</td></tr><tr><td>IMDB</td><td>34.2%</td><td colspan="2">33.8% 35.6%</td><td>44.7%</td></tr><tr><td>MNIST</td><td>3.448</td><td>1.369</td><td>1.936</td><td>3.526</td></tr></table>

Evaluation of approximator and rationale fidelity

Table content: <table><tr><td/><td/><td/><td/><td/><td colspan="2">Approximator Fidelity</td><td/><td>Rationale Fidelity</td></tr><tr><td/><td colspan="2">chunk size k</td><td>Saliency</td><td>LIME</td><td>SHAP</td><td>L2X</td><td>VIBI (Ours)</td><td>L2X</td><td>VIBI (Ours)</td></tr><tr><td/><td>sentence</td><td colspan="5">1 38.7 ± 0.9 72.7 ± 0.8 49.5 ± 1.0 87.6 ± 0.6</td><td>87.7 ± 0.6</td><td>72.7 ± 0.8</td><td>73.1 ± 0.8</td></tr><tr><td>IMDB</td><td>word 5 words</td><td colspan="5">5 41.9 ± 0.9 75.6 ± 0.8 50.1 ± 1.0 73.8 ± 0.8 1 42.4 ± 0.9 29.0 ± 0.8 49.7 ± 1.0 75.9 ± 0.7</td><td>74.4 ± 0.8 76.4 ± 0.7</td><td>63.8 ± 0.8 60.1 ± 0.9</td><td>65.7 ± 0.8 63.2 ± 0.8</td></tr><tr><td/><td>5 words</td><td colspan="5">3 41.4 ± 0.9 67.9 ± 0.8 49.1 ± 1.0 83.3 ± 0.7</td><td>83.5 ± 0.7</td><td>69.4 ± 0.8</td><td>66.0 ± 0.8</td></tr><tr><td/><td>2 × 2</td><td colspan="5">16 91.2 ± 0.6 77.0 ± 0.8 94.2 ± 0.5 93.4 ± 0.5</td><td>94.8 ± 0.4</td><td>73.5 ± 0.9</td><td>77.1 ± 0.8</td></tr><tr><td/><td>2 × 2</td><td colspan="5">24 93.8 ± 0.5 80.7 ± 0.8 95.4 ± 0.4 95.1 ± 0.4</td><td>95.3 ± 0.4</td><td>77.6 ± 0.8</td><td>85.6 ± 0.7</td></tr><tr><td>MNIST</td><td>2 × 2 4 × 4</td><td colspan="5">40 95.7 ± 0.4 85.9 ± 0.7 95.4 ± 0.4 96.7 ± 0.4 4 86.3 ± 0.7 60.9 ± 1.0 94.8 ± 0.4 95.3 ± 0.4</td><td>96.2 ± 0.4 94.8 ± 0.4</td><td>81.1 ± 0.8 65.0 ± 0.9</td><td>91.5 ± 0.5 77.5 ± 0.8</td></tr><tr><td/><td>4 × 4</td><td/><td/><td/><td/><td/><td/></tr></table>

Evaluation of Interpretability on an LSTM movie sentiment prediction model using IMDB.

Table content: <table><tr><td colspan="3">C INTERPRETABILITY EVALUATED BY HUMANS</td><td/><td/><td/></tr><tr><td colspan="5">C.1 LSTM MOVIE SENTIMENT PREDICTION MODEL USING IMDB DATASET</td><td/></tr><tr><td>EVALUATION</td><td/><td/><td/><td/><td/></tr><tr><td>Black-Box Output</td><td>Recognized by Mturk worker</td><td colspan="4">Saliency LIME L2X VIBI (Ours)</td></tr><tr><td>Positive</td><td>Positive</td><td>19.8</td><td>17.4</td><td>17.6</td><td>24.3</td></tr><tr><td>Positive</td><td>Negative</td><td>12.6</td><td>6.8</td><td>7.2</td><td>16.9</td></tr><tr><td>Positive</td><td>Neutral</td><td>25.6</td><td>18.8</td><td>24.2</td><td>11.1</td></tr><tr><td>Negative</td><td>Positive</td><td>11.0</td><td>10.6</td><td>11.6</td><td>16.2</td></tr><tr><td>Negative</td><td>Negative</td><td>14.4</td><td>16.4</td><td>18.0</td><td>20.4</td></tr><tr><td>Negative</td><td>Neutral</td><td>16.6</td><td>30.0</td><td>21.4</td><td>11.2</td></tr></table>

, the percentage of samples belongs to each combination of the black-box output and the sentiment recognized by the workers are showed. VIBI has the highest percentage of samples belonging to the Positive/Positive or Negative/Negataive and the lowest percentage of samples belonging to the Positive/Neutral or Negative/Neutral. LIME has the lowest percentage of samples belonging to the Positive/Negative or Negative/Positive, but it is because LIME tends to select words such as 'that', 'the', 'is' so that most of samples are recognized as Neutral. SURVEY EXAMPLE FOR IMDB Title: Label sentiment given a few words. Description: Recognize the primary sentiment of the movie review given a few words only.Figure 5: A survey example of MTurk evaluation on the LSTM movie sentiment prediction model C.2 CNN DIGIT RECOGNITION MODEL USING MNIST EVALUATION Evaluation of Interpretability on a 2d CNN digit recognition model using MNIST.

Table content: <table><tr><td>MNIST Digit</td><td colspan="3">Saliency LIME L2X VIBI (Ours)</td></tr><tr><td>0</td><td>3.200</td><td>2.000 2.333</td><td>3.000</td></tr><tr><td>1</td><td>4.393</td><td>0.795 1.263</td><td>3.913</td></tr><tr><td>2</td><td>3.125</td><td>1.200 1.400</td><td>3.200</td></tr><tr><td>3</td><td>3.286</td><td>1.833 2.429</td><td>3.625</td></tr><tr><td>4</td><td>3.333</td><td>1.000 1.857</td><td>3.857</td></tr><tr><td>5</td><td>3.167</td><td>1.381 2.000</td><td>2.875</td></tr><tr><td>6</td><td>3.333</td><td>1.000 1.889</td><td>3.625</td></tr><tr><td>7</td><td>3.667</td><td>2.000 1.667</td><td>4.000</td></tr><tr><td>8</td><td>3.750</td><td>1.333 2.667</td><td>3.500</td></tr><tr><td>9</td><td>3.222</td><td>1.143 1.857</td><td>3.667</td></tr><tr><td>Ave. over digits</td><td>3.448</td><td>1.369 1.936</td><td>3.526</td></tr></table>

Evaluation of rationale fidelity on LSTM movie sentiment prediction model using IMDB.Rationale fidelity quantifies ability of the selected chunks to infer the black-box output. A large rationale fidelity implies that the selected chunks account for a large portion of the approximator fidelity. Prediction accuracy and F1-score of the approximator for the CNN model are shown. (β = 0, 0.001, 0.01, 0.1, 1, 10, 100)

Table content: <table><tr><td/><td/><td/><td>L2X</td><td/><td colspan="2">VIBI (Ours)</td></tr><tr><td/><td colspan="2">chunk size k</td><td>0</td><td>0.001 0.01</td><td>0.1</td><td>1</td><td>10</td><td>100</td></tr><tr><td/><td>sentence</td><td colspan="6">1 0.727 0.693 0.711 0.731 0.729 0.734 0.734</td></tr><tr><td>Accuracy</td><td>word 5 words</td><td colspan="6">5 0.638 0.657 0.666 0.657 0.648 0.640 0.654 1 0.601 0.630 0.624 0.632 0.628 0.623 0.628</td></tr><tr><td/><td>5 words</td><td colspan="6">3 0.694 0.660 0.662 0.660 0.662 0.660 0.660</td></tr><tr><td/><td>sentence</td><td colspan="6">1 0.581 0.547 0.562 0.567 0.585 0.586 0.586</td></tr><tr><td>F1-score</td><td>word 5 words</td><td colspan="6">5 0.486 0.521 0.551 0.512 0.516 0.508 0.526 1 0.478 0.506 0.500 0.540 0.501 0.498 0.504</td></tr><tr><td/><td>5 words</td><td colspan="6">3 0.551 0.528 0.529 0.522 0.529 0.528 0.525</td></tr></table>

Evaluation of approximator fidelity on LSTM movie sentiment prediction model using IMDB.

Table content: <table><tr><td/><td/><td/><td colspan="2">Saliency LIME SHAP L2X</td><td/><td colspan="2">VIBI (Ours)</td></tr><tr><td/><td colspan="2">chunk size k</td><td/><td>0</td><td>0.001 0.01</td><td>0.1</td><td>1</td><td>10</td><td>100</td></tr><tr><td/><td>sentence</td><td>1</td><td>0.387</td><td colspan="4">0.727 0.495 0.876 0.877 0.869 0.877 0.879 0.879 0.884</td></tr><tr><td>Accuracy</td><td>word 5 words</td><td>5 1</td><td>0.419 0.424</td><td colspan="4">0.756 0.501 0.738 0.766 0.772 0.744 0.773 0.763 0.767 0.290 0.496 0.759 0.784 0.780 0.764 0.774 0.778 0.774</td></tr><tr><td/><td>5 words</td><td>3</td><td>0.414</td><td colspan="4">0.679 0.491 0.833 0.836 0.831 0.835 0.834 0.830 0.833</td></tr><tr><td/><td>sentence</td><td>1</td><td>0.331</td><td colspan="4">0.564 0.400 0.721 0.693 0.707 0.730 0.730 0.727 0.734</td></tr><tr><td>F1-score</td><td>word 5 words</td><td>5 1</td><td>0.350 0.360</td><td colspan="4">0.585 0.413 0.565 0.607 0.616 0.594 0.620 0.609 0.612 0.302 0.418 0.621 0.641 0.622 0.624 0.615 0.622 0.616</td></tr><tr><td/><td>5 words</td><td>3</td><td>0.352</td><td colspan="4">0.523 0.409 0.680 0.683 0.674 0.681 0.677 0.669 0.682</td></tr><tr><td colspan="8">Approximator fidelity quantifies ability of the approximator to imitate the behaviour of a black-box. Prediction accuracy and F1-score</td></tr><tr><td colspan="6">of the approximator for the LSTM model are shown. (β = 0, 0.001, 0.01, 0.1, 1, 10, 100)</td><td/></tr></table>

Evaluation of the rationale fidelity on CNN digit recognition model using MNIST.

Table content: <table><tr><td/><td/><td/><td>L2X</td><td/><td colspan="2">VIBI (Ours)</td><td/></tr><tr><td/><td>chunk size</td><td>k</td><td>0</td><td>0.001 0.01</td><td>0.1</td><td>1</td><td>10</td><td>100</td></tr><tr><td/><td>1 × 1</td><td colspan="7">64 0.694 0.690 0.726 0.689 0.742 0.729 0.766</td></tr><tr><td/><td>1 × 1</td><td colspan="7">96 0.814 0.831 0.780 0.806 0.859 0.765 0.826</td></tr><tr><td/><td>1 × 1</td><td colspan="7">160 0.903 0.907 0.905 0.917 0.917 0.928 0.902</td></tr><tr><td/><td>2 × 2</td><td colspan="7">16 0.735 0.795 0.750 0.771 0.732 0.753 0.769</td></tr><tr><td/><td>2 × 2</td><td colspan="7">24 0.776 0.855 0.834 0.856 0.868 0.854 0.847</td></tr><tr><td>Accuracy</td><td>2 × 2</td><td colspan="7">40 0.811 0.914 0.915 0.903 0.918 0.935</td></tr><tr><td/><td>2 × 2</td><td colspan="7">80 0.905 0.949 0.940 0.939 0.962 0.941 0.923</td></tr><tr><td/><td>4 × 4</td><td>4</td><td colspan="6">0.650 0.655 0.650 0.775 0.717 0.682 0.681</td></tr><tr><td/><td>4 × 4</td><td>6</td><td colspan="6">0.511 0.858 0.706 0.701 0.708 0.690 0.730</td></tr><tr><td/><td>4 × 4</td><td colspan="7">10 0.835 0.835 0.824 0.933 0.875 0.854 0.782</td></tr><tr><td/><td>4 × 4</td><td colspan="7">20 0.954 0.962 0.815 0.934 0.929 0.946 0.943</td></tr><tr><td/><td>1 × 1</td><td colspan="7">64 0.684 0.679 0.716 0.670 0.734 0.710 0.755</td></tr><tr><td/><td>1 × 1</td><td colspan="7">96 0.808 0.825 0.750 0.803 0.854 0.750 0.820</td></tr><tr><td/><td>1 × 1</td><td colspan="7">160 0.898 0.902 0.899 0.912 0.913 0.924 0.897</td></tr><tr><td/><td>2 × 2</td><td colspan="7">16 0.720 0.786 0.738 0.761 0.723 0.744 0.769</td></tr><tr><td/><td>2 × 2</td><td colspan="7">24 0.766 0.848 0.836 0.851 0.858 0.859 0.840</td></tr><tr><td>F1-score</td><td>2 × 2</td><td colspan="7">40 0.798 0.914 0.910 0.910 0.898 0.914 0.931</td></tr><tr><td/><td>2 × 2</td><td colspan="7">80 0.901 0.946 0.936 0.930 0.959 0.938 0.918</td></tr><tr><td/><td>4 × 4</td><td>4</td><td colspan="6">0.634 0.658 0.637 0.763 0.704 0.671 0.669</td></tr><tr><td/><td>4 × 4</td><td>6</td><td colspan="6">0.493 0.852 0.693 0.687 0.692 0.675 0.720</td></tr><tr><td/><td>4 × 4</td><td colspan="7">10 0.828 0.827 0.816 0.928 0.869 0.849 0.773</td></tr><tr><td/><td>4 × 4</td><td colspan="7">20 0.950 0.959 0.806 0.931 0.926 0.942 0.940</td></tr><tr><td colspan="9">Rationale fidelity quantifies ability of the selected chunks to infer the black-box output. A large</td></tr><tr><td colspan="9">rationale fidelity implies that the selected chunks account for a large portion of the approximator</td></tr><tr><td colspan="9">fidelity. Prediction accuracy and F1-score of the approximator for the CNN model are shown. (β =</td></tr><tr><td colspan="3">0, 0.001, 0.01, 0.1, 1, 10, 100)</td><td/><td/><td/><td/><td/></tr></table>

Evaluation of the approximator fidelity on CNN digit recognition model using MNIST.

Table content: <table><tr><td/><td/><td/><td colspan="2">Saliency LIME SHAP L2X</td><td/><td colspan="2">VIBI (Ours)</td></tr><tr><td/><td>chunk size</td><td>k</td><td/><td>0</td><td>0.001 0.01</td><td>0.1</td><td>1</td><td>10</td><td>100</td></tr><tr><td/><td>1 × 1</td><td>64</td><td>0.944</td><td colspan="5">0.982 0.955 0.933 0.959 0.962 0.959 0.960 0.952 0.953</td></tr><tr><td/><td>1 × 1</td><td>96</td><td>0.956</td><td colspan="5">0.986 0.950 0.963 0.963 0.951 0.967 0.968 0.953 0.962</td></tr><tr><td/><td>1 × 1</td><td>160</td><td>0.964</td><td colspan="5">0.989 0.958 0.970 0.967 0.973 0.974 0.974 0.974 0.967</td></tr><tr><td/><td>2 × 2</td><td>16</td><td>0.912</td><td colspan="5">0.770 0.942 0.934 0.945 0.941 0.948 0.938 0.939 0.940</td></tr><tr><td/><td>2 × 2</td><td>24</td><td>0.938</td><td colspan="5">0.807 0.954 0.951 0.956 0.955 0.953 0.953 0.953 0.960</td></tr><tr><td>Accuracy</td><td>2 × 2</td><td>40</td><td>0.957</td><td colspan="5">0.859 0.954 0.967 0.965 0.966 0.962 0.967 0.965 0.967</td></tr><tr><td/><td>2 × 2</td><td>80</td><td>0.966</td><td colspan="5">0.897 0.966 0.976 0.977 0.974 0.972 0.977 0.971 0.973</td></tr><tr><td/><td>4 × 4</td><td>4</td><td>0.863</td><td colspan="5">0.609 0.948 0.953 0.922 0.928 0.948 0.942 0.942 0.953</td></tr><tr><td/><td>4 × 4</td><td>6</td><td>0.906</td><td colspan="5">0.637 0.936 0.957 0.963 0.954 0.956 0.953 0.963 0.962</td></tr><tr><td/><td>4 × 4</td><td>10</td><td>0.949</td><td colspan="5">0.705 0.951 0.965 0.971 0.959 0.967 0.961 0.969 0.964</td></tr><tr><td/><td>4 × 4</td><td>20</td><td>0.963</td><td colspan="5">0.771 0.955 0.974 0.977 0.975 0.975 0.973 0.975 0.974</td></tr><tr><td/><td>1 × 1</td><td>64</td><td>0.938</td><td colspan="5">0.981 0.952 0.930 0.956 0.960 0.956 0.957 0.950 0.950</td></tr><tr><td/><td>1 × 1</td><td>96</td><td>0.950</td><td colspan="5">0.985 0.948 0.961 0.961 0.954 0.965 0.966 0.951 0.960</td></tr><tr><td/><td>1 × 1</td><td>160</td><td>0.959</td><td colspan="5">0.989 0.954 0.969 0.965 0.971 0.973 0.972 0.972 0.966</td></tr><tr><td/><td>2 × 2</td><td>16</td><td>0.902</td><td colspan="5">0.755 0.938 0.930 0.942 0.936 0.944 0.934 0.936 0.936</td></tr><tr><td/><td>2 × 2</td><td>24</td><td>0.932</td><td colspan="5">0.795 0.951 0.949 0.954 0.952 0.950 0.951 0.950 0.958</td></tr><tr><td>F1-score</td><td>2 × 2</td><td>40</td><td>0.952</td><td colspan="5">0.853 0.946 0.965 0.963 0.964 0.961 0.965 0.963 0.965</td></tr><tr><td/><td>2 × 2</td><td>80</td><td>0.962</td><td colspan="5">0.892 0.963 0.974 0.976 0.973 0.972 0.975 0.969 0.971</td></tr><tr><td/><td>4 × 4</td><td>4</td><td>0.849</td><td colspan="5">0.588 0.943 0.951 0.917 0.923 0.944 0.938 0.939 0.950</td></tr><tr><td/><td>4 × 4</td><td>6</td><td>0.895</td><td colspan="5">0.621 0.920 0.954 0.961 0.951 0.953 0.950 0.960 0.959</td></tr><tr><td/><td>4 × 4</td><td>10</td><td>0.943</td><td colspan="5">0.689 0.946 0.963 0.969 0.956 0.965 0.958 0.968 0.961</td></tr><tr><td/><td>4 × 4</td><td>20</td><td>0.959</td><td colspan="5">0.763 0.952 0.972 0.976 0.972 0.974 0.972 0.973 0.972</td></tr><tr><td/><td/><td/><td/><td>xxi</td><td/><td/><td/></tr></table>

Fidelity and interpretability under different approximators.

Table content: <table><tr><td/><td colspan="2">Approximator Fidelity</td><td colspan="2">Rationale Fidelity</td><td colspan="2">Interpretability</td></tr><tr><td>chunk size k</td><td>CNN</td><td>LSTM</td><td>CNN</td><td>LSTM</td><td colspan="2">CNN LSTM</td></tr><tr><td colspan="6">sentence 1 87.7 ± 0.6 87.8 ± 1.0 73.1 ± 0.8 71.7 ± 2.4 45.1</td><td>65.9</td></tr><tr><td colspan="6">word 5 74.4 ± 0.8 61.3 ± 3.0 65.7 ± 0.8 57.1 ± 2.4 49.4</td><td>51.3</td></tr></table>

Peer review (iclr2020):
[Due to the rebuttal, my score was raised from a weak reject to a weak accept] Summary The paper addresses the problem of interpreting predictions/decisions of a black-box classifier/regressor by masking the parts of the input that were most relevant. The proposed approach consists of, first, manually designing “cognitive chunks” of input data, e.g. individual words for sentiment classification or fixed-size image-patches for image-classification. Then, a variational IB framework is used to infer which of these chunks are relevant for the classifier’s decision. Additionally, there is a (hard) constraint, making sure that only a fixed (small) number of chunks is used. The bottleneck variable, in this case, is a sparse-chunk representation of the data. The latter is obviously a more compressed representation of the data, but importantly it is a more compressed representation that contains the largest possible amount of relevant information about the decision (because of properties of the information-bottleneck objective). Both factors together, according to the paper, constitute a “good” (i.e. brief but comprehensive) explanation which allows for interpretability and attribution of the black-box system’s decision. The method is evaluated on three tasks (sentiment prediction, image classification, TCR to epitope binding prediction) and performance is reported to be on-par or better than state-of-the-art methods. Contributions -) Application of the IB-method for generating summaries of decision-relevant input-data, which are good candidates for interpretability. The theoretical properties of the IB objective are appealing for producing interpretable data-summaries. -) Adaptation of the variational IB framework, using bits and pieces reported in the literature such that the bottleneck variable is a sparse, binary vector over “cognitive chunks”. -) Experimental evaluation, where human judges rate the “interpretability“ of various state-of-the-art attribution methods. Quality, Clarity, Novelty, Impact The paper addresses a timely and important problem, particularly the IB framework could add some solid theoretical footing (the “theory of relevant information”) to the field of interpetability methods. The paper is well written (though it needs another pass for typos, etc.), related methods and literature are discussed and compared against, and the specific variational IB objective is introduced nicely. Large parts of the method (deep variational IB, VI with categorical variables) have been published before, but these parts are combined in a novel and original way. My main issues with the current paper are (I) interpretability and comprehensiveness are not necessarily the same as maximum compression of maximally relevant information, (II) the method (in theory) depends strongly on the quality of the approximator, this is currently not mentioned and not explored, (III) the experimental section is currently not very strong, in particular the MNIST experiment. See more details for the main issues below. Overall, I personally think that the main idea of the paper is interesting, mature and fleshed out enough for a publication, however, the experimental section is somewhat lacking and (II) is missing from the current manuscript. While the method has theoretical advantages, empirically it seems to perform more or less equal to L2X (but the chunks produced seem qualitatively different which is interesting). I am therefore slightly leaning towards suggesting a major revision of the paper, but I am happy to be convinced otherwise by the other reviewers and the authors during discussion/rebuttal. (I would rate the paper as *borderline*, but it seems that this year*s review system only allows for *weak reject* or *weak accept*, so I*ll go for *weak reject* for now). Improvements / major issues (I) Good compression of highly relevant information is not (always) the same as good interpretability/comprehensiveness. In the limit, the bottleneck variable captures a minimal sufficient statistic, i.e. a maximally compressed version of all relevant information - for finite beta, the bottleneck approximates such a minimal sufficient statistic. From a theoretical point of view this is very appealing, since it is guaranteed to cover a maximal amount of information (given a certain level of compression). But the way this information is represented matters a lot for interpretability - any reversible mapping of the bottleneck variable does not change its information content but can have substantial effects on interpretability, e.g. consider encrypting or randomly perturbing elements of the explanation (i.e. the selected cognitive chunks). This is a major open problem, and some theoretical grounding in the IB framework helps by talking about this problem in very concrete terms. While I would not expect the paper to solve the problem in full generality, some discussion, and perhaps adding a “shortcomings” section would be nice. (II) The relevance of information is measured via I(t;y), which ultimately boils down to the approximation q(y|t). The quality of this approximation is crucial, which can of course be seen by how it influences the tightness of the bound. While I appreciate that the paper investigates the quality of the approximation to some degree (by inspecting the approximator fidelity), I would highly appreciate a thorough discussion of this issue (because ultimately the method will produce cognitive chunks that are relevant for q(y|t), not p(y|x) - the interpretations can be trusted only if q matches fairly well). It would be very interesting to see how quickly interpretability degrades with lower-quality q(y|t) - the latter would of course require more experiments with human “interpreters” which I would not expect to be easily feasible within the rebuttal period. Another interesting experiment to test the match between q(y|t) and p(y|x) would be to “minimally intervene” on the input-chunks suggested by the method and see whether that actually affects the predictions of the black-box models. E.g. do small random perturbations to the selected cognitive chunks in the MNIST digits change the prediction of the black-box classifier? Compare this against small random perturbations in arbitrary chunks of the input. (III) Experimental section: I’m fairly happy with the IMDB experiment, and the TCR to Epitope binding is a nice non-standard application but I find the quality and significance of the results a bit hard to judge. My main concern though is the MNIST experiment: what I would have expected was the following: cognitive chunks are shown to participants and they need to guess the correct number (just like in the IMDB experiment). In the experiment reported in the paper, I’m afraid that there’s a certain bias for judges favoring explanations that lie on the digits rather than off digits. It remains unclear whether they simply prefer the chunks selected by VIBI over other methods, or whether they have actually gained more understanding of how the black-box makes decisions. (IV) Table 2 can easily be misleading because entries with highest mean-accuracy are marked in bold, regardless of whether confidence intervals overlap with other entries or not. Please fix this by either only marking entries in bold where the error bars don’t overlap with an entry in the same row, or marking all entries in bold that lie within the error bars of the best-performing entry. Particularly for “Approximate Fidelity” VIBI often does not perform significantly better than L2X but performs roughly equally well. Of course it would also help to run more repetitions to potentially shrink confidence intervals. (V) Please state the (parametric form) of the prior r(z*) used for the experiments. Also state the analytical expression for the KL-term in the final objective that this prior leads to. Minor comments a) Please add some discussion on how the method depends on hand-crafting cognitive chunks, and how hard/easy this might be for different domains. b) Rather than fixing r(z), other papers have proposed to optimize the prior as well (typically in the context of VAEs / VIB) which is well justified from an IB perspective. It might be interesting to explore these possibilities for VIBI as well in the future. [1] Fixing a broken ELBO. Alemi et al. 2017 [2] The beta-VAE’s Implicit Prior. Hoffman et al. 2017 c) Instead of fixing the number of cognitive chunks in advance, it could also be interesting to infer that number as well (as a future extension of the method). This could either be achieved via a sparsity-inducing prior r(z), or perhaps by borrowing some ideas from the Deterministic IB [3], and its variational version. [3] The deterministic information bottleneck. Strouse and Schwab. 2016