MAGNEx: A Model Agnostic Global Neural Explainer
Authors: Nikolaos Manginas
Prodromos Malakasiotis
Eirini Spyropoulou
Ion Androutsopoulos
Georgios Paliouras
Venue: ICLR 2022 Submitted (2022)
Keywords: Explainability
Neural Explainer
Faithfullness
Global
Post-hoc

Abstract: Black-box decision models have been widely adopted both in industry and academia due to their excellent performance across many challenging tasks and domains. However, much criticism has been raised around modern AI systems, to a large extent due to their inability to produce explainable decisions that both their end-users and their developers can trust. The need for such decisions, i.e., decisions accompanied by a rationale for why they are made, has ignited much recent research. We propose MAGNEx, a global algorithm that leverages neural-network based explainers to produce rationales for any black-box decision model, neural or not. MAGNEx is model-agnostic, and thus easily generalizable across domains and applications. More importantly, MAGNEx is global, i.e., it learns to create rationales by optimizing for a number of instances at once, contrary to local methods that aim at explaining a single example. The global nature of MAGNEx has two advantages over local methods: i) it generalizes across instances hence producing more faithful explanations, ii) it is computationally more efficient during inference. Our experiments confirm that MAGNEx outperforms popular explainability algorithms both in explanation quality and in computational efficiency.

Under review as a conference paper at ICLR 2022
MAGNEX: A MODEL AGNOSTIC GLOBAL NEURAL EXPLAINER
Anonymous authors Paper under double-blind review
ABSTRACT
Black-box decision models have been widely adopted both in industry and academia due to their excellent performance across many challenging tasks and domains. However, much criticism has been raised around modern AI systems, to a large extent due to their inability produce explainable decisions that both their end-users and their developers can trust. The need for such decisions, i.c., deci- sions accompanied by a rationale for why they are made, has ignited much recent research. We propose MAGNEX, a global algorithm that leverages neural-network based explainers to produce rationales for any black-box decision model, neural not. MAGNEX is model-agnostic, and thus easily generalizable across domains and applications. More importantly, MAGNEX is global, i.e., it learns to create ra- tionales by optimizing for number of instances at once, contrary to local methods that aim at explaining a single example. The global nature of MAGNEX has two advantages over local methods: i) it generalizes across instances hence producing more faithful explanations, ii) is computationally more efficient during infer- ence. Our experiments confirm that MAGNEX outperforms popular explainability algorithms both in explanation quality and in computational efficiency.
INTRODUCTION
Black-box decision models have for some time now posed dilemma between power and inter- pretability, For use cases where explanations are necessary, the inability of black-box models to supply them is often a deterrent adoption. However, even in low-risk scenarios this lack of ex- plainability often causes distrust to both the developers and the users of these models, who are often puzzled about how decisions emerge (Ribeiro al.] 2016). Also, explainability an important mechanism when investigating if black-box models act fairly and without bias (Sun et al. 2019
In this paper, we propose MAGNEX, model-agnostic neural explainer that globally learns to ex- plain an already trained model, neural or not. In this post-hoc interpretability setting, most methods (Ribeiro et al. 2016; Sundararajan et al. 2017: Lundberg & Lee] 20177] [uo et al.) 2020) create feature-based explanations, i.e., explanations that assign score to each feature of the input based on how important the feature is to the model's 's decision. This importance score may rely on some in- ternal mechanism of the model
wish to explain, e.g.. gradients (Sundararajan al.) 2017 Shriku- mar et al. 2017) or attention (Jain & Wallace 2019 Wiegreffe & Pinter 2019). ). Such methods have limitations to the types of model they explain, e.g., maddeent-based methods work only with dif- ferentiable models. Perturbation-
methods, e.g., LIME (Ribeiro et al., (016)). SHAP (Lundberg) & Lee 2017). drop combinations of features for specific input the model and observe its out- put. Omitted features that have a large impact on the output of the model across perturbations are deemed important, while other features are considered unimportant. This allows perturbation- based methods to be model -agnosic,,, but adds severe computational complexity, since a large number of perturbations is required per input instance to create quality explanations. The search for an optimal solution by erasure (feature drop) is combinatorial and practically infeasible for even small feature spaces; the search space the power set of the feature set resulting a complexity of O(2") for n features. Therefore, pertubation-banl based methods find approximate (sub-optimal) solutions, but even this process is cumbersome, especially when the input is large (i.c., contains many features) and the explainability method is local (a different search must be performed for each input instance).
We use explainability and interpretability interchangeably as there is clear consensus the literature.

---

Under review as a conference paper at ICLR 2022
original input: i
sufficiency
f1
f2
83
84
f5
f8
f7
Model (M) y
Ysu (3/ 9
Model (M)
Explainer (E)
B(i)
81
82
83
84
85
86
87
(explainer scores)
p1t, 8) (masked input)
f
m
fs
F4
m
m
m
Isp (4(i, 8))
sparsity
Figure I: A high-level illustration of MAGNEX. The features of input are scored by the explainer. The scores are used create a binary mask (Eq. 2 representing the parts of be retained. The original and masked input are both passed through the pre-trained model and the outputs, y and B, are used calculate the sufficiency score. The explainer aims to maximize sufficiency while also masking many features as possible, thus also maximizing sparsity.
MAGNEX also creates feature-based explanations, but contrary most methods, its explainer is a neural network that globally learns to assign feature importance scores. This is similar to the work of De Cao al. (2020), which relies on gradients to produce feature importance scores; their explainer is shallow network directly attached to the model being explained, requiring the gradients to flow through both the explainer and the model. This limits the approach of Cao et al. (2020) to explaining only differentiable models, while increasing the memory and computational complexity.
Neural explanation modules have also been used explainable by-design models (Lei et al. 2016 Bastings et al.] [019] Yu et al.) [209 Chang et al.) [019] [halkidis et al., (021). These models mainly focus on human-centric explainability and jointly train rationale extractor and a classifier. This setting poses an extra burden the training procedure and disincentivizes exploring parts of the space of possible solutions the model can arrive to, possibly leading to loss model performance at the expense of interpretability. MAGNEX disentangles the explainer from the model it wishes to explain, aiming to produce explanations that are faithful, i.e., accurately reflect the features consid- ered important by the model (Lipton 2018 Jacovi & Goldberg 2020). The generated explanations do not have to agree with human annotated rationales (c.g., gold text snippets), which are often used as an optimal solution in explainable by-design methods. MAGNEX's goal is to faithfully reveal in- formation about model's inner workings to its developers and if its explanations agree with human intuition, then MAGNEX can also be used as an explanation algorithm for end-users nations do not agree with human intuition, MAGNEX can reveal weaknesses of the explained model and/or of the data used to train it. For instance, training data may not diverse enough, allowing a model to overfit to insignificant features in its input.
the expla-
The explainer of MAGNEX is shallow neural network that attributes scores to input features (Figure [}. Its training is dictated by two-fold objective that can be classified under constrained optimiza- tion. The scoring of the explainer must be primarily sufficient, i.c., the explanations when fed to the underlying model must result in the same output (De Young al. 2020). Also, among all possible sufficient explanations the explainer is tasked with finding one that is maximally sparse, i.e., utilizes the minimum number of features (Lei et al.) 2016). This constrained objective, relaxed version of it, has been also used by perturbation-basee systems (De Cao et al. 2020 Ribeiro et al. 2016).
Our contributions are as follows:
We propose MAGNEX method that explains the inner workings of pre-trained model in a post-hoc manner. MAGNEX, is completely model agnostic and thus can explain any pre-trained model across modalities (e.g., vision, text) and applications.
The proposed approach is global, allowing MAGNEX to generalize across instances; learn- ing from many samples has regularization effeet alleviating hindsight bias (Fischhoff & Beyth 1975) which a common phenomenon machine learning problems (Mahdavi & Rahimian 2017) and more specificaly in perturbation-base explainability (De Cao al.
2

---

Under review as a conference paper at ICLR 2022
2020
Also, the global nature of MAGNEX transfers the computationally expensive feature search to the training stage allowing for more efficient inference than its competitors.
The experiments showed that MAGNEX produces explanations of better quality than popular explainability methods (LIME (Ribeiro et al. 2016). IG (Sundararajan et al. 2017)) while also being more stable across instances and much more efficient during inference.
METHODOLOGY
2.1 FORMULATION
Let M 1 + O be the model wish explain. Each imput i E is set of imput features such that i = f where n is the number of features. An explainer, E, operates on i and associates each feature in with an importance score 8j E [0, for (1,
E = {s1, $2..... $2.....
(1)
masking function μ. operates on an imput = {ff, f2..... fn and its corresponding scoring {81, , 8n) produce masked version of é:
p(i, = fei E (fi.m), E {f2. ff.mm........ E [f,mm)
(2)
where m is a feature which carries no information for M and is chosen according to the task, c.g., black pixel in computer vision realized in different ways in training and inference. During training whether feature fy will be replaced by is determined by performing biased coin flip based 8j (the score of fy), while during inference is realized as:
pad token in natural language processing. In practise, μ is
fz. if >
u(i, 8),
(3)
m, otherwise
Given an input and scoring s Eii, we compute the quality of the explainer's scoring in terms of sufficiency (su) and sparsity (sp), defined as follows:
su = 9su (M),,, M (u(2,
(4)
sp 9sp (u(t, s))
(5)
where, you 1xx0x - [0, 1] is a function measuring to what extent the output of M when presented with u 8) resembles the output of M when presented with i, and Isp computes the percentage of features which have been replaced with m. Since we opt for maximally sparse inputs with high sufficiency, the total quality score for the explanation computed as:
su + sp, if > -
q=
(6)
su,
otherwise
Note that we force explanations that have sufficiency of at least e to ensure a minimum quality.
Unless otherwise specified, use ysu tailored classification where assume O [0, 1 and is the number of classes. Classification sufficiency is then calculated as:
Jau (3) 9 = (max(y) - 1
where p argmax(y) the predicted class in y E O and you is bounded in [0, 1] with higher scores signaling more faithful explanations. The term max(yy -p measures the divergence between the top probability estimate across classes y and the probability estimate for the same class in ÿ.
2.2 LEARNING
Our explainer is realized as neural network with parameters 0 throughout. We aim to find the opti- mal values which maximize q (Eq, across a number of training examples (1z, ......, fm } C I. However, the standard backpropagation optimization approach falls short in this case since it is im- possible to produce gradients to update the explainer. While the explainer itself is differentiable, in order for sufficiency score be computed for some input i hard choice must be made
3

---

Under review as a conference paper at ICLR 2022
which features in i retain and which to substitute with m. The masking function μ is therefore non-differentiable More importantly, our approach is completely model-agnostic and we make no assumptions about whether the model we wish to explain (M) is differentiable not.
In the simplest scenario, where only the masking function is non-differentiable a number of ap- proaches have attempted to produce gradient estimations with methods other than the score estima- tor, based on REINFORCE (Williams) 1992). The most common of these approaches are the straight- through estimator (Chang et al. [019 Chalkidis al., 2021) and relaxation to binary variables (Louizos et al. [2118] Bastings et al. 2019 De Cao et al. 2020) which leverages the reparamelriza- fion trick (Kingma&
Welling 2013). Both of these approaches can be used only when the masking function is the only non non- con-differentiable component, i.c., they require M to be differentiable, thus breaking the model-agnostic nature of the explainer, which is a requirement in MAGNEX.
To retain the model-agnostic nature of our formulation and alleviate large computational strain to our method we opt to train our model by estimating gradients for updates to our explainer network with the score estimator (REINFORCE). For training on single input i we create multi-variable policy using the output of our explainer.
(o)
(8)
where B is the Bernoulli distribution. Sampling from this policy is equivalent to sampling from each of the Bernoulli distributions independently. Therefore a sample t ~ ( is sequence of binary variables indicating the presence the absence of the feature at position j. We train our method using the score estimator which in this case can be written as:
VoJ((o(()) (t)) (o we can rewrite the above gradient in the form:
(9)
Following Williams
1992
(T)(((()) = (t) (V log P(T|(; 0) q(r)]
(10)
where (i = n
==
We can easily approximate Eq. 10 by Monte-Carlo sam-
pling We further add a baseline in order to reduce the variance of the gradient estimator:
(BJ( = Ex o log (i 0) (q(T) - b)]
(II)
In Mnih et al. (2044) which bears some similarities to our setting a learned baseline was used. Here, we use a moving average baseline which seems to be sufficient for our use cases.
While similar methods relying on differentiable relaxation to binary variables have been shown to outperform REINFORCE in some cases (Bastings et al.) 2219;; De Cao et al., 2020), we choose this gradient estimator for number of reasons. Firstly, we want our method to remain purely model- agnostic, a requirement which cannot satisfied by the estimators in [astings et al. (2019) and De Cao et al. (2020) which support only differentiable models. Secondly, the fact that the estimator works by simple exposure to a scalar metric (Eq. (). which does not need to have a gradient, greatly reduces the space complexity of the method, allowing higher degree of parallelism on the same hardware, and in practice allowing very complex models to be explained in reasonable time. Lastly, this problem involves constrained optimization. Looking back the definition of our total metric (Eq. (6) we can see that we are optimizing sparsity subject to sufficiency being higher than some threshold. This objective is therefore non-differentiable and De Cao al. (2020) and Bastings et al.] (2019) employ Lagrangian relaxation to approximate the constrained objective in differentiable manner. This adds new hyperparameter, the Lagrangian multiplier, which needs to be tuned, adding further overhead to the explainer's 's development procedure and is in all cases an approximation of the true constrained objective.
EXPERIMENTS
We tested MAGNEX in three challenging settings across modalities, i.e., image classification, sen- timent classification, and question answering. Although, some of these tasks are simple to handle with modern models, they pose challenging setting for coturbattionbbsse algorithms, since feature spaces are large compared to previous work (Ribeiro et al. 2016); Lei et al. 2016 Bastings et al.) 2019 De Cao et al.) 2020). This makes it difficult for such methods to identify subsets of the input which are sparse and yet expressive enough to lead to the same output.
"ee Appendix A.2 for proof.
4

---

Under review as a conference paper at ICLR 2022
3.1 BASELINES AND EVALUATION
We compare MAGNEX against Lime (Ribeiro et al. 2016) and Integrated Gradients (IG) (Sundarara- jan et al. 2017). two popular post-hoc explainability methods. Recall that the explainer of MAGNEX (E) outputs score sj E [0, 1] for each feature f, of an input i. This score can be interpreted as the probability of fj being important for a model produce the same output. During training these probabilistic scores are converted to a binary value with biased coin flip. During inference we cast a feature as important if its respective explainer score is above a treshold e (0.5 in our experiments). On the other hand, both LIME and IG compute relative feature importance (i.e. whether a feature is more important than another) but have no explicit threshold to decide which features to keep. This is left up the user a post-processing step. We therefore evaluate both of these methods by selecting the top l most highly scored features, where l is the number of features selected by E' (MAGNEX) for the same input. In other words, evaluate the sufficiency of all explanations the sparsity level achieved by F. For each method we also report the time required to produce explanations averaged across inputs. Further, we intentionally make no attempt to compare explanations against human annotated rationales. Our main goal is to produce explanations which are faithful to the underlying model, i.e., accurately reflect the features in an input which are important for the model (Lipton 2018 Jacovi & Goldberg 2020), which is generally ensured when the sufficiency scores are high. Whether these faithful explanations align with what human would consider a correct explanation is an open question and beyond the scope of this work
3.2 TECHNICAL DETAILS
For IG, following Sundararajan et al. (217)), we use 50 steps in the approximation of the integral throughout | Concerning LIME, shallow models allow a larger number of perturbations to be drawn. Therefore, in image classification where use MAGNEX explain shallow models, we allow 1,500 perturbations to be drawn F On the other hand, for sentiment classification the model complexity increases and we allow 1,000 perturbations, to ensure that the explanations will be created in rea- sonable amount of time (less than 10 seconds per input). For similar reasons, in question answering where we try to explain the most complex model in this work, we allow only 700 perturbations per example which again keeps explanation generation time around the 10 second mark per input 6
3.3 IMAGE CLASSIFICATION
For image classification we use the popular MNIST dataset (Deng 2012). The input features are pixels, and we choose E to be a two-layer Convolutional Neural Network (CNN) (LeCun al.) 1990) which creates vector representation for each pixel, followed by shared linear projection to output a single score per pixel. The feature space size the number of pixels in an image (28 784)
3.3.1 EXPLAINING RANDOM FOREST
We train a random decision forest Ho 1995) to perform digit classification from raw pixels. The model achieves perfect accuracy on the test set. In this case, we only compare against LIME because random forests are not differentiable and IG is unable to produce explanations. Table [ shows the results. MAGNEX is able to achieve much higher sufficiency than LIME while being faster during inference. Explanations are also more stable (smaller standard deviation across inputs).
3.3.2 EXPLAINING A CNN
We also train two-layer CNN on the same task and compare MAGNEX against LIME and IG since the CNN is able to produce gradients The CNN reaches near-perfect accuracy on the test set (99%). Our results can be seen in Table 2 All methods have very high sufficiency scores, producing faithful
We report results for the best of two runs because the differences were small (< 1%). Sundararajan al. (017)) proposed value between 20 and 300.
We found that increasing this number further does not yield more sufficient explanations.
'We only perform mild manual tuning for both baselines due to their complexity.
7 Reffer to Appendix A.3 for sample rationales.
We also attempted to use simple gradient approach (Jacobian) but the results were significantly inferior.
5

---

Under review as a conference paper at ICLR 2022
Method
Sparsity
Sufficiency
Time (s)
MAGNEX (ours)
0.81 + 0.05
0.99 0.02
0.02 0.00
LIME
0.83 0.15
0.30 + 0.02
Table I: Performance of MAGNEX and LIME when explaining a random forest model trained for digit recognition. MAGNEX outperforms LIME in both sufficiency and inference time.
BERT Embeddlngg
GRU Embeddiigg
Single GRU Embedding
$1
Scorc
2
BERT
GRU
Figure 2: MAGNEX for experiments with text. The input is passed through BERT fine-tuned for the task and then a GRU further contextualizes the embeddings before each of them is passed through shared linear layer to produce a score for each token. BERT not updated during training.
explanations. This could be attributed to the fact that the pre-trained CNN is much more robust to slight changes in the input images. It appears have generalized better than the random forests and can be more easily explained, as LIME performs well here while it was had difficulties when explaining the random forests. This is further supported by the fact that the sparsity here higher, meaning that we can mask a larger percentage of each input while retaining very high sufficiency. When it comes to efficiency our method remains much faster during inference than its competitors.
Method
Sparsity
Sufficiency
Time (s)
MAGNEX (ours)
0.88 + 0.04
0.99
0.05
0.01 + 0.0
LIME
0.99
0.06
0.57 + 0.1
IG
0.99 + 0.06
0.60 + 0.0
Performance of MAGNEX. LIME, and IG when explaining CNN trained for digit recogni- tion. All methods generate sufficient explanations, but MAGNEX more efficient during inference.
Table 2:
3.4 SENTIMENT CLASSIFICATION
We fine-tune BERT (Dellin et al. 2019) for binary sentiment classification on the IMDB review dataset (Maas et al. 2011). In this task. the input is sequence of tokens. Again choose dataset with a large number of features (tokens) per instance, to ensure our method is tested for its scalability to realistic scenarios. On the validation set, the mean number of tokens per review is 303. The maximum allowed number of tokens is 512 (13% of validation instances). BERT achieves 94% accuracy on the test set. Here is single-layer GRU model (Cho al. 2014) operating on the contexualized embeddings drawn from the last layer of the fine-tuned BERT model. In effect the GRU further contexualizes these embeddings before passing them to a shared linear projection to create the score for each token. Updates are only performed on the GRU and the linear projection, i.e., the fine-tuned BERT remains frozen. The whole architecture of MAGNEX for this setting can be seen in Figure 2 As mentioned in Section 2.2 reason for choosing REINFORCE is the alleviation of lot of computational stress from the training procedure. we were to use differentiable masking (relaxation to binary variables) to create semi-hard scores for each token, would have to produce gradients for the feedback given back from the pre-trained model. The computation graph this case would be further tasked with tracking gradients for the BERT model and all its intermediate computation throughout the explainer's 's training procedure.
Our results can be seen in Table MAGNEX produces comparable better explanations than its competitors, while being faster during inference. One additional advantage of MAGNEX that alleviates hindsight bias, which often be observed in perturbation-bar based explainability. When

---

Under review as a conference paper at ICLR 2022
Method
Sparsity
Suffiiinccy
Time (8)
MAGNEX
0.94 + 0.06
0.95 + 0.10
0.05 0.03
LIME
0.95 + 0.09
9.33 = 5.23
IG
0.90 = 0.16
1.32 + 0.70
Table 3: Performance of MAGNEX, LIME, and IG, explaining BERT fine-tuned for sentiment classi- fication. MAGNEX outperforms the baselines both suffuciency and inference time.
this was great movie for being only 67 minutes long there was an aspect of film noir
contained this movie and i am glad that nolan picked to film in black and white the plot is simple yet entertaining that keeps you engaged even the dialogue was good along with the acting reminded me what was to come me ##mento by not being chronological order liked how the main character tried to use what cobb taught him for example saying everyone has box which he put his personal things into also on the writer door was the batman logo which seemed ironic because christopher nolan would later direct batman begins and the dark knight, two other great movies there is great twist in the end which i m not going spoil for anyone who hasn seen even though kind of figured what would happen when cobb gave the young man d lloyd ##s credit card also liked how the writer had copy of the republic by plato one of my favorite philosophical books this is definitely a movie you need to watch more than once to get the full aspect of it plus it only being hour long there is also circular aspect to it by ending where it began which i thought was pretty brilliant...
this was a great movie for being only 67 minutes long there was an aspect of film noir contained in this movie and am glad that nolan picked to film it in black and white the plot simple yet entertaining that keeps you engaged even the dialogue was good along with the acting reminded me of what was to come in me ##mentto by not being in chronological
order i liked how the main character tried to use what cobb taught him for example saying everyone has box which he put his personal things into also on the writer door was the batman logo which seemed ironic because christopher nolan would later direct batman begins and the dark knight
other great movies there great twist in the end which m not going to spoil for anyone who hasn seen it even though kind of figured what would happen when cobb gave the young man d lloyd ##s credit card also liked how the writer had a copy the republic by plato one my favorite philosophical books this is definitely a movie you need to watch more than once to get the full aspect of it plus it only being hour long there is also circular aspect to it by ending where began which thought was pretty brilliant.
(b)
Figure 3: Explanations produced by MAGNEX (a) and LIME (b) for a review in the IMDB test set. Both explanations have high sufficiency, but we believe that (a) is a more faithful interpretation of the model's computation. LIME appears to have fallen victim of hindsight bias identifying a single feature (great) as important. This does not mean that the underlying model ignores everything else. MAGNEX considers more features to be important as its global training acts against hindsight bias.
performing erasure (i.e., dropping features from the input to a model) there is no guarantee that low scored features are indeed not useful for the underlying model. The token great in Figure 3b scored very highly in comparison to all other tokens. Retaining this token alone would lead to the same output. This does not necessarily mean that the pre-trained model does not consider any

---

Under review as a conference paper at ICLR 2022
other tokens when casting decision. Although we employ explicit mechanism to tackle this issue, we believe that we make steps towards the right direction due to two characteristics of our method. Firstly, MAGNEX is global, i.., it learns model across large number of instances, which we believe to act as a regularizer against hindsight bias. The exact optimum of erasure for a specific instance easily fall victim hindsight bias and therefore approximations of this exact opti- mum (e.g., LIME) can often exhibit this same behaviour. On the other hand, global context allows MAGNEX to reveal patterns that generalize across instances instead of local ones, For instance, great being associated with positive prediction is a pattern local to an instance that cannot be established globally. It is therefore, disincentivised as solution MAGNEX'S global training regime. We do not argue that such patterns are non-existent, rather that they are important only when they can be established across a variety of instances and can otherwise create extremely misleading explanations (which score very well according to our metrics, i.e., they are sufficient while also being very sparse) but are at the same time not faithful explanations. Secondly, E operates in the space hidden rep- resentations, i.c., BERT token embeddings, which arguably capture contextualized norpho-syntactis and semantic information, allowing MAGNEX to learn more general explanation patterns.
3.5 QUESTION ANSWERING
For this task trained and fine-tuned on SQUAD (Rajpurkar et al. (2016)). The model attempts to predict answer spans within some context given question. It achieves an FI score of 93.2% and an exact match score of 86.9%. We show the results of various explainers on the task in Table 4 Our explainer continues to produce explanations that have higher sufficiency scores than the other methods while also being considerably faster. A side note here is that for this task sufficiency is realized as the (f1, fe...... fn
a large variant of BERT (24 layers, 16 attention heads, 1024 hidden size), pre-
Jaccard index. So between two predicted spans y {f1, f,...., } and sufficiency is measured as su(y, 9) = (y) (1)) The Jaccard index severely punishes small mistakes. For example, the Jaccard index for the spans 'Stanford University' and 'the Standford University' is 2/3 despite the answers being almost identical. To some extent this explains the large standard deviations (Table reported for all methods. Finally, similarly the sentiment analysis task, LIME suffers from the hindsight bias problem. Figure shows an indicative example where the explanations produced by MAGNEX are more well formed than those of LIME.
Method
Sparsity
Sufficiency
Time (s)
MAGNEX
0.76 + 0.12
0.88 + 0.30
0.09 + 0.03
LIME
0.82 0.32
12.91 + 5.22
IG
0.34 0.40
5.76 + 2.24
Table 4: Performance of MAGNEX, LIME, and IG, explaining a large variant of BERT in question answering. MAGNEX outperforms the baselines in both suffuciency and inference time.
4 RELATED WORK
The outburst in the field of explainability in machine learning started with Ribeiro et al. (0166) who proposed LIME, model-agnostic perturbation-base algorithm that generates explanations for ma- chine learning models. Since then, apart from perturbation- based methods, gradient based methods were explored (Shrikumar al. 2017| Sundararajan et al.) 2017)). More recently, Dain & Wallace (2019) and Wiegreffe & Pinter (019)). driven by the increasing popularity of Transformer-base models (Vaswani et al. 2017)
number of tasks across modalities (Devlin et al.) 2019 [Schnei-
der et al.) [019 Dosovitskiy et al., 202II, have initiated the discussion whether the attention mechanism in these models can provide quality explanations.
In parallel, lot of work focused on creating explainable neural networks by-design (Lei et al.) 2016 Bastings et al., 2019 Yu et al., [019 Chang et al., 2019 Chalkidis et al.] 2021). These sys- tems typically contain two components. The first component selects rationales (subsets of the input) which are then fed to the second component for classification. They are typically trained with sim- ilar objectives perturbation- based methods, i.e., sufficiency and sparsity, other objectives such as
8

---

Under review as a conference paper at ICLR 2022
who wrote about the great pest
Witlle #ffnce in 1893 the historian francis aidan gas #fqueet
wrote about the great pest #iile ##nce in 1893 and suggested that it would appear to be some form of the ordinary eastern or bu #ilbonn #ilicc plague " he was able to adopt the ep ##ide ##mi ##ology of the bu ##bon ##ic plague for the black death for the second edition in 1908 imp ##lica ##ttigg rats and flea ##s in the process and his interpretation was widely accepted for other ancient and medieval epidemic ##s such as the justin ##ian plague that was prevalent in the eastern roman empire from 54 ##1 to 700 ce
(a)
who wrote about the great pest #Hlle ##nce in 1893 the historian francis aidan gas ##quuet wrote about the great pest #tile ##nce in 1893 and suggested that " it would appear to be some form of the ordinary eastern or bu #ilboon #ific plague he was able to adopt the ep ##ide ##mi ##ology of the bu ##bon #tc plague for the black death for the second edition in 1908 imp ##lica ##ting rats and flea ##s in the process and his interpretation was widely accepted for other ancient and medieval epidemic ##s such as the justin #sian plague that was prevalent in the eastern roman empire from 54 ##1 700 ce
Figure 4: Explanations produced by MAGNEX (a) and LIME (b) for a question (green box) of SQUAD validation set. Filtering these to produce binary scores as we do during evaluation creates the reduced inputs i) who wrote about the great pest #ffile #incce in 1893 ? the historian francis aidan gas ##qquet wrote about the great pest #tfile and ii) who wrote about thile #innce 1893 the francis gas #itquet wrote great 1893 suggested some adopt black interpretation other roman ##1. Global context allows MAGNEX to produce comparatively well formed inputs to present to the pre-trained model.
continuity Lei et al. 2016) and comprehenssieenss Yu et al. (2019); Chalkidis et al. (2021) have also been tested. However, the initial system (Lei al. (016) performed gradient estimation for the rationale extractor with REINFORCE without baseline reduction making the system unstable due to the nature of the learning algorithm. [Bastings et al. (2019) relaxed this binary rationale extrac- tion process by following Louizos et al. (2018) and therefore making the objective differentiable. Nonetheless, all further work done this area (Yu et al., 2019 Chang al.) 2019) optimizes at least two neural networks concurrently and uses some sort of gradient estimation either through REIN- FORCE, the reparametrization trick, or straight- trouhh estimators. Naturally, these systems are often very hard to train due to their complexity and their innate inability produce exact gradients due to their stochastic nodes. Finally, recent work (Jain et al., 22200) Situ et al.] [202T) has attempted to use auxiliary explanations generated by some attribution method (such LIME integrated gradients)
as supervision. For instance, one of the methods explored by (Situ et al. (2021) was to train a ratio- nale extractor on explanations produced by LIME. They also experimented with integrated gradients, sumilarly Jain et al. (2020) to train a neural network to create explanations in the post-hoc setting.
CONCLUSION & FUTURE WORK
We introduced MAGNEX, post-hoc explainability algorithm, that is global and completely model- agnostic. We experimentally showed that our approach outperforms popular post-hoc algorithms in terms of the faithfulness of its explanations and in computational complexity across tasks and modalities. In addition, the global nature of MAGNEX seems to alleviate the hindsight bias problem that seemed to trouble local perturbation-bast -aaee explainability methods.
In the future we aim to pursue number of interesting directions. We would like to combine our method with online learning to produce higher quality explanations. The explanations learned offline can be used as a starting point and further search can be performed on a per instance basis. We are also planning on performing human evaluation between MAGNEX and its baselines to determine how informative its explanations in the eyes of human. Lastly, we will explore counterfactual explainability (Goyal et al. 2019; Elazar et al. 202T) and how can be used to improve both the training regime of MAGNEX well as evaluation framework.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Jasmijn Bastings, Wilker Aziz, and Ivan Titov, Interpretable neural predictions with differentiable binary variables. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2963- 2977,, Florence, Italy, July 2019. Association for Computational Linguis- tics. doi: 10.1 (0.18653/11/P199 1284. URL
Iaa Chalkidis, Manos Fergadiotis, Dimitrios Tsarapatsanis, Nikolaos Aletras, Ion Androutsopou- los, and Prodromos Malakasiotis. Paragraph-level rationale extraction through regularization: A case study on European court of human rights cases. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, pp. 226-241, Online, June 2021. Association for Computational Linguis- tics. doi: 20.18633/v1/2022 http://.....aaallnnnaaaaa mnacl-mmiin22
URL https: lttpps://aclantclloyy... org/2001
naacl ma mmain.2. 11.22 22
Shiyu Chang, Yang Zhang, Mo Yu, and Tommi Jaakkola. A game theoretic approach to class-wise selective rationalization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buce E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL
nww.ips......... pdf
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. In Proceedings of the 2014 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734 Doha, Qatar, Oc- tober 2014. Association for Computational Linguistics. doi: 10.3115/w1/D441119, URL wgw.oo/g/....0--1100//
Nicola De Cao, Michael Sejr Schlichtkrull, Wilker Aziz, and Ivan Titov. How do decisions emerge across layers in neural models? interpretation with differentiable masking. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP),, pp. 3243- 3255, Online, November 2020. Association for Computational Linguistics. doi: 10. 18653vvl
2020.emnlp-main.26. URL https: http:///aaaaaaatacaaoooo ott.aaattttooooooooo220000000000 org//020. emnlp main. 262
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142 2012.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 1111666////N11-14233 URL https: //aclanthology owg.1y9-ogNNN111990000
Jay Yo Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. ERASER: A benchmark to evaluate rationalized NLP models. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4443-4458, Online. July 2020. Association for Computational Linguisties. doi: 10. 10.18653/11/0200 acll main. 408. URL https::pp www. acwweb. crq htt.aaawww...aaaattttooooo////..
main...n.....mmmmm.........mmmnm 408
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: //openreview. wtt.///wwww..eeeoooeeeee........
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals. Transactions of the Association for Computational Linguistics, 9: 9:160-175, 03 2021. ISSN 2307-387X. doi: 10.1 1 162/tacl a 00359. URL h /00 //doi.org/10.111
003359
10

---

Under review as a conference paper at ICLR 2022
Baruch Fischhoff and Ruth Beyth. knew it would happen: Remembered probabilities of
once-furee things. Organizational Behavior and Human Performance. 13(1):1-16, 1975.
ISSN 0030-5073. doi:
URL https: ://www.
sci Rcceecedi com scmm/scii cttp:///sa.ieeaaaaaaaaaaaa/////.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. AllenNLP: A deep semantic natural language processing platform. In Proceedings Workshop for NLP Open Source Software (NLP- pp. 1-6, Melbourne, Australia, July 2018. Association for Computational I Linguistics. doi: 10.18653/
v1/W18-2501 URL
Yash Goyal, Uri Shalit, and Been Kim. Explaining classifiers with causal concept effect (cace). CoRR, abs/1907.07166 166,,, 2019. URL
Tin Kam Ho. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition, volume 1, pp. 278-282. IEFE, 1995.
Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable NLP systems: How should define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, pp. 4198 -298, Online, July 2020. Association for Computa- tional Linguistics. doi: 10. 10.18863/v1/2000 acl maa.mmmain.88... Utthttps:: wwww www.wawlwwb.coo org/
mathology/20253 aww.maan.mmmmmm.. 386
Sarthak Jain and Byron C. Wallace. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 3543-3556, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-133 URL
1357
Sarthak Jain, Sarah Wiegreffe, Yuval Pinter, and Byron C. Wallace. Learning faithfully rational- ize by construction. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pp. 4459-4473, Online, July 2020. Association for Computational Linguis- URL
tics. doi:
acl acl-main.40s
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015. URL
/as/g//////11...../// /1412. 6980
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
arXiv preprint
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz- Richardson. Captum: unified and generic model interpretability library for pytorch, 2020.
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. Jackel. Hand- written digit recognition with back- propagapaantinnn network. In David Touretzky (ed.), Advances in Neural Information Processing Systems (NIPS 1989), volume 2, Denver, CO, 1990. Morgan Kaufman.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 107-117, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/ D16-1011.
Zachary C. Lipton. The mythos of model interpretability. Commun. ACM, 61(1))36434 Septem ber 2018. ISSN 0001-0782. doi: 10.1145/3233231. URL htttps://dei..org/11.11111
3233231
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL btttps://ppenzeeeew.eee.t wtw.:e//wwww....ee///rmme.......
Bkg Cqy7

---

Under review as a conference paper at ICLR 2022
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through ly regularization. In International Conference on Learning Representations, 2018. URL hetps:
/ (ageea/ww.ew..attfffoooomooaeae.
Scott M Lundberg and Su-In Lee. unified approach to interpreting model predictions. In Proceed- ings of the 31st international conference on neural information processing systems, pp. 4768- 4777, 2017.
Dongsheng Luo, Wei Cheng, Dongkuan Xu, Wenchao Yu, Bo Zong, Haifeng Chen, and Xiang Zhang. Parameterized explainer for graph neural network. Advances in Neural Information Pro- cessing Systems, 33, 2020.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150 2011.
Shaudi Mahdavi and M. Amin Rahimian. Hindsight bias impedes learning. In Tatiana V. Guy, Miroslav Kárný. David Riss-Insua, and David H. Wolpert (eds.). Proceedings of the NIPS 2016 Workshop Imperfect Decision Makers, volume 58 of Proceedings of Machine Learning Re- search, pp. 117- PMLR, 09 Dec 2017. URL htttps://proceeeeee .ml.iinnnssmm......mmmm.mmrrmmmm mlr. htw.anngssmm.....ss//..
v58/mahdavi 7a. html.
Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In Advances in neural information processing systems, pp. 2204-2212, 2014.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http: /ttp:://pappr..ee peapperssneuuipee neurips cc/paper/ 9015 pat---yttrcc-aaa an -nt--aaratmeeeeeeeeee sttlestyeeeeee weigh-pprffrmancco wewpaaanneeeeeeeeeeeeeeaeeeeeeee learning libraryy......
pdf
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16- 1264.
URL
Sxq/D16-1264
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1 1135-114,, 2016.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition. In Gernot Kubin and Zdravko Kacic (eds.), Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pp. 3465 3469. ISCA, 2019. doi: 10.1145/heerppechh20991883 URL https: .or. ......21330 21437 20e2211333nnteee 102.gg/g.e.0023333/teeeeeeee.ee
1873
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, pp. 3145- 3153. PMLR, 2017.
Xuelin Situ, Ingrid Zukerman, Cecile Paris, Sameen Maruf, and Gholamreza Haffari. Learning to explain: Generating stable explanations fast. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural I Language Processing (Volume I: Long Papers), pp. 5340-5355, 2021.
12

---

Under review as a conference paper at ICLR 2022
Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Eliz- abeth Belding, Kai-Wes Chang, and William Yang Wang. Mitigating gender bias in natural lan- guage processing: Literature review. In Proceedings of the 57th Anmual Meeting of the Associ- ation for Computational Linguistics, pp. 1630-1640, Florence, Italy, July 2019. Association for URL
Computational Linguistics. doi: 10. 10.18653/11/119-111
org/ 11599
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pp. 3319-3328. PMLR, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 11-20, Hong Kong. China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/D11-1100
URL /tEtps///acLanttooooy..oooo
Ronald J Williams. Simple statistical follinnt algorithms for connectionist reinforcement learning. Machine learning. 8(3):229-2) 22,66,9-256, 1992.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemnite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-at natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Associ- ation for Computational Linguistics. doi: 10.18653/v http:////222.eeee 20000...mllp-dmo..... URL https:
7/aclanthology org emn p demos. 6
Mo Yu, Shiyu Chang, Yang Zhang, and Tommi Jaakkola. Rethinking cooperative rationalization: Introspective extraction and complement control. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence Natural Language Processing (EMNLP-IJCNLP),, pp. 4094-4103, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-14220 URL http::///ccaanthoooooooo////////
A APPENDIX
A.1 IMPLEMENTATION DETAILS
We implemented all models in AllenNIP Gardner et al. 2018) and PyTorch (Paszke et al. 2019
For experiments with text (Sections 3.4 and 3.5) we used the Transformers library of Huggingface (Wolf et al.) 2020). For MAGNEX we found automatic tuning to be slow in developement since REINFORCE tends to diverge very quickly when the learning rate is non-optimal. We therefore man- ually tuned the hyper-parameters in each experiment by allowing small number of batches to flow through the model and observing its metrics. Non-promising runs were terminated very quickly. We performed this procedure for sentiment classification and question answering for learning rates E (10-1, 10-5). The optimal learning rates were 10-1 for sentiment classification and 10-5 for question answering. For image classification (Section 3.3) we used a learning rate of 10-3 Opti- mization was carried out with Adam (Kingma & Bal 2015) throughout the experiments except when fine-tuning BERT-base in sentiment classification where AdamW (Loshchilov & Hutter 2019) was used. Finally, for our baselines (LIME, IG) we use Captum (Kokhlikyan et al. 2020).
A.2 SCORE ESTIMATOR PROOF
We begin by creating a policy parametrized by 0. For some input the policy can be written as:
(i) (B))
(12)
13

---

Under review as a conference paper at ICLR 2022
Our loss is the expected return across samples drawn from Te (T (i)) with gradient:
(=J(Fo())) = (+][TT]
(13)
The probability of trajectory which is sequence of binary variables being sampled from (t is:
P) = II ((T))=0(0),)
(14)
==1
We can rewrite Eq. 13 by taking the definition of expectation
VaJ(=a((t) = V / P(T():)))(())
(15)
We can add the gradient Eq. under the integral:
voJ(Fo(t)) = / VoP(T((:0)aTT)
(16)
and rewrite Eq. 16 by utilizing log(s(x))) = ((xx)/((x)) as:
VaJ(mo(1)) P(T log P(T)i:))g(T)
(17)
By reverting back expectation form we arrive at:
(TT((t(()) (i)) V log P(rli; q(r)]
(18)
This allows to sample T Te (i) and compute the gradient with Monte Carlo Sampling.
A.3 IMAGE CLASSIFICATION RATIONALES
We show rationales for MNIST classification when the model being explained is Random Forest (Figure 5) or CNN (6 MAGNEX produces explanations of higher quality than LIME.
5-
0
5-
D
5-
15
15
15
25.
25
30
15
30
20
(a)
(b)
(c)
Figure 5: Explanations for a sample image (a) in the MNIST dataset with explanations generated by MAGNEX (b) and LIME (c) when explaining a Random Forest.
14

---

Under review as a conference paper at ICLR 2022
8
15
25-
25
(a)
(b)
(c)
Figure 6: Explanations for sample image (a) in the MNIST dataset with explanations generated by
MAGNEX (b) and LIME (c) when explaining a CNN.
15


Reviews:
[{"review_id": "vv9I_BSqDms", "reviewer": "Reviewer_9rPK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposal of the paper is interesting. However, it has some weaknesses especially in the experimental part that should be fixed before publication. In particular, the evaluation is not convincing because: a) the sparsity metrics takes always the same values for all the methods tested and there are not tests of statistical significance; b) the evaluation is done with respect to two measures optimized by the MAGNEX (it would be odd that it get low results on them). However, it should be tested also with respect to other evaluation measures; c) the main advantage of MAGNEX seems to be the low runtime but this aspect is not the focus of the paper; d) there are no examples of explanations for images or question-answering; e) the approach should be tested also on the most simple data type, i.e., tabular data; f) I recommend adding a large array of baselines such as surrogate decision trees and random forests returning as explanation the features importance, SHAP, Trepan. Finally, I recommend investigating how changes the performance of these baselines when varying instead of assigned to it the same retrieve by MAGNEX.\n\nMinor. Some mission related works:\n- Setzu, Mattia, et al. \"GLocalX-From Local to Global Explanations of Black Box AI Models.\" Artificial Intelligence 294 (2021): 103457.\n- ElShawi, Radwa, et al. \"ILIME: Local and Global Interpretable Model-Agnostic Explainer of Black-Box Decision.\" European Conference on Advances in Databases and Information Systems. Springer, Cham, 2019.\n- Spinner, Thilo, et al. \"explAIner: A visual analytics framework for interactive and explainable machine learning.\" IEEE transactions on visualization and computer graphics 26.1 (2019): 1064-1074.", "summary_of_the_paper": "The paper proposes a global model-agnostic neural network-based explainer.", "summary_of_the_review": "Experiments are not sound nor convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SX-7zNsUnps", "reviewer": "Reviewer_Mi4V", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n1.\tThe proposed explainer is model-agnostic and fairly efficient, which has great potential in applications. Moreover, such universality is validated by various models, including Random Forest, BERT, and CNN.\n\n**Weakness**\n1.\tI find the paper hard to read, and lots of its parts seem to be wordy and lack clarity. I suggest the authors thoroughly reorganize the lines of writing, especially for the introduction and method section. \n2.\tAs suggested in the introduction part, the novelty of this works partly comes from the global feature of datasets instead of a single instance, which is not new in the literature, e.g., [1]. \n3.\tRegardless of the variety of models in the experiments,  more competitive baselines are necessary to justify the superiority of MAGNEX.  \n4.\tI think more clarifications for the experimental conclusions are needed. For example, why would the sparsity values be the same for all the methods? Plus, it would be nice to provide a more in-depth study about the performance gain of MAGNEX.\n\n\n[1] Parameterized Explainer for Graph Neural Network. Dongsheng Luo, Wei Cheng, Dongkuan Xu, Wenchao Yu, Bo Zong, Haifeng Chen, Xiang Zhang.", "summary_of_the_paper": "This paper proposes a post-hoc explainability algorithm that is global and model-agnostic. The global comes from training from batches of data while the model-agnostic property is achieved by a gradient estimator.  Experiments show the proposed method outperforms IG and LIME from aspects of sparsity, sufficiency, and time. Some case studies also validate its effectiveness.", "summary_of_the_review": "I think this work is not ready for publication, thus I lean towards rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OfOi6WeFJbj", "reviewer": "Reviewer_eFGa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an interesting paper that discusses an important and timely topic, generating fast and reliable model-agnostic explanations. The authors propose a novel and interesting approach that compares favorably with popular methods, and present experiments across many modalities and tasks. They also discuss and analyze the methods performance, and provide a detailed review of their technical approach.\n\nWhile I do find the paper useful and clear, I have some issues that I would like to see addressed for it to be published. \n\nFirst and foremost, I find the lack of any discussion about causal explanation and especially counterfactuals to be highly problematic. Masking parts of the input and observing the model’s output does not causally explain the model’s decisions, even if the prediction of the model is the same.\n\nThe authors note that they intentionally do not compare to humans as we should care only about what the model is doing. However, they do not extend this reasoning to trying to causally estimate how decisions are made. I recommend the authors address the causal explanation literature, and explain why masking tokens or replacing existing pixels with black pixels is sufficient in terms of counterfactuals [1,2,3].\n\nSecond, I don’t see how is Figure 3 helpful for analysis. As I read it, this seems to be just the authors’ opinion regarding the model’s internal state, without any comparison to counterfactuals. Also, MAGNex mostly highlights “great” in this particular example, so I don’t fully agree with the intuitive explanation. Same critique stands for Figure 4 as well.\n\nThird, the empirical results are not as convincing as the authors claim. Apart from time, MAGNEx doesn’t seem to be improving much in the first three experiments, and the fourth has huge standard deviations. In the BERT experiment, if tested on significantly shorter sentences, even the time advantage might not be true. Also, the CNN experiments should be done on a substantially more challenging dataset\n\n\n[1] Explaining Classifiers with Causal Concept Effect (CaCE)\n[2] Causalm: Causal model explanation through counterfactual language models\n[3] Amnesic probing: Behavioral explanation with amnesic counterfactuals\n", "summary_of_the_paper": "This paper proposes a global model-agnostic explanation method. The method relies on a neural model that learns to predict which input features are important for the original model’s predictions. Through substantial experimentation, the authors demonstrate that their approach outperforms LIME and Integrated Gradients. They compare between explanation methods \nin terms of the faithfulness of its explanations and in computational complexity.\n\nThe main contribution of this paper is MAGNEx, a neural model-agnostic explanation method that is substantially faster than popular methods on high-dimensional inputs.\n", "summary_of_the_review": "While I do find the paper interesting and find some of the contributions meaningful, I have some issues with the paper in its current form. First and foremost, I find the lack of any discussion about causal explanation and especially counterfactuals to be highly problematic. Second, the qualitative analysis of the output of MAGNEx compared with LIME is unconvincing. Third, the empirical results are not as strong as the authors claim, apart from the important and substantial gain in time efficiency.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "N3L5A9xvT9B", "reviewer": "Reviewer_A419", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the approach in this paper is fairly intuitive. The results\nare considerably better wrt execution time. One thing that was not\nclear to me: is prediction time for the trimmed input going through\nthe original model included in the total execution time?\n\nIn the introduction, there is a statement that I'm not sure I agree\nwith. Last phrase of the first paragraph: \"Furthermore, explainability\nis an important mechanism to ensure black-box models act fairly and\nwithout bias\". I'm not sure that once we understand model predictions\nwe understand its fairness characteristics. Perhaps undersstanding\nmodel predictions could help with understanding its fairness.\n\nIn Figure 4, it's not clear what the question is.\n\nI think the paper would have a much stronger contribution if a small\nsample of the results for the explainer and LIME would be judged by\nhumans for how helpful they are for the human to appreciate the\nresults of the model. As it stands now, the execution time is the main advantage (albeit, a strong one). \n", "summary_of_the_paper": "This paper proposes a global explainers for black-box models. Original\ninputs/features are fed in parallel with the black-box model to an\nexplainer model that is in charge of deciding important scores for\neach feature. Depending on the scores, features are dropped and the\nremaining input is also bed to the original model. The outputs for the\noriginal input and the trimmed version with the explainer's help are\ncompared. The explainer is trained such that the original input and\nthe trimmed version produce similar scores when fed to the original\nmodel. In addition, the sparsity of the trimmed input is maximized.\n\nThe new explainer is compared to LIME and Integrated Gradients in\nterms of how faithful the produced explanations are (how close the\nresults of the original input and the trimmed one are when passed\nthrough the black-box model) and the execution time on three different\ntasks: image classification, sentiment analysis and question\nanswering. The new explainer produces the results much faster and with\nsimilar and sometimes higher quality (as measured by faithfullness).\n\n", "summary_of_the_review": "I think this paper introduces a straightforward technique for global explainers that has clear execution time advantages over existing explainers. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a global model-agnostic explanation method. The method relies on a neural model that learns to predict which input features are important for the original model’s predictions. Using experimentation, the authors demonstrate that their approach outperforms LIME and Integrated Gradients. They compare the explanation methods in terms of the faithfulness of explanations and computational complexity. While the premise of the work is interesting, reviewers have suggested several areas for improvement: i) writing can use more clarity. writing seems verbose and hard to understand especially in intro and methods section ii) several points have been raised about empirical evaluation including lack of user studies, a more extensive set of baselines and data modalities. Given this, we are unable to recommend an acceptance at this time. We hope the authors find the reviews helpful.
Decision: Reject