MAGNEx: A Model Agnostic Global Neural Explainer

1 INTRODUCTION . Black-box decision models have for some time now posed a dilemma between power and interpretability . For use cases where explanations are necessary , the inability of black-box models to supply them is often a deterrent to adoption . However , even in low-risk scenarios this lack of explainability often causes distrust to both the developers and the users of these models , who are often puzzled about how decisions emerge ( Ribeiro et al. , 2016 ) . Also , explainability is an important mechanism when investigating if black-box models act fairly and without bias ( Sun et al. , 2019 ) .1 In this paper , we propose MAGNEx , a model-agnostic neural explainer that globally learns to explain an already trained model , neural or not . In this post-hoc interpretability setting , most methods ( Ribeiro et al. , 2016 ; Sundararajan et al. , 2017 ; Lundberg & Lee , 2017 ; Luo et al. , 2020 ) create feature-based explanations , i.e. , explanations that assign a score to each feature of the input based on how important the feature is to the model ’ s decision . This importance score may rely on some internal mechanism of the model we wish to explain , e.g. , gradients ( Sundararajan et al. , 2017 ; Shrikumar et al. , 2017 ) or attention ( Jain & Wallace , 2019 ; Wiegreffe & Pinter , 2019 ) . Such methods have limitations to the types of model they can explain , e.g. , gradient-based methods work only with differentiable models . Perturbation-based methods , e.g. , LIME ( Ribeiro et al. , 2016 ) , SHAP ( Lundberg & Lee , 2017 ) , drop combinations of features for a specific input to the model and observe its output . Omitted features that have a large impact on the output of the model across perturbations are deemed important , while other features are considered unimportant . This allows perturbation-based methods to be model-agnostic , but adds severe computational complexity , since a large number of perturbations is required per input instance to create quality explanations . The search for an optimal solution by erasure ( feature drop ) is combinatorial and practically infeasible for even small feature spaces ; the search space is the power set of the feature set resulting in a complexity of O ( 2n ) for n features . Therefore , pertubation-based methods find approximate ( sub-optimal ) solutions , but even this process is cumbersome , especially when the input is large ( i.e. , contains many features ) and the explainability method is local ( a different search must be performed for each input instance ) . 1We use explainability and interpretability interchangeably as there is no clear consensus in the literature . MAGNEx also creates feature-based explanations , but contrary to most methods , its explainer is a neural network that globally learns to assign feature importance scores . This is similar to the work of De Cao et al . ( 2020 ) , which relies on gradients to produce feature importance scores ; their explainer is a shallow network directly attached to the model being explained , requiring the gradients to flow through both the explainer and the model . This limits the approach of De Cao et al . ( 2020 ) to explaining only differentiable models , while increasing the memory and computational complexity . Neural explanation modules have also been used in explainable by-design models ( Lei et al. , 2016 ; Bastings et al. , 2019 ; Yu et al. , 2019 ; Chang et al. , 2019 ; Chalkidis et al. , 2021 ) . These models mainly focus on human-centric explainability and jointly train a rationale extractor and a classifier . This setting poses an extra burden on the training procedure and disincentivizes exploring parts of the space of possible solutions the model can arrive to , possibly leading to loss of model performance at the expense of interpretability . MAGNEx disentangles the explainer from the model it wishes to explain , aiming to produce explanations that are faithful , i.e. , accurately reflect the features considered important by the model ( Lipton , 2018 ; Jacovi & Goldberg , 2020 ) . The generated explanations do not have to agree with human annotated rationales ( e.g. , gold text snippets ) , which are often used as an optimal solution in explainable by-design methods . MAGNEx ’ s goal is to faithfully reveal information about a model ’ s inner workings to its developers and if its explanations agree with human intuition , then MAGNEx can also be used as an explanation algorithm for end-users . If the explanations do not agree with human intuition , MAGNEx can reveal weaknesses of the explained model and/or of the data used to train it . For instance , training data may not be diverse enough , allowing a model to overfit to insignificant features in its input . The explainer of MAGNEx is a shallow neural network that attributes scores to input features ( Figure 1 ) . Its training is dictated by a two-fold objective that can be classified under constrained optimization . The scoring of the explainer must be primarily sufficient , i.e. , the explanations when fed to the underlying model must result in the same output ( DeYoung et al. , 2020 ) . Also , among all possible sufficient explanations the explainer is tasked with finding one that is maximally sparse , i.e. , utilizes the minimum number of features ( Lei et al. , 2016 ) . This constrained objective , or a relaxed version of it , has been also used by perturbation-based systems ( De Cao et al. , 2020 ; Ribeiro et al. , 2016 ) . Our contributions are as follows : • We propose MAGNEx a method that explains the inner workings of a pre-trained model in a post-hoc manner . MAGNEx , is completely model agnostic and thus can explain any pre-trained model across modalities ( e.g. , vision , text ) and applications . • The proposed approach is global , allowing MAGNEx to generalize across instances ; learning from many samples has a regularization effect alleviating hindsight bias ( Fischhoff & Beyth , 1975 ) which is a common phenomenon in machine learning problems ( Mahdavi & Rahimian , 2017 ) and more specificaly in perturbation-based explainability ( De Cao et al. , 2020 ) . Also , the global nature of MAGNEx transfers the computationally expensive feature search to the training stage allowing for more efficient inference than its competitors . • The experiments showed that MAGNEx produces explanations of better quality than popular explainability methods ( LIME ( Ribeiro et al. , 2016 ) , IG ( Sundararajan et al. , 2017 ) ) , while also being more stable across instances and much more efficient during inference . 2 METHODOLOGY . 2.1 FORMULATION . Let M : I → O be the model we wish to explain . Each input i ∈ I is a set of input features such that i = { f1 , f2 , . . . , fn } , where n is the number of features . An explainer , E , operates on i and associates each feature in i with an importance score sj ∈ [ 0 , 1 ] for j ∈ { 1 , 2 , . . . , n } : E ( i ) = { s1 , s2 , . . . , sn } ( 1 ) A masking function µ , operates on an input i = { f1 , f2 , . . . , fn } and its corresponding scoring s = { s1 , s2 , . . . , sn } to produce a masked version of i : µ ( i , s ) = { e1 ∈ { f1 , m } , e2 ∈ { f2 , m } , . . . , en ∈ { fn , m } } ( 2 ) where m is a feature which carries no information for M and is chosen according to the task , e.g. , a black pixel in computer vision or a pad token in natural language processing . In practise , µ is realized in different ways in training and inference . During training whether a feature fj will be replaced by m is determined by performing a biased coin flip based on sj ( the score of fj ) , while during inference µ is realized as : µ ( i , s ) j = { fj , if sj > m , otherwise ( 3 ) Given an input i and a scoring s = E ( i ) , we compute the quality of the explainer ’ s scoring in terms of sufficiency ( su ) and sparsity ( sp ) , defined as follows : su = gsu ( M ( i ) , M ( µ ( i , s ) ) ) ( 4 ) sp = gsp ( µ ( i , s ) ) ( 5 ) where , gsu : O×O → [ 0 , 1 ] is a function measuring to what extent the output of M when presented with µ ( i , s ) resembles the output of M when presented with i , and gsp computes the percentage of features which have been replaced with m. Since we opt for maximally sparse inputs with high sufficiency , the total quality score for the explanation is computed as : q = { su + sp , if su > υ su , otherwise ( 6 ) Note that we force explanations that have a sufficiency of at least υ to ensure a minimum quality . Unless otherwise specified , we use a gsu tailored to classification where we assume O = [ 0 , 1 ] c and c is the number of classes . Classification sufficiency is then calculated as : gsu ( y , ŷ ) = 1− ( max ( y ) − ŷp ) ( 7 ) where p = argmax ( y ) is the predicted class in y ∈ O and gsu is bounded in [ 0 , 1 ] with higher scores signaling more faithful explanations . The term max ( y ) − ŷp measures the divergence between the top probability estimate across classes in y and the probability estimate for the same class in ŷ . 2.2 LEARNING . Our explainer is realized as a neural network with parameters θ throughout . We aim to find the optimal values θ∗ which maximize q ( Eq . 6 ) across a number of training examples { i1 , i2 , . . . , im } ⊆ I . However , the standard backpropagation optimization approach falls short in this case since it is impossible to produce gradients to update the explainer . While the explainer itself is differentiable , in order for a sufficiency score to be computed for some input i a hard choice must be made on which features in i to retain and which to substitute with m. The masking function µ is therefore non-differentiable . More importantly , our approach is completely model-agnostic and we make no assumptions about whether the model we wish to explain ( M ) is differentiable or not . In the simplest scenario , where only the masking function is non-differentiable , a number of approaches have attempted to produce gradient estimations with methods other than the score estimator , based on REINFORCE ( Williams , 1992 ) . The most common of these approaches are the straightthrough estimator ( Chang et al. , 2019 ; Chalkidis et al. , 2021 ) and relaxation to binary variables ( Louizos et al. , 2018 ; Bastings et al. , 2019 ; De Cao et al. , 2020 ) which leverages the reparametrization trick ( Kingma & Welling , 2013 ) . Both of these approaches can be used only when the masking function is the only non-differentiable component , i.e. , they require M to be differentiable , thus breaking the model-agnostic nature of the explainer , which is a requirement in MAGNEx . To retain the model-agnostic nature of our formulation and alleviate large computational strain to our method we opt to train our model by estimating gradients for updates to our explainer network with the score estimator ( REINFORCE ) . For training on a single input i we create a multi-variable policy using the output of our explainer . πθ ( i ) = { B ( Eθ ( i ) j ) } |i|j=1 ( 8 ) where B is the Bernoulli distribution . Sampling from this policy is equivalent to sampling from each of the Bernoulli distributions independently . Therefore a sample τ ∼ πθ ( i ) is a sequence of binary variables indicating the presence or the absence of the feature at position j . We train our method using the score estimator which in this case can be written as : ∇θJ ( πθ ( i ) ) = ∇θEτ∼πθ ( i ) [ q ( τ ) ] ( 9 ) Following Williams ( 1992 ) we can rewrite the above gradient in the form : ∇θJ ( πθ ( i ) ) = Eτ∼πθ ( i ) [ ∇θ log P ( τ |i ; θ ) q ( τ ) ] ( 10 ) where P ( τ |i ; θ ) = ∏|i| j=1 P ( τj |πθ ( i ) j ) . We can easily approximate Eq . 10 by Monte-Carlo sampling.2 We further add a baseline in order to reduce the variance of the gradient estimator : ∇θJ ( πθ ) = Eτ∼πθ [ ∇θ log P ( τ |i ; θ ) ( q ( τ ) − b ) ] ( 11 ) In Mnih et al . ( 2014 ) , which bears some similarities to our setting a learned baseline was used . Here , we use a moving average baseline which seems to be sufficient for our use cases . While similar methods relying on differentiable relaxation to binary variables have been shown to outperform REINFORCE in some cases ( Bastings et al. , 2019 ; De Cao et al. , 2020 ) , we choose this gradient estimator for a number of reasons . Firstly , we want our method to remain purely modelagnostic , a requirement which can not be satisfied by the estimators in Bastings et al . ( 2019 ) and De Cao et al . ( 2020 ) which support only differentiable models . Secondly , the fact that the estimator works by simple exposure to a scalar metric ( Eq . 6 ) , which does not need to have a gradient , greatly reduces the space complexity of the method , allowing a higher degree of parallelism on the same hardware , and in practice allowing very complex models to be explained in reasonable time . Lastly , this problem involves constrained optimization . Looking back at the definition of our total metric ( Eq . 6 ) we can see that we are optimizing sparsity subject to sufficiency being higher than some threshold . This objective is therefore non-differentiable and De Cao et al . ( 2020 ) and Bastings et al . ( 2019 ) employ Lagrangian relaxation to approximate the constrained objective in a differentiable manner . This adds a new hyperparameter , the Lagrangian multiplier , which needs to be tuned , adding further overhead to the explainer ’ s development procedure and is in all cases an approximation of the true constrained objective .

Summary: This paper proposes a post-hoc explainability algorithm that is global and model-agnostic. The global comes from training from batches of data while the model-agnostic property is achieved by a gradient estimator.  Experiments show the proposed method outperforms IG and LIME from aspects of sparsity, sufficiency, and time. Some case studies also validate its effectiveness.