Human-interpretable model explainability on high-dimensional data

1 INTRODUCTION . The explainability of AI systems is important , both for model development and model assurance . This importance continues to rise as AI models – and the data on which they are trained – become ever more complex . Moreover , methods for AI explainability must be adapted to maintain the human-interpretability of explanations in the regime of highly complex data . Many explainability methods exist in the literature . Model-specific techniques refer to the internal structure of a model in formulating explanations ( Chen & Guestrin , 2016 ; Shrikumar et al. , 2017 ) , while model-agnostic methods are based solely on input-output relationships and treat the model as a black-box ( Breiman , 2001 ; Ribeiro et al. , 2016 ) . Model-agnostic methods offer wide applicability and , importantly , fix a common language for explanations across different model types . The Shapley framework for model-agnostic explainability stands out , due to its theoretically principled foundation and incorporation of interaction effects between the data ’ s features ( Shapley , 1953 ; Lundberg & Lee , 2017 ) . The Shapley framework has been used for explainability in machine learning for years ( Lipovetsky & Conklin , 2001 ; Kononenko et al. , 2010 ; Štrumbelj & Kononenko , 2014 ; Datta et al. , 2016 ) . Unfortunately , the combinatorics required to capture interaction effects make Shapley values computationally intensive and thus ill-suited for high-dimensional data . More computationally-efficient methods have been developed to explain model predictions on highdimensional data . Gradient- and perturbation-based methods measure a model prediction ’ s sensitivity to each of its raw input features ( Selvaraju et al. , 2020 ; Zhou et al. , 2016 ; Zintgraf et al. , 2017 ) . Other methods estimate the mutual information between input features and the model ’ s prediction ( Chen et al. , 2018a ; Schulz et al. , 2020 ) , or generate counterfactual feature values that change the model ’ s prediction ( Chang et al. , 2019 ; Goyal et al. , 2019 ; Wang & Vasconcelos , 2020 ) . See Fig . 1 for explanations produced by several of these methods ( with details given in Sec . 3.5 ) . When intricately understood by the practitioner , these methods for model explainability can be useful , e.g . for model development . However , many alternative methods exist to achieve broadly the same goal ( i.e . to monitor how outputs change as inputs vary ) with alternative design choices that Under review as a conference paper at ICLR 2021 Figure 1 : Pixel-based explanations of a model trained to predict the attractiveness label in CelebA . < latexit sha1_base64= '' uVbZDjeKN2CUg8cuBY25CqEibBw= '' > AAAB8HicjVDLSgMxFL1TX7W+qi7dBIvgqsyIoO6KblxWsA9ph5JJ77ShSWZIMkIp/Qo3LhRx6+e482/MtF2oKHggcDjnXu7JiVLBjfX9D6+wtLyyulZcL21sbm3vlHf3mibJNMMGS0Si2xE1KLjChuVWYDvVSGUksBWNrnK/dY/a8ETd2nGKoaQDxWPOqHXSXVdSO4xiEvfKlaDqz0D+JhVYoN4rv3f7CcskKssENaYT+KkNJ1RbzgROS93MYErZiA6w46iiEk04mQWekiOn9EmcaPeUJTP168aESmPGMnKTeUDz08vF37xOZuPzcMJVmllUbH4ozgSxCcl/T/pcI7Ni7AhlmrushA2ppsy6jkr/K6F5Ug1Oqxc3p5Xa5aKOIhzAIRxDAGdQg2uoQwMYSHiAJ3j2tPfovXiv89GCt9jZh2/w3j4BebeQNw== < /latexit > < latexit sha1_base64= '' 48lxewtfvoTjglqBN3Xn2cbIFVI= '' > AAAB8HicjVDLSgMxFL2pr1pfVZdugkVwVWakoO6KblxWsA9ph5JJM21okhmSjFiGfoUbF4q49XPc+Tdm2i5UFDwQOJxzL/fkhIngxnreByosLa+srhXXSxubW9s75d29lolTTVmTxiLWnZAYJrhiTcutYJ1EMyJDwdrh+DL323dMGx6rGztJWCDJUPGIU2KddNuTxI7CCN/3yxW/6s2A/yYVWKDRL7/3BjFNJVOWCmJM1/cSG2REW04Fm5Z6qWEJoWMyZF1HFZHMBNks8BQfOWWAo1i7pyyeqV83MiKNmcjQTeYBzU8vF3/zuqmNzoKMqyS1TNH5oSgV2MY4/z0ecM2oFRNHCNXcZcV0RDSh1nVU+l8JrZOqX6ueX9cq9YtFHUU4gEM4Bh9OoQ5X0IAmUJDwAE/wjDR6RC/odT5aQIudffgG9PYJlP+QSQ== < /latexit > < latexit sha1_base64= '' K7cr1icZ8HzhwI560g3J0QOAQ0o= '' > AAAB8HicjVDLSgMxFL2pr1pfVZdugkVwVWakoO6KblxWsA9ph5JJM21okhmSjFCHfoUbF4q49XPc+Tdm2i5UFDwQOJxzL/fkhIngxnreByosLa+srhXXSxubW9s75d29lolTTVmTxiLWnZAYJrhiTcutYJ1EMyJDwdrh+DL323dMGx6rGztJWCDJUPGIU2KddNuTxI7CCN/3yxW/6s2A/yYVWKDRL7/3BjFNJVOWCmJM1/cSG2REW04Fm5Z6qWEJoWMyZF1HFZHMBNks8BQfOWWAo1i7pyyeqV83MiKNmcjQTeYBzU8vF3/zuqmNzoKMqyS1TNH5oSgV2MY4/z0ecM2oFRNHCNXcZcV0RDSh1nVU+l8JrZOqX6ueX9cq9YtFHUU4gEM4Bh9OoQ5X0IAmUJDwAE/wjDR6RC/odT5aQIudffgG9PYJmAeQSw== < /latexit > < latexit sha1_base64= '' KmEypMHsM7yk2RUtZ+c6bXbVosA= '' > AAAB/nicbVDLSsNAFJ3UV62vqLhyM1iEuimJFNRd0Y3LCvYBTSiTyU07dPJgZiKWUPBX3LhQxK3f4c6/cdJmoa0HBg7n3Ms9c7yEM6ks69sorayurW+UNytb2zu7e+b+QUfGqaDQpjGPRc8jEjiLoK2Y4tBLBJDQ49D1xje5330AIVkc3atJAm5IhhELGCVKSwPzyAmJGnkBzoKaoxj3AT+eTQdm1apbM+BlYhekigq0BuaX48c0DSFSlBMp+7aVKDcjQjHKYVpxUgkJoWMyhL6mEQlButks/hSfasXHQSz0ixSeqb83MhJKOQk9PZmHlYteLv7n9VMVXLoZi5JUQUTnh4KUYxXjvAvsMwFU8YkmhAqms2I6IoJQpRur6BLsxS8vk8553W7Ur+4a1eZ1UUcZHaMTVEM2ukBNdItaqI0oytAzekVvxpPxYrwbH/PRklHsHKI/MD5/AHIalSw= < /latexit > < latexit sha1_base64= '' WqAssZMNrO3tfegI87RjqSrSAbQ= '' > AAACAXicbVDLSsNAFJ34rPUVdSO4GSxCuymJFNSNFNy4rGAf0IYymUzaoZMHMzdiDXXjr7hxoYhb/8Kdf+OkzUJbD1w4nHMv997jxoIrsKxvY2l5ZXVtvbBR3Nza3tk19/ZbKkokZU0aiUh2XKKY4CFrAgfBOrFkJHAFa7ujq8xv3zGpeBTewjhmTkAGIfc5JaClvnnYCwgMXR+nfrkHXHgM35cfKpVJ3yxZVWsKvEjsnJRQjkbf/Op5EU0CFgIVRKmubcXgpEQCp4JNir1EsZjQERmwrqYhCZhy0ukHE3yiFQ/7kdQVAp6qvydSEig1Dlzdmd2r5r1M/M/rJuCfOykP4wRYSGeL/ERgiHAWB/a4ZBTEWBNCJde3YjokklDQoRV1CPb8y4ukdVq1a9WLm1qpfpnHUUBH6BiVkY3OUB1dowZqIooe0TN6RW/Gk/FivBsfs9YlI585QH9gfP4AK46WEQ== < /latexit > < latexit sha1_base64= '' 249iHdhtu7W0+U31BonYLaM+rVg= '' > AAAB+3icjVDLSsNAFL2pr1pfsS7dDBbBVUlEUHdFNy4r2Ac0oUwmk3boZBJmJtIS8ituXCji1h9x5984abtQUfDAhcM593IPJ0g5U9pxPqzKyura+kZ1s7a1vbO7Z+/XuyrJJKEdkvBE9gOsKGeCdjTTnPZTSXEccNoLJtel37unUrFE3OlZSv0YjwSLGMHaSEO77sVYj4MI5Z5mPKRoWgzthtt05kB/kwYs0R7a716YkCymQhOOlRq4Tqr9HEvNCKdFzcsUTTGZ4BEdGCpwTJWfz7MX6NgoIYoSaUZoNFe/XuQ4VmoWB2azTKp+eqX4mzfIdHTh50ykmaaCLB5FGUc6QWURKGSSEs1nhmAimcmKyBhLTLSpq/a/ErqnTfeseXl71mhdLeuowiEcwQm4cA4tuIE2dIDAFB7gCZ6twnq0XqzXxWrFWt4cwDdYb5/eNJRa < /latexit > make their explanations uncomparable to a general user : e.g . the distinct explanations in Fig . 1 describe the same model prediction . Ideally , a set of axioms ( agreed upon or debated ) would constrain the space of explanations , thus leading to a framework of curated methods that the user can choose from based on which axioms are relevant to the application . A further challenge on high-dimensional data is the sheer complexity of an explanation : in the methods described above , explanations have the same dimensionality as the data itself . Moreover , the importance of raw input features ( e.g . pixels ) are not individually meaningful to the user . Even when structured patterns emerge in an explanation ( e.g . in Fig . 1 ) this is not sufficient to answer higher-level questions . For example , did the subject ’ s protected attributes ( e.g . gender , age , or ethnicity ) have any influence on the model ’ s decision ? In this work , we develop methods for explaining predictions in terms of a digestible number of semantically meaningful concepts . We provide several options for transforming from the highdimensional raw features to a lower-dimensional latent space , which allow varying levels of user control . Regardless of the method used , transformation to a low-dimensional human-interpretable basis is a useful step , if explanations are to satisfy experts and non-experts alike . Once a set of semantic latent features is selected , one must choose an explainability algorithm to obtain quantitative information about why a certain model prediction was made . Fortunately , since the set of latent features is low-dimensional by construction , a Shapley-based approach becomes once again viable . In this work , we develop a method to apply Shapley explainability at the level of semantic latent features , thus providing a theoretically-controlled , model-agnostic foundation for explainability on high-dimensional data . Our main contributions are : • We introduce an approach to model explainability on high-dimensional data that involves encoding the raw input features into a digestible number of semantically meaningful latent features . We develop a procedure to apply Shapley explainability in this context , obtaining Shapley values that describe the high-dimensional model ’ s dependence on each semantic latent feature . • We demonstrate 3 methods to extract semantic features for the explanations : Fourier transforms , disentangled representations , and image-to-image translation . We benchmark our approach on dSprites – with known latent space – and showcase its effectiveness in computer vision tasks such as MNIST , CIFAR-10 , ImageNet , Describable Textures , and CelebA . 2 SEMANTIC SHAPLEY EXPLAINABILITY . In this section , we present a simple modular framework for obtaining meaningful low-dimensional explanations of model predictions on high-dimensional data . The framework contains two modules : ( i ) a mechanism for transforming from the high-dimensional space of raw model inputs to a lowdimensional space of semantic latent features , and ( ii ) an algorithm for generating explanations of the model ’ s predictions in terms of these semantic features . See Fig . 2 . We will begin by describing module ( ii ) in Sec . 2.1 , where we will show how to adapt Shapley explainability to latent features . Then we will describe several options for module ( i ) in Sec . 2.2 . 2.1 SHAPLEY VALUES FOR LATENT FEATURES . Shapley values ( Shapley , 1953 ) were developed in cooperative game theory to distribute the value v ( N ) earned by a team N = { 1 , 2 , . . . , n } among its players . The Shapley value φv ( i ) represents the marginal value added by player i upon joining the team , averaged over all orderings in which the team can form . In particular , φv ( i ) = ∑ S⊆N\ { i } |S| ! ( n− |S| − 1 ) ! n ! [ v ( S ∪ { i } ) − v ( S ) ] ( 1 ) where v ( S ) represents the value that a coalition S obtains without the rest of their teammates . Shapley values are the unique attribution method satisfying 4 natural axioms ( Shapley , 1953 ) . For example , they sum to the total value earned : ∑ i φv ( i ) = v ( N ) − v ( { } ) , and they are symmetric if two players i and j are functionally interchangeable : φv ( i ) = φv ( j ) . Shapley values thus serve as a well-founded explanation of an output ( the earned value ) in terms of inputs ( the players ) . The method can be adapted to explain the output of a machine learning model by interpreting the model ’ s input features x = ( x1 , . . . , xn ) as the players of a game . Consider a classification task , and let fy ( x ) be the model ’ s predicted probability that data point x belongs to class y . To apply Shapley explainability , one must define a value function representing the model ’ s expected output given only a subset of the input features xS . The most common choice is vfy ( x ) ( S ) = Ep ( x′ ) [ fy ( xS t x′S̄ ) ] ( 2 ) where p ( x′ ) is the distribution from which the data is drawn , S̄ is the complement of S , and xS tx′S̄ represents the spliced data point with in-coalition features from x and out-of-coalition features from x′ . Then , inserting the value function of Eq . ( 2 ) into the definition of Eq . ( 1 ) , one obtains Shapley values φfy ( x ) ( i ) representing the portion of the prediction fy ( x ) attributable to feature xi . The Shapley values presented above provide a local explanation of the model ’ s behaviour on an individual data point . For a global explanation , local values can be aggregated ( Frye et al. , 2020 ) Φf ( i ) = Ep ( x , y ) [ φfy ( x ) ( i ) ] ( 3 ) where p ( x , y ) is the joint distribution from which the labelled data is drawn . This aggregation preserves the Shapley axioms and is motivated by the sum rule : ∑ i Φf ( i ) = Ep ( x , y ) [ fy ( x ) ] − Ep ( x′ ) p ( y ) [ fy ( x ′ ) ] ( 4 ) which can be interpreted as the model accuracy above a class-balance baseline . Global Shapley values thus represent the portion of model accuracy attributable to each feature . To adapt the Shapley framework to latent features , suppose ( as in Fig . 2 ) that a mapping x → z exists to transform the raw model inputs x into a semantically meaningful representation z ( x ) , and that an ( approximate ) inverse mapping z → x̃ exists as well . Then we can obtain an explanation of the model prediction fy ( x ) in terms of the latent features z ( x ) by applying Shapley explainability instead to the function fy ( x̃ ( z ) ) at the point z = z ( x ) . To be precise , we define a value function ṽfy ( x ) ( S ) = Ep ( x′ ) [ fy ( x̃ ( zS ( x ) t zS̄ ( x′ ) ) ) ] ( 5 ) which represents the marginalisation of fy ( x̃ ( z ) ) over out-of-coalition features zS̄ . Here zS ( x ) is the in-coalition slice of z ( x ) , and zS̄ ( x′ ) is the out-of-coalition slice corresponding to a different data point . These get spliced together in latent space before transforming back to model-input-space and feeding into the model . Inserting the value function of Eq . ( 5 ) into the definition of Eq . ( 1 ) produces semantic Shapley values that explain fy ( x ) in terms of latent features zi . 2.2 LANDSCAPE OF SEMANTIC REPRESENTATIONS . A wide variety of methods exist to transform from the high-dimensional set of raw model inputs to an alternative set of features that offer semantic insight into the data being modelled . In this section , we consider several options for this semantic component of our approach to explainability .

Summary: This paper develops a Shapley value approach to explanation that uses low-dimensional latent features to explain the original input. In high-dimensional settings, Shapley values can be computationally intractable; as such, the authors adapt the characteristic (aka value) function $v(\cdot)$ to consider coalitions defined in latent space. This helps ensure that only feasible and plausible (i.e., semantically meaningful) perturbations are made. They consider three different latent encodings: Fourier Transforms, Disentanglement, and image-to-image translation (to isolate factors of variation).