Shapley Explanation Networks

Shapley values have become one of the most popular feature attribution explanation methods . However , most prior work has focused on post-hoc Shapley explanations , which can be computationally demanding due to its exponential time complexity and preclude model regularization based on Shapley explanations during training . Thus , we propose to incorporate Shapley values themselves as latent representations in deep models—thereby making Shapley explanations first-class citizens in the modeling paradigm . This intrinsic explanation approach enables layer-wise explanations , explanation regularization of the model during training , and fast explanation computation at test time . We define the Shapley transform that transforms the input into a Shapley representation given a specific function . We operationalize the Shapley transform as a neural network module and construct both shallow and deep networks , called SHAPNETs , by composing Shapley modules . We prove that our Shallow SHAPNETs compute the exact Shapley values and our Deep SHAPNETs maintain the missingness and accuracy properties of Shapley values . We demonstrate on synthetic and real-world datasets that our SHAPNETs enable layer-wise Shapley explanations , novel Shapley regularizations during training , and fast computation while maintaining reasonable performance . Code is available at https : //github.com/inouye-lab/ShapleyExplanationNetworks . 1 INTRODUCTION . Explaining the predictions of machine learning models has become increasingly important for many crucial applications such as healthcare , recidivism prediction , or loan assessment . Explanations based on feature importance are one key approach to explaining a model prediction . More specifically , additive feature importance explanations have become popular , and in Lundberg & Lee ( 2017 ) , the authors argue for theoretically-grounded additive explanation method called SHAP based on Shapley values—a way to assign credit to members of a group developed in cooperative game theory ( Shapley , 1953 ) . Lundberg & Lee ( 2017 ) defined three intuitive theoretical properties called local accuracy , missingness , and consistency , and proved that only SHAP explanations satisfy all three properties . Despite these elegant theoretically-grounded properties , exact Shapley value computation has exponential time complexity in the general case . To alleviate the computational issue , several methods have been proposed to approximate Shapley values via sampling ( Strumbelj & Kononenko , 2010 ) and weighted regression ( Kernel SHAP ) , a modified backpropagation step ( Deep SHAP ) ( Lundberg & Lee , 2017 ) , utilization of the expectation of summations ( Ancona et al. , 2019 ) , or making assumptions on underlying data structures ( Chen et al. , 2019 ) . To avoid approximation , the model class could be restricted to allow for simpler computation . Along this line , Lundberg et al . ( 2020 ) propose a method for computing exact Shapley values for tree-based models such as random forests or gradient boosted trees . However , even if this drawback is overcome , prior Shapley work has focused on post-hoc explanations , and thus , the explanation approach can not aid in model design or training . On the other hand , Generalized Additive Models ( GAM ) as explored in Lou et al . ( 2012 ; 2013 ) ; Caruana et al . ( 2015 ) ( via tree boosting ) , Chen et al . ( 2017a ) ( via kernel methods ) , and Wang et al . ( 2018 ) ; Agarwal et al . ( 2020 ) ( via neural networks ) can be seen as interpretable model class that exposes the exact Shapley explanation directly . In particular , the output of a GAM model is simply the summation of interaction-free functions : fGAM ( x ) = ∑ s fs ( xs ) , where fs ( · ) are univariate functions Published as a conference paper at ICLR 2021 < latexit sha1_base64= '' L+sbllxo615ueQ9HeA5HU0gtQjU= '' > AAAGhHicpVTLbtNQEJ0GMKW8UliysYgqFZEGO1DRDaiCDQsWBTVtpT4i27lprfhVPyBVlC2fwoot/At/AH/BmbHTpE3iVsJXTefOnXPmzuvakecmqWH8XqjcuHlLu714Z+nuvfsPHlaXH+0kYRY7quWEXhjv2VaiPDdQrdRNPbUXxcrybU/t2r33fL77RcWJGwbb6VmkDn3rOHC7rmOlULWrene13zbrer/dfPbmIDmN0wH2R83nUBw1h2tmu1ozGoZ8+rRgFkKNim8rXK58pwPqUEgOZeSTooBSyB5ZlGDtk0kGRdAd0gC6GJIr54qGtARsBisFCwvaHn6PsdsvtAH2zJkI2oEXD38xkDqtABPCLobM3nQ5z4SZtfO4B8LJdzvDf7vg8qFN6QTaq3Ajy+vjbFiVx5pSlzYkRhcxR6Lh6J3CSyZZ48j0iahTMETQsdzBeQzZEeSoDrpgEskN596S8z9iyVreO4VtRn9Lo2CfnsTxP5GwT0+q+BVybr8GhD5hP59/G93E3F3ggwvM+VoRzkTytAFeG6ycBcbp56iyKE/QFXlOOUfdCe6xh7GNJ7fuXStv+pzFjAl4fCDzCrfoM32U+uZ+mDWVDgjAWdZnIVZP+sKGxay7R7hNiLiUTIArGeGKrNEpWC2JiP3qBUveP1fNxDizs3zaMp+M4htyH3agHU2Ofj6JPN1lvgJ5ObgqeSfP8sVvgF8g0ynuvAd4Xkx6QU2qQ1LyXjVK/Pril/M2qshwyu9lmxDYWOaTu2U+N88ty3zDPg3x/pqXX9tpYafZMNcbxqdXtc13xUu8SE/oKa0irte0SR9oC13k0Df6QT/pl6Zpde2ltp6bVhYKzGO68Glv/wFXaTvj < /latexit > f ( x1 , x2 ) = q x21 + x 2 2 1 < latexit sha1_base64= '' a9KOP9WfSfBx818hxz6GtxeECuY= '' > AAAGg3icpVTLbtNQEJ0Gakp5pbBkgUVUqYg22FEQ3VSqYMOCRUFNW6mpItu5aa34hR+QKsqSX2HDFj6GP4C/4MzYadImcSvhqzjjuXPO3HldO/LcJDWM30uVW7eXtTsrd1fv3X/w8FF17fFBEmaxo1pO6IXxkW0lynMD1Urd1FNHUaws3/bUod1/x/uHX1ScuGGwn55H6sS3TgO35zpWClWn+qy3MeiYm/qg03ixA6mdur5K2okbDJvQjTrVmlE35NFnBbMQalQ8e+Fa5Tu1qUshOZSRT4oCSiF7ZFGCdUwmGRRBd0JD6GJIruwrGtEqsBmsFCwsaPt4n+LruNAG+GbORNAOvHj4xUDqtA5MCLsYMnvTZT8TZtYu4h4KJ5/tHP92weVDm9IZtNfhxpY3x9mwKo81pR5tS4wuYo5Ew9E7hZdMssaR6VNRp2CIoGO5i/0YsiPIcR10wSSSG869Jft/xJK1/O0Uthn9LY2CfXoSx/9Ewj49qeJXyLn9FhD6lP1i/n10E3P3gA8uMedrXTgTydM2eG2wchYYp1+gyqI8Q1fkOeUc9aa4Jx4mNp6cun+jvOkLFjMm4PGBzCvcok/0Qeqb+2HWVDogAGdZn4VYfekLGxbzzh7hNCHiUjIBrmSEK7JFn8FqSUTsVy9Y8v65biYmmZ3n05b5ZBSfkPuwC+14cvSLSeTpLvMVyM3BVck7eZ4vvgP8ApnOcOc9wPNi0itq0CYkJfdVvcSvL345b+OKjGb8XrUJgY1lPrlbFnPz3LLMJxwQ37/m1dt2Vjho1M3XdeNjs7b7triJV+gpPacNxPWGduk97aGLHPpGP+gn/dKWtZdaQ2vmppWlAvOELj3azj9aejxi < /latexit > f ( x1 , x2 ) = x1 ⇥ sin 4x2 < latexit sha1_base64= '' f+iIV9NkU0Rk1FNjiXyQwwz7rcU= '' > AAAGZXicpVTNbtNAEJ4Gakqh0ALiwgGLqBIHGuwIRI8VXDhwKNC0lUpV2c6mtbL+wV5DqyqPwBXegHfiCYC34Jux06RN4lbCqzjj2fm+2flbP9Vhbhzn11zj2vV568bCzcVbt5fu3F1eubedJ0UWqE6Q6CTb9b1c6TBWHRMarXbTTHmRr9WO33/D+ztfVJaHSbxlTlK1H3mHcdgLA89A9fH4oH2w3HRajjz2pOBWQpOqZzNZafykT9SlhAIqKCJFMRnImjzKsfbIJYdS6PbpFLoMUij7iga0CGwBKwULD9o+3of42qu0Mb6ZMxd0AC8avwxIm1aBSWCXQWZvtuwXwszaWdynwslnO8G/X3FF0Bo6gvYy3NDy6jgfVvWxGurRusQYIuZUNBx9UHkpJGscmT0WtQFDCh3LXexnkANBDutgCyaX3HDuPdn/I5as5e+gsi3ob20U7FNLHP8TCfvUUsWvkEv7NSDsMfvZ/FvoJubuAR+fYy7XqnDmkqd18Ppg5Swwzj5D1UV5hK4oc8o56o1xjzyMbLScun+lvNkzFjPm4ImALCvcoQ/0Tupb+mFWIx0Qg7OuzxKsvvSFD4tpZ09xmgRxKZmAUDLCFVmjz2D1JCL2a1csZf9cNhOjzE7z6ct8MopPyH3YhXY4OfbZJPJ01/mK5ebgqpSdPM0X3wFRhTQT3GUP8Ly49Jza9AySkvuqVeM3Er+ct2FFBhN+L9okwGYyn9wts7l5blnmEx7TAPeve/G2nRS22y33Zct5/6K58bq6iRfoET2hp4jrFW3QW9pEFwXg/Ubf6cf8b2vJemA9LE0bcxXmPp17rMf/AG7jMm8= < /latexit > x2 °2 0 2 °2 °1 0 1 2 ci rc le Input °0.5 0.0 0.5 °0.5 0.0 0.5 Shapley Representation °2 0 2 °2 °1 0 1 2 Li ne ar £ si n Input °1 0 1 °1 0 1 Shapley Representation °0.90 °0.45 0.00 0.45 0.90 1.35 1.80 °0.90 °0.45 0.00 0.45 0.90 1.35 1.80 °1.8 °1.2 °0.6 0.0 0.6 1.2 1.8 °3.2 °2.4 °1.6 °0.8 0.0 0.8 1.6 2.4 3.2 °2 0 2 °2 °1 0 1 2 ci rc le Input 0.5 1.0 1.5 0.5 1.0 1.5 Shapley Representation °2 0 2 °2 °1 0 1 2 Li ne ar £ si n Input °1 0 1 °1 0 1 Shapley Representation °0.90 °0.45 0.00 0.45 0.90 1.35 1.80 °0.8 °0.4 0.0 0.4 0.8 1.2 1.6 2.0 2.4 2.8 °1.8 °1.2 °0.6 0.0 0.6 1.2 1.8 °3.2 °2.4 °1.6 °0.8 0.0 0.8 1.6 2.4 3.2 < latexit sha1_base64= '' EzP2813inJZ9gcRf6vdtsAy3GNQ= '' > AAAGZXicpVTNTtRQFD6MUhFFQY0bFzZOSFzI2BKILIluXLhAZYAECWk7d6CZ2x/bW4WQeQS3+ga+k0+gvoXfOe0wAzNTSOzNdE7PPd937vm7fqrD3DjOr5nGjZuz1q252/N37i7cu7+49GAnT4osUO0g0Um253u50mGs2iY0Wu2lmfIiX6tdv/eG93e/qCwPk3jbnKbqIPKO4rAbBp6B6uPJoXu42HRajjz2uOBWQpOqZytZavykT9ShhAIqKCJFMRnImjzKsfbJJYdS6A7oDLoMUij7ivo0D2wBKwULD9oe3kf42q+0Mb6ZMxd0AC8avwxIm5aBSWCXQWZvtuwXwszaadxnwslnO8W/X3FF0Bo6hvYq3MDy+jgfVvWxGurShsQYIuZUNBx9UHkpJGscmT0StQFDCh3LHexnkANBDupgCyaX3HDuPdn/I5as5e+gsi3ob20U7FNLHP8TCfvUUsWvkEv7FSDsEfvp/NvoJubuAh9fYC7XsnDmkqcN8Ppg5Swwzj5H1UV5jK4oc8o56o5wDz0MbbScunetvNlTFjPm4ImALCvcpg/0Tupb+mFWIx0Qg7OuzxKsnvSFD4tJZ09xmgRxKZmAUDLCFVmhz2D1JCL2a1csZf9cNRPDzE7y6ct8MopPyH3YgXYwOfb5JPJ01/mK5ebgqpSdPMkX3wFRhTRj3GUP8Ly49JJW6QUkJfdVq8ZvJH45b4OK9Mf8XrZJgM1kPrlbpnPz3LLMJzyhPu5f9/JtOy7srLbc9Zbzfq25+bq6iefoCT2j54jrFW3SW9pCFwXg/Ubf6cfsb2vBemQ9Lk0bMxXmIV14rKf/AGjkMm4= < /latexit > x1 < latexit sha1_base64= '' +PQNwZwMPRqf1dadH50k+pNYOr0= '' > AAAGdHicpVTLbtNQEJ0GGkp5pbCEhUUUqUhtalcguqxgw4JFQU1bqaki27lprPiFfV1aRfkTtvSf+AP4AdacGTtN2iRuJXwVZzx3zpk7r+vEvpdq0/y1VLl3f7n6YOXh6qPHT54+q609P0ijLHFVy438KDly7FT5Xqha2tO+OooTZQeOrw6dwUfePzxTSepF4b6+iNVJYJ+GXs9zbQ1Vp1Zrx32vY633NtpOMDwfvenU6mbTlMeYFaxCqFPx7EVrlUtqU5cicimjgBSFpCH7ZFOKdUwWmRRDd0JD6BJInuwrGtEqsBmsFCxsaAd4n+LruNCG+GbOVNAuvPj4JUAa1AAmgl0Cmb0Zsp8JM2sXcQ+Fk892gX+n4Aqg1dSH9jbc2PLuOAdW5bFq6tGOxOgh5lg0HL1beMkkaxyZMRW1BkMMHctd7CeQXUGO62AIJpXccO5t2f8tlqzlb7ewzehPaRTs05c4/icS9ulLFb9Dzu03gTCm7Bfz76ObmLsHfHiNOV8N4UwlTzvgdcDKWWCccYUqi7KPrshzyjnqTXFPPExsfDn14E55MxYsZkzBEwCZV7hFX+mz1Df3w6xaOiAEZ1mfRVgD6QsHFvPOHuM0EeJSMgGeZIQrsknfwGpLROzXKFjy/rltJiaZnefTkflkFJ+Q+7AL7XhyjKtJ5Oku8xXKzcFVyTt5ni++A4ICqWe48x7gebFoi7ZpA5KS+6pZ4jcQv5y3cUVGM35v2kTAJjKf3C2LuXluWeYTntMI969187adFQ62m9a7pvnlbX33Q3ETr9BLek3riOs97dIn2kMXuXRGP+gnXS7/rb6q1quN3LSyVGBe0LWn2vwHfkY3xA== < /latexit > 1 ( f , x ) < latexit sha1_base64= '' f+iIV9NkU0Rk1FNjiXyQwwz7rcU= '' > AAAGZXicpVTNbtNAEJ4Gakqh0ALiwgGLqBIHGuwIRI8VXDhwKNC0lUpV2c6mtbL+wV5DqyqPwBXegHfiCYC34Jux06RN4lbCqzjj2fm+2flbP9Vhbhzn11zj2vV568bCzcVbt5fu3F1eubedJ0UWqE6Q6CTb9b1c6TBWHRMarXbTTHmRr9WO33/D+ztfVJaHSbxlTlK1H3mHcdgLA89A9fH4oH2w3HRajjz2pOBWQpOqZzNZafykT9SlhAIqKCJFMRnImjzKsfbIJYdS6PbpFLoMUij7iga0CGwBKwULD9o+3of42qu0Mb6ZMxd0AC8avwxIm1aBSWCXQWZvtuwXwszaWdynwslnO8G/X3FF0Bo6gvYy3NDy6jgfVvWxGurRusQYIuZUNBx9UHkpJGscmT0WtQFDCh3LXexnkANBDutgCyaX3HDuPdn/I5as5e+gsi3ob20U7FNLHP8TCfvUUsWvkEv7NSDsMfvZ/FvoJubuAR+fYy7XqnDmkqd18Ppg5Swwzj5D1UV5hK4oc8o56o1xjzyMbLScun+lvNkzFjPm4ImALCvcoQ/0Tupb+mFWIx0Qg7OuzxKsvvSFD4tpZ09xmgRxKZmAUDLCFVmjz2D1JCL2a1csZf9cNhOjzE7z6ct8MopPyH3YhXY4OfbZJPJ01/mK5ebgqpSdPM0X3wFRhTQT3GUP8Ly49Jza9AySkvuqVeM3Er+ct2FFBhN+L9okwGYyn9wts7l5blnmEx7TAPeve/G2nRS22y33Zct5/6K58bq6iRfoET2hp4jrFW3QW9pEFwXg/Ubf6cf8b2vJemA9LE0bcxXmPp17rMf/AG7jMm8= < /latexit > x2 < latexit sha1_base64= '' EzP2813inJZ9gcRf6vdtsAy3GNQ= '' > AAAGZXicpVTNTtRQFD6MUhFFQY0bFzZOSFzI2BKILIluXLhAZYAECWk7d6CZ2x/bW4WQeQS3+ga+k0+gvoXfOe0wAzNTSOzNdE7PPd937vm7fqrD3DjOr5nGjZuz1q252/N37i7cu7+49GAnT4osUO0g0Um253u50mGs2iY0Wu2lmfIiX6tdv/eG93e/qCwPk3jbnKbqIPKO4rAbBp6B6uPJoXu42HRajjz2uOBWQpOqZytZavykT9ShhAIqKCJFMRnImjzKsfbJJYdS6A7oDLoMUij7ivo0D2wBKwULD9oe3kf42q+0Mb6ZMxd0AC8avwxIm5aBSWCXQWZvtuwXwszaadxnwslnO8W/X3FF0Bo6hvYq3MDy+jgfVvWxGurShsQYIuZUNBx9UHkpJGscmT0StQFDCh3LHexnkANBDupgCyaX3HDuPdn/I5as5e+gsi3ob20U7FNLHP8TCfvUUsWvkEv7FSDsEfvp/NvoJubuAh9fYC7XsnDmkqcN8Ppg5Swwzj5H1UV5jK4oc8o56o5wDz0MbbScunetvNlTFjPm4ImALCvcpg/0Tupb+mFWIx0Qg7OuzxKsnvSFD4tJZ09xmgRxKZmAUDLCFVmhz2D1JCL2a1csZf9cNRPDzE7y6ct8MopPyH3YgXYwOfb5JPJ01/mK5ebgqpSdPMkX3wFRhTRj3GUP8Ly49JJW6QUkJfdVq8ZvJH45b4OK9Mf8XrZJgM1kPrlbpnPz3LLMJzyhPu5f9/JtOy7srLbc9Zbzfq25+bq6iefoCT2j54jrFW3SW9pCFwXg/Ubf6cfsb2vBemQ9Lk0bMxXmIV14rKf/AGjkMm4= < /latexit > x1 < latexit sha1_base64= '' XegFC4q7LlcHkMh76AKV9FRgFVo= '' > AAAGdHicpVTLTttQEB3SklL6Cu2yXViNIlEJghMVwRK1my66oBUBJIIi27khVvyqfU1BUf6k2/JP/YP2B7rumbFDAkkMUn0VZzx3zpk7r2tHnpto0/y1VHrwcLn8aOXx6pOnz56/qKy9PEzCNHZUywm9MD62rUR5bqBa2tWeOo5iZfm2p47swUfePzpXceKGwYG+jNSpb50Fbs91LA1Vp1JpR32301zvbbRtf3gxetepVM26KY8xKzRyoUr5sx+ula6oTV0KyaGUfFIUkIbskUUJ1gk1yKQIulMaQhdDcmVf0YhWgU1hpWBhQTvA+wxfJ7k2wDdzJoJ24MXDLwbSoBowIexiyOzNkP1UmFm7iHsonHy2S/zbOZcPraY+tHfhxpb3x9mwKo5VU492JUYXMUei4eid3EsqWePIjKmoNRgi6FjuYj+G7AhyXAdDMInkhnNvyf5vsWQtfzu5bUp/CqNgn57E8T+RsE9Pqvgdcma/CYQxZb+Y/wDdxNw94IMbzNmqCWciedoFrw1WzgLjjGtUUZR9dEWWU85Rb4p74mFi48mpB/fKm7FgMWMCHh/IrMIt+kqfpb6ZH2bV0gEBOIv6LMQaSF/YsJh39ginCRGXkglwJSNckU36BlZLImK/Rs6S9c9dMzHJ7Dyftswno/iE3IddaMeTY1xPIk93ka9Abg6uStbJ83zxHeDnSD3DnfUAz0uDtqhJG5CU3Ff1Ar+++OW8jSsymvF72yYENpb55G5ZzM1zyzKf8IJGuH8bt2/bWeGwWW9s180v76t7H/KbeIVe01taR1w7tEefaB9d5NA5/aCfdLX8t/ymXC3XMtPSUo55RTeecv0fhE83xQ== < /latexit > 2 ( f , x ) < latexit sha1_base64= '' +PQNwZwMPRqf1dadH50k+pNYOr0= '' > AAAGdHicpVTLbtNQEJ0GGkp5pbCEhUUUqUhtalcguqxgw4JFQU1bqaki27lprPiFfV1aRfkTtvSf+AP4AdacGTtN2iRuJXwVZzx3zpk7r+vEvpdq0/y1VLl3f7n6YOXh6qPHT54+q609P0ijLHFVy438KDly7FT5Xqha2tO+OooTZQeOrw6dwUfePzxTSepF4b6+iNVJYJ+GXs9zbQ1Vp1Zrx32vY633NtpOMDwfvenU6mbTlMeYFaxCqFPx7EVrlUtqU5cicimjgBSFpCH7ZFOKdUwWmRRDd0JD6BJInuwrGtEqsBmsFCxsaAd4n+LruNCG+GbOVNAuvPj4JUAa1AAmgl0Cmb0Zsp8JM2sXcQ+Fk892gX+n4Aqg1dSH9jbc2PLuOAdW5bFq6tGOxOgh5lg0HL1beMkkaxyZMRW1BkMMHctd7CeQXUGO62AIJpXccO5t2f8tlqzlb7ewzehPaRTs05c4/icS9ulLFb9Dzu03gTCm7Bfz76ObmLsHfHiNOV8N4UwlTzvgdcDKWWCccYUqi7KPrshzyjnqTXFPPExsfDn14E55MxYsZkzBEwCZV7hFX+mz1Df3w6xaOiAEZ1mfRVgD6QsHFvPOHuM0EeJSMgGeZIQrsknfwGpLROzXKFjy/rltJiaZnefTkflkFJ+Q+7AL7XhyjKtJ5Oku8xXKzcFVyTt5ni++A4ICqWe48x7gebFoi7ZpA5KS+6pZ4jcQv5y3cUVGM35v2kTAJjKf3C2LuXluWeYTntMI969187adFQ62m9a7pvnlbX33Q3ETr9BLek3riOs97dIn2kMXuXRGP+gnXS7/rb6q1quN3LSyVGBe0LWn2vwHfkY3xA== < /latexit > 1 ( f , x ) < latexit sha1_base64= '' XegFC4q7LlcHkMh76AKV9FRgFVo= '' > AAAGdHicpVTLTttQEB3SklL6Cu2yXViNIlEJghMVwRK1my66oBUBJIIi27khVvyqfU1BUf6k2/JP/YP2B7rumbFDAkkMUn0VZzx3zpk7r2tHnpto0/y1VHrwcLn8aOXx6pOnz56/qKy9PEzCNHZUywm9MD62rUR5bqBa2tWeOo5iZfm2p47swUfePzpXceKGwYG+jNSpb50Fbs91LA1Vp1JpR32301zvbbRtf3gxetepVM26KY8xKzRyoUr5sx+ula6oTV0KyaGUfFIUkIbskUUJ1gk1yKQIulMaQhdDcmVf0YhWgU1hpWBhQTvA+wxfJ7k2wDdzJoJ24MXDLwbSoBowIexiyOzNkP1UmFm7iHsonHy2S/zbOZcPraY+tHfhxpb3x9mwKo5VU492JUYXMUei4eid3EsqWePIjKmoNRgi6FjuYj+G7AhyXAdDMInkhnNvyf5vsWQtfzu5bUp/CqNgn57E8T+RsE9Pqvgdcma/CYQxZb+Y/wDdxNw94IMbzNmqCWciedoFrw1WzgLjjGtUUZR9dEWWU85Rb4p74mFi48mpB/fKm7FgMWMCHh/IrMIt+kqfpb6ZH2bV0gEBOIv6LMQaSF/YsJh39ginCRGXkglwJSNckU36BlZLImK/Rs6S9c9dMzHJ7Dyftswno/iE3IddaMeTY1xPIk93ka9Abg6uStbJ83zxHeDnSD3DnfUAz0uDtqhJG5CU3Ff1Ar+++OW8jSsymvF72yYENpb55G5ZzM1zyzKf8IJGuH8bt2/bWeGwWW9s180v76t7H/KbeIVe01taR1w7tEefaB9d5NA5/aCfdLX8t/ymXC3XMtPSUo55RTeecv0fhE83xQ== < /latexit > 2 ( f , x ) < latexit sha1_base64= '' f+iIV9NkU0Rk1FNjiXyQwwz7rcU= '' > AAAGZXicpVTNbtNAEJ4Gakqh0ALiwgGLqBIHGuwIRI8VXDhwKNC0lUpV2c6mtbL+wV5DqyqPwBXegHfiCYC34Jux06RN4lbCqzjj2fm+2flbP9Vhbhzn11zj2vV568bCzcVbt5fu3F1eubedJ0UWqE6Q6CTb9b1c6TBWHRMarXbTTHmRr9WO33/D+ztfVJaHSbxlTlK1H3mHcdgLA89A9fH4oH2w3HRajjz2pOBWQpOqZzNZafykT9SlhAIqKCJFMRnImjzKsfbIJYdS6PbpFLoMUij7iga0CGwBKwULD9o+3of42qu0Mb6ZMxd0AC8avwxIm1aBSWCXQWZvtuwXwszaWdynwslnO8G/X3FF0Bo6gvYy3NDy6jgfVvWxGurRusQYIuZUNBx9UHkpJGscmT0WtQFDCh3LXexnkANBDutgCyaX3HDuPdn/I5as5e+gsi3ob20U7FNLHP8TCfvUUsWvkEv7NSDsMfvZ/FvoJubuAR+fYy7XqnDmkqd18Ppg5Swwzj5D1UV5hK4oc8o56o1xjzyMbLScun+lvNkzFjPm4ImALCvcoQ/0Tupb+mFWIx0Qg7OuzxKsvvSFD4tpZ09xmgRxKZmAUDLCFVmjz2D1JCL2a1csZf9cNhOjzE7z6ct8MopPyH3YhXY4OfbZJPJ01/mK5ebgqpSdPM0X3wFRhTQT3GUP8Ly49Jza9AySkvuqVeM3Er+ct2FFBhN+L9okwGYyn9wts7l5blnmEx7TAPeve/G2nRS22y33Zct5/6K58bq6iRfoET2hp4jrFW3QW9pEFwXg/Ubf6cf8b2vJemA9LE0bcxXmPp17rMf/AG7jMm8= < /latexit > x2 < latexit sha1_base64= '' dOzJUIAhwdtY7mtaVsXQVGoLNyY= '' > AAAGhnicpVRLTxNRFD6g1ooPii7d3NCQYAJ1hkggrhrduHCBhAIJEDIzvYVJ5+XMHYQ03ftLXLrVv+I/0H/hd85MabEPSJybTs+ce77v3PO6bhL4mbGsX3Pz9+4/qDysPlp4/OTps8Xa0vP9LM5TT7e8OIjTQ9fJdOBHumV8E+jDJNVO6Ab6wO2+5/2DC51mfhztmatEn4TOWeR3fM8xUJ3Wlo+NvjQ9ebud3q7uNNSFE+Q6679V/VV7zX51WqvbDUseZY0Jg606lc9OvDT/jY6pTTF5lFNImiIykANyKMM6IpssSqA7oR50KSRf9jX1aQHYHFYaFg60XbzP8HVUaiN8M2cmaA9eAvxSIBWtABPDLoXM3pTs58LM2mncPeHks13h3y25QmgNnUN7G25geXecC6vZsRrq0LbE6CPmRDQcvVd6ySVrHJkaidqAIYGO5Tb2U8ieIAd1UILJJDece0f2f4sla/nbK21z+jMzCvYZSBz/Ewn7DKSKXyAX9utAqBH76fx76Cbm7gAf3WAu1opwZpKnbfC6YOUsME5do2ZFeY6uKHLKOeqMcA89DG0COXX3TnlTUxYzZuAJgSwq3KJd+ij1Lfwwq5EOiMA5q89irK70hQuLSWdPcJoYcWmZAF8ywhVZp89gdSQi9qtKlqJ/bpuJYWYn+XRlPhnFJ+Q+bEM7mBx1PYk83bN8RXJzcFWKTp7ki++AsESaMe6iB3hebHpNG7QGSct91ZjhNxS/nLdBRfpjfv+1iYFNZT65W6Zz89yyzCe8pP7o/Ttd2N9o2JsN69ObevNdeRNX6SUt0yri2qImfaAddJFHX+k7/aCflWqlUdmsbBWm83Ml5gXdeCrNvw6uPf0= < /latexit > Ref . values : ( 1 , 1 ) < latexit sha1_base64= '' Fi2qoCAxSadKqu9LvFFLL5Ggwxg= '' > AAAGhnicpVTLTttQEB1om6b0QWiX3VhESFQC10FFoK6idtNFFxQRQAKEbOcarPhV+5qCouz7JV122/5K/6D9i54ZOyQ0iUGqr+KM5845c+d1nSTwM21Zv+bm791/UHtYf7Tw+MnTZ4uNpef7WZynruq4cRCnh46dqcCPVEf7OlCHSars0AnUgdN7z/sHFyrN/Dja01eJOgnts8j3fNfWUJ02lo+1utR9eTtef1d5pnFhB7nKBm+Nwaq11np12mi2TEsew5oQhltNKp+deGn+Gx1Tl2JyKaeQFEWkIQdkU4Z1RC2yKIHuhPrQpZB82Vc0oAVgc1gpWNjQ9vA+w9dRqY3wzZyZoF14CfBLgTRoBZgYdilk9mbIfi7MrJ3F3RdOPtsV/p2SK4RW0zm0t+GGlnfHObCqjlWTR9sSo4+YE9Fw9G7pJZescWTGWNQaDAl0LHexn0J2BTmsgyGYTHLDubdl/7dYspa/3dI2pz+VUbDPQOL4n0jYZyBV/AK5sF8Hwhizn82/h25ibg/46AZzsVaEM5M8bYPXAStngXHGNaoqynN0RZFTzpE3xj3yMLIJ5NS9O+XNmLGYMQNPCGRR4Q7t0kepb+GHWbV0QATOqj6LsXrSFw4spp09wWlixKVkAnzJCFdknT6D1ZaI2K9RshT9c9tMjDI7zacj88koPiH3YRfa4eQY15PI013lK5Kbg6tSdPI0X3wHhCVST3AXPcDz0qLXtEFrkJTcV2aF31D8ct6GFRlM+P3XJgY2lfnkbpnNzXPLMp/wkgbj9+9sYX/DbG2a1qc3zfa78iau00taplXEtUVt+kA76CKXvtJ3+kE/a/WaWdusbRWm83Ml5gXdeGrtvwisPfw= < /latexit > Ref . values : ( 0 , 1 ) Figure 1 : Shapley representations span beyond one-dimensional manifolds and depend on both the inner function and the reference values . In both groups , the gray-scale background indicates the respective function value while the rainbow-scale color indicates correspondence between input ( left ) a d Shapley representation ( right ) along with function values—red means highest and purple means lowest function values . The red cross represents the reference values . More details in subsection 2.1. that can be arbitrarily complex . Interestingly , the Shapley explanation values , often denoted φs ( x , f ) , for a GAM are exactly the values of the independent function , i.e. , ∀s , φs ( x , fGAM ) = fs ( xs ) . Hence , the prediction and the corresponding exact SHAP explanation can be computed simultaneously for GAM models . However , GAM models are inherently limited in their representational power , particularly for perceptual data such as images in which deep networks are state-of-the-art . Thus , prior work is either post-hoc ( which precludes leveraging the method during training ) or limited in its representational power ( e.g. , GAMs ) . To overcome these drawbacks , we propose to incorporate Shapley values themselves as learned latent representations ( as opposed to post-hoc ) in deep models—thereby making Shapley explanations first-class citizens in the modeling paradigm . Intuitive illustrations of such representation are provided in Fig . 1 and detailed discussion in subsection 2.1 . We summarize our core contributions as follows : • We formally define the Shapley transform and prove a simple but useful linearity property for constructing networks . • We develop a novel network architecture , the SHAPNETs , that includes Shallow and Deep SHAPNETs and that intrinsically provides layer-wise explanations ( i.e. , explanations at every layer of the network ) in the same forward pass as the prediction . • We prove that Shallow SHAPNET explanations are the exact Shapley values—thus satisfying all three SHAP properties—and prove that Deep SHAPNET explanations maintain the missingness and local accuracy properties . • To reduce computation , we propose an instance-specific dynamic pruning method for Deep SHAPNETs that can skip unnecessary computation . • We enable explanation regularization based on Shapley values during training because the explanation is a latent representation in our model . • We demonstrate empirically that our SHAPNETs can provide these new capabilities while maintaining comparable performance to other deep models . Dedicated Related Works Section We present related works above and in the text where appropriate . Due to space limit , we refer to Appendix D for a dedicated literature review section . 2 SHAPLEY EXPLANATION NETWORKS . Background We give a short introduction to SHAP explanations and their properties as originally introduced in Lundberg & Lee ( 2017 ) . Given a model f : Rd 7→ R that is not inherently interpretable ( e.g. , neural nets ) , additive feature-attribution methods form a linear approximation of the function over simplified binary inputs , denoted z ∈ { 0 , 1 } d , indicating the “ presence ” and “ absence ” of each feature , respectively : i.e. , a local linear approximation η ( z ) = a0 + ∑d i=1 aizi . While there are different ways to model “ absence ” and “ presence ” , in this work , we take a simplified viewpoint : “ presence ” means that we keep the original value whereas “ absence ” means we replace the original value with a reference value , which has been validated in Sundararajan & Najmi ( 2020 ) as Baseline Shapley . If we denote the reference vector for all features by r , then we can define a simple mapping function between z and x as Ψx , r ( z ) = z x + ( 1 − z ) r , where denotes element-wise product ( eg , Ψx , r ( [ 0 , 1 , 0 , 1 , 1 ] ) = [ r1 , x2 , r3 , x4 , x5 ] ) . A simple generalization is to group certain features together and consider including or removing all features in the group . Lundberg & Lee ( 2017 ) propose three properties that additive feature attribution methods should intuitively satisfy . The first property called local accuracy states that the approximate model η at z = 1 should match the output of the model f at the corresponding x , i.e. , f ( x ) = η ( 1 ) = ∑d i=0 ai . The second property called missingness formalizes the idea that features that are “ missing ” from the input x ( or correspondingly the zeros in z ) should have zero attributed effect on the output of the approximation , i.e. , zi = 0⇒ ai = 0 . Finally , the third property called consistency formalizes the idea that if one model always sees a larger effect when removing a feature , the attribution value should be larger ( see ( Lundberg & Lee , 2017 ) for full definitions of the three properties and their discussions ) . Definition 1 ( SHAP Values ( Lundberg & Lee , 2017 ) ) . SHAP values are defined as : φi ( f , x ) , 1 d ∑ z∈Zi ( d− 1 ‖z‖1 ) −1 [ f ( Ψx , r ( z ∪ { i } ) ) − f ( Ψx , r ( z ) ) ] , ( 1 ) where ‖z‖1 is the ` 1 norm of z given a set Zi = { z ∈ { 0 , 1 } d : zi = 0 } and z ∪ { i } represents setting the i-th element in z to be 1 instead of 0 . 2.1 SHAPLEY TRANSFORM AND SHAPLEY REPRESENTATION . For the sake of generality , we will define the Shapley transform in terms of tensors ( or multidimensional arrays ) . Below in Def . 2 we make a distinction between tensor dimensions that need to be explained and other tensor dimensions . For example , in image classification , usually an explanation ( or attribution ) value is required for each pixel ( i.e. , the spatial dimensions ) but all channels are grouped together ( i.e. , an attribution is provided for each pixel but not for each channel of each pixel ) . We generalize this idea in the following definitions of explainable and channel dimensions . Definition 2 ( Explainable Dimensions and Channel Dimensions ) . Given a tensor representation x ∈ RD×C ≡ R ( d1×d2×··· ) × ( c1×c2×··· ) , the tensor dimensions D ≡ d1 × d2 × · · · that require attribution will be called explainable dimensions and the other tensor dimensions C ≡ c1 × c2 × · · · will be called the channel dimensions . As a simplest example , if the input is a vector x ∈ Rd×1 , then D = d and C = 1 , i.e. , we have one ( since C = 1 ) importance value assigned to each feature . For images in the space Rh×w×c where h , w denote the height and width , respectively , the explainable dimensions correspond to the spatial dimensions ( i.e. , D = h× w ) and the channel dimensions correspond to the single channel dimension ( i.e. , C = c ) . While the previous examples discuss tensor dimensions of the input , our Shapley transforms can also operate on latent representations ( e.g. , in a neural net ) . For example , our latent representations for image models could be in the space Rw×h×c1×c2 , where the explainable dimensions correspond to spatial dimensions ( i.e. , D = h×w ) and there are two channel dimensions ( i.e. , C = c1 × c2 ) . Given this distinction between explainable and channel dimensions , we can now define the Shapley transform and Shapley representation . Definition 3 ( Shapley Transform ) . Given an arbitrary function f ∈ F : RD×C 7→ RC′ the Shapley transform Ω : RD×C ×F 7→ RD×C′ is defined as : [ Ω ( x , f ) ] i , j = φi ( x , fj ) , ∀i ∈ ID , j ∈ IC′ , ( 2 ) where ID denotes the set of all possible indices for values captured by the explainable dimensions D ( and similarly for IC′ ) and fj denotes the scalar function corresponding to the j-th output of f . Definition 4 ( Shapley Representation ) . Given a function f ∈ F as in Def . 3 and a tensor x ∈ RD×C , we simply define the Shapley representation to be : Z , Ω ( x , f ) ∈ RD×C′ . Notice that in the Shapley transform , we always keep the explainable dimensions D unchanged . However , the channel dimensions of the output representation space , i.e. , C ′ , are determined by the co-domain of the function f . For example , if f is a scalar function ( i.e. , C ′ = 1 ) , then the attribution for the explainable dimensions is a scalar . However , if f is a vector-valued function , then the attribution is a vector ( corresponding to the vector output of f ) . A multi-class classifier is a simple example of an f that is a vector-valued function ( e.g. , C ′ = 10 for 10-class classification tasks ) . In summary , the Shapley transform maintains the explainable tensor dimensions , but each explainable element of D may be associated to a tensor of attribution values corresponding to each output of f .

Summary: The paper proposes to incorporate Shapley values as latent representations in deep models. Specifically, the paper constructs Shallow SHAPNETs that computes the exact Shapley values. The paper also constructs Deep SHAPNETs that maintain the missingness and accuracy properties of Shapley values. The effectiveness of the proposed SHAPNETs is demonstrated through experiments on synthetic and real-world data.