Representer Point Selection via Local Jacobian Expansion for Post-hoc Classifier Explanation of Deep Neural Networks and Ensemble Models

1 Introduction . Deep learning as well as ensemble methods such as XGBoost [ 2 ] have revolutionized the field of machine learning and led to unprecedented levels of accuracy in a variety of data-driven prediction applications . However , it can be extremely challenging to debug these complex methodologies when they make incorrect predictions . The first step in this debugging process is explaining the prediction , which can stem from a variety of interpretive processes . For example , we may seek to understand the key features [ 13 , 16 ] or salient regions [ 18 , 20 ] that led to a prediction , or in a more recent line of work , to understand the influence of the training data on test data predictions [ 1 , 11 , 15 , 23 ] . In this paper we focus on a particularly appealing ( and efficient ) method for understanding the impact of training data on test predictions , namely Representer Point Selection ( RPS ) [ 23 ] , which leverages an application of the representer theorem [ 17 ] to deep neural networks . Specifically , it uses the dual form of l2 regularized optimization in the last layer of the neural network to identify the contribution of training data to the test prediction . Compared to model-agnostic approaches [ 1 , 9 , 10 ] that estimate ∗Contributions were made while the author was at the University of Toronto . †Affiliate to Vector Institute of Artificial Intelligence , Toronto . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . data influence after optimization , the RPS approach directly integrates with the prediction model to provide a high-fidelity white-box interpretation of the prediction . Compared to the influence function-based approach [ 11 ] , the RPS is more computationally efficient as it focuses on the last layer of a neural network instead of all parameters . While holding many advantages , we note the existing RPS approach faces two key drawbacks . First , as the RPS approach requires fine-tuning the last layer of the neural network with l2 regularization , it leads to a disagreement between the originally trained network and the l2 regularized network and is furthermore highly sensitive to l2 regularization strength ( cf . Figure 1 ( a ) ) . Thus , it is hard to justify if the explanation produced by RPS is for the original model or the modified model . Second , we observe that the explanations produced by RPS are nearly identical for all test samples that are classified into the same category , which provides more of a class-level rather than instance-level explanation for RPS ( cf . Figure 1 ( b ) ) . While the RPS approach is appealing in principle , both of these drawbacks significantly harm the utility of the RPS explanation approach in practice . To address these issues , this paper presents an alternative derivation for Representer Point Selection ( RPS ) based on a Local Jacobian Taylor expansion ( LJE ) , which corrects for both aforementioned problems . We empirically compared RPS-LJE to the RPS-l2 on image classification ( with ResNet [ 6 ] ) , text classification ( with Bi-LSTM ) , and credit analysis ( with XGBoost [ 2 ] ) tasks . Quantitatively , we show RPS-LJE outperforms RPS-l2 and other state-of-the-art data explanation methods by to up 3 % on a data debugging task . Qualitatively — and perhaps most importantly — the RPS-LJE provides stable ( i.e. , no need for l2 tuning ) and diverse explanations that are more coherent to the test data . 2 Representer Points Selection for Explaining Deep Neural Networks . 2.1 Preliminaries . In a machine learning context , representer theorems [ 17 ] loosely state that under certain conditions , model prediction ŷt of a test sample xt can be expressed as a linear combination of kernel evaluations K ( xi , xt ) between each training point xi and the test sample xt such that ŷt = n∑ i αiK ( xi , xt ) , ( 1 ) where αi is the weight of the training data point i that is independent from the test point t. As the representer theorem linearly separates the contribution of training samples to the prediction , it has been introduced into the deep learning model interpretation research [ 23 ] for tracking the training data influence on predictions . Concretely , the current Representer Point Selection ( RPS ) approach [ 23 ] introduces an l2 regularization term ( for the last layer ’ s parameters ΘL ⊂ Θ ) into a model ’ s objective function , which facilities its derivation such that it fulfills Equation 1 by setting αi = − 1 2λn ∂L ( xi , yi , Θ ) ∂ΘLφ ( xi ) and K ( xi , xt ) = φ ( xi ) Tφ ( xt ) , ( 2 ) where λ denotes the hyper-parameter of l2 regularization term , n denotes the number of total training samples , and φ ( xi ) and φ ( xt ) are the representations of input xi and xt at layerL . As this explanation approach works on the last layer of a network and is efficient to compute , it demonstrates a significant advantage over Influence Function-based approaches [ 11 ] in terms of computational efficiency . 2.2 Caveats of Current Representer Points Selection . As mentioned , to facilitate its derivation , the current RPS approach introduces an l2 regularization term into the computation , which inevitably violates the saddle point assumption its derivation relies on . To address this , the RPS conducts fine-tuning on the trained model with a new objective function Θ∗L = arg min ΘL { 1 n n∑ i L ( Φ ( xi , Θ ) , Φ ( xi , Θgiven ) ) + λ‖ΘL‖2 } , ( 3 ) where Φ ( xi , Θgiven ) and Φ ( xi , Θ ) represents the post-activation prediction of xi of the given model and fine-tuned model , respectively . Correspondingly , the kernel weights in the expression of Equation 2 are updated as αi = − 1 2λn ∂L ( xi , yi , Θ∗ ) ∂Θ∗Lφ ( xi ) . ( 4 ) As the data importance factor αi is computed with respect to the l2-modified parameter set Θ∗ , the explanations correspond to the modified model instead of the original model . Indeed , the gap between the two sets of parameters is sensitive to the hyper-parameter setting of the l2 regularizer . This reflects our previous observation in Figure 1 ( a ) showing that prediction explanations can vary significantly based on different l2 regularizer weightings . Also , by inspecting the expression of training data contribution Θ∗Lφ ( xt ) = αiK ( xi , xt ) = − 1 2λn ∂L ( xi , yi , Θ∗ ) ∂Θ∗Lφ ( xi ) φ ( xi ) Tφ ( xt ) = − 1 2λn ∂L ( xi , yi , Θ∗ ) ∂Θ∗Lφ ( xi ) ∂Θ∗Lφ ( xi ) ∂Θ∗L φ ( xt ) = − 1 2λn ∂L ( xi , yi , Θ∗ ) ∂Θ∗L︸ ︷︷ ︸ dominant term φ ( xt ) , ( 5 ) we note the first-order derivative of the loss of a single training example is the dominant term that suppresses a small difference of test example representation φ ( xt ) among test samples in the same class . Figure 2 ( a ) shows the strong positive correlation between the dominant term and the relevant order of the training data contribution . This supports our observation in Figure 1 ( b ) that explanations tend to be identical for the test samples being classified into the same category making RPS more of a class-level than instance-level explanation that defeats the intent of uncovering instance-level prediction errors ! 3 Representer Point Selection via Local Jacobian Expansion . In an effort to preserve the conceptual and computational advantages of the previously discussed RPSl2 methodology while improving fidelity of the explanations to the original model and encouraging test instance-level explanation , we now present a novel derivation of Representer Point Selection through a Local Jacobian Expansion ( RPS-LJE ) . Similar to RPS-l2 , RPS-LJE also expresses the pre-activation prediction outcome of a well-trained classification model ( near the saddle point ) as a linear combination of kernel evaluations between the test sample and the training points . Specifically , we use a first-order Taylor expansion on the Jacobian matrix in our derivation , which avoids the problems introduced by imposing the additional l2 regularization term in RPS-l2 . First , we begin with the formal problem definition . Consider a classifierMΘ† ( the target model in the following context ) that has learned to map input observation xt ∈ IRd to an output space yt ∈ { 1 · · · k } , whose pre-activation prediction3 M̄Θ† could be formulated as ŷt = M̄Θ† ( xt ) = Θ † Lφ ( xt ) , ( 6 ) where bold yi ∈ IRk represents the prediction vector ( one element for a class ) . Our goal of prediction explanation is to reformulate Equation 6 into the format of Equation 1 such that the contribution of each training data point on the prediction is linearly separable . While the above setting appears restricted , it represents a large group of machine learning models commonly used in practice ; many well-known models , such as ResNet [ 6 ] for images , Transformers [ 22 ] for text , and even XGBoost [ 2 ] for tabular classification tasks , can be expressed in this simple formulation leveraging a feature embedding stage followed by a pre-activation linear prediction stage . 3.1 First-order Taylor Expansion on Jacobian Matrix . We begin by presuming we are given a well-trained target modelMΘ† , whose loss derivative with respect to the decision making parameter ΘL is close to a saddle point such that 0 ≈ n∑ i=1 ∂L ( xi , yi , Θ ) ∂ΘL ∣∣∣∣∣ ΘL=Θ † L . ( 7 ) Here , we assume that the loss term L is twice-differentiable and strictly convex with respect to the last linear layer parameters ΘL . In the case of non-convexity , we can adopt the approach proposed in the Influence Function [ 11 ] to form a convex quadratic approximation of the loss by introducing a damping term . With any parameter assignment Θ∗L close to the model parameter Θ † L , we can further rewrite Equation 7 via a first-order Taylor expansion such that 0 ≈ 1 n n∑ i=1 ∂L ( xi , yi , Θ ) ∂ΘL ∣∣∣∣ ΘL=Θ∗L︸ ︷︷ ︸ ∇ΘLL ( xi , yi , Θ∗ ) + ( Θ†L −Θ ∗ L ) [ 1 n n∑ i=1 ∂L2 ( xi , yi , Θ ) ∂Θ2L ∣∣∣∣ ΘL=Θ∗L ] ︸ ︷︷ ︸ HΘ∗ L def = 1n ∑n i=1∇2ΘLL ( xi , yi , Θ ∗ ) . ( 8 ) This expansion yields the model parameter Θ†L as a linear combination of functions of each training data point Θ†L = Θ ∗ L − n∑ i=1 1 n H−1Θ∗L∇ΘLL ( xi , yi , Θ ∗ ) + ξ , ( 9 ) 3For the trained classifier , the predictions before and after activation are consistent . where ξ is a negligible error term . Intuitively , this equation shows the model parameter Θ†L could be reconstructed through a one-step gradient descent from the nearby parameter assignment Θ∗L with a dynamic learning rate 1nH −1 Θ∗L ( as a matrix ) . We remark that the expression in Equation 9 represents the contribution of each data point as a linearly separable function , which is fundamentally different from recording gradients at training time as a data importance score [ 15 ] . That is , the gradients during model training are sensitive to both training order and optimizer settings . For Equation 9 to hold after the Taylor expansion , Θ∗L has to be close to the model parameter Θ † L. Therefore , we propose to estimate Θ∗L through a one-step stochastic gradient ascent from the trained model using any optimizer ( Adam , RMSProp , etc ) , such that n∑ i=1 L ( xi , yi , Θ∗ ) > n∑ i=1 L ( xi , yi , Θ† ) ( 10 ) and Θ∗L is maintained close to the original model parameter Θ † L with a small loss shift .

Summary: This paper addresses the problem of model explainability by looking at which training examples were responsible for the model’s predictions. The contributed model takes inspiration from RPS and changes its computation of the data importance factor such that it becomes less sensitive to the choice of hyper-parameters and leads to higher diversity in the explanations. The proposed RPS-LJE is compared against two baseline approaches from the state of the art.