Representer Point Selection via Local Jacobian Expansion for Post-hoc Classifier Explanation of Deep Neural Networks and Ensemble Models

1 Introduction . Deep learning as well as ensemble methods such as XGBoost [ 2 ] have revolutionized the field of machine learning and led to unprecedented levels of accuracy in a variety of data-driven prediction applications . However , it can be extremely challenging to debug these complex methodologies when they make incorrect predictions . The first step in this debugging process is explaining the prediction , which can stem from a variety of interpretive processes . For example , we may seek to understand the key features [ 13 , 16 ] or salient regions [ 18 , 20 ] that led to a prediction , or in a more recent line of work , to understand the influence of the training data on test data predictions [ 1 , 11 , 15 , 23 ] . In this paper we focus on a particularly appealing ( and efficient ) method for understanding the impact of training data on test predictions , namely Representer Point Selection ( RPS ) [ 23 ] , which leverages an application of the representer theorem [ 17 ] to deep neural networks . Specifically , it uses the dual form of l2 regularized optimization in the last layer of the neural network to identify the contribution of training data to the test prediction . Compared to model-agnostic approaches [ 1 , 9 , 10 ] that estimate ∗Contributions were made while the author was at the University of Toronto . †Affiliate to Vector Institute of Artificial Intelligence , Toronto . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . data influence after optimization , the RPS approach directly integrates with the prediction model to provide a high-fidelity white-box interpretation of the prediction . Compared to the influence function-based approach [ 11 ] , the RPS is more computationally efficient as it focuses on the last layer of a neural network instead of all parameters . While holding many advantages , we note the existing RPS approach faces two key drawbacks . First , as the RPS approach requires fine-tuning the last layer of the neural network with l2 regularization , it leads to a disagreement between the originally trained network and the l2 regularized network and is furthermore highly sensitive to l2 regularization strength ( cf . Figure 1 ( a ) ) . Thus , it is hard to justify if the explanation produced by RPS is for the original model or the modified model . Second , we observe that the explanations produced by RPS are nearly identical for all test samples that are classified into the same category , which provides more of a class-level rather than instance-level explanation for RPS ( cf . Figure 1 ( b ) ) . While the RPS approach is appealing in principle , both of these drawbacks significantly harm the utility of the RPS explanation approach in practice . To address these issues , this paper presents an alternative derivation for Representer Point Selection ( RPS ) based on a Local Jacobian Taylor expansion ( LJE ) , which corrects for both aforementioned problems . We empirically compared RPS-LJE to the RPS-l2 on image classification ( with ResNet [ 6 ] ) , text classification ( with Bi-LSTM ) , and credit analysis ( with XGBoost [ 2 ] ) tasks . Quantitatively , we show RPS-LJE outperforms RPS-l2 and other state-of-the-art data explanation methods by to up 3 % on a data debugging task . Qualitatively — and perhaps most importantly — the RPS-LJE provides stable ( i.e. , no need for l2 tuning ) and diverse explanations that are more coherent to the test data . 2 Representer Points Selection for Explaining Deep Neural Networks . 2.1 Preliminaries . In a machine learning context , representer theorems [ 17 ] loosely state that under certain conditions , model prediction ŷt of a test sample xt can be expressed as a linear combination of kernel evaluations K ( xi , xt ) between each training point xi and the test sample xt such that ŷt = n∑ i αiK ( xi , xt ) , ( 1 ) where αi is the weight of the training data point i that is independent from the test point t. As the representer theorem linearly separates the contribution of training samples to the prediction , it has been introduced into the deep learning model interpretation research [ 23 ] for tracking the training data influence on predictions . Concretely , the current Representer Point Selection ( RPS ) approach [ 23 ] introduces an l2 regularization term ( for the last layer ’ s parameters ΘL ⊂ Θ ) into a model ’ s objective function , which facilities its derivation such that it fulfills Equation 1 by setting αi = − 1 2λn ∂L ( xi , yi , Θ ) ∂ΘLφ ( xi ) and K ( xi , xt ) = φ ( xi ) Tφ ( xt ) , ( 2 ) where λ denotes the hyper-parameter of l2 regularization term , n denotes the number of total training samples , and φ ( xi ) and φ ( xt ) are the representations of input xi and xt at layerL . As this explanation approach works on the last layer of a network and is efficient to compute , it demonstrates a significant advantage over Influence Function-based approaches [ 11 ] in terms of computational efficiency . 2.2 Caveats of Current Representer Points Selection . As mentioned , to facilitate its derivation , the current RPS approach introduces an l2 regularization term into the computation , which inevitably violates the saddle point assumption its derivation relies on . To address this , the RPS conducts fine-tuning on the trained model with a new objective function Θ∗L = arg min ΘL { 1 n n∑ i L ( Φ ( xi , Θ ) , Φ ( xi , Θgiven ) ) + λ‖ΘL‖2 } , ( 3 ) where Φ ( xi , Θgiven ) and Φ ( xi , Θ ) represents the post-activation prediction of xi of the given model and fine-tuned model , respectively . Correspondingly , the kernel weights in the expression of Equation 2 are updated as αi = − 1 2λn ∂L ( xi , yi , Θ∗ ) ∂Θ∗Lφ ( xi ) . ( 4 ) As the data importance factor αi is computed with respect to the l2-modified parameter set Θ∗ , the explanations correspond to the modified model instead of the original model . Indeed , the gap between the two sets of parameters is sensitive to the hyper-parameter setting of the l2 regularizer . This reflects our previous observation in Figure 1 ( a ) showing that prediction explanations can vary significantly based on different l2 regularizer weightings . Also , by inspecting the expression of training data contribution Θ∗Lφ ( xt ) = αiK ( xi , xt ) = − 1 2λn ∂L ( xi , yi , Θ∗ ) ∂Θ∗Lφ ( xi ) φ ( xi ) Tφ ( xt ) = − 1 2λn ∂L ( xi , yi , Θ∗ ) ∂Θ∗Lφ ( xi ) ∂Θ∗Lφ ( xi ) ∂Θ∗L φ ( xt ) = − 1 2λn ∂L ( xi , yi , Θ∗ ) ∂Θ∗L︸ ︷︷ ︸ dominant term φ ( xt ) , ( 5 ) we note the first-order derivative of the loss of a single training example is the dominant term that suppresses a small difference of test example representation φ ( xt ) among test samples in the same class . Figure 2 ( a ) shows the strong positive correlation between the dominant term and the relevant order of the training data contribution . This supports our observation in Figure 1 ( b ) that explanations tend to be identical for the test samples being classified into the same category making RPS more of a class-level than instance-level explanation that defeats the intent of uncovering instance-level prediction errors ! 3 Representer Point Selection via Local Jacobian Expansion . In an effort to preserve the conceptual and computational advantages of the previously discussed RPSl2 methodology while improving fidelity of the explanations to the original model and encouraging test instance-level explanation , we now present a novel derivation of Representer Point Selection through a Local Jacobian Expansion ( RPS-LJE ) . Similar to RPS-l2 , RPS-LJE also expresses the pre-activation prediction outcome of a well-trained classification model ( near the saddle point ) as a linear combination of kernel evaluations between the test sample and the training points . Specifically , we use a first-order Taylor expansion on the Jacobian matrix in our derivation , which avoids the problems introduced by imposing the additional l2 regularization term in RPS-l2 . First , we begin with the formal problem definition . Consider a classifierMΘ† ( the target model in the following context ) that has learned to map input observation xt ∈ IRd to an output space yt ∈ { 1 · · · k } , whose pre-activation prediction3 M̄Θ† could be formulated as ŷt = M̄Θ† ( xt ) = Θ † Lφ ( xt ) , ( 6 ) where bold yi ∈ IRk represents the prediction vector ( one element for a class ) . Our goal of prediction explanation is to reformulate Equation 6 into the format of Equation 1 such that the contribution of each training data point on the prediction is linearly separable . While the above setting appears restricted , it represents a large group of machine learning models commonly used in practice ; many well-known models , such as ResNet [ 6 ] for images , Transformers [ 22 ] for text , and even XGBoost [ 2 ] for tabular classification tasks , can be expressed in this simple formulation leveraging a feature embedding stage followed by a pre-activation linear prediction stage . 3.1 First-order Taylor Expansion on Jacobian Matrix . We begin by presuming we are given a well-trained target modelMΘ† , whose loss derivative with respect to the decision making parameter ΘL is close to a saddle point such that 0 ≈ n∑ i=1 ∂L ( xi , yi , Θ ) ∂ΘL ∣∣∣∣∣ ΘL=Θ † L . ( 7 ) Here , we assume that the loss term L is twice-differentiable and strictly convex with respect to the last linear layer parameters ΘL . In the case of non-convexity , we can adopt the approach proposed in the Influence Function [ 11 ] to form a convex quadratic approximation of the loss by introducing a damping term . With any parameter assignment Θ∗L close to the model parameter Θ † L , we can further rewrite Equation 7 via a first-order Taylor expansion such that 0 ≈ 1 n n∑ i=1 ∂L ( xi , yi , Θ ) ∂ΘL ∣∣∣∣ ΘL=Θ∗L︸ ︷︷ ︸ ∇ΘLL ( xi , yi , Θ∗ ) + ( Θ†L −Θ ∗ L ) [ 1 n n∑ i=1 ∂L2 ( xi , yi , Θ ) ∂Θ2L ∣∣∣∣ ΘL=Θ∗L ] ︸ ︷︷ ︸ HΘ∗ L def = 1n ∑n i=1∇2ΘLL ( xi , yi , Θ ∗ ) . ( 8 ) This expansion yields the model parameter Θ†L as a linear combination of functions of each training data point Θ†L = Θ ∗ L − n∑ i=1 1 n H−1Θ∗L∇ΘLL ( xi , yi , Θ ∗ ) + ξ , ( 9 ) 3For the trained classifier , the predictions before and after activation are consistent . where ξ is a negligible error term . Intuitively , this equation shows the model parameter Θ†L could be reconstructed through a one-step gradient descent from the nearby parameter assignment Θ∗L with a dynamic learning rate 1nH −1 Θ∗L ( as a matrix ) . We remark that the expression in Equation 9 represents the contribution of each data point as a linearly separable function , which is fundamentally different from recording gradients at training time as a data importance score [ 15 ] . That is , the gradients during model training are sensitive to both training order and optimizer settings . For Equation 9 to hold after the Taylor expansion , Θ∗L has to be close to the model parameter Θ † L. Therefore , we propose to estimate Θ∗L through a one-step stochastic gradient ascent from the trained model using any optimizer ( Adam , RMSProp , etc ) , such that n∑ i=1 L ( xi , yi , Θ∗ ) > n∑ i=1 L ( xi , yi , Θ† ) ( 10 ) and Θ∗L is maintained close to the original model parameter Θ † L with a small loss shift .

Summary: The aim of this paper is to identify the contribution of the training data to the prediction. This work highlights the short comings of the existing Represented Point Selection (RPS-$l_2$) both empirically and by analyzing the derivation of RPS$-l_2$, and proposes an improved method called RPS-LJE. RPS-LJE is a clear improvement over RPS-$-l_2$ both qualitatively and quantitatively. Qualitatively, it is observed that the explanations produced by RPS-LJE isn't repetitive like RPS$-l_2$. Quantitatively, it performs comparably to, or better that the existing data-explanation approaches.