Title: Sample based Explanations via Generalized Representers
Venue: NeurIPS 2023
Accepted: yes
Keywords: explainable machine learning

sample based explanation

representer point
Abstract: We propose a general class of sample based explanations of machine learning models, which we term generalized representers. To measure the effect of a training sample on a model's test prediction, generalized representers use two components: a global sample importance that quantifies the importance of the training point to the model and is invariant to test samples, and a local sample importance that measures similarity between the training sample and the test point with a kernel. A key contribution of the paper is to show that generalized representers are the only class of sample based explanations satisfying a natural set of axiomatic properties. We discuss approaches to extract global importances given a kernel, and also natural choices of kernels given modern non-linear models. As we show, many popular existing sample based explanations could be cast as generalized representers with particular choices of kernels and approaches to extract global importances. Additionally, we conduct empirical comparisons of different generalized representers on two image classification datasets.

Sample based Explanations via Generalized Representers

Abstract

We propose a general class of sample based explanations of machine learning models, which we term generalized representers . To measure the effect of a training sample on a model's test prediction, generalized representers use two components: a global sample importance that quantifies the importance of the training point to the model and is invariant to test samples, and a local sample importance that measures similarity between the training sample and the test point with a kernel. A key contribution of the paper is to show that generalized representers are the only class of sample based explanations satisfying a natural set of axiomatic properties. We discuss approaches to extract global importances given a kernel, and also natural choices of kernels given modern non-linear models. As we show, many popular existing sample based explanations could be cast as generalized representers with particular choices of kernels and approaches to extract global importances. Additionally, we conduct empirical comparisons of different generalized representers on two image and two text classification datasets.

1 Introduction

As machine learning becomes increasingly integrated into various aspects of human life, the demand for understanding, interpreting, and explaining the decisions made by complex AI and machine learning models has grown. Consequently, numerous approaches have been proposed in the field of Explainable AI (XAI). Feature based explanations interpret models by identifying the most relevant input features [1-4], while sample based explanations do so via the most relevant training samples [5-8]. Although different methods emphasize different aspects of the model, some may even have conflicting philosophies [9]. To address this issue, there have been growing calls within the XAI community for more objective or normative approaches [10-12], which could help align XAI techniques more effectively with human needs.

One of the most straightforward approaches to assess the effectiveness of explanations is by evaluating their utility in downstream applications [13, 14]. However, such evaluations can be costly, particularly during the development stages of explanations, as they often necessitate the involvement of real human users. As a result, a well-grounded, axiom-based evaluation can be beneficial for designing and selecting explanations for implementation. Axioms can be viewed as theoretical constraints that dictate how explanations should behave in response to specific inputs. A notable example is the Shapley value [15], which originated in cooperative game theory and has gained popularity in XAI due to its appealing axiomatic properties. Nonetheless, while axiomatic approaches have been widely applied in identifying significant features [4, 16, 17], feature interactions [18, 19], and high-level concepts [20], they have not been extensively discussed in sample based explanations.

In this work, we propose a set of desirable axioms for sample based explanations. We then show that any sample based attributions that satisfy (a subset of) these axioms are necessarily the product of two components: a global sample importance , and a local sample importance that is a kernel similarity between a training sample and the test point. We term the explanations in this form generalized representers .

We note that the efficiency axiom (detailed in the sequel) can only be satisfied if the model function lies in the RKHS subspace spanned by the training kernel representers, which is indeed typically the case. Otherwise, we could ask for the smallest error in satisfying the efficiency axiom which can be cast as an RKHS regression problem. Thus, given a kernel, extracting global importances can be cast as solving an RKHS regression problem, by recourse to RKHS representer theorems [21]. We additionally also propose tracking representers that scalably compute the global importance by tracking kernel gradient descent trajectories.

The class of generalized representers allow for the user to specify a natural kernel given the model they wish to explain, perhaps drawn from domain knowledge. We discuss some natural automated choices given modern non-linear models. Specifically, we discuss the kernel with feature maps specified by last-layer embeddings, neural tangent kernels [22], and influence function kernels [5]. Many existing sample-based explanation methods such as representer point selection [7], and influence functions [5] can be viewed as specific instances of the broad class of generalized representers. As we show, TracIn [6] could be viewed as a natural extension of generalized representers that uses multiple kernels, and computes multiple corresponding global and local importances. We empirically compare different choices of generalized representers for neural networks on two image and two text classification datasets.

1.1 Related work

Axiomatic attribution in XAI: The most common line of work that incorporates axioms in the design of explanations is the family of Shapley values [15], that originates from cooperative game theory. This line of work first tackles the attribution problem by converting it into a set value function and then applying the Shapley value. The Shapley value is widely used in feature-based explanations [4, 16, 23], feature-interaction explanations [18, 19], concept-based explanations [20] and sample-based explanations [8].

We note that the axiomatic framework for sample-based explanations in data Shapley [8] has distinct purposes and interpretations to ours. Generalized representers assess the significance of training samples with respect to a specific test sample's prediction. In contrast, data Shapley assesses training sample importance via training loss and can be directly adapted to orginal Shapley value axioms. Consequently, while there are shared axiomatic principles between the two frameworks, the generalized representers require a different approach due to their additional focus on a specific test sample. We will delve into a detailed comparison of these distinctions in Section 3.

Sample based explanations: Existing sample-based explanation approaches can be separated into retraining-based and gradient-based approaches [24]. Retraining-based approaches are based on the measurement of the difference between a model prediction with and without a group of training samples [8, 25-34]. Gradient-based methods estimate data influence based on similarities between gradients. The majority of methods build upon the three theoretical frameworks, namely: (1) Representer theorems [7, 35-37], (2) Hessian-based influence functions [5, 38-45], and (3) Decomposing the training loss trajectory [6, 46-48]. In this work, we show that most gradient-based methods can be viewed as generalized representers.

2 Problem Definition

We consider the task of explaining a supervised machine learning model f : R d ↦→ R , given inputs x ∈ R d , where d is the input dimension. 2 We are interested in explaining such models f ( · ) in terms of the training points D := { ( x i , y i ) } n i =1 with each training sample ( x i , y i ) ∈ R d × R . We denote a sample explanation functional φ D : F × D × R d ↦→ R n , as a mapping that takes a real-valued

model f ∈ F , training data points D , and an arbitrary test data point x ∈ R d as input, and outputs a vector of explanation weights [ φ D ( f, ( x 1 , y 1 ) , x ) , · · · , φ D ( f, ( x n , y n ) , x )] ∈ R n with each value corresponding to an importance score of each training sample to the test point. In the sequel, we will suppress the explicit dependence on the entire set of training points in the notation for the explanation functional and the dependence on the training label y i . Also, to make clear that the first data point argument is the training sample, and the second is the test sample, we will use φ ( f, x i → x ) ∈ R to denote the sample explanation weight for x i to explain the prediction of the model f ( · ) for the test point x .

3 Axioms for Sample based Explanations

In this section, we begin by presenting a collection of axioms that describe various desirable characteristics of sample based explanations.

Definition 1 (Efficiency Axiom) . For any model f , and test point x ∈ R d , a sample based explanation φ ( · ) satisfies the efficiency axiom iff:

The efficiency axiom entails that the sum of the attributions to each training sample together adds up to the model prediction at the test point. This is a natural counterpart of the efficiency axioms used in the Shapley values [49].

Definition 2 (Self-Explanation Axiom) . Asample based explanation φ ( · ) satisfies the self-explanation axiom iff there exists any training point x i having no effect on itself, i.e. φ ( f, x i → x i ) = 0 , the training point should not impact any other points, i.e. φ ( f, x i → x ) = 0 for all x ∈ R d .

The self-explanation axiom states that if the label y i does not even have an impact on the model's prediction for x i , it should not impact other test predictions. This axiom shares a similar intuition as the dummy axiom in the Shapley values [15] since both axioms dictate that explanations should be zero if a training sample has no impact on the model. However, the self-explanation axiom requires a different theoretical treatments due to the additional focus in generalized representers of explaining a model prediction on a particular test sample.

Definition 3 (Symmetric Zero Axiom) . A sample explanation φ ( · ) satisfies the symmetric zero axiom iff any two training points x i , x j such that if φ ( f, x i → x i ) = 0 and φ ( f, x j → x j ) = 0 , then

/negationslash

The symmetric-zero axiom underscores the bidirectional nature of "orthogonality'. It emphasizes that if a sample has no impact on another sample, this lack of correlation is mutual and implies that they are orthogonal.

Definition 4 (Symmetric Cycle Axiom) . A sample explanation φ ( · ) satisfies the symmetric cycle axiom iff for any set of training points x t 1 , ... x t k , with possible duplicates, and x t k +1 = x t 1 , it holds that:

Let us first consider the vacuous case of two points: x 1 , x 2 , for which the axiom is the tautology that: φ ( f, x 1 → x 2 ) φ ( f, x 2 → x 1 ) = φ ( f, x 2 → x 1 ) φ ( f, x 1 → x 2 ) . Let us next look at the case with three points: x 1 , x 2 , x 3 , for which the axiom entails:

It can be seen that this is a generalization of simply requiring that the explanations be symmetric as in the symmetry axiom in the Shapley values. In fact, the unique explanation satisfying this and other listed axioms is in general not symmetric. The axiom could also be viewed as a conservation or path independence law, in that the flow of explanation based information from a point x i to itself in a cycle is invariant to the path taken.

/negationslash

Definition 5 (Continuity Axiom) . A sample based explanation φ ( · ) satisfies the continuity axiom iff it is continuous wrt the test data point x , for any fixed training point x i :

Such continuity is a minimal requirement on the regularity of the explanation functional, and which ensures that infinitesimal changes to the test point would not incur large changes to the explanation functional.

Definition 6 (Irreducibility Axiom) . A sample explanation φ ( · ) satisfies the irreducibility axiom iff for any number of training points x 1 , ..., x k ,

A sufficient condition for an explanation φ ( · ) to satisfy the irreducibility axiom is for

/negationslash

since this makes the matrix above strictly diagonally dominant, and since the diagonal entries are non-negative, by the Gershgorin circle theorem, the eigenvalues are all non-negative as well, so that the determinant in turn is non-negative.

The continuity and irreducibility axiom primarily serves a function-analytic purpose by providing sufficient and necessary conditions of a kernel being a Mercer kernel, which requires that the kernel function be continuous and positive semi-definite.

We are now ready to investigate the class of explanations that satisfy the axioms introduced above.

Theorem 7. An explanation functional φ ( f, · , · ) satisfies the continuity, self-explanation, symmetric zero, symmetric cycle, and irreducibility axioms for any training samples D containing n training samples ( x i , y i ) ∈ R d × R for all i ∈ [ n ] iff

for some α ∈ R n and some continuous positive-definite kernel K : R d × R d ↦→ R .

This suggests that a sample explanation φ ( f, x i → x ) = α i K ( x i , x ) has two components: a weight α i associated with just the training point x i independent of test points, and a similarity K ( x i , x ) between the training and test points specified by a Mercer kernel. Following Yeh et al. [7], we term the first component the global importance of the training sample x i and the second component the local importance that measures similarities between training and test samples.

Once we couple this observation together with the efficiency axiom, one explanation that satisfies these properties is:

This can be seen to hold only if the target function f lies in the RKHS subspace spanned by the kernel evaluations of training points. When this is not necessarily the case, then the efficiency axiom (where the sum of training sample importances equals the function value) exactly, cannot be satisfied exactly. We can however satisfy the efficiency axiom approximately with the approximation error arising from projecting the target function f onto the RKHS subspace spanned by training representers.

This thus provides a very simple and natural framework for specifying sample explanations: (1) specify a Mercer kernel K ( · , · ) so that the target function can be well approximated by the corresponding kernel machine, and (2) project the given model onto the RKHS subspace spanned by kernel evaluations on the training points. Each of the sample explanation weights then has a natural specification in terms of global importance associated with each training point (arising from the projection of the function onto the RKHS subspace, which naturally does not depend on any test points), as well as a localized component that is precisely the kernel similarity between the training and test points.

4 Deriving Global Importance Given Kernel Functions

The previous section showed that the class of sample explanations that satisfy a set of key axioms naturally correspond to an RKHS subspace. Thus, all one needs, in order to specify the sample explanations, is to specify a Mercer kernel function K and solve for the corresponding global importance weights α . In this section, we focus on the latter problem, and present three methods to compute the global importance weights given some kernel K .

4.1 Method 1: Projecting Target Function onto RKHS Subspace

The first method is to project the target function onto the RKHS subspace spanned by kernel evaluations on the training points. Given the target function f , loss function L : R × R ↦→ R and training dataset D = { ( x i , y i ) } n i =1 (potentially, though not necessarily used to train f ), and a user-specified Mercer kernel K , our goal is to find a projection ˆ f K of the target model onto the RKHS subspace defined by H n = span ( { K ( x i , · ) } n i =1 )) . To accomplish this, we formulate it as a RKHS regression problem:

where H K as the RKHS defined by kernel K , ‖ · ‖ H K : H K ↦→ R is the RKHS norm, and λ is a regularization parameter that controls the faithfulness and complexity of the function ˆ f K . The loss function L can be chosen as the objective function used to train the target function f to closely emulate the behavior of target function f and its dependence on the training samples D . By the representer theorem [21], the regularization term ‖ f K ‖ 2 H K added here ensures that the solution lies in the RKHS subspace H n spanned by kernel evaluations. Indeed, by the representer theorem [21], the minimizer of Eqn.(4) can be represented as ˆ f K ( · ) = ∑ n i =1 α i K ( x i , · ) for some α ∈ R n , which allows us to reparameterize Eqn.(4):

where K ∈ R n × n is the kernel gram matrix defined as K ij = K ( x i , x j ) for i, j ∈ [ n ] , and we use the fact that ‖ f K ‖ H K = 〈 ∑ n i =1 α i K ( x i , · ) , ∑ n i =1 α i K ( · , x i ) 〉 1 2 = √ α /latticetop K α . By solving the first-order optimality condition, the global importance α must be in the following form:

Proposition 8. (Surrogate derivative) The minimizer of Eqn. (4) can be represented as ˆ f K = ∑ n i =1 ˆ α i K ( x i , · ) , where

We call α ∗ i the surrogate derivative since it is the derivative of the loss function with respect to the surrogate function prediction.

α ∗ i can be interpreted as the measure of how sensitive ˆ f K ( x i ) is to changes in the loss function. Although the global importance α solved via Eqn.(5) may not be unique as indicated by the above results, the following proposition ensures that all ˆ α ∈ { α ∗ + v | v ∈ null ( K ) } result in the same surrogate function ˆ f K = ∑ n i =1 α ∗ i K ( x i , · ) .

Proposition 9. For any v ∈ null ( K ) , the function f v = ∑ n i =1 v i K ( x i , · ) specified by the span of kernel evaluations with weights v is a zero fucntion, such that f v ( x ) = 0 for all x ∈ R d .

The proposition posits that adding any v ∈ null ( K ) to α ∗ has no effect on the function ˆ f K . Therefore, we use α ∗ to represent the global importance as it captures the sensitivity of the loss function to the prediction of the surrogate function.

4.2 Method 2: Approximation Using the Target Function

Given the derivation of global importance weights α ∗ in Eqn.(6), we next consider a variant replacing the surrogate function prediction ˆ f K ( x i ) with the target function prediction f ( x i ) :

Definition 10 (Target derivative) . The global importance computed with derivatives of the loss function with respect to the target function prediction is defined as:

where L ( · , · ) is the loss function used to train the target function.

A crucial advantage of this variant is that we no longer need solve for an RKHS regression problem. There are several reasons why this approximation is reasonable. Firstly, the loss function in Eqn.(4) encourages the surrogate function to produce similar outputs as the target function, so that ˆ f K ( x i ) is approximately equal to f ( x i ) . Secondly, when the target function exhibits low training error, which is often the case for overparameterized neural networks that are typically in an interpolation regime, we can assume that f ( x i ) is close to y i . Consequently, the target derivative can serve as an approximation of the surrogate derivative in Eqn.(6). As we will show below, the influence function approach [5] is indeed as the product between the target derivative and the influence function kernel.

4.3 Method 3: Tracking Gradient Descent Trajectories

Here, we propose a more scalable variant we term tracking representers that accumulates changes in the global importance during kernel gradient descent updates when solving Eqn.(4). Let Φ : R d ↦→ H be a feature map corresponding to the kernel K , so that K ( x , x ′ ) = 〈 Φ ( x ) , Φ ( x ′ ) 〉 . We can then cast any function in the RKHS as f K ( x ) = 〈 θ , Φ ( x ) 〉 for some parameter θ ∈ H . Suppose we solve the unregularized projection problem in Eqn.(4) via stochastic gradient descent updates on the parameter θ : θ ( t ) = θ ( t -1) -η ( t ) | B ( t ) | ∑ i ∈ B ( t ) ∇ θ L ( f θ ( x i ) , f ( x i )) Φ ( x i ) | θ = θ ( t -1) , where we use B ( t ) and η ( t ) to denote the minibatch and the learning rate. The corresponding updates to the function is then given by 'kernel gradient descent' updates: f ( t ) K ( x ) = f ( t -1) K ( x ) -α it K ( x i , x ) , where α it = η ( t ) | B ( t ) | ∑ i ∈ B ( t ) ∂ L ( f ( t -1) K ( x i ) ,f ( x i )) ∂ f ( t -1) K ( x i ) . The function at step T can then be represented as:

Definition 11 (Tracking representers) . Given a finite set of steps T , we term the global importance weights obtained via tracking kernel gradient descent as tracking representers:

We note that one can draw from standard correspondences between gradient descent with finite stopping and ridge regularization (e.g. [50]), to in turn relate the iterates of the kernel gradient descent updates for any finite stopping at T iterations to regularized RKHS regression solutions for some penalty λ . The above procedure thus provides a potentially scalable approach to compute the corresponding global importances: in order to calculate the global importance α ( T ) i , we need to simply monitor the evolution of α ( t ) i when the sample x i is utilized at iteration t . In our experiment, we use the following relaxation for further speed up:

where we assume the target model is trained with (stochastic) gradient descent, f ( t ) ( x i ) denotes the target model at t th iteration during training, and B ( t ) and η ( t ) are the corresponding mini-batch and learning rate. Similar to the intuition of replacing the surrogate derivative with to target derivative, we track the target model's training trajectory directly instead of solving Eqn.(4) with kernel gradient descent.

5 Choice of Kernels for Generalized Representers

Previously, we discussed approaches for deriving global importance given user-specified kernels, which can in general be specified by domain knowledge relating to the model and the application domain. In this section, we discuss natural choices of kernels for modern non-linear models. Moreover, we show that existing sample based explanation methods such as representer points [7] and influence functions [5] could be viewed as making particular choices of kernels when computing general representers. We also discuss TracIn [6] as a natural extension of our framework to multiple rather than a single kernel.

5.1 Kernel 1: Penultimate-layer Embeddings

A common method for extracting a random feature map from a neural network is to use the embeddings of its penultimate layer [7, 51, 52]. Let Φ Θ 1 : R d ↦→ R /lscript denote the mapping from the input to the second last layer. The target model f can be represented as

where Θ 2 ∈ R /lscript is the weight matrix of the last layer. That is, we treat the deep neural network as a linear machine on top of a learned feature map. The kernel function is then defined as K LL ( x , z ) = 〈 Φ Θ 1 ( x ) , Φ Θ 1 ( z ) 〉 , ∀ x , z ∈ R d . This is the case with most deep neural network architectures, where the feature map Φ Θ 1 is specified via deep compositions of parameterized layers that take the form of fully connected layers, convolutional layers, or attention layers among others. While the last-layer weight matrix Θ 2 may not lie in the span of { Φ θ 1 ( x i ) } n i =1 , we may solve the its explanatory surrogate function using Eqn.(4).

Corollary 12. (Representer point selection [7]) The minimizer of Eqn. (4) , instantiated with K LL ( x , z ) = 〈 Φ Θ 1 ( x ) , Φ Θ 1 ( z ) 〉 , ∀ x , z ∈ R d , can be represented as

The above corollary implies that ˆ Θ 2 = ∑ n i =1 α i Φ θ 1 ( x i ) . In other words, the RKHS regularization in Eqn.(4) can be expressed as ‖ f K ‖ 2 H K = ‖ Θ 2 ‖ 2 , which is equivalent to L2 regularization. Consequently, the representer point selection method proposed in Yeh et al. [7] can be generalized to our framework when we use last-layer embeddings as feature maps.

5.2 Kernel 2: Neural Tangent Kernels

Although freezing all layers except for the last layer is a straightforward way to simplify neural networks to linear machines, last-layer representers may overlook influential behavior that is present in other layers. For example, Yeh et al. [53] shows that representation in the last layer leads to inferior results for language models. On the other hand, neural tangent kernels (NTK) [22] have been demonstrated as a more accurate approximation of neural networks [54-56]. By using NTKs, we use gradients with respect to model parameters as feature maps and approximate neural networks with the corresponding kernel machines. This formulation enables us to derive a generalized representer that captures gradient information of all layers.

For a neural network with scalar output f θ : R d ↦→ R parameterized by a vector of parameters θ ∈ R p , the NTK is a kernel K : R d × R d ↦→ R defined by the feature maps Φ θ ( x ) = ∂ f θ ( x ) ∂θ :

Connection to TracIn [6]: TracIn measures the change in model parameters from the start to the end of training . While it is intractable due to the need to store model parameters of all iterations, Pruthi et al. [6] used checkpoints(CP) as a practical relaxation: given model parameters θ ( t ) and learning rates η ( t ) at all model checkpoints t = 0 , · · · , T , the formulation of TracInCP is given

below 3 :

When the learning rate is constant throughout the training process, TracInCP can be viewed as a generalized representer instantiated with target derivative as global importances and NTK (Eqn.(13)) as the kernel function, but uses different kernels on different checkpoints.

5.3 Kernel 3: Influence Function Kernel

The influence functions [5] can also be represented as a generalized representer with the following kernel:

where H θ = 1 n ∑ n i =1 ∂ 2 L ( f θ ( x i ) ,y i ) ∂θ 2 is the Hessian matrix with respect to target model parameters. The influence function then can be written as:

Therefore, the influence function can be seen as a member of generalized representers with target derivative global importance (Definition 10) and the influence function kernel. Influence functions [57] were designed to measure how would the model's predictions change if a training input were perturbed for convex models trained with empirical risk minimization. Consequently, the inversed Hessian matrix describes the sensitivity of the model parameters in each direction.

6 Experiments

In the experiment, we compare different representers within our proposed framework on both vision and language classification tasks. We use convolutional neural networks (CNN) since they are widely recognized deep neural network architectur. We compare perforamnce of different choices of kernels and different ways to compute global importance. Existing generalized representers, such as influence functions [5], representer point selections [53], and TracIn [6], are included in our experiment.

6.1 Experimental Setups

Evaluation metrics: We use case deletion diagnostics [53, 57, 58], DEL -( x , k, φ ) , as our primary evaluation metric. The metric measures the difference between models' prediction score on x when we remove topk negative impact samples given by method φ and the prediction scores of the original models. We expect DEL -to be positive since models' prediction scores should increase when we remove negative impact samples. To evaluate deletion metric at different k , we follow Yeh et al. [53] and report area under the curve (AUC): AUC-DEL -= ∑ m i =1 DEL -( x , k i , φ ) /m , where k 1 < k 2 < · · · < k m is a predefined sequence of k .

We choose k i = 0 . 02 in for i = 0 , 1 , · · · , 5 with n as the size of the training set. The average of each metric is calculated across 50 / 200 randomly initialized neural networks for vision/language data. For every neural network, sample-based explanation methods are computed for 10 randomly selected testing samples.

Datasets and models being explained: For image classification, we follow Pruthi et al. [6] and use MNIST [59] and CIFAR-10 [60] datasets. For text classification, we follow Yeh et al. [53] and use Toxicity 4 and AGnews 5 datasets, which contain toxicity comments and news of different categories respectively. Due to computational challenges in computing deletion diagnostics, we subsample the datasets by transforming them into binary classification problems with each class containing around 6 , 000 training samples. The CNNs we use for the four datasets comprise 3 layers. For vision datasets, the models contain around 95 , 000 parameters. For text datasets, the total number of parameters in the model is 1 , 602 , 257 with over 90% of the parameters residing in the word embedding layer that contains 30 , 522 trainable word embeddings of dimensions 48 . Please refer to the Appendix C for more details on the implementation of generalized representers and dataset constructions.

6.2 Experimental Results

The results are shown in Table 1. We also provide deletion curves we compute AUC-DEL -for in the Appendix.

I. Comparison of different global importance: In the first experiment, we fix the kernel to be the NTK computed on final model parameters, and we compare different methods for computing global importance in Section 4. We do not compute the surrogate derivative on the text datasets since the total numbers of parameters are too large, making the computation infeasible.

We observe that tracking has the best performance, followed by target derivative and then surrogate derivative . This could be due to the loss flattening when converged and the loss gradients becoming less informative. Consequently, accumulating loss gradients during training is the most effective approach. Moreover, if tracking is not feasible when training trajectories are not accessible, we may use target derivative instead of surrogate derivative as an alternative to explain neural networks since they have similar performance.

II. Comparison of different kernels: Next, we fix the global importance to tracking and compare different kernels in Section 5. We employ the tracking representers to compute global importance since it showed the best performance in the previous experiment. We can see that the influence function kernel performs the best in the vision data sets, and the NTK-final kernel has the best performance in language data sets. Note that influence functions exhibit distinctly contrasting performances on image and text data, which could be attributed to our reliance solely on last-layer parameters for influence function computation on language datasets. This finding aligns with the conclusions of Yeh et al. [53], who suggest that the last layer gradients provide less informative insights for text classifiers.

In summary, these findings indicate that NTK-final is a dependable kernel selection due to its consistent high performance across all four datasets, while also offering a computational efficiency advantage over the influence function kernel. These results also demonstrate that accessing target model checkpoints for computing kernels is unnecessary since NTK and influence function on the final model already provide informative feature maps.

III. Comparison of different generalized representers: Finally, we compare the new generalized representer with other existing generalized representers. We categorize TracInCP, the influence function, and the representer point as existing generalized representers: TracInCP can be viewed as an ensemble of generalized representers with target derivatives using the Neural Tangent Kernel. The influence function can be expressed as the influence function kernel with the target derivative. Lastly, the representer point can be seen as a form of generalized representer that utilizes the last-layer kernel and the surrogate derivative.

We find that the Inf-final has comparable performance to TracInCP and they outperform other approaches. Although TracInCP has the best performance on MNIST, it requires accessing different checkpoints, which requires a significant amount of memory and time complexity. In contrast, the NTK and Inf tracking representers are more efficient since they only require tracking gradient descent trajectories during training without the need for storing checkpoints.

7 Conclusion and Future work

In this work, we present generalized representers that are the only class of sample based explanations that satisfy a set of desirable axiomatic properties. We explore various techniques for computing generalized representers in the context of modern non-linear machine learning models and show that many popular existing methods fall into this category. Additionally, we propose tracking representers that track sample importance along the gradient descent trajectory. In future work, it would be of interest to derive different generalized representers by altering different global importance and choices of kernels, as well as investigating their applicability to diverse machine learning models and modalities.

References

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1135-1144. ACM, 2016.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantamand, Devi Parikh, and Dhruv Parikh. Grad-cam: Visual explanations from deep networks via gradientbased localization. International conference on computer vision , 2017.

Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825 , 2017.

Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems , pages 4765-4774, 2017.

Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 1885-1894. JMLR. org, 2017.

Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems , 33, 2020.

Chih-Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. In Advances in Neural Information Processing Systems , pages 9291-9301, 2018.

Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning , pages 2242-2251. PMLR, 2019.

Satyapriya Krishna, Tessa Han, Alex Gu, Javin Pombra, Shahin Jabbari, Steven Wu, and Himabindu Lakkaraju. The disagreement problem in explainable machine learning: A practitioner's perspective. arXiv preprint arXiv:2202.01602 , 2022.

WJames Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences , 116(44):22071-22080, 2019.

Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 , 2017.

Blair Bilodeau, Natasha Jaques, Pang Wei Koh, and Been Kim. Impossibility theorems for feature attribution. arXiv preprint arXiv:2212.11870 , 2022.

Valerie Chen, Nari Johnson, Nicholay Topin, Gregory Plumb, and Ameet Talwalkar. Use-casegrounded simulations for explanation evaluation. arXiv preprint arXiv:2206.02256 , 2022.

Kasun Amarasinghe, Kit T Rodolfa, Sérgio Jesus, Valerie Chen, Vladimir Balayan, Pedro Saleiro, Pedro Bizarro, Ameet Talwalkar, and Rayid Ghani. On the importance of application-grounded experimental design for evaluating explainable ml methods. arXiv preprint arXiv:2206.13503 , 2022.

Lloyd S Shapley. A value for n-person games. Contributions to the Theory of Games , 2(28): 307-317, 1953.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning , 2017.

Chih-Kuan Yeh, Cheng-Yu Hsieh, Arun Suggala, David I Inouye, and Pradeep K Ravikumar. On the (in) fidelity and sensitivity of explanations. Advances in Neural Information Processing Systems , 32, 2019.

Che-Ping Tsai, Chih-Kuan Yeh, and Pradeep Ravikumar. Faith-shap: The faithful shapley interaction index. Journal of Machine Learning Research , 24(94):1-42, 2023.

Mukund Sundararajan, Kedar Dhamdhere, and Ashish Agarwal. The shapley taylor interaction index. In International Conference on Machine Learning , pages 9259-9268. PMLR, 2020.

Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-based explanations in deep neural networks. Advances in Neural Information Processing Systems , 33:20554-20565, 2020.

Bernhard Schölkopf, Ralf Herbrich, and Alex J Smola. A generalized representer theorem. In International conference on computational learning theory , pages 416-426. Springer, 2001.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

Scott M Lundberg, Gabriel G Erion, and Su-In Lee. Consistent individualized feature attribution for tree ensembles. arXiv preprint arXiv:1802.03888 , 2018.

Zayd Hammoudeh and Daniel Lowd. Training data influence analysis and estimation: A survey. arXiv preprint arXiv:2212.04612 , 2022.

Tianhao Wang and Ruoxi Jia. Data banzhaf: A data valuation framework with maximal robustness to learning stochasticity. arXiv preprint arXiv:2205.15466 , 2022.

Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Datamodels: Predicting predictions from training data. arXiv preprint arXiv:2202.00622 , 2022.

Yongchan Kwon and James Zou. Beta shapley: a unified and noise-reduced data valuation framework for machine learning. arXiv preprint arXiv:2110.14049 , 2021.

Tom Yan and Ariel D Procaccia. If you like shapley then you'll love the core. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 5751-5759, 2021.

Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. arXiv preprint arXiv:2112.12938 , 2021.

Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regularities of labeled data in overparameterized models. arXiv preprint arXiv:2002.03206 , 2020.

Ruoxi Jia, Fan Wu, Xuehui Sun, Jiacen Xu, David Dao, Bhavya Kailkhura, Ce Zhang, Bo Li, and Dawn Song. Scalability vs. utility: Do we have to sacrifice one for the other in data importance quantification? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8239-8247, 2021.

Jinsung Yoon, Sercan Arik, and Tomas Pfister. Data valuation using reinforcement learning. In International Conference on Machine Learning , pages 10842-10851. PMLR, 2020.

Amirata Ghorbani, Michael Kim, and James Zou. A distributional framework for data valuation. In International Conference on Machine Learning , pages 3535-3544. PMLR, 2020.

Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J Spanos. Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1167-1176. PMLR, 2019.

Yuanyuan Chen, Boyang Li, Han Yu, Pengcheng Wu, and Chunyan Miao. Hydra: Hypergradient data relevance analysis for interpreting deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 7081-7089, 2021.

Yi Sui, Ga Wu, and Scott Sanner. Representer point selection via local jacobian expansion for post-hoc classifier explanation of deep neural networks and ensemble models. Advances in neural information processing systems , 34:23347-23358, 2021.

Jonathan Brophy, Zayd Hammoudeh, and Daniel Lowd. Adapting and evaluating influenceestimation methods for gradient-boosted decision trees. arXiv preprint arXiv:2205.00359 , 2022.

Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. arXiv preprint arXiv:2303.14186 , 2023.

Andrew Silva, Rohit Chopra, and Matthew Gombolay. Cross-loss influence functions to explain deep network representations. In International Conference on Artificial Intelligence and Statistics , pages 1-17. PMLR, 2022.

Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8179-8186, 2022.

Rui Zhang and Shihua Zhang. Rethinking influence functions of neural networks in the over-parameterized regime. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 9082-9090, 2022.

Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning are fragile. arXiv preprint arXiv:2006.14651 , 2020.

Han Guo, Nazneen Fatema Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. Fastif: Scalable influence functions for efficient model interpretation and debugging. arXiv preprint arXiv:2012.15781 , 2020.

Elnaz Barshan, Marc-Etienne Brunet, and Gintare Karolina Dziugaite. Relatif: Identifying explanatory training samples via relative influence. In International Conference on Artificial Intelligence and Statistics , pages 1899-1909. PMLR, 2020.

Samyadeep Basu, Xuchen You, and Soheil Feizi. On second-order group influence functions for black-box predictions. In International Conference on Machine Learning , pages 715-724. PMLR, 2020.

Hugo Thimonier, Fabrice Popineau, Arpad Rimmel, Bich-Liên Doan, and Fabrice Daniel. Tracinad: Measuring influence for anomaly detection. In 2022 International Joint Conference on Neural Networks (IJCNN) , pages 1-6. IEEE, 2022.

Satoshi Hara, Atsushi Nitanda, and Takanori Maehara. Data cleansing for models trained with sgd. Advances in Neural Information Processing Systems , 32, 2019.

Maximilian Mozes, Tolga Bolukbasi, Ann Yuan, Frederick Liu, Nithum Thain, and Lucas Dixon. Gradient-based automated iterative recovery for parameter-efficient tuning. arXiv preprint arXiv:2302.06598 , 2023.

Lloyd S. Shapley. A value for n-person games , page 31-40. Cambridge University Press, 1988. doi: 10.1017/CBO9780511528446.003.

Arun Suggala, Adarsh Prasad, and Pradeep K Ravikumar. Connecting optimization and regularization paths. Advances in Neural Information Processing Systems , 31, 2018.

Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165 , 2017.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Chih-Kuan Yeh, Ankur Taly, Mukund Sundararajan, Frederick Liu, and Pradeep Ravikumar. First is better than last for training data influence. arXiv preprint arXiv:2202.11844 , 2022.

Alexander Wei, Wei Hu, and Jacob Steinhardt. More than a toy: Random matrix models predict how real-world neural representations generalize. In International Conference on Machine Learning , pages 23549-23588. PMLR, 2022.

Philip M Long. Properties of the after kernel. arXiv preprint arXiv:2105.10585 , 2021.

Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. arXiv preprint arXiv:2210.05643 , 2022.

R Dennis Cook and Sanford Weisberg. Residuals and influence in regression . New York: Chapman and Hall, 1982.

Xiaochuang Han, Byron C Wallace, and Yulia Tsvetkov. Explaining black box predictions and unveiling data artifacts through influence functions. arXiv preprint arXiv:2005.06676 , 2020.

Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http: //yann.lecun.com/exdb/mnist/ .

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Naman Agarwal, Brian Bullins, and Elad Hazan. Second-order stochastic optimization in linear time. stat , 1050:15, 2016.

James Mercer. Xvi. functions of positive and negative type, and their connection the theory of integral equations. Philosophical transactions of the royal society of London. Series A, containing papers of a mathematical or physical character , 209(441-458):415-446, 1909.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a unifying framework for sample-based explanation methods via generalised representers. The framework proposes to approximate a general nonlinear predictive function using a surrogate from the Reproducing Kernel Hilbert Space (RKHS) (Surrogate function $f(x)=\\sum_{i=1}\\phi(x_i,x)=\\sum_{i=1}\\alpha k(x_i,x)$). The underlying explanation function $\\phi$ is both necessary and sufficient to satisfy a given set of axioms (aka efficiency, continuity, self-explanation, symmetric zero, symmetric cycle, irreducibility axioms) and can be decomposed in two parts, namely a global component, viz. the alpha coefficients (constant over the whole input space), and a local one, viz. the kernel similarity between the evaluating point and each training instance (which clearly depends on the input). Existing sample-based explanation frameworks, including TracInCP, influence functions and representer points, are shown to be particular instantiations of the proposed framework depending on the choice of the alpha coefficients and the kernel function. Additionally, the framework suggests a new sample-based explanation strategy to reduce the computational storage burden of TracInCP. Experiments are conducted over a CNN model on CIFAR10 and MNIST, thus empirically highlighting the benefits of different choices of alpha coefficients and kernel functions.\n\nSTRENGTHS:\n1. The idea of unifying existing sample-based explanation approaches is elegant, original and novel ( **Originality** )\n2. Overall, the paper is clear and self-contained ( **Clarity** ). Perhaps, the presentation can be improved by taking into account some of the suggestions highlighted in the Questions section.\n\nWEAKNESSES:\n1. Code is not available. It would be good to release the code for replicating the analysis ( **Reproducibility** ).\n2. While the theory unifying different sample-based explanation approaches is nice, the observations that can be drawn from it are rather limited to the analysis on the choice of different alpha/kernel functions ( **Significance** ).\n3. The experimental analysis uses rather small and simple neural networks and represents a simple proof of concept (**Quality**). Since the authors compare against TracInCP, they should consider using the same experimental settings of the main paper (namely using ResNet architecture).\n\nQUESTIONS:\nOverall, I liked the overall formalisation of sample-based explanation techniques. Can you please elaborate more on the following two aspects?\n1. Additional considerations/insights that can be drawn from the analysis.\n2. Scope of validity of experimental results. Are the observations valid for instance on other and larger network architectures than simple CNNs?\n\nPlease, find below a list of possible points to improve the presentation:\n1. Is Method 2 in Section 4.2 necessary for the presentation?\n2. Line 206 should $\\nabla_{\\theta}$  be replaced with $\\nabla_{f_\\theta}$?\n3. Line 222 replace the correct reference and Line 224 remove “show”\n4. Would it better to summarise sample-based explanation strategies in the form of a table, thus highlight the different choices of alpha and kernel functions and the difference with the proposed strategy?\n5. Can you please highlight more the difference between TracInCP and your strategy?\n\nLIMITATIONS:\nNo additional suggestions to improve the paper", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this study, the authors conducted an axiomatic analysis of a measure that quantifies the influence of a given training data on predictions.\nUnder several axioms, the authors demonstrated that an effective measure of influence is limited to the form of a suitable coefficient multiplied by a continuous and positive definite kernel function.\nBased on this finding, the authors showed that many existing influence metrics can actually be expressed in the form of a suitable coefficient multiplied by a kernel function.\nFurthermore, the authors proposed a new measure by combining Representer Point Selection and Neural Tangent Kernel.\n\nSTRENGTHS:\nThe strength of this study lies in the axiomatic analysis of the measure of data influence.\nUnder Continuity Axiom, Self-Explanation Axiom, Symmetric Zero Axiom, Symmetric Cycle Axiom, and Irreducibility Axiom, the authors demonstrated that an effective measure of influence is limited to the form of a suitable coefficient multiplied by a continuous and positive definite kernel function.\nFurthermore, based on this finding, the authors showed that many existing metrics for measuring influence can indeed be expressed in the form of a suitable coefficient multiplied by a kernel function.\nThe reorganization of these existing influence metrics from an axiomatic perspective represents a novel and significant contribution of this study.\n\nWEAKNESSES:\nAn essential weakness of this study is the insufficient discussion regarding the validity of various axioms.\nWhile Continuity Axiom appears to naturally require the continuity of the measure, the validity of the other axioms, Self-Explanation Axiom, Symmetric Zero Axiom, Symmetric Cycle Axiom, and Irreducibility Axiom, is not necessarily evident from the current discussions in the paper.\nIn fact, Data Shapley [8] employs different axioms.\nSince the choice of axioms determines the appropriate measure, the discussion of the validity of these axioms becomes crucial in the axiomatic analysis.\nWhile the authors provide some intuitive explanations, they seem insufficient as a discussion on the validity of these axioms.\nFor example, what are the similarities and differences between the axioms employed in Data Shapley [8] and the axioms considered in this study?\n\nQUESTIONS:\n* Please discuss the validity of the axioms introduced in the paper. When they are appropriate and when they may be not?\n* What are the similarities and differences between the axioms employed in Data Shapley [8] and the axioms considered in this study?\n\n---\nI have read the authors' rebuttal.\nThe difference of the current study and Data Shapley [8] is partly solved.\nI strongly believe it should be discussed in detail in the paper.\n\nLIMITATIONS:\nThe authors mentioned some possible future directions that are not addressed in the current study.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a new framework for generating sample-based explanations for black-box machine learning algorithms. To explain a black-box model, the basic idea of sample-based explanations is to quantify how each training data is influencing the prediction of certain test data. The main contribution of this work is to give a natural set of axioms for defining explanation functional, and showed the equivalence of every explanation function that satisfies these axioms with Mercer kernels. Under this framework, the paper studies how to define the importance function based on given kernel functions and discusses some popular choices of kernel functions in the context of deep learning.\n\nSTRENGTHS:\nThis paper studies a very interesting and important question within the realm of interpretable (explainable) machine learning, making a very neat connection between a natural set of axioms for the explanation functional and kernel functions. The study is substantial and showcases promising results from numerical studies.\n\nWEAKNESSES:\nThe authors may benefit from expanding their discussion on the selection of kernels. For instance, the Inf-final kernel outperforms all other kernels in Table 1 but lags significantly in Table 2, according to the supplementary material. Given this inconsistency, it seems slightly misleading to simply promote the influential kernel in the main body of the paper without a more thorough discussion.\n\nQUESTIONS:\nThe paper would also be strengthened by additional discourse on the axioms, which are a key element of the study. How do existing sample-based representers fare against these axioms? Furthermore, is the axiom set minimal in its essence? For example, is it possible that the continuity axiom may not hold in some practical situation? The kernel function doesn't have to be continuous without the continuity axiom.\n\nAlthough the authors present the tracking representer as \"a more scalable variant\" due to its reduced computational burden, it consistently outperforms the other two approaches. Could this be partially attributed to early stopping? In addition, the target derivative is proposed as an approximation to avoid solving the Reproducing Kernel Hilbert Space (RKHS) regression, but it also consistently outperforms the original surrogate derivative. It would be beneficial to have further commentary on these points.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies and proposes a set of desirable axioms for sample based explanations. This further demonstrates that the only solution satisfying the set of desirable axioms has the form of $\\alpha_i K(x_i,x)$ (i.e., the product of two components: a global sample importance, and a local sample importance that is a kernel similarity between a training sample and the test point) where $K(.,.)$ is a psd kernel function. Moreover, Many existing sample-based explanation methods such as representer point selection [7], influence functions [5], and TracIn [6]  can be viewed as specific instances of the broad class of generalized representers.\n\nSTRENGTHS:\nThe set of desirable axioms makes sense and the link between kernel theory and sample based explanations through set of desirable axioms is interesting.\n\nThe viewpoints to connect the sample based explanations through the set of desirable axioms and the existing sample-based explanation methods is interesting.\n\nWEAKNESSES:\nTraining and learning a kernel model to approximate a given black-box function are challenging and computationally expensive.\n\nMethod 2: approximation using the target function is not convincing to me because it has a loose connection to the kernel theory and the  local sample importance is unclear in this method.\n\nMethod 3 seems to be most practical, but it requires the feature map $\\Phi$ is fixed. \n\nRegarding, Kernel 1: Penultimate-layer Embeddings, it is still unclear to me how to store $\\alpha$ and compute the score $\\alpha_i K(x_i,x)$ if the feature map $\\Phi_{\\theta_1}$ is updated all the times and we have multiple kernels along with the training progress.\n\nFor Kernel 2 and 3, the proposed approach serves as a tool to explain [5] and [6] rather than introduce a new approach. \n\nThe experiments conduct only for binary classification with limited data although the conclusion is interesting.\n\nQUESTIONS:\nPlease address my questions in the weakness section.\n\nLIMITATIONS:\nThe authors did not address the limitations and potential negative societal impact of their work", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}