Faithful and Efficient Explanations for Neural Networks via Neural Tangent Kernel Surrogate Models
Authors: Andrew William Engel
Zhichao Wang
Natalie Frank
Ioana Dumitriu
Sutanay Choudhury
Anand Sarwate
Tony Chiang
Venue: ICLR 2024 spotlight (2024)
Keywords: Explainability
Surrogate Models
Neural Tangent Kernel
Deep Learning
Attribution
Primary area: general machine learning (i.e., none of the above)
TL;DR: We evaluate new approximate neural tangent kernel, including random projection variants that cost less to compute and store.

Abstract: A recent trend in explainable AI research has focused on surrogate modeling, where neural networks are approximated as simpler ML algorithms such as kernel machines. A second trend has been to utilize kernel functions in various explain-by-example or data attribution tasks. In this work, we combine these two trends to analyze approximate empirical neural tangent kernels (eNTK) for data attribution. Approximation is critical for eNTK analysis due to the high computational cost to compute the eNTK. We define new approximate eNTK and perform novel analysis on how well the resulting kernel machine surrogate models correlate with the underlying neural network. We introduce two new random projection variants of approximate eNTK which allow users to tune the time and memory complexity of their calculation. We conclude that kernel machines using approximate neural tangent kernel as the kernel function are effective surrogate models, with the introduced trace NTK the most consistent performer.

Published as a conference paper at ICLR 2024
FAITHFUL AND EFFICIENT EXPLANATIONS FOR NEU- RAL NETWORKS VIA NEURAL TANGENT KERNEL SUR- ROGATE MODELS
Andrew Engel'
Zhichao Wang"
Natalie S. Frank Ioana Dumitriu
Sutanay Choudhury' Anand Sarwate"
Tony Chiang 1,,6
Pacific Northwest National Laboratory
UUniversity of California. San Diego
"Courant Institute, NYU *Rutgers University Uniivrsitty of Washington University Texas, El Paso (andrew. engel, autanay choudhury, tony, .an..o. chiang] #ppn1.gov. (2hw036, i dddmmtrrii @uccdu edu; nf nf10660nyu. edu; ads sooe. rutgers............. .du
ABSTRACT
A recent trend in explainable AI research has focused on surrogate modeling, where neural networks are approximated as simpler ML algorithms such as kernel machines. second trend has been to utilize kernel functions in various explain- by-example data attribution tasks. In this work, combine these two trends analyze approximate empirical neural tangent kernels (eNTK) for data attribution. Approximation is critical for eNTK analysis due the high computational cost compute the eNTK. We define new approximate eNTK and perform novel analysis on how well the resulting kernel machine surrogate models correlate with the underlying neural network. We introduce two new random projection variants of approximate eNTK which allow users tune the time and memory complexity of their calculation. We conclude that kernel machines using approximate neural tangent kernel as the kernel function are effective surrogate models, with the intro- duced trace NTK the most consistent performer. Open source software allowing users efficiently calculate kernel functions in the PyTorch framework is available here".
INTRODUCTION
Explainability remains a critical open problem for applications of deep neural networks (NNs) (Leavitt & Morcos, 2020). Explain-by-example techniques (Lai et al., 2021; Yang al., 2020) have emerged as a major category of algorithms for explainability, including prototype examples (Chen al., 201 Deep K-Nearest Neighbors (Papernot & McDaniel, 2018; Wang et al., 2021; Dziedzic et al., 2022), and Representer Points (Yeh et al., 2018; Tsai et al., 2023). These techniques explain models by providing example(s) that capture model behavior on new data. Kernel functions (Alvarez et al., 2011) are natural choice for building explain- by example algorithms (Yeh et al., 2018) 8);; kernel measures the similarity between individual data points via an inner product a reproducing kernel Hilbert space (RKHS) (Hilbert, 1912; Ghojogh et al., 2021) I). A RKHS that faithfully represents a linearized NN feature space can used in kernel machine to explain (model) the NN decision as weighted sum of similarities to training data.
In this work, we investigate computationally efficient approximations to the empirical neural tangent kernel (eNTK), which is kemnel function motivated by advances in the theory of deep learning (Jacot et al., 2018). It is well established that NNs trained using gradient descent are equivalent to kernel machines (Schölkopf & Smola, 2002) with kernel constructed from sum over eNTK (Lee et al., 2020) computed at each gradient step (Domingos, 2020; Bell et al., 2023). Given this equivalence, would like evaluate the eNTK as the kernel function for explain-by-exampe algorithm; however, computing eNTK is computationally expensive (Novak et al., 2022; Chen et al., 2022),
hhtps//gghhha..oom//mm/mmcccnitn

---

Published as a conference paper at ICLR 2024
so low computational cost approximations have been developed instead (Mohamadi & Sutherland, 2022). We are the first to define and evaluate one such approximate kernel, the trace neural tangent kernel (tNTTK). Additionally, we build from the work of Park et al. (2023) to provide software to compute andom-projeetto cost over traditional eNTK. Using these approximations, we build low-cost and faithful surrogate models for neural network classifiers.
variants that can be computed and stored with lower time and memory
Our methodology improves over the past evaluation of kernel surrogate models. We measure the faith- fulness of a kernel function by assessing how well a kernel generalized linear model (kGLM) (Hof- mann et al., 2007) correlates with the softmax probabilities of the original NN using a rank correlation. Previous evaluations relied on test accuracy (Mohamadi & Sutherland, 2022; Long, 2021), or having high similarity to the correct class (Hanawa et al., 2021), which are both flawed. Our approach and accompanying code-repository will allow users to evaluate how close their own NNs are to kernel machines in the PyTorch framework with limited overhead (Paszke et al., 2019).
CONTRIBUTIONS
We make three major contributions in this work:
1. We define and evaluate new kernel functions for faithful approximation of an underlying neural network; we are the first to analyze random projection variants that permit tuning the computational and memory expense of upproximate eNTK.
2. We are the first to show that approximate eNTK kernel surrogate models are consistently correlated to the underlying neural network across experiments including R8sNet18 on CIFARIO and Bert-base COLA.
3. We compare explanations of NN decisions generated from each kernel function through a data attribution strategy and through an explain-by-ewt by-emample strategy; this is the first such qualitative evaluation between approximate eNTK.
RELATED WORK
Surrogate Models for Explaining Neural Network Behavior. Recent work in explainable Al has focused on determining when NÑs are exactly equivalent to other common ML algorithms (Lee et al., 2018; Balestriero & Baraniuk, 2018; Schmitz et al., 1999), including kernel machines. It has been shown that infinitely wide NNs are equivalent to a kernel machine with kernel function chosen as the neural tangent kernel (Jacot et al., 2018). These infinitely wide models, however, do not replicate the feature learning behavior seen in tnith- networks (Chizat et al., 2018; Yang & Hu, 2021; Wang et al., 2022). Subsequently, researchers turned to investigate properties of finite-width models with NTK computed at various checkpoints (Domingos, 2020; Bell et al., 2023) and/or after training (Long, 2021). 1). This framework was used explore inductive biases (Ortiz-Jiméne) Jiménez et al., 2021), feature learning (Radhakrishnan et al., 2022), learning dynamics (Fort et al., 2020; Atanasov al., 2022), and adversarial faithfulness (Tsilivis & Kempe, 2023; Loo et al., 2022). Support vector machines ( (Vapnik, 1999) using eNTK or approximate eNTK kernels computed after training were shown to achieve the same test accuracy as the underlying NN (Atanasov et al., 2022: Long, 2021; Vyas et al., 2022; Mohamadi & Sutherland, 2022). Our work builds upon this by evaluating whether kernel machines can approximate the underlying neural network function itself, rather than simply reproduce the same test accuracy.
Kernels for Explainability. Kernel functions defined from various RKHS have been proposed to explain the behavior of NN in different contexts, (Park et al., 2023; Koh & Liang, 2017; Pruthi et al., 2020; Akyürek et al., 2023), but in each of these works the kernel studied is loss-based and relies upon the availability of labels at inference time. We differ in that our goal is to model/cxplain the classification behavior on any new data, including unlabeled data where the loss is incalculable. Most relevant to work, Yeh al. (2018) (hereafter Representer Points) used kernel formed from the NN final embedding in what we call the data attribution task (see section 2). We build from Representer Points by evaluating their assumptions under new approximate cNTK kernels.
Computationally Feasible Approximations of the eNTK The computational cost of the eNTK is prohibitively high for large models and datasets. Advances this issue have been two-pronged:
2

---

Published as a conference paper at ICLR 2024
Some groups focus on algorithmic improvements to calculate the eNTK directly (Novak et al., 2022). An alternative strategy has been to avoid eNTK calculation and instead compute kernel functions that share similar structure to the eNTK (Mohamadi & Sutherland, 2022). One such approximate kernel was introduced quietly in Chen et al. (2022) which we refer to as the trace-NTK (trNTK). We are the first to explicitly investigate the trNTK's properties. Finally, Park et al. (2023), hereafter TRAK, utilized random projection matrices to scale the computation of a loss-based kernel function. We modify TRAK to compute projected variants of approximate eNTK.
Evaluating Kernel Attribution. In this paper, we use three evaluation strategies. The first focuses on evaluating the faithfulness of the surrogate model through rank correlation. The second evaluates surrogate model performance on a data-attribution task. We follow the methodology in Shan et al. (2022) to evaluate the model via precision and recall in tracing decisions on poisoned test data back to poisoned training data. Finally, we compare kernels qualitatively via explain-by-exane Previous work evaluated kernels through whether the attributions trace training data of the correct class (Hanawa et al., 2021), whether surrogate models replicate NN test accuracy (Mohamadi & Sutherland, 2022; Long, 2021). These are insufficient: our goal is that kernel functions reflect the neural network behavior, but test accuracy is invariant the specific classification datapoints. Representer Points used Pearson correlation as a faithfulness measure, but Pearson correlation can conflate covariance with faithfulness (see Appendix H). We will demonstrate that our methodology is more secure measurement of faithfulness.
individual
2 PRELIMINARIES
Neural Networks for Classification. We consider the supervised classification problem with C classes. Consider a data input æ E X R" with n the dimensionality of inputs, and a one-hot encoded data label vector the output space y _ RC is an intermediary step in our classification called a "logit." " The NN F(ar; 0) is parameterized by the vector 0 and was learned via back-propagation to minimize the cross entropy loss between the target label vector z and softmax probability vector a [ (F(:;;)) with a: y + Z the softmax function. We denote the e-th scalar output of the network as F We interpret the predicted confidence for the e-th class for input æ as (F(x: ))
2 CC We define neural network F(x;0) X + 3 where
Kernel functions implicitly map the data vector æ to a feature vector p(æ) in higher dimensional RKHS V for which the kernel function n²(.,-) evaluates the inner product of two feature vectors in V. We will notate the data matrix X- I1...... NN E IR N xn with N the number of training samples. With some abuse of notation, we will write k( X) E R for the vector whose j-th component is is (2) and x(X X) E RNNNN for the matrix whose (i, entry is << ..
Kernel Functions.
Kernel General Linear Models as Surrogate Models We limit our investigation of surrogate models to kernel general linear models. We define general kernel linear model kGLM X + )/ asa
kGLM(()) == W (x X) b,
where W& E CCxN is a learnable weight matrix, is the kernel function, and b E RC is a learnable bias vector. We compute classifications from kGLM by mapping the final activations to softmax confidences. The parameters W and b are learned using an optimizer to minimize the cross entropy loss using the same dataset upon which the NN is trained, Given an input æ, the softmax activation o, and NN F(x;; 0), the ideal surrogate modeling goal is to find a kGLM that satisfies:
o(GGLM())) a ((x,)))))
(2)
for all x. Keeping this ideal in mind is useful for building intuition, but in practice, we will relax from this ideal goal for reasons described below.
Data Attribution with Kernels. Our main motivation is explain neural networks through data attribution, i.e., by computing "a score for each training datapoint indicating its importance to the output of interest" (TRAK). Given the choice of kernel function k. the scalar valued data attribution for the e-th class for a test input x and a training datapoint æ; is given by:
A(x, az = We, k( x.) + b
(3)
N
3

---

Published as a conference paper at ICLR 2024
be term is necessary to ensure that the sum the attributions for the entire training Where the N
dataset equal to the kGLM's logit for class c, Σ a( kz) If the kGLM is an
ideal surrogate model Eq. 2, then the softmax function applied to the vector created from each class attribution will equal the NN confidence in each class. Consequently, we will have decomposed the reasoning for the NN's specific confidence in each class linear combination of similarities between a and each training datapoint 2;- We emphasize that Eq. 3 is our definition of data attribution. Attribution a weighted sum of kernel/similarity values.
3 METHODS
We now turn towards the novel work of this research. In the following sections we describe our measure of faithfulness then introduce the kernel functions.
Evaluating the Faithfulness of Surrogate Models. Given many choices of kernel functions we require measure determine which surrogate models have higher approximation quality (i.e.. faithfulness) to the NN. We relax from the ideal surrogate model goal Eq. 2 and instead evaluate kernel functions by how well they are correlated with the neural network using the Kendall-7 rank correlation.
To assess the faithfulness of surrogate model, we compute Tк between the softmax probability of the neuron representing the correct class, a (F(x: 0)); and the kGLM softmax probability for the output representing the correct class, (x)) тK was chosen for two reasons; First, тк has range [] 1] with + I representing monotonic relationship and value of 0 representing no correlation. Second, if the relationship between the kGLM and NN is strictly monotonic, then an invertible mapping function exists between the kGLM softmax probabilities and the NN's (Bartle & Sherbert, 2011). Therefore, for a тк = 1 we would recover the one-to-one ideal surrogate model relationship given by Eq. 2. In Appendix L, we demonstrate how to find these mapping functions with iterative optimizers (Virtanen et al., 2020). We provide a formal definition of Kendall-7 rank correlation in appendix G.
We additionally report two more complementary metrics. While we have argued that the test accuracy flawed measure faithfulness, we will report the test accuracy differential be complete with prior works. We define test accuracy differential (TAD) as:
TAD == Test.Aco.GLM TestACCNN.
A fundamental limitation of TK is that it can only be computed over set of scalar outputs so does not take advantage of the vectorized output of classification networks. To compensate, we will also report the misclassification coincidence rate, (Rmiss), which captures whether two models both misclassify the same datapoints as the same class, which is an intuitive property TK misses. A formal definition of Riiss is available in appendix G. We turn to defining the specific kernel functions we evaluate.
Trace Neural Tangent Kernel. For any two data inputs æ; and æy., we define the Jacobian of the NN's e-th output neuron with respect to O at datapoint æ; as ( 0) Fx 0). Then, for choice of class e and c', the eNTK is a kernel function defined as:
eNTK (2i, ar)) = (g (a:;; 0)g²' (æ.;; 0))-
For C' classes and N datapoints, the full eNTK can be evaluated for each choice of (c,c') and (i,j) resulting in a large NC' x NC total size matrix. This matrix is often too expensive to compute or manipulate in memory, leading researchers to seek approximations.
We introduce now the trace neural tangent kernel (trNTK) approximation, which removes the C2 scaling memory by effectively performing a "block-trace" operation on the original eNTK. The trNTK is kernel function defined as:
S ( (x.; ; 0), (x; 0))
trNTK (z 2.y)
=
(5)
Σ (m: 0) 0))))> Σ | (m,; 0))
4

---

Published as a conference paper at ICLR 2024
The denominator of Eq. 5 is normalization that makes the trNTK a kernel of cosine-similarii values. has been suggested that this normalization helps smooth out kernel mass over the entire training dataset (Akyürek et al., 2022). The normalization ensures that two identical inputs always have maximum similarity value 1. Additional intuition about how this kernel relates to the geometry of the neural network function surface is available in Appendix C. We provide additional details about these definitions in Appendix D. In the following section, we relate this kernel to another approximate eNTK kernel, the pseudo neural tangent kernel.
Wei et al. (2022)
Relationship to the Pseudo Neural Tangent Kernel. We can understand the motivation for the trNTK in the context of another approximate eNTK, called the pseudo neural tangent kernel (pNTK). The pNTK computed between inputs æ; and æj a kernel function defined as:
(N 1 ( Vo E F(a::0)))
Ve Σ F(z;;00 (
(6)
Mohamadi & Sutherland (2022) showed that the product of the (N mz) with the C x C identity matrix is bounded Frobenius norm to the eNTK by O (.)).. with n the width parameter of feed forward fully connected NN with ReLU activation (Nair & Hinton, 2010; Glorot et al., 2011) and He-normal (He al., 20) initialization, with high probability over random initialization.
We can frame the critical differences between the pNTK and trNTK by how each approximate the eNTK. The pNTK approximates the eNTK as a constant diagonal matrix with constant equal to the scalar kernel function given in Eq. 6. In contrast, the LrNTK allows the diagonal elements of the eNTK approximation to vary, and in fact, calculates these values directly. Both the pNTK and trNTK perform simplifying
over the diagonal elements, which reduces the memory footprint of the approximations by a factor C² compared to the eNTK. We choose not to compare directly with the pNTK because the trNTK is higher cost, but more precise, approximation of the eNTK. Instead, we focus our comparisons to much lower cost alternatives, including a projection variant of the pNTK.
Projection (rNTK and Projection pNTK. For large number of parameters P and large datasets N, computing approximate eNTK remain expensive, therefore, we explore random projection variant that allows us to effectively choose P regardless of architecture studied. Let P be a random projection matrix P IK x K P. with all entries drawn from either the Gaussian A (O, 1) or Rademacher (with p=0.5 for all entries) distribution. K is hyperparameter setting the projection matrix dimension. We set K 10240 for all experiments. We use to project the Jacobian matrices to a lower dimension, which reduces the memory needed to store the Jacobians and reduce the time complexity scaling. The Johnson--Lindensrraus lemma ensures that most of the information in the original Jacobians is preserved when embedded into the lower dimensional space (Johnson & Lindenstrauss, 1984). We define the proj-trNTK and proj-p.TTK as random projection variants of the trNTK and pNTK:
C P P
(z;
0).
P.P. (z.z 0)
Σ
proj-pNTK((;; 2r,)
(x,, 0)
P
g g 0)
E (m; 0), (a
proj-teNTK(E,,233)
(8)
(a, 0) 12)
Σ
[P;
where both definitions include the cosine- mesine-oomalliztion.
Random projection variants can improve the time complexity scaling for computing approximate
cNTK under large dataset size and large number of parameters. Assuming computation via Jacobian contraction and time [) for a forward pass, the eNTK time complexity is: NCFPP] + C²C²p P (Novak et al., 2022). The pNTK computation reduces this to N[FP] 2 while the trNTK computation only reduces to NC N In contrast, the PrK costs N[F P N2K
5

---

Published as a conference paper at ICLR 2024
NKP, and the proj-trNTK costs NC[FP] CN CNKP. The final term in the projection variants is the cost of the extra matrix multiplication with the random projection matrix P and the Jacobian matrix. For K < P and N large, projection variants reduce the time complexity.
Additional Kernel Functions. We also evaluate the conjugate kernel (CK) formed from the Gram matrix of the final embedding vector (Fan & Wang. 2020; Yeh et al., 201 8), the un-normalized trNTK (trNTK") which is equal to the numerator of Eq. 5, and the embedding kernel (Akyürek et al., 2023), formed from a sum over the Gram matrices of embedding vectors from various layers in the network architecture. See Appendix B for formal definition of these kernels.
4 RESULTS
Experiments. Classification NNs with architectures and datasets (MNIST (Lecun al., 1998), FM-
NIST (Xiao et al., 2017), CIFARIO (Krizhevsky & Hinton, 2009), and COLA (Warstadt et al., 2018)) shown in Table are trained using standard techniques. Additional details regarding datasets are pro- vided in Appendix K.1. Models that have value of more than in the column '# Models' in Table 1 are trained multiple times with different seeds to generate uncertainty estimates. The ResNet18 (He et al., 201 5b),,, ResNet34, and MobileNetV2 (Sandler et al., 2018) models were trained by an indepen- dent research group with weights downloaded from an online repository (Phan, 2021). Bert-base (De- vlin et al., 2019) weights were downloaded from the HuggingFace (Wolf et al., 2019) repository then transferred onto the COLA dataset, as is common practice for foundation models (Bommasani et al., 2021). After training, we calculate the trNTK and alternative kernels using Py Torch automatic differentiation (Paszke et al., 2019). We train a kGLM (sklearn SCD ccDclec ass i fier) (Pedregosa et al., 2011) for each k using the same training dataset for training the NN model. All computation was completed on a single A100 GPU with 40GB memory. Details such as specifics of architecture and choice of hyperparameters are available in Appendix K.
Faithful Surrogate Modeling via NrN We calculate the Tk correlation between the surrogate model and underlying NN and report the results in Table We find that the efficacy of our surrogate model as measured by the correlation to the NN changes depending on architecture and dataset; though remarkably, TK is consistently high, with a lower bound value of 0.7 across all experiments, indicating high faithfulness. To demonstrate high тк implies we can achieve a point-for-poing linear realization of the NN, we learn non-linear mapping from the kGLM to the NN (Figure for Bert-base. (Additional visualizations for the remainder of experiments are available in Appendix L.) Finally, we observe that the kGl.M with choice of = LNNTK achieves comparable test accuracy as the underlying NN, which replicates the observations of prior work (Long, 2021; Vyas et al., 2022; Mohamadi & Sutherland, 2022) using our trN
Data Attribution with NTK. Accepting that the trNTK is a faithful kernel function for a kGLM surrogate model, we can use the data attribution formalism to analyze the importance of individual training datapoints to the classification. In Figure 2 we present the visualization of data attribution for one test input and provide additional visualizations in Appendix M. The distribution of attribution follows regular pattern in every visualization generated: the central value of attribution mass for each logit from each class is centered on the distribution of all training data from that class. We emphasize that in no cases have we observed a sparse number of training datapoints dominate the data attribution.
Comparison of Faithfulness between Kernels Functions. For 8 and Bert-base models, we evaluate our choice of trNTK against alternative kernel functions, reporting Tx and test accuracy differential in Table 2. Across both R and Bert-base experiments, we observe that the trNTK forms surrogate models with the highest correlation to the underlying NN decision function and furthermore consistent in replicating the performance of these networks (TAD nearly 0). The embedding kernel (Em) does not perform consistently between both tasks, but for its intuitive connection to the internal representation of the neural network may warrant further investigation.
Faithful Surrogates in Data Poisoning Regime. Next, we evaluate whether surrogate models can be extended to analyze network behavior on poisoned data. We train a 21 -lyye CNN (details available in Appendix K.2.5) using BadNet CIFAR10 data (Gu et al., 2019; Shan et al., 2022). We randomly perturb training data by placing a yellow square in a tenth of training images from CIFAR1O and modify the label of these perturbed images to targeted label (see example in Appendix N). We

---

Published as a conference paper at ICLR 2024
Table 1: Choice of trNTK faithfully forms surrogate model of underlying NN. We perform each experiment with '# Models' independent seeds. For each model and dataset we train and extract the trNTK, train kGLM. then calculate and report the correlation between the kGLM softmax probability and NN softmax probability for the correct class. The NN test accuracy column shows that training terminates with highly performant model, and the test accuracy differential (TAD) columns reports the difference between the MGLM test accuracy and NN test accuracy. We report the leading digit of emor (standard error of the mean) as a parenthetical, when available.
Model (Dataset)
# Models
NN test acc (%)
TAD (%)
TK
MLP (MNIST2)
100
99.64(1)
+0.03(5)
0.708(3)
CNN (MNIST2)
100
98.4(1)
-0.2(2)
0,857(7)
CNN (CIFAR2)
100
94.94(5)
-2.1(5)
0.711(3)
CNN (FMNIST2)
100
97.95(4)
-2.((2)
0,882(3)
ResNet18 (CIFARIO)
93.07
-0.28
0.776
ResNet34 (CIFAR10)
1
93.33
-0.29
0.786
MobileNetV2 (CIFARIO)
93.91
-0.4
0.700
BERT base (COLA)
4
83.4(1)
-0.1(3)
0.78(2)
BERT-base NN Logits over Linearized KGLM Model
=².0.94
44 T=0.82
-² T=0.73
-280.89 T= 0.83
NN Logits
Parity
True Class Z = 0
True Class
kGLM Linearizing-Mapper Logits
Figure 1: Linear Realization of Bert-base Model. Each panel shows linearization of Bert-base transfer model, initialized from a different seed. An invertible mapping fit between the kGLM kGLM's final activations to the NN's, described in Appendix L.. Both Tk and the Coefficient Determination (R²) are shown for cach model.
transform
Table 2: Comparison across surrogate feature spaces. For ResNet18 and Bert-base experiments we report the faithfulness as TK. restaccuracy-diffee differeentali (TAD), misclassification coincidence rate (Ryus) for each kernel function: the trace-NTI NTK (trNTK), unnormalized trace-NTK (trNTK", the projection trace NTK pproj-trNTKK the projection pscudo NTK (proj-pNTK)) the embedding kemel (Em) and the conjugate kemel (CK). If available, we report leading digit of error (standard error of the mean) as parenthetical.
Exp Name
Metric
trNTK
trNTK
proj-trNTK
proj-pNTK
Em
CK
TK
0.776
0.658
0.737
0.407
0.768
0.630
ResNet18
TAD (%)
-0.30
-0.52
-0.20
-0.30
-0.32
-0.20
RMiss
0.75
0.65
0.77
0.71
0.80
0.73
0.809(9)
0.5(1)
0,800(9)
0.72(2)
0.65(2)
0.52(4)
Bert-base
TAD (%)
+0.1(3)
+0.6(2)
+0.1(2)
+0.5(2)
-0.3(5)
-0.1(1)
RMiss
0.67(2)
0.71(5)
0.61(2)
0.86(3)
0.86(2)
0.91(2)
create a "clean" test dataset from CIFAR10's normal test dataset, and a "poisoned" test dataset by placing yellow squares into each image of CIFAR10's s test dataset. At test time, perturbed test data tricks the model into producing labels of the targeted label. We train a model on this poisoned dataset, compute each kernel function, measure faithfulness, and report results in Table 3. We find that the
7

---

Published as a conference paper at ICLR 2024
Analyzing Distribution of Attribution
A)
B)
C)
predicts
describing
dog
1e
trNTK dog
Training Point Attribution In Logit= Logit=doo dog
Image: corr=dog NN=dog
Mean Attribution
iDog
Ngann
Woon
Rova
Norve
Now
Hout
Trining Data Class
plane
bird
deier
frog
Figure 2: Overview of Using Kernel Functions for Data Attribution A) An image from the test dataset of CIFARIO is chosen. B) We propagate the test image through the NN and plot the mean attribution of the training points from each class for each output neuron. C) Zooming into the neuron representing class "dog". view the distribution attributions as modified box-plot with central lines mean and outliers shown flier points. The mean lines are always observed to be within the inner quartile, suggesting that no sparse number of datapoints dominate the central value, and therefore, do not dominate the data attribution.
trNTK is most faithful to the NN on the clean test data, but the proj-pNTK is most faithful when evaluated on the poisoned test data. Overall comparison to the non-poisoned set of experiments each kGLM is less faithful, except for the proj-pNTK. We also point out that the kGLM with overall highest faithfulness are the kernel functions with our assine-normalization applied.
In addition, show application of surrogate modeling approach enabled by kernel-techniques. Forensics models trace NN behavior on unseen poisoned data to the poisoned data source a training set (Shan al., 2022). We treat each kernel poisoned test dataset we compute the top 5 most similar training datapoints. If 3/5 of these training datapoints are poisoned we flag the image as poisoned. In doing so, we can filter poisoned images from clean images. We report the performance of our forensic models using precision and recall (see Appendix G) in table 3. Each kernel, except for the conjugate kernel, are all comparable in performance forensics models. Appendix N provides examples of multiple forensic models acting on poisoned and clean versions of CIFARIO 10 data.
forensic model: for each image in the clean and
Table Poisoned data attribution forensics. compute each kernel function between poisoned training data and the clean test dataset. We report 7x. TAD, and Risss between the kGLM and NN for both the poisoned (poi.) and clean set of unseen test images. Finally, we evaluate each kernel as filter for identifying unseen poisoned data through high similarity to poisoned training data and report the performance Precision and Recall.
Method
Precision (%)
Recall (%)
TAD (%)
PMiss
pui. Tk
poi. TAD(%)
poi.
Rytiss
trNTK 99,99
100.00
0.643
+0.45
0.44
0.569
+0.09
0.12
trNTK<
99.99
99.97
0,344
+0.87
0.20
0.125
+0.13
0.01
proj-trNTK
99.99
99.97
0,565
+0.09
0.45
0.418
+1.3
0.12
Proj-pNTK
99.99
100.00
0,554
+0.07
0.59
0.665
-1.3
0.11
Embedding
99.71
100.00
0.430
-2.73
0.07
0.261
-13.98
0.22
CK
1.65
50.61
0.552
-3.50
0.38
0.454
-81.25
0.00
8

---

Published as a conference paper at ICLR 2024
5 SUMMARY AND CONCLUSIONS
Impact of Linear Surrogate Modeling for Explainability. We have shown evidence supporting the choice of the trNTK as a consistently faithful choice of kernel function for surrogate model (table 1). We made this determination by measuring the correlation between the kGLM surrogate and the NN, which is an improvement over past methodologies. Our choice of a linear model surrogate model allows us to separate the attribution terms from each training datapoint, and ensures the central value of the attribution distribution is coupled the kGLM's s logit, and therefore the NN which approximates (Section 2). We observed that the highest attributed images from the trNTK have relatively small mass compared to the bulk contribution, suggesting that the properties of the bulk, rather than a few outliers, are the main source driving decision making. We believe this is result of the cosine normalization we apply in our definition of the trNTK, as the unnormalized tNNTK shows a much tighter IQR of attribution (see appendix M.1.2), and in fact, this pattern exists between all normalized vs normmallld kernel functions. This directly visualizes the intuition that the cosine normalization "smooths- outto the attribution (Akyürek et al., 2022). Because the properties of the bulk drive classification, we conclude that presenting the top highest attribution training images without the context of the entire distribution attribution is potentially misleading as a form of explanation, strategies is misguided.
i.e., the assumption of sparsity in
Comparison of Kernel Functions for Surrogate Models. Our quantitative experiments showed the tNNTK as more consistently correlated to the NN model compared to the unnormalized trNT,, Embedding kernel, and CK. We observe qualitative differences between these kernel's attributions (Appendix M. and which training datapoints have highest similarity (Appendix N). As qualitative comparison between kernel functions, in Appendix M.2 we visualize the top-5 most similar datapoints evaluated by each kernel function. This further reveals the similarities and differences between kernel functions. Overall, we observe that the (rNTK is more sensitive to conceptual similarities between test and train examples than the CK. The embedding kernel consistently sensitive to background pixel values, though this may be an artifact from our specific choice of layers to sample from. The proj-trNTK,, as expected, follows closely with the regular NTK. These differences could be used to tied to interesting phenomena: for example, because the CK is computed from the final embedding it is likely more sensitive to the effects of neural-collapse (Papyan al., 2020) than the NTK, which is computed from Jacobians of weight tensors across the entire architecture. We believe this fact explains why the highest similar images measured by the LNNTK are more conceptually tied to the specific test image, while the CK has collapsed that inner-class variance away.
Computational Feasibility. Finally, we comment on the computational feasibility of each of the kernel functions. Table 4 reports the time to compute each kernel, and Appendix F shows that the empirical residual distribution between the trNTK and proj-trNTK falls exponentially. The projection-trNTKK and projection-pNTK have efficient computation thanks to software made available in Park et al. (2023). The full trNTK is by far the slowest. As implemented, our trNTK computation was layerwise (see Appendix D), except in the Poisoning experiment, which we now believe is sub-optimal. Both the (rNTK and projection-trNTK computation scales with the number of output neurons linearly, so for models with large output space the projection-pNIT may remain the only feasible option. Finally, because the residuals between the trNTK and proj-trNTK are small and decay rapidly, we believe using the projected variants are well justified. In total, we believe the differences between the trNTK and proj-trNTK are small enough that for small number of outputs, our recommendation is to utilize the proj-trNTK. Finally, see Appendix A for limitations.
Table 4: Computational Complexity of Large Model Experiments. We report time to compute each of the trNTK, Proj-trNTK, and proj-pNTT for the large model large dataset experiments are shown.
Exp Name
trNTK
proj-trNTK
proj-pNTK
ResNet18
389h
1.12h
7.4m
BertBase
1200h
22m
12m
Poisoning
50h
9.3m
1m
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The authors thank Panos Stinis, Mark Raugas, Saad Qadeer, Adam Tsou, Emma Drobina, Amit Harlev, Ian Meyer, and Luke Gosink for varied discussions while preparing the draft. This work would not have been possible without the help from Wendy Cowley in helping navigate the release protocol. The authors thank Davis Brown for discussions regarding TRAK. A. W.E. Z.W., S.C., N.F., and T.C. were partially supported by the Mathematics for Artificial Reasoning in Science (MARS) initiative via the Laboratory Directed Research and Development (LDRD) Program at PNNI. and A.D.S. and T.C. were partially supported by the Statistical Inference Generates kNowledge for Artificial Learners (SIGNAL) Program at PNNL. A.D.S. was partially supported by the US NSF under award CNS- 2148104. PNNL is a (DOE) by Battelle Memorial Institute under Contract No. DE-ACO5-76RLO-I
national laboratory operated for the U.S. Department of Energy
REFERENCES
Ekin Akyürek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. Towards tracing knowledge in language models back to the training data. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2429-2446, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL hutp://aclanthooogy.oogg wwwpg///2..ininnnooooooe 180.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algo- rithm is in-context learning? investigations with linear models. In The Eleventh International Con- ference on Learning Representations, 2023. URL
Mauricio A. Alvarez, Lorenzo Rosasco, and Neil D. Lawrence. Kernels for Vector Valued Functions: Review. arXiv e-prints, art. arXiv: 106. 6251, June 2011. doi: 10.48550/arXiv. 106.6251.
Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect. In International Conference on Learning Representations, 2022. URL
https://ppnneviee..eefffommm Pid= Ni
Randall Balestriero and Richard Baraniuk. A spline theory of deep networks. In International Conference on Machine Learning, 2018.
Robert G. Bartle and Donald R. Sherbert. Introduction to Real Analysis (4th Edition). Wiley, 2011.
Brian Bell, Michael Geyer, David Glickenstein, Amanda Fernandez, and Juston Moore. An exact kernel equivalence for finite classification models, 2023.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman. Neel Guha Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang, Thomas Ieard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair. Avanika Narayan. Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr. Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the Opportunities and Risks of Foundation Models, arXiv e-prints, art. arXiv:2 100.21088.07558 August 2021. doi: 10.48550/ar 2108.0728..
10

---

Published as a conference paper at ICLR 2024
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019.
Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When Vision Transformers Outperform ResNets without Pre-training Strong Data Augmentations. In International Conference Learning Representations, 2022. URL http:://peeeeeeeeveeeeeoonmndddd
Lénaïc Chizat, Edouard Oyallon, and Francis R. Bach. On lazy training in differentiable programming. In Neural Information Processing Systems, 2018.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp. 4171 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18663/11/119114233. URL https: 1wwp:/hooooggooo//1---9999-
Pedro Domingos. Every model learned by gradient descent is approximately kernel machine. arXiv preprint arXiv:20/2.00152 2020.
Adam Dziedzic, Stephan Rabanser, Mohammad Yaghini, Armin Ale, Murat A. Erdogdu, and Nicolas Papernot. p- D-DKNN: Duttt--Distributiin Detection Through Statistical Testing of Deep Representations. arXiv e-prints, art. arXiv:2207.125453 July 2022. doi: 10.48550/rX1w.22207. 12545.
Zhou Fan and Zhichao Wang. Spectra of the conjugate kernel and neural tangent kernel for linear- width neural networks. Advances in neural information processing systems, 33:7710-7722 2020.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33:5850-5861, 2020.
Benyamin Ghojogh, Ali Ghodsi, Fakhri Karray, and Mark Crowley. Reproducing kernel hilbert space, mercer's theorem, eigenfunctions, nystr\." om method, and use of kernels in machine learning: Tutorial and survey. arXiv preprint arXiv:2106.08445 2021.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Interna- tional Conference on Artificial Intelligence and Statistics, 2011. URL https://api.semanticchhhaa Org/CorpusID:2233473...
Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. BadNets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230-47244, 01 2019. doi: 10.1 ACCCESSS 2019.2909068.
Kazuaki Hanawa, Sho Yokoi, Satoshi Hara, and Kentaro Inui. Evaluation of similarity-based explanations. In International Conference on Learning Representations, 2021. URL https:
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: level performance on imagenet classification. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034, 2015a. URL
Surpassing human-
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2015b.
David Hilbert. Grundzüge einer allgemeinen Theorie der linearen Integralgleichunger. Number 3. BG Teubner, 1912.
Thomas Hofmann, Bernhard Schölkopf, and Alex Smola. Kernel methods in machine learning. Annals of Statistics, 36:1 171-1220, 2007.
11

---

Published as a conference paper at ICLR 2024
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
William B. Johnson and Joram Lindenstrauss. Extensions of lipschitz mappings into hilbert space. Contemporary Mathematicx, 26, 1984. URL http://stanforr..cuu/llss////// /tt.a/////laassss///eaaannng//// //rre//ings//--
pdf.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, pp. 1885-1894. PMLR, 2017.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Vivian Lai, Chacha Chen, Q Vera Liao, Alison Smith-Renner, and Chenhao Tan. Towards science of human-ai decision making: a survey empirical studies. arXiv preprint arXiv:2112.1443 2021.
Matthew L. Leavitt and Ari Morcos. Towards falsifiable interpretability research. arXiv e-prints, art. arXiv:2010. 12016, October 2020. doi: (0.48550/arXi.2201000 20160.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2,22,,,278-2324, 1998. doi: 10. 10.1109/5.72679
Jachoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak. Sam Schoenholz, and Yasaman Bahri. Deep neural networks as gaussian processes. In International Conference on Learning Representations, 2018. URL http:s//pppeneeeeeeeeefooomt BiEAnn//---BB---OO
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Journal of Statistical Mechanics: Theory and Experiment, 2020(12): 124002, December 2020. doi: 10. 11100//744-546///b0b0e
Rongmei Lin, Weiyang Liu, Zhen Liu, Chen Feng. Zhiding Yu, James M. Rehg, Li Xiong, and Le Song. Regularizing neural networks via minimizing hyperspherical energy. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 691 6916-6925, 2019.
Philip M Long. Properties of the after kernel. arXiv preprint arXiv:2105.10585, 2021.
Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Evolution of neural tangent kernels under benign and adversarial training, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. arXiv e-prints, art. 1.X0v:1711.05101, November 2017. doi: 10550/arXiv..711.05555
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. ICLR, 2019.
Mohamad Amin Mohamadi and Danica J. Sutherland. A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernel. arXiv e-prints, art, arXiv:2206.12543, June 2022. doi: 10.485550/rxiwv22206 12543.
Vinod Nair and Geoffrey F. Hinton. Rectified linear units improve restricted Boltzmann machines. In International Conference on Machine Learning, 2010.
Roman Novak, Jascha Sohl-Dickstein. and Samuel Schoenholz. Fast finite width neural tangent kernel. In International Conference on Machine Learning, pp. 17018-17044, PMLR, 2022.
Guillermo Ortiz-Jiménez, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. What can linearized neural networks actually say about generalization? Advances in Neural Information Processing Systems, 14:8998-9011 2021.
Nicolas Papernot and Patrick McDaniel. Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning. arXiv e-prints, art. arXiv: 100.::1800.0411.. 880.004765 March 2018
12

---

Published as a conference paper at ICLR 2024
Nicolas Papernot, Patrick D. McDaniel, and lan J. Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. CoRR, 2016a.
Nicolas Papernot, Patrick D. McDaniel, lan J. Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against deep learning systems using adversarial examples. CoRR, 2016b.
Vardan Papyan, Xuemei Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences of the United States of America, 117:24652 24663, 2020.
Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. TRAK: Attributing Model Behavior Scale. arXiv e-prints, art. arXiv:2303. 14186, March 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, segh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg. J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Huy Phan. huyvnphan/pytorch. 1ttpf/yotttoocccii11..0 January 2021. URL https///oriogg//..22/////eeeeeee
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In Advances in Neural Information Processing Systems, volume 33, pp. 19920-1 19930. Curran Associates, Inc., 2020.
Saad Qadeer, Andrew Engel, Adam Tsou, Max Vargas, Panos Stinis, and Tony Chiang. Efficient kernel surrogates for neural network-based regression. ArXiv, abs/2310.18612, 2023. URL
Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. Feature learning neural networks and kernel machines that recursively learn features. arXiv preprint 2rX.::2212.138811 2022.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chies Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018
G.P.J. Schmitz, C. Aldrich, and F.S. Gouws. Ann-dt: an algorithm for extraction of decision trees from artificial neural networks. IEEE Transactions Neural Networks, 1066):1392-140.. 1999. doi: 10.1 10918.99788809884
Bernhard Schölkopf and Alexander J Smola. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2002.
Shawn Shan, Arjun Nitin Bhagoji, Haitao Zheng, and Ben Y Zhao. Poison forensics: data poisoning attacks in neural networks. In 3 USENIX Security Symposium (USENIX Security 22), pp. 3575-3592, 2022.
Traceback of
Antonio Torralba, Rob Fergus, and William T. Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1970 2008.
Che-Ping Tsai, Chih-Kuan Ych, and Pradeep Ravikumar. Sample based explanations via generalized representers. ArXiv, 0bs/2310.18522 2023. URL http:://ap.scmmaatccchhhooooooop 264590303.
Nikolaos Tsilivis and Julia Kempe. What can the neural tangent kernel tell us about adversarial robustness?, 2023.
13

---

Published as a conference paper at ICLR 2024
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Breu, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, Ilhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, António H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SeiPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. doi:
Nikhil Vyas, Yamini Bansal, and Preetum Nakkiran. Limitations of the NTK for Understanding Generalization in Deep Learning. arXiv e-prints, art. arXiv:2206.10012, June 2022.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019. URL http:///oenmiiiv.ee//fouuu id=rJ4km2R5t7.
Ren Wang, Tianqi Chen, and Alfred Hero. Deep Adversarially--nnhance k-Nearest Neighbors. arXiv e-prints, art.
August 2021. doi: 10448550/arXii22108.000
Zhichao Wang, Andrew Engel, Anand Sarwate, Ioana Dumitriu, and Tony Chiang. Spectral evolution and invariance in linthr-width neural networks. arXiv preprint arXiv:221 2022.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2018.
Alexander Wei, Wei Hu, and Jacob Steinhardt. More than toy: Random matrix models predict how real-world neural representations generalize. In International Conference Machine Learning. pp. 33549-23588 PMLR, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. HuggingFace's Transformers: State-of- the-art Natural Language Processing, arXiv e-prints, art. arXiv:1910.03771. October 2019, doi: 10.48550/arXiv. htt.:/aaaiii..11..0.... 1910.03771.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a novel image dataset for bench- marking machine learning algorithms. ArXiv, abs/1708.07744 2017.
Fumeng Yang, Zhuanyi Huang, Jean Scholtz, and Dustin L Arendt. How do visual explanations foster end users' appropriate trust in machine learning? In Proceedings of the 25th International Conference Intelligent User Interfaces, pp. 189 209, 2020.
Greg Yang and Edward Hu. Tensor Programs IV: Feature learning in infinite-width neural networks. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11727-11737. PMLR, 18-24 Jul 2021.
Chih- Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. Advances in neural information processing systems, 31, 2018
14

---

Published as a conference paper at ICLR 2024
A LIMITATIONS
We point out previous works using support vector machines (SVM) kernel surrogate models report limitations that we believe extend to kGLM models. We know of two such limitations. We found that SVM surrogate models fail to replicate NN behavior under gradient-based adversarial attacks Appendix In addition, SVM surrogate models do not have the same scaling relationships as underlying NNs (Vyas et al., 2022). Our conclusions are limited to kGLM surrogate models; an interesting follow-on work would investigate using kernel functions in K-Nearest Neighbors surrogate models which may recover sparse explanation.
fundamental limitation of our choice of Kendall-r was discussed in section and we expand upon it here. Kendall-r requires set of scalars, which forces us to reduce the naturally vector output space of classification networks to a single value. We choose to use the logit representing the correct ground-truth class. This is reasonable given that the confidence given by the neural network in the correct class is an interesting behavior with consequences to the classification task; however, this choice does not leverage the total amount of information given by the output soft-max vector. To compensate for this, we report the misclassification coincidence rate, RMiss, which utilizes the intuition that coupled models should also be wrong in the same way, the same time. While this added metric provides an additional powerful line of evidence demonstrating the coupling between kGLM and NN, it also clouds our analysis on which Kernel function represents the best choice. Therefore future work should continue to improve upon Kendall-r as a metric for faithfulness.
While many explainability techniques now exist, its not always clear how useful any technique actually is until a human reviewer attempts to utilize the technique. In this work we do not perform any human subjects testing to evaluate each kernel, but principle this would be an interesting direction for future work.
This work's 's premise is limited in that we have no guarantee that the surrogate model is performing "reasoning" in the same manner as the underlying neural network. We have only worked to establish that the kGLMs are highly coupled to NNs and evaluate this coupling between different choices of kernel functions. Because we find evidence for a high correlation between NN and kGLM models, we suggest that structure of kGLMs serve as a potential explanation of NNs in a way that connect decisions made new inputs to specific training data. In the most limited view of this work, this is simply a fundamental assumption that must be empirically evaluated for each new network-kGLM pair. Follow on work could look compute the eNTK multiple times throughout training to form an approximation to the path kernel (Domingos, 2020).
Finally, our evaluations between the trNTK and the pNTK are limited in extent to which either a true approximation of the eNTK. For example, we are guaranteed that the Lr (r(tNTTK) (r(eNTK) all times, but the (r(pNTK) does direction of future work would be to evaluate to what extent the trNTK reproduced the eNTK in similar manner as Mohamadi & Sutherland (2022). any case, given the computational difficulty of the eNTK we believe the more interesting questions are for what behavior/phenomena are the approximations "close-enough" to model the eNTK. This has recently been explored in Qadeer et al. (2023).
necessarily equal the Ir(eNTK) all times. An interesting
B DEFINITION OF KERNELS
In this Appendix we provide the definition of each of the kernel functions evaluated. For convenience we restate the definition of the trNTK
trNTK Recall the definition of the total gradient with respect to 0 at datapoint o. by
0) =
0).
Then the trN evaluated at datapoints x; and xy is given by
Σ (g 0), 9 (z;;
trNTK (x); ==
Σ g (~:; 0) ())) Σ (m); 0))
15

---

Published as a conference paper at ICLR 2024
We provide additional details about the exact calculation in Appendix D.
Projection Trace Neural Tangent Kernel. We restate our definition of the proj-trNTK kernel function:
Σ (o 0), Pg"
proj-trNTK((x,z))) =
(a (0:
Pg 0)133)
We remind the reader that P is Rademacher Gaussian random projection matrix RXXP with hyperparameter, P the number of model parameters, and K chosen to be K < P. In all experiments K = 10240.
Projection Pseudo Neural Tangent Kernel.
(P Σ c=1
g 0). g (m, 0))
(z
(r ==
ga, 0) ||P
Σ (z, 0)1
Embedding Akyürek et al. (2022) defines the embedding kernel, which restate here. The embedding kernel is computed from the correlation of the activations following each layer. Let xe(x;;) 0) be the output of the f-th hidden layer of F(x;; 0). We denote the ε-th embedding kernel datapoints 2; and æ; by
EE 2,) = (x 0), A (z, 0))
[a ; 0)1 (m O)]l
Let the full embedding kernel be defined by the normalized over the unnormalized embedding kernel at each layer of the NN
(l (e,)e(ar;;)))
B(D4,
EEE : 1) (ee 0)
Embedding kernels are interesting comparison for the data attribution task when we consider the prominent role they play transfer learning and auto-encoding paradigms. both, finding an embedding that can be utilized in down-stream tasks is the objective.
Conjugate Kernel We utilize an the empirical conjugate kernel (CK) to compare to the trNTK. Let the normalized CK be defined by
(Az (xx :)) CK(2,, x,) (x;;;;;;)) Oz (x, 0)
The CK is interesting comparison for couple of reasons: first, for any network that ends in fully connected layer, the CK is actually an additive component of the trNTK; therefore, we can evaluate whether smaller amount of the total trNTK CK is computed from the final feature vector before a network makes a decision; the NN is exactly a linear model with respect this final feature vector. NN architectures typically contain bottlenecks that project down to this final feature vector. These projections remove information. While that information might be of no use to the classification task, may be useful for the attribution task. We can think of the the final information presented to the NN as the CK, and the information contained before these projections as the trNTK, though more work needed to formalize and explore this comparison.
accomplish the same task. Second, the
Unnormalized Pseudo Neural Tangent Kernel To evaluate the effect of the normalization in the LNNTK definition we will evaluate the kernel without normalizing, let the unnormalized (rNTK be defined as:
NNN æ.)) g( 2,; 0) g g(x,; ;
While neural tangent kernels are not typically cosine-nonee cosine-nommlized kernels we were drawn to investigate such normalized kernels for few reasons: Akyürek al. (2022) remarked that cosine normalization
16

---

Published as a conference paper at ICLR 2024
F(m::
F 8)
trNTK (i; x3) = cos
VaF(m,, Ø)
VeF(z,, 0)
Figure 3: Geometric intuition behind the trNTK. A NN function is evaluated two points creating surfaces F F 0) and F(zy;; (0) These surfaces are shown with a tangent hyper plane at the same point (0) in parameter space coinciding with the end of training. The Jacobian vector defines the tangent hyperplane's ' orientation in parameter space. The INNTK is kernel whose (i, (etj) element the cosine angle between averaged Jacobian vectors. The more similar the local geometry between 2; and zy local to in parameter-space the higher the value of trNTK(Zz..;))
could prevent training data with large magnitude Jacobian vectors from dominating the kernel, and Hanawa et al. (2021) notes a cosine-similarity kernel achieves best performance among alternative kernels data attribution task. Key motivators for study included that the cosine normalized values are intuitive geometrically, and that standard practice to ensure feature matrices such in small range (such as [-1,1]) for machine learning.
C GEOMETRIC INTUITION BEHIND NEURAL TANGENT KERNELS
In figure 3 we provide pictorial representation of the geometric interpretation behind the trNTK.
D ADDITIONAL DETAILS REGARDING THE TRACE NEURAL TANGENT KERNEL
In this Appendix provide an expanded definition of the trNTK that highlights how the trNTK is actually computed from a series individual contributions from each learnable tensor. This layerwise decomposition has been pointed out in previous work (Novak et al., 2022). Let O₂ be the parameter vector consisting of only the parameters from the I-th layer. Let the number of parameters in the 1-th layer be pt. Jacobian is vector of first-order partial derivatives of the NN with respect the parameters. We will specify each Jacobian through the e-th scalar function (equivalently, c-th output neuron) for the parameters in the 1-th layer as:
gf (ax.) (F(;;;;) 0)
20; E IXP
Note that we have intentionally broken our notation for the vector by using the Gothic capital g for the Jacobian vector. We do this to avoid confusion with the lowercase j used as an index. Let (m;;) be the concatenation of all such gf (x:;) for all e (1,2,...,CC
g:(;))
[gt (æ:), 87 (w:), of (xx)) RIXCR
(10)
Let Jx) (xX)r be the matrix formed from column vectors (x) over each training data point a, where E NN:
( [g (x))
[xN) E CPPN
(11)
17

---

Published as a conference paper at ICLR 2024
Let the 1-th unnormalized gpeudo-Neurs Tangent Kernel, or trNTK, be the Gram matrix formed from the products of ()) matrices:
TNT,, (G) E NXN
(12)
As Gram matrix, tNNTK/ is symmetric and positive semi-definite. Let trNTK RNN be the matrix formed from summing the contributions from all lrNTK] Consider
trNTK Σ INTTR RRNN
(13)
Here, trNTK itself is symmetric, as the sum of symmetric matrices is symmetric. Finally, we must apply the normalization. Let the matrix B be defined as the element-wise product the trNTK with the identity:
B = 1 O trNTK
(14)
Then the normalized trNTK can be computed form the unnormalized trNTK by the following relationship:
trNTK = = trNTK B
(15)
The relationship between the full neural tangent kernel and the trNTK is described in Appendix E.
E RELATIONSHIP TO THE EMPIRICAL NTK
To calculate the full eNTK, first find the e-th class Jacobian vector, g", with respect to 0 backwards through the network for each æ; in the data matrix X. Explicitly, the c-th logit's Jacobian i-th column-vector corresponds datapoint æ; and is defined:
0 (F 0)
gx
(16)
00
From which we can define the Jacobian matrix as:
G (g())))) (x1),, (((1),... (xN))
(17)
The eNTK is the block-matrix whose (k.j)-th block, where both j. = (1.2...., C}, is the linear kernel formed between the Jacobians of the (k,))-th logits:
NTK (G") (G*).
(18)
The NTK therefore a matrix E RCNXCN The relationship between the unnormalized (rNTK and the NTK is simply
INNTK Σ NTTK....
(19)
We chose to study the trNTK instead of the NTK for simplicity, computational efficiency, and reduced memory footprint. Follow on work could attempt to use the entire NTK to form the surrogate models. We were additionally motivated by the approach taken in Chen et al. (2022) and Chen et al. (2022), and we refer the reader Mohamadi Sutherland (2022) for a deeper discussion of the qualities of similar approximations.
F NOTES ON THE PROJECTED VARIANTS OF THE NTK
For TRAK, Park et al. (2023) utilized the Johnson Lindenstrauss lemma (Johnson & Lindenstrauss, 1984) to justify the use of the projection matrix K. The Johnson Lindenstrauss lemma bounds the error between any two vectors and the same two vectors projected under projection matrix P. The lemma can be used to show a bound on the cosine similarity between two vectors and two projected vectors (Lin et al., 2019). However, this bound relates the probability of the residual for all vectors being less than some small e. From an applied perspective we might care only that the residuals of cosine similarity are small with high probability. We empirically observe that the
18

---

Published as a conference paper at ICLR 2024
ResNet18 Residuals between trNTK proj-trNTK
Bert-basc Residuals between trNTK and proj-trNTK
0.8
68-
Curmulative
0.8
Cumullaiive
0.6
Abs(Rediu [Ing]
Abs/Residual) (log)
(a)
(h)
Figure 4: tNNTK and proj-trNTK cosine-similarity residuals fall exponentially. For both ResNet18 Eq. 4a and Bert-base Eq. 4b we plot the cumulative histogram of residuals between the lNNTK and proK. The orange line an exponential function with k=10240. The orange line is fit "by eye" rather than some best-fit, the objective being to reference the exponential shape of the residual distribution.
absolute residuals of of the trace-NTK and proj-trNTK fall away as exp(-x: (3), where is the decay rate. In Figures 4a and 4b, show the residuals for our ResNet18 and Bert-base experiments, with an overlaid exponential decay model for reference. We are unaware of a formal proof that would dictate the form of the distribution of residuals, but the exploration of the projected-variany as close approximations for the original kernels with large enough K. Intuitively, we expect that there a trade-off between size of the dataset, size the model, and K.
these plots empirically justify
G FORMAL DEFINITION OF EVALUATION METRICS
In this Appendix we restate all the metrics used throughout this study. Kendall-7 rank correlation
For a paired sequence S, = {a1 b2) ........ (ax, by)} a pair (a,, by) and ba with i & are concordant if either both a; > aj and b; > by a; aj and b; by. Otherwise, the pair is discordant. We count the total number of concordant, NC, number of discordant pairs, ND. Then, Tk is defined
-NC-ND T(S,) NC + ND
Test accuracy differential (TAD) We track the test accuracy differential, or TAD, given by the difference between the kGLM and NN's test accuracy,
TAD TestAACk Test ACONN,
(20)
to demonstrate that kGLM have similar performance to the underlying NN. A value of 0 preferred.
Misclassification Coincidence Rate we compute the intersection of misclassifications between each kGLM model and the NN where both the NN and kGLM predict the same class, over the union of all misclassifications of either the NN kGLM models decimal. value of 1.0 indicates that in all cases where the NN is wrong, the kGLM is also wrong and predicts the same class as the NN.
(V 0) * 2,) n (KGLM(#.)) + 2 (a; 0) kGLM (x:))
RMiss
(21)
Iff 0) F 2 (KGLM(2.) + 21
19

---

Published as a conference paper at ICLR 2024
Precision and Recall To evaluate whether attributions are performant discriminating between perturbed and unperturbed test datapoints, we use precision as a measure of how valid the flags given by our attribution model are, and recall as measure of how complete these attributions were at identifying poisoned test data. A perfect model would have both precision and recall = 1. Precision and recall are defined:
TP
Precision
(TP + FP) TP
Recall
(TP (TP+FN) FN)
where TP is the true positive rate, FP is the false positive rate, and FN is the false negative rate.
Coefficient of Determination R² The coefficient of determination is used goodness-of-fi to assess the viability of our linearization of the NN (described below in Appendix L). It is possible to have high тk but small 12 if the choice of invertible mapping function is wrong if the fit of said function does not converge. Such cases can be inspected visually to determine the relationship between the logits.
For a sequence of observations (in the context of this paper, the natural logarithm of probability of the correct class for the NN and kGLM) Sqa= {(T1, ...,..., ... (1 ]. let the sample average of the
y/ observations be y == Σ. N y:- Then let the total sum of squares be SS = ΣS" (y+ g)? and
the sum of squared residuals be SS,es defined by
Σi (y: x.)2. Then let the goodness-of-fi 12 function be
SSres
(8R=) =1
SSeor
H ALTERNATIVE MEASURES OF CORRELATION
To justify our choice of Kendall-T as the measure of correlation, we compare to other choices of correlation, the Pearson-R and Spearman-pe We wrote that Pearson-R is unsuitable as measure of correlation because it conflates the covariance between models with the correlation between models. Consider the thought experiment to see this is true: FA and FB are independent models, both of which for any input X; are correct models are correct, the output is Y; + N o), with Y; {0, 1}. and when incorrect are |Y, 1| + N (0, r). Furthermore, assume an even class distribution, and that a < The result the paired set of evaluations from FA and FB is a point cloud with most points centered at 0 and 1, in figure 5. Because both models are correct with high probability, the probability that FB 's output is centered at zero is high if F 's output is centered at zero; likewise, the probability that F3 's output is centered at one is high if Fx's 's output is centered at one. These point clouds act as anchor points that sway the Pearson-R correlation to values of even though there is no real coupling between the models. To the point: because the kGLM and NN are highly performant models, we must distinguish from correlation from this fact and their independence, from true kGLM dependence on the NN itself. While rank- bankd correlations are sensitive to this phenomena, the expected value of Kendall-7 would only be 0.5 in this experiment.
rate PA and PB. with PA and PB nearly one. When the
To complete the thought experiment, consider if FB is dependent on FA: FB(X,) FA(X,) + (O a) (visualized right panel of figure 5). In the limit - O, we would like to choose the correlation measure that most slowly converges to 1. This is because we want to maximize the interval over which out faithfulness measure discriminates between models. We complete the numerical experiment and visualize the result in figure 6, showing the Kendall-r converges to value one slowest.
I USE THE NN OUTPUTS FOR THE KGLM TARGETS
To evaluate methodology of training the kGLM using the ground truth labels, we compare to training using the neural network model output as the label for the kGLM. This reasonable choice, frameing the surrogate model's learning teacher-student model. Contemporaneous work investigates kernel based data attribution using this method, (Tsai et al., 2023). We report the result of this experiment in table 5. Compared methodology (table 2), We generally training with
20

---

Published as a conference paper at ICLR 2024
Independent Model
Dependent Model
P( (FAMM
Fx (x(X) +N(0,o)
1.50 Pearson-R: 0.98
14
Kendall-Tau Correlation:
Pearson-R: 0.96
Spearman Rho: 0.92
68
Tendall-Tau: 0.48
1.00
Spearman-Rha
(A(X)
0.25
-0.25
06 10 -0.2 0.2
12
F B (X,)
F (X.)
Figure 5: Distinguishing between independence but high covariance from true dependence Left: Plotting the confidence-confidens scatter plot using two independent models which both have high probability of correct classification results in a point cloud with high density (0.0) and (1,1). These point clouds act as anchors that force the Pearson correlation measure to be nearly 1, but because there is underlying structure the aank-corellttion dependent case, which is an ideal form of our surrogate model definition. We see that the anchor point structure still present forcing the Pearson be nearly and now the rank correlation has grown to 0.75, Our main point is that Kendall-7 is not so affected by the issue of separating covariance from dependence as Pearson.
only 0.5. Right: We visualize the
Rate of Convergence For Each Correlation Measure
0.9
Correlation Value
0.7
0.5
0.4
Pearson
Kendall
0.3
Spearman
10-1
10-3
10-5
10-7
10
Error
Figure 6: Comparison of Rate of Convergence of Correlation Measures. Using the dependent models thought experiment, we reduce the a, or error (x-axis), and plot the correlation value. The Kendall-r is the slowest converge values of 1, meaning its the most sensitive measure of correlation over the interval studied.
21

---

Published as a conference paper at ICLR 2024
Table 5: Using the NN outputs as labels for training kGLM. We report our modified experiment results for ResNet18 8 and Bert-base. For the Bert-base model where multiple models are trained, we report the leading digit of the standard error of the mean as a parenthetical.
Exp Name
Metric
K
trNTK
trNTK
proj-trNTK
proj-pNTK
Em
CK
ResNet18
TK
0.44166
*
0.4443
0.6707
0,47159
0.62874 -0.02
TAD (%)
-0.66
*
-0.68
-0.01
-0.18
Bert-base
TK
0.50(3)
0.31(2)
0.50(3)
0.43(4)
0.40(4)
0.38(4)
TAD (%)
0(2)
0.1(2)
0(2)
-0.2(1)
-0.9(2)
-0.3(2)
the original ground truth labels increases Kendall-r. We speculate this is because the optimization problem are shared between the kGLM and the NN training if the original ground truth labels are utilized.
J ADVERSARIAL ATTACKS
We trained NN models on the MNIST dataset. In order to avoid combinatorial considerations, the classifier was trained on just two classes- we used 7's and I's because these digits look similar. Subse- quently, we extracted the NTKs and used these kernels to train SVMs. To attack both types of models, considered foo perturbations, computed using the projective gradient descent algorithm (Madry et al., 2019) with steps (PGD-7). Our experiments leverage PyTorch'ss 's auto-differentiation engine to compute second-order derivatives effectively attack the SVMs. In contrast, prior work (Tsilivis Kempe, 2023) derived an optimal one-step attack for the NTK at the limit and and used this approxi- mation compute adversarial examples. To compare neural nets with kernel regression, (Tsilivis Kempe, 2023) compute the cosine similarity between the FGSM adversarial attack and the optimal I-step attack for kernel machine, computed analytically by taking the limit for an infinitely wide neural net. Their results show (Figures 3 and 7 of (Tsilivis & Kempe, 2023)) that throughout training, cosine similarity of this optimal 1-step attack and the empirical attack on the neural net decreases. This observation suggests that in practice, the NTK limit is not a good surrogate model for neural net under an adversarial attack. Our plots (Figure 7) confirm this observation as SVMs are much more vulnerable to attacks that the associated neural nets. To better compare with prior work, trained our SVMs using NTKs rather than pNTKs.
In considering security of neural nets, attacks are categorized either white-box or black-box. White- box attacks assume that the adversary has access to all the weights of a neural net while black box attacks do not assume that an adversary has this information. common strategy for creating a black box attack is training an independent NN and then using perturbations calculated from attacking this new NN to attack the model in question. Such attacks are called transfer attacks; see (Papernot et al., 2016b;a) for examples of successful black-box and transfer attacks.
In line with this framework, we our models against two white-box attacks and a black box attack. First, we test neural nets and SVMs by directly attacking the models. Next, to better understand the similarities between a neural net and the associated SVM, we evaluate the SVM on attacks generated from the associated neural net and the neural net on attacks generated from the associated SVM. For the black box attacks, we test: 1) neural nets on adversarial examples generated from independently trained neural nets, 2) SVMs adversarial examples from SVMs trained with an NTK from an independently trained neural net, 3) Neural nets on adversarial examples from SVMs trained with NTK from and independently trained neural net, 4) SVMs adversarial examples from independently trained neural nets.
The error bars for all three figures are 10 trials. For the black box figure, each model tested against 9 other independently trained models; the plotted quantities are the average of all these black box attacks.
22

---

Published as a conference paper at ICLR 2024
Eror Uoder Thaathrr ritsct Neb end Amookkeed CVPe
yorr 205 cin
02
333
251
(a)
Figure 7: Error Under Adversarial Attacks: Eq. 7a White-box attack: Attacking a neural net and the associated NTK SVM directly, Eq. 7b White-box attack: Attacking neural net using perturbed examples for the associated SVM and attacking an NTK SVM by using perturbed examples for the associated neural net. Eq. 7c Black-box attack: Attacking neural nets and SVMs using perturbed examples from independently trained SVMs and neural nets. This demonstrates limitation of our surrogate model method: we find that the SVM's performance does not scale the same the NN's performance with increasing perturbation radius, across multiple kinds of attack.
J.1 ADVERSARIAL EXPERIMENT DETAILS
When performing PGD to find adversarial examples to our models, we did not restrict pixel values of the attacked images to the interval [0, 1]. See (Madry et al., 2019) for more information on using the PGD algorithm in adversarial context. Notice that the PGD algorithm, attacking the SVM trained with the NTK involves computing second derivatives of the neural net. Due to this consideration, using ReLUs as neurons in this experiment was impractical- the second derivative of piecewise linear function at any point is either zero or non-existent. Hence the nets are constructed from sigmoid neurons.
The model architecture was 3 fully connected layers of 100 neurons. The models were trained for 100 epochs and with learning rate 10 with AdamW optimizer and minibatches of size 64 in Py Torch on the cross-entropy loss. The error bars in both Figures 7a and 7b figures are computed from the standard deviation calculated from 10 independent experimental trials set with different random
seeds.
The SVMs were trained using sklearn's SVM package.
K ADDITIONAL EXPERIMENTAL DETAILS
In this Appendix we detail the specific choice of architecture, hyperparameters, and training times for each experiment.
K. DATASETS
Our experiments utilize common benchmark datasets: MNIST, FMNIST, CIFARIO, and COLA. we will quickly introduce each in turn. The Modified National Institute of Science Technology (MNIST) (Lecun et al., 1998) handwritten digit database is a grey-scale image classification dataset comprised of handwritenn numerical digits and label pairs created from combining the National Institute of Science and Technology special datasets 1 and 3. MNIST has over 50,000 training and (FMNIST) (Xiao al., 2017) is another image
10,000 testing data-label pairs. "Fashion"-MNNI classification dataset that was specifically introduced to serve drop in replacement to MNIST. was created by reducing images from an online European fashion catalogue to the same 28x28 pixel resolution MNIST and to grey-scale. FMNIST has 10 classes of different kinds of garments, with 7,000 examples of each garment, split into 60,000 training and 10,000 test data. Canadian Institute for Advanced Research- 10 (CIFARIO 10) is 10-class supervised image classification dataset comprised of 32x32 pixel 3-color channel hand-labeled subset the TinyImages dataset (Torralba al., 2008) featuring everyday objects and animals. CIFAR1O is composed of 50,000 training and
23

---

Published as a conference paper at ICLR 2024
10,000 test data, evenly split among the 10 classes. Finally, the Corpus of Linguistic Acceptability
(CoLA) (Warstadt et al., 2018) 8) is a dataset composed of sentences and labels corresponding to the grammatical correctness of the sentence compiled from texts on grammar. CoLA includes 9515 training sentences and 1049 test sentences. CoLA was included in the original GLUE (Wang et al., 2019) set of benchmarks for NLP, which became the de-facto benchmark set of tasks for general language modeling.
K.2 EXPERIMENTS
K.2.1 100 FULLY CONNECTED MNIST2 MODELS
Using the first two classes of MNIST, (MNIST2), we train 100 independent 4-layer fully connected NNs using PyTorch. The network layer widths were [100,100, 100.1], and each had Rectified Linear Unit (ReI.U) activation function, except for the final layer. We define all of our networks to terminate without final activation for the sake of calculating function to map the activations onto a value we interpret as probability of class 1. As is typical in NTK parameterization, we divided each activation map by the square root of the preceding layer's width. The input space of MNIST was modeled as 784-size feature vector that we preprocessed to have values between 0 and 1 by dividing by the maximum pixel value of 255. For simplicity, we down sampled the test dataset to share an equal amount of class O and class I examples, giving 980 examples of each class. We initialized the layers using the normal distribution.
trNTK; however,
the sigmoid link
Each model instance had the same hyperparameters, architecture, and approximate training time. The only differences were the initialization given by seed and the stochastic sequence of datapoints from standard PyTorch data-loader. We trained our model to minimize the binary cross entropy loss between the true labels and the prediction function. We chose to optimize our model using stochastic gradient descent with no momentum and static learning rate 1e-3. Training 100 models sequentially takes approximately 8 hours on a single A100 GPU.
K.2.2 100 CNN MNIST2, FMNIST2, AND CIFAR2 MODELS
We use the same CNN architecture for our 100 MNIST2, FMNIST2, and CIFAR2 models; for brevity, we will describe the model once. Each model is a 12-layer NN where the first 9 layers are a sequence of 2D convolutional layers and 2D Batch Normalization layers. The final 3 layers are fully connected. The first nine layers are split into three sections operating equal feature map sizes (achieved with padding). The first layer in each section is a convolutional layer with kernel size 3 and padding size followed by batch normalization layer, followed by second convolutional layer with kernel size 3 and padding size but with stride 2 to reduce the feature map in half. The number of filters steadily increases throughout each convolutional layer as 16,24,44,4448864) After the convolutional layers, a flattening operation reduces the image dimensions into a 1-dimensional vector. Next, fully connected layers of widths [256, 256, 1] are applied. After each convolutional layer and fully connected layer we apply the rectified linear unit (RcLU) activation. Training times for 100 models on MNIST2, CIFAR2, and FMNIST 2 were 15 hours (100 epochs), hours (100 epochs), and 48 hours (200 epochs), respectfully, on single A100 GPU. The difference in times can be explained by the different choices of batch size and number of epochs, which were 4, 64, and 4, respectfully. We chose these batch sizes, and all other hyperparameters, by hand after small search that stopped after achieving comparable performance the many examples of models available online for these benchmark tasks. One oddity we believe worth mentioning is that subtract the initial model's final activation vector for the CIFAR2 model, after observing that this lead to a modest improvement. Initial LRs were 1e-3 for each model, but the optimizers were chosen as SGD, Adam, and Adam for MNIST2, CIFAR2, and FMNIST2, respectfully.
K.2.3 COLA BERT-BASE MODELS
To train the BERT-base models, we downloaded pre-trained weights available on the HuggingFace repository for BERT-base no capitalization. We then replaced the last layer with two-neuron output fully connected layer using HuggingFace's 's API for classification tasks. We set different seeds for each model instance, which sets the random initialization for the final layer. We train our model on the COLA dataset for binary classification of sentence grammatical correctness. We train our model using the the AdamW optimizer (Loshchilov & Hutter, 2017) 7) with initial learning rate n
24

---

Published as a conference paper at ICLR 2024
2e-5. We allow every layer to update. Training is done over 10 epochs after which the training accuracy is seen to exceed 99% performance on each model. Training takes a few minutes on an A100 GPU. Calculating the NTK is achieved by splitting the parameter vector into each learnable tensor's 's contribution, then parallelizing across each tensor. Fach tensor's trNTK computation time depends upon the tensor's size. In total the computation takes 1200 GPU hours, single A100 GPUs.
K.2.4 LARGE COMPUTER VISION MODELS
We downloaded 3 pre-trained model weights files from an independent online repository (Phan, 2021). ResNet18 and Resnet34 architectures can be found described in He et al. (2015b), and MobileNetV2 can be found described in Sandler et al. (2018). Each model's INNTK was computed by parallelizing the trNTK computation across each learnable tensor. the computation time varies function of the learnable tensor's s size, but the total time to compute each of ResNet18, ResNet34, and MobilenetV2 was 389, 1371, and 539 GPU hours, respectfully, on single A100 GPUs.
K.2.5 CNN FOR POISONED DATA EXPERIMENT
We trained 22 layer CNN with architecture described in the repository alongside Shan al. (2022) and restated here. The architecture's ' first 15 layers are composed of a 5 layer repeating sequence of convolution, batch normalization, convolution, batch normalization, and max pooling. After the 15th layer, we flatten the feature vector, apply another max pooling operation, and then apply dropout with probability 0.2. The next parameterized layers consist of the sequence fully connected layer, batch normalization, fully connected layer, batch normalization and final fully connected layer. A ReL.U activation is applied between each hidden layer. The repository of Shan al. (2022) generates BadNet cifar 10 images as data artifact. We translate their architecture to Py PyTorch and train our own model. The model was trained to minimize the cross entropy loss on the poisoned image dataset with stochastic gradient descent with an initial learning rate of le-2. The total number of parameters for this model is 820394. We take a different approach to calculate the trNTK of this model and choose not to parallelize the computation across each learnable tensor. The total trNTK calculation completed in 8 hours on a single A100 GPU.
K.3 COMPUTING EMBEDDING KERNELS
To compute an embedding kernel we must make choice of what constitutes a "layer". This has some slight nuance, as for example, the most complete Embedding kemel would be computed after every modification the feature space. In a typical fully connected layer there would be 2-3 modifications that occur: 1) the weight matrix multiplication; 2) the bias vector addition; 3) the activation function. Typically, we would take each of these modifications as part of the same fully connected layer and sample activation for the Embedding following all three. Next, consider residual blocks and similar non-feed forward or branching architectures. We must make a choice of where to sample in the branch that may have impact how the final Embedding kernel behaves. In this Appendix, we list our choice of layers to sample the activation for each experiment. We chose to balance completeness and computation time. Follow on work could investigate how these choices affect the final embedding kernel.
K.3.1 RESNET18
Table 6 shows where the components of the embedding kernel were calculated.
K.3.2
BERT-BASE
The layers used to calculate Bert-base embedding kernel are shown in Table 7.
K.3.3 POISONED CNN
Table 8 shows after which modules the embedding kernel was calculated for the data poisoning CNN.
25

---

Published as a conference paper at ICLR 2024
Table 6: Embedding Layers ResNet18 with æ E (1,2,3,4)
Layername
conv1
bnl
maxpool
layer.z:
layer.r .
layer.z.0.conv
layer.z.0.bnl
layer.c.0.conv2
layer.x.0.bn2
layer.z.1
layer.x.I.conv
layer.z.1.bbn
layer.x.1.com
layer.x.1.br.
av avgpool
fc
Table Bert-base Layers with Embedding Kernel calculation, æ E 3,,,,, 10,
Layername
bert.embeddings
bert.embeddings.c word cereddddmeddgss
www.eeebbbiiggspssiiiiiiii embeddings
Ber..embeddingsttoeen type_typpeepp ttt__ttpe_________ee_ embeddings
bert. www.eeedeo..eeeee.e.e
pert.encoder.layent enncoder.laeex..tteeniii.
bert. www.eeodeee.lleex.tteeeiiioooooo self
bert. www.cooee..aaacc...tttttiii..... queru
bert. www.coodr..ayecc..ttetni....... wel.ff.oosseeoo.o......o...... key
bert. bert. wwr..nncderrlaeee..tteenoooo outpuu
value
oorr.eencoeer.lyeeessattttttooon http::/attttetttoooouuuu.uu.eee output.dense
berr.eencoeerraaayeeriiimmmemee
bert..encoleclayyrra.nimmmaaaat dense
fn
bert. port.encoderrllyyex..uuuuuuu
bert. www.eecoeerrlleee..uuuuu.....ede
bert,poole
bert, pert.pooler.denss.
classifier
26

---

Published as a conference paper at ICLR 2024
Table 8: Embedding Layers Poisoned CNN
Layername
conv2d
aalch_normalization
conv2d_1
batch_
Baacch_ommalzation,
max maa__maaplllll
_aa__pooing2d
conv2d_2
batch_nooe aatch_normalization, 2
conv2d_3
batch ww..mmmmzzaaiinn.
max_ _ax__polligg2__
conv2d_4 4
batch normalization 4
conv2d_5
batch_normalization.
max _au__pooing2d 2
max_poolingld
dense
batch Baaichnormalization__
dense_1
natch_normalization_]
dense_2
27

---

Published as a conference paper at ICLR 2024
L METHODOLOGY FOR LINEARIZING NNS VIA KGLMS
We describe the procedure to achieve a linearization of the NN via a kGLM surrogate model. First, we fit supervised NN using standard techniques. Next, we compute the trNTK. This kernel acts as the feature space of the kGLM. we fit the kGLM (Pedregosa et al., 2011) (sklearn. linear model. _anear_model___ ing data as the NN is trained upon. The dimensionality of the output vector from the kGLM will be the same as the NN, and equal the number of classes.
using the kernels computed from the same train-
We are concerned with demonstrating that after applying an invertible mapping function Φ, the NN decision function approximately equal to the kGLM decision function. Because the decision function is typically only a function of the probabilities of each class, this objective can be achieved by showing the following approximation holds:
oF 0)) ((GLM((z))
Across many models and datasets we generally observed that the trend between the NN activation and the kGLM activation was "S-shaped", or else was already linear. The analytic class function that are "S "S-shaped" are sometimes called sigmoid functions. The following three functions used to map the kGLM to the NN.
4 (x) vr + μi
exp(
2 (x(x)= 1 exp(
+ p
x-a
8
4 (x()) arctan
V
--
+ M.
π
23
②b is linear re-scaling. Both 2 and a are sigmoid-shaper functions that map (-o0, oo) to (0,1). All choices of & are invertible. We made these choices for şi after observing the relationship between the kGLM and the NN. We & functions with an iterative optimizer (Virtanen et al., 2020) on the 1.2 loss between F X ()cc and ((GGLM(XX) classification (we describe changes necessary for multi-classification below). Fits are completed over a partition of half the test dataset and evaluated on the remaining half. The linearizations are visualized in Appendix L.I.
where e is chosen to be class in the case of binary
To visualize we use scale using the logit function. We define the logit function as the scalar-valued function that acts on the softmax probability p E (0, of a single class and outputs "logit":
logitfn(x) log
Using the logit creates better visualization of the probabilities themselves by smoothing out the distribution of values across the visualized axes. As final implementation note, we observed some numerical instability due to values being so close to p=1 that errors occur in re- re-mapping back into logits. We choose to mask out these values from our fit, our visualization, and the R² metric.
L.I VISUALIZATIONS OF POINT-FOR-PONT LINEAR REALIZATIONS FOR EACH EXPERIMENT
What follows is the visualization of the linearizations of the NN logits with respect to the kGLM logits. A perfect fit would line up with parity, shown diagonal dashed line in each plot. The coefficient of determination or R² is shown in the text for each plot. Seeds are shown in each panel's title. For the classification models ResNet18, ResNet34, and MobileNetV2. we flatten out the regressed vector and choose to plot the distribution as a KDE estimate of the correct class and incorrect classes instead of scatter plot, due the large number of points.
28

---

Published as a conference paper at ICLR 2024
MNIST2 MLP
seed
T-
18
-30
-80
422
Ri-0
T-0.57
Tw
T-0.69
-9
405
Seod
K7-0 94
Rt 93
87-0.09
T--975
2-0.75
r-0.21
-37
Tue Class=] True Clasz-0
Figure 8: MNIST2 MLP Linearization
29

---

Published as a conference paper at ICLR 2024
MNIST2 CNN
5000
1068
-0.0..0
R7-0
R7.0007
Tu0.00
67-00 AM
87-0.85
8/+0 96
T=0.01
Tw0.87
Tw0,65
-3-
Feed
60-w0 88
RTric 48
two
per0
Tue Class=] Tue Cless-0 Partty
Figure 9: MNIST2 CNN Linearization
30

---

Published as a conference paper at ICLR 2024
CIFAR2 CNN
Sood
500d
Bred
Afled 09
45-
-80
-00
Tw:375
1:0.11
Tw0.72
-50
-00
-49
Sood
Sand
67a0.63
R7v0.00
Teb
Tue Cluss=]
Tree Clasz-0
Figure 10: CIFAR2 CNN Linearization
31

---

Published as a conference paper at ICLR 2024
FMNIST2 CNN
Sooil
3ood
Alw0.05
R1+0 95
Tw0-41
T-0.85
TmC.87
-00
R/w00:0 97
Rf-n 48
R7-o 56
Two
[wC B0)
-12
1.5
-35
-20
Sced
#7-0.06
CHC.922
Tue Class=] Tre Class-O Partyy
Figure 11: FMNIST2 CNN Linearization
32

---

Published as a conference paper at ICLR 2024
M ATTRIBUTION VISUALIZATIONS EXPLAINED
In this Appendix, we describe the methodology used to visualize the attribution in greater detail. Our kGLM architecture gives each kernel value unique weight for each output neuron in the NN. For example, in our visualized CIFARIO 10 ResNet18 network, there are 10 learned weights for each kernel value. For each column, we plot line representing the average attribution given by training examples in that class. By design, multiplying the average attribution from each class by the number of points each class (in CIFAR this is uniform 5,000 for each class) and summing will result in the logit value of the kGLM in that class. We can therefore use these visualizations to quickly compare this:
N
N E
AA A(2, -GGLM(a)).
(22)
When visualizing, we choose to hide the attribution from each training datapoint to the activation of the class if the training datapoint's true label not e, by slightly modifying the attribution. Let Ne be the number of datapoints in class Let S. be the set of training datapoint indices with true label x = c. Let Sy be the set of training datapoint indices with true label & # c. Finally, assume the classes are balanced, as is the case for CIFARIO. Therefore, the length of the set S. N Ne- Then Aviz gives the attribution we visualize for i E Sc:
No
N. Σ JESy
B.
Aviz (x) x..)
We. x.) +
+
We, k(),
(23)
No
No
SES.
In other words, we have evenly distributed the attribution from training datapoints not in class c to the training datapoints in class Future work can investigate the human-Al interaction from different methods of visualization to determine the most informative visualization technique.
M.1 ADDITIONAL ATTRIBUTION VISUALIZATIONS
In the following subsection, we visualize additional examples of attribution from the R CIFARIO. In the first subsection, we visualize the mean value of attribution for each logit. In the second subsection, we focus on the correct logit and visualize the distribution of attribution explaining that logit's value. In the final subsection, we visualize the highest similar images from each kernel function.
M.1.1 MEAN VALUE OF ATTRIBUTION IN EACHI LOGIT
In the following plots, we visualize the mean attribution value (y-axis) from each class (different colors) to each logit (x-axis) evaluated on the test datapoint shown. We compare these values across each of the kernel functions. Because the number of datapoints each class are an equal 5000, one interpretation of these plots are that each mean value times 5000 summed over each contributing class is equivalent to the logit value in that column. Overall, we see that typically the training data representing the same class the logit have the highest attribution, expected. Because attribution can be negative, high similarity with a class can also remove total attribution a logit. We notice that in some fraction of misclassifications seemingly random choice of prediction is the result of high and off-setting similarity to two classes, that leave a third class with initially low attribution as having the highest mass, and therefore logit value.
M.1.2 VISUALIZING PREDICTED CLASS ATTRIBUTION MASS
Each figure shows the attribution distribution from each training data class for the predicted logit. Each sub-panel shows a different kernel function with the logit visualized labeled in the title. Each sub-panel is boxplot with dark line representing the mean contribution of attribution mass from that class. For our most consistent performing trNTK kernel function, the mean contribution is within the inner quartile range for every test image.
33

---

Published as a conference paper at ICLR 2024
Visualizing Distribution of Attributions
tYNTK truck
IrNTKO plane
pPojAtNIK in horse
1.0
plane
Tast corr=horse MN-horse
horce
0.0
-0.5
trock
PNTK horee
Em horse
C horse
An image of horse with human handler (right side) standing in front of trailer. The NN correctly classifies the image as a horse with close runner-up secondary classification as a truck, which we might consider excusable given the presence of both horse and the trailer in the image. The trNTK classifies as truck, with high activations for cat, dog, horse and truck. While cat is the second highest activation, the dog attribution in the cat logit subtracts from the total logit value.
Figure 12:
Visualizing Distribution of Attributions Across Kernels
projtrNTK
1.
plare
1.0
Test correcar NN-car
horas
<TD
Irokk
PNTK
Em
OK =
0.75
0.25
0.00
0.30
+0.75
Figure 13: An image of silver car is correctly classified as car. This perfect example of high confidence classification. In each logit (i.e., column), the orange tabs represent total attribution across the entire car class. In the column, this attribution adds to the logit; in the remaining columns a high attribution to car yields negative contribution to the logits (i.e., we trained on mutually exclusive classes, so the strong presence one class should remove confidence in another class).
34

---

Published as a conference paper at ICLR 2024
Visualizing Distribution of Attributions Across Kemelss
1.
deer
Test Image: corr-bird NN=bird
0.5
[rog
pro) phTK bird
Em brd
CK - bird
Figure 14: A bird against blue sky is correctly classified as bird. This is another example of strong correct classification, but unlike the previous example, the contributions the remaining logits are somewhat elevated. The negative contribution of bird in these classes ensures the logit remains small compared to the bird logit.
Visualizing Distribution of Attributions Across Kernels
1.25
1.5
1.0
dea*
Test Image: correship MNmplane
0.50
Trog
0.25
0.0
pr ..oo-pNTTC pNTK plane
Em plane
X plane
Figure 15: A small boat resting grass is incorrectly classified as plane by the NN. We show that many kernels also follow the network misclassification. which an important property for surrogate model. We see strong positive attribution plane that is me-mediated by any of the other classes.
35

---

Published as a conference paper at ICLR 2024
Visualizing Distribution of Attributions Across Kernels
LnNTK - plane
M/MT plane
PropttNTK - plane
1.25
plans
1.00
0.75
0.53
0.2
Test imagaa NN-plane
0.25
0.0
0.0
402
4..5
0.4
0.50
proj-pNTK
captionA bird resting on wire is misclassified plane by the NN. We again demonstrate that the kernels misclassify as the same incorrect class. We again see the reason why is strong positive attribution to plane.
Visualizing Distribution of Attributions Across Kemels
trNTK dog
trNIKO
projtrNTK dog
plane
Nr
doc
0.5
Test Image: cortwaod
Inick
Figure 16: dog in a pink background frame classified correctly as dog. Similar to the other high confidence classifications, this image shows how positive attribution in one logit acts subtract confidence in another image. It particularly highlights how high similarity to dog subtracts greatly from cat. This is important idea to explain some misclassification explore below.
36

---

Published as a conference paper at ICLR 2024
Visualizing Distribution of Attributions Across Kemels
trNTI IrNTKO projerniTK
Nir
cot
Test Image: cor--iid Mindee
proj-piTK deer
deef
Figure 17: A large bird, possibly an ostrich, is misclassified by the NN as deer. The kernels all have the misclassification, with high confidence in deer, bird, and
Visualizing Distribution of Attributions Across Kerness
frog
frog
projetNVT frog
plare
Test corrsship NWwtrog
horas
<Tpp
0.25
Irock
0.5
pTT = frog
Em fngg
CK mog
Figure 18: An inflatable boat is misclassified a frog by the NN. This is an interesting example, and we focus in on the trNTK. The cat attribution is actually the highest, but unlike previous examples, the attribution in the cat logit from the remaining classes subtracts enough away from the logit such that the highest remaining class is frog.
37

---

Published as a conference paper at ICLR 2024
Visualizing Distribution of Attributions Across Kernels
trTTK bidd
U/NK bird
proj-en[TK bid
1.25
plore
1.00
0.75
1.0
Gt
liss
dea
030
Test Image:
0.25
0.0
tnock
0.50
proj-pNKK
Figure 19: A car elevated on a platform against a white sky is misclassified as bird by the NN, The plane and bird class both highly activated across each kernel function.
Visualizing Distribution of Attributions Across Kernels
tiri
1.0
000
0.3
Test Image: corne dog NN=dog
-05
projeNIKK dog
Em dag
Figure 20: A dog with blurry text overhead is correctly classified as dog. Each kernel function, except the CK, follows the correct classification, and it can be explained by the high attribution the dog training data.
38

---

Published as a conference paper at ICLR 2024
Visualizing Distribution of Attributions Across Kernels
Test Image:
corr-plane MN-horse
[rog
horse
proj-pNTT PNTK hore
Em horse
O horse
Figure 21: A person sitting the nose of large plane faces the camera and is misclassified as horse. There a high positive attribution to both plane and horse.
Visualizing Distribution of Attributions Across Kernels
plane
Test image: cortcharse NNN
projghIKK hore
horse
LK
Figure 22: We visualize the entire distribution of attribution through each kernel's 's predicted class, (shown in sub-title). Focusing on the trNTK, the highest attributed images are truck. Compared to the previous section's plots, we now see structure of individual points from the other classes adding constructively to the Truck class logit, some examples from each class. The mean value of attribution from each class visualized by the colored bar.
39

---

Published as a conference paper at ICLR 2024
Visualizing Distribution of Attributions Across Kernels
trNTK
trNTKC
properr TK
image:
den
deg
hog
Tocc
S1P
thick
pmp-atiXK
0:00
00 00
Figure 23: We visualize the entire distribution of attribution through each kernel's predicted class, (shown in sub-title). Each kernel's predicted class is car. Focusing on the trNTK: we see the distribution of cars represented as box-plot quite high, establishing that many car examples contribute to classify this image correctly, rather than sparse few.
Visualizing Distribution Attributions Across Kernels
pro-ttNTK bird
MrNK - bird
UNTIO - bird
plarc
Gr
16d
cor-=bird
C3T
NN=bird
drg
Fro
21D
Je. prop-stl
Le-2
20.3
000
C20
Figure 24: We visualize the entire distribution of attribution through each kernel's predicted class, (shown sub-title). Focusing on the trNTK emphasizes how both that the distribution Bird is high compared the similarity of other classes, but that there are also some plane examples with high positive attribution. We might expect planes that are on blue-sky backgrounds to positively share features with birds. We delve into this example deeper the next section.
Visualizing Distribution of Attributions Across Kernels
HNTT
trl XOO
proj UNM plane
image:
C37
dee"
NN-plane
deg
hog
lsse
210
ti.kk
Daanc
Em = plore
CK plone
Figure 25: We visualize the entire distribution of attribution through each kernel's predicted class, (shown in sub-title). Focusing onto the trNTK, we see both car, ship, and truck have examples with high attribution supporting plane.
40

---

Published as a conference paper at ICLR 2024
Visualizing Distribution of Attributions Across Kernels
brTT
Ir XO
proy MMMK plane
image: corr-bird NNlpp
ded
Fog
S1p
thick
Em plare
CK
plone
04
100000
Figure 26: We visualize the entire distribution of attribution through each kernel's predicted class, (shown sub-title). Focusing onto the tNNTK, we see that there are additional bird and car examples positively attributing to the plane logit. We explore this misclassification in more detail in the section below.
Visualizing Distribution Attributions Across Kernels
triTK - doc
tYNIKK dog
Pa-y btNTK dog
cor-=cog
NN-doo
pryj pMTK
OK
0000
Figure 27: We visualize the entire distribution of attribution through each kernel's predicted class,
(shown sub-title). Focusing onto the trNTK, many dog examples have high attribution resulting in a clear and correct classification of dog.
Visualizing Distribution Attributions Across Kernels
proj-riTTS deer
DNTK deer
UNTOO deer
dlarc
1ird
corr-bidd
image:
cat
dec
NN-doer
0og
Prg
xTp
000
Figure 28:
We visualize the entire distribution of attribution through each kernel's predicted class, (shown in sub-title). Focusing onto the trNTK, there higher variance to the distributions of bird, dog, deer, and horse compared to plane, car, ship and truck. Despite these variances, the distributions of the living classes are still centred on zero, so that the net contribution from the other classes is slightly negative.
41

---

Published as a conference paper at ICLR 2024
Visualizinng Distribution of Attributions Across Kernels
trNTKD proj-trTTK frog
1ed
Test image: rar-=ship NN-ffrog
kkg
frg
Iruck
proy pNTK frog
Em frog
CK - frog
Figure 29; We visualize the entire distribution of attribution through each kernel's predicted class, (shown in sub-title). Focusing onto the trNTK, we see higher variance of the dog and frog classes compared the the remaining classes.
Visualizing Distribution Attributions Across Kernels
tNNF - bird
trV/1K - bird
pro;
Test cor--car
NN NN=bird
bird
brrd
CK
0:30
Figure 30: We visualize the entire distribution of attribution through each kernel's predicted class, (shown in sub-title). Focusing onto the trNTK, there singular examples from the frog and horse class that stand as outliers of positive contribution while the average contribution from these classes slightly negative (colored bar).
Visualizing Distribution of Attributions Across Kernels
spro
Test image: com--cogg
NN-dog
Pro
Jtpp track
proo oNTK dog
Em dog
CK
horse
Figure 31: We visualize the entire distribution of attribution through each kernel's predicted class, (shown in sub-title). Focusing onto the tNNTK, the attribution for dog, horse and deer have higher variance, though only dog has positive attribution.
42

---

Published as a conference paper at ICLR 2024
Visualizing Distribution Attributions Across Kernels
LNTK fros
heraw
praptrMTK
140
1ed
corr:plane NN-sorse
Fog
projgNTKK hore
b--
We visualize the entire distribution of attribution through each kernel's predicted class, (shown in sub-title). Focusing onto the trNTK, both plane and deer have example which contribute some positive attribution, but overall the effect of these classes are slightly negative to the classificaiton of horse.
Figure 32:
43

---

Published as a conference paper at ICLR 2024
M.2 TOP FIVE EXEMPLAR ATTRIBUTION VISUALIZATIONS
In the following plots, we visualize the qualitative differences between kernels by plotting the top five most similar training images for the same selection of images in the last Appendix. We emphasize that here, we are using the kernel function as a measure of similarity. Qualitatively, we observe that test data often share conceptual similarities with the most similar training data as evaluated by the trNTK, and that what is chosen as most similar often reveal something about the kernel itself. For example, the CK kernel is created from the final representation of the neural network. For NN trained until convergence this final representation should have all inner-class variance collapsed (Papyan et al., 2020). Therefore, we expect the CK to mostly show that the test image highly similar to ALL training images of the predicted class. Because the top most similar are not tied directly to our kernel surrogate model any explanations we generate from these visualizations are admittedly up to interpretation. Future work could endeavor evaluate different kernel surrogate models such as a K-Nearest Neighbors, which would tie these visualizations directly to the surrogate model's prediction. This would be way to recover explain-b--exampe with sparse number of exemplars. We also can visually confirm that the most of the highest similar images are shared between the trNTK and proj-trNTK, as expected. We notice that many proj-pNTIk examples seem shared with the CK, which we did not expect. In fact, much of the evidence presented throughout this work suggests that the proj-pNTK and CK share similar properties.
Visualizing Most Similar Across Kemels ResNet18
WNTK
tNTTK
praj-trNTK
Test com-horse Nishorse
Figure 33: A horse stands next to a human and in front of trailer or truck correctly classified as horse by the NN model. Many of the attributed animals are shown in profile, as the subject horse of the original image stands.
44

---

Published as a conference paper at ICLR 2024
Visualizing Most Similar Across Kemels ResNet18
WNKK
(aa--INTK projaNIK
Test Image com-eaa
Figure 34: A silver car is correctly classified by the NN. Many similar images (seemingly the same image with different crops) exist the training dataset.
Visualizing Most Similar Across Kernels ResNet18
INNK
tNTT>
praj trNT proj PNTK
Em
CK
Test Image com- bird MNmbird
A
Figure 35: A bird flies with wings spread in blue sky background and is correctly classified by the NN. Many of the bids attributed to by the evaluated kernels are also flying in similar manner in blue sky background.
Visualizing Most Similar Across Kemels ResNet18
UNT
proj-trNTK
proj
Em
CK
Test Image
corr-ship NN=plane
Figure 36: A boat resting grass is misclassified plane by the NN. The most similar attributions are varied, perhaps demonstrating a weakness this kind of visualization.
45

---

Published as a conference paper at ICLR 2024
Visualizing Most Similar Across Kernels ResNet18
*NNI
tnMm
praj-trMiX
Test Image come bird NN
Figure 37: A bird resting on wire that spans the image diagonally misclassified as a plane. Many of the highest attributed images from the trNTK and trNTK have similar diagonal quality, the underlying class of the subject of the image is much different than the true or classified class.
Visualizing Most Similar Across Kernels ResNet18 ora proj PNTK
Image
corr-dog
Figure 38: A small puppy in pink background looking out of the screen ("at the camera") is correctly classified dog. Many of the most similar images dogs that look out of the screen. The Embedding kernel seems very focused on the background pixel values, as many of the attributions pink centered.
46

---

Published as a conference paper at ICLR 2024
Visualizing Most Similar Across Kemels ResNet18
UNTK
aes--IENTK projeNIKK
Image
Figure 39: A large bird is misclassified as a deer. The attributed images are varied, perhaps demonstrating weakness in this kind of visualization.
Visualizing Similar
INM
prajorNNIK projppikK
E'm
CK
Test Image
corrsship NN-trog
Figure 40: A white inflatable boat is misclassified as a frog. The attributed images varied, perhaps demonstrating weakness in this kind of visualization.
Visualizing Most Similar Across Kernels ResNet18
*NTK
trNTe"
tTNT
Test
Mi-bird
Figure 41: A car resting on raised platform is misclassified bird. Many of the bird attributed by the trNTK and trNTK are large bird with rotund black bodies and stalky legs, perhaps suggesting a pathway for the misclassification.
47

---

Published as a conference paper at ICLR 2024
Visualizing Most Similar Across Kereess ResNet18
UNTK
trTTK"
pray trNTK proj PNTK
Em
CK
Test Image
cor--dog MN-dog
Figure 42: A dog with blurry text overhead is correctly classified as dog. The attributed images in the trNTK and \NNTK are mostly images of animals with white fur "looking left" mirroring the test image of the dog "looking right".
Visualizing Most Similar Across Kernels ResNet18
*NN
trNn<?
praj-trMIK
proj-pNIKK
Emn
Test Image cort=plane plane MN-horse
Figure 43: An image of person sitting on the nose of plane facing towards the camera. The NN misclassified this example as horse. The trNTK shows many example of people riding horses, mirroring the person "riding" the plane.
48

---

Published as a conference paper at ICLR 2024
N ADDITIONAL DATA POISONING ATTRIBUTION VISUALIZATIONS
In this Appendix, we provide additional visualization for the data poisoning experiment attributions. We show the same selection of images as the previous section for comparison. Because the NN classifies nearly every poisoned image as the targeted class deer, we expect that a good surrogate model would reflect this fact by attributing highly to poisoned examples. Because the model trained is a different architecture than the ResNet, it can be interesting to compare the top attributions to the previous Appendix section.
Visualizing Poisoned Similarity Across Kernels
Original, corrmhorse. NNmhorse
Palsoned,
projttrtiKKmjpeitc
targeted lebel=deer. NNzdeer
Posooned Similte mmmiterty toy
Figure 44
Visualizing Poisoned Similarity Across Kernels
Original, correcar,
Simlarty
Poisoned.
targeted label=deer. MN=deer
Postoned Smittrr
Figure 45
49

---

Published as a conference paper at ICLR 2024
Visualizing Poisoned Similarity Kernels
aMre pro.-uniiKpraepiiic
Orignnal
targeted labaledear, NNN-deer
Paisened,
proj
Figure 46
Visualizing Poisoned Similarity Across Kernels
UNTKond piTK
Original, com-ship, NN=ship
Poisoned,
proj NNTC
targeted label-deer, NN-ddeer
potomned
Figure 47
50

---

Published as a conference paper at ICLR 2024
Visualizing Poisoned Similarity Across Kernels
Original, corr-bird, NN-plane
simjjrriy
targeted label=dear, dear, NNN-deer
Poisoned,
proj
Figure 48
Visualizing Poisoned Similarity Across Kernels
bMine UNTKondi
Original, carredog, NN-dog
Smilany
Poisoned,
targeted label=deer, NNNedeer
Rotomed Symuu bortr
Figure 49
51

---

Published as a conference paper at ICLR 2024
Visualizing Poisoned Similarity Across Kernels
MMre
Original, corr-bird, NN-horse
targeted deal- NNN-deer
Poisoned,
proj
Noscona
Figure 50
Visualizing Poisoned Similarity Across Kernels
UNTKond pNTK
Original, correship.
smijary
Poisoned,
proj popp
targeted label=deer, NIN-deer
peromedd Semmun Aoae
Figure 51
52

---

Published as a conference paper at ICLR 2024
Visualizing Poisoned Similarity Kernels
orrrnal corr-car, MN-plane
Unporturbee samjaarn mamlarrt
targeted labeledear, NNNedeer
Poisened,
proj
Nessoee prrmect Aaa
Figure 52
Visualizing Poisoned Similarity Across Kernels
UNTKKrsi pNTK
Original, carredog, NN-dog
Poisoned,
targeted labeledeer, NN-ddeer
Figure 53
53

---

Published as a conference paper at ICLR 2024
Visualizing Poisaned Similarity Across Kernels
WMT
UNMTK pNTK
Original, carr-plane, NN-horse
Unperturtes
Poisoned,
proj UNTKondj PNTK
targeted labeledoer, NN-deer
paponeed smittony
Figure 54
54


Reviews:
[{"review_id": "Fvy0f7tIYb", "reviewer": "Reviewer_yHxi", "summary": "This paper investigates kernel-based surrogate models based on various approximations of the Neural Tangent Kernel (NTK) to provide explanations for deep neural networks. A primary contribution is showing that computationally-feasible approximations to the empirical NTK provide high-fidelity surrogate models, and that much cheaper projection-based approximations provide accurate estimates of the empirical NTK. Appealing to existing literature on explanation-by-example, the paper develops a simple score for data attribution. A synthetic data experiment shows that the proposed attribution score accurately attributes erroneous model predictions to poisoned data, giving some confidence that the proposed score is capturing some notion of similarity between data points.", "questions": "1. Have you considered using the kernel function directly to evaluate sample similarity? Why do you choose to include the weights from the kernel machine in all attribution scores?\n2. You fit the parameters $W, b$ on the ground-truth labels $z$ from the  training dataset. If the goal is to create a surrogate that emulates a neural network, why don’t you fit these parameters on cross-entropy loss with the class probabilities predicted by the NN? This would be consistent with your objective in Eq. 2.\n3. It is not totally clear how the Kendall-$\\tau$ statistic is computed. A couple of sentences would make this portion more reproducible. Do you take the matrix of all the logits produced on a test set ($N x C$) for the NN and for the surrogate model, flatten these two matrices into vectors, and compute the rank correlation? Is the rank correlation computed per-test-output and averaged over the test set?\n4. Why is trNTK initially introduced with cosine normalization and projNTK not? Don’t your experiments include cosine normalization for all kernels?\n5. Is there any theoretical statement you can add about the variance of the projection-based kernel estimates, e.g. based on JLS? The choice of 10240 dimensions seems arbitrary and model-dependent.\n6. Does the introduction of cosine normalization explain the experimental result “that the highest attributed images from the trNTK (and furthermore all evaluate kernel functions) have relatively small mass compared to the bulk contribution, suggesting that the properties of the bulk”?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper provides a potential solution to a very important problem, i.e. data attribution based on a trained neural network checkpoint.\n* The authors draw a very important distinction—which is somewhat obvious but not well-reflected in the literature—between difference in test accuracy (TAD) and correlation of model outputs. The addition of Kentall-$\\tau$ is important, and will hopefully shift the way future works evaluate the fidelity of surrogate models.\n* The paper provides a clear definition of proposed kernel approximations, with strong computational justification for the speedup e.g. of the trace approximation. Equation 2 clearly defines the working definition of a “high-fidelity model” and adds to the clarity of exposition.\n* The experiments include a sufficient diversity of alternative kernel estimates to demonstrate the value of the proposed projected-trace-NTK approach. The inclusion of uncertainty based on multiple runs is highly appreciated.\n* The inclusion of experiments on Bert-base significantly strengthens the paper, indicating that the method is not specific to the computer vision domain.\n* The discussion that explanations are not sparse is an important acknowledgement of the proposed data attribution method. In particular, the following statement is poignant: “presenting the top highest attribution training images without the context of the entire distribution of attribution is probably misleading.”\n* The paper’s title is very strong and well reflects the work’s primary contributions.", "weaknesses": "* The paper relies on previous work to establish credibility of attribution-based scores for neural network explanation. It doesn’t seem obvious that attribution is the same as similarity for learned kernel functions.\n* I find the second sentence in the abstract confusing. I expected this trend to have to do with using kernel-based models for data attribution rather than to “investigate a diverse set of neural network behavior”. Isn’t the goal of your paper exactly to apply kernel models to investigate network behavior?\n* The 3rd experiment on qualitative evaluation of attribution is weak. A user study is probably beyond the scope of this paper, and I believe the work is strong enough to stand without such a study. However, the paper would significantly benefit from some discussions about how these attributions could be better qualitatively evaluated in the future.\n* The claim about Peason correlation is not very well explained: “These point clouds serve as anchors that force the covariance, and therefore Pearson correlation, to be large. We require a measure that does not conflate the covariance with faithfulness.” Is the problem here that correlation is not computed between model logits for each test point?\n* The paper never explicitly defines the empirical NTK in its own notation. Could you add this prior to defining the trNTK or pNTK in order to allow an easier discussion of the approximations introduced?\n* The take-away from Figure 2 is not exactly clear. Is this just meant to show that attribution scores are not sparse?\n* Your Chen ICML’22 reference is duplicated. Did you intend to cite two different papers?\n* The notation is non standard. Most papers use $y$ not $z$ for ground-truth labels. I can see this causing some readers mild confusion.\n\nSmall issues:\n* In the last paragraph of the “Relationship to the Pseudo Neural Tangent Kernel” section, the reference to Eq 3 is to the wrong equation.\n* The inclusion of all four panes in Figure 1 seem a bit superfluous, I’m not sure what this is supposed to show that cannot be shown in a single figure.\n* Given that one of the 3 experiments claimed in the paper is a qualitative evaluation, I think it is important to include one of the data attribution figures from the supplemental material in the main text.\n* The main text cites Figure 4.a, which is in Supplemental Material.\n\nSmall typos:\n* 2nd paragraph of the Introduction: “Its well established” should be “it’s” (or better yet spell out “it is” to be less colloquial.\n* Undefined reference (?) at the bottom of page 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vL5v90bcV2", "reviewer": "Reviewer_zBfF", "summary": "The paper proposes new variants of eNTK and implements faster approximate versions as well, and then evaluates them on a few different tasks / visualizations.", "questions": "- Confused by what the rank correlation is measured over exactly. I understand it's measured between the truth model outputs and surrogate model outputs, but what is it varied over? Are you measuring across different inputs x?\n- A bit confused by what the message/takeaway of the box/distribution plots are. Can the authors elaborate?\n- It seems that the eNTK is only defined in Appendix D, so it's a bit hard to contextualize pNTK and trNTK when they are introduced", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Paper evaluates variants of eNTK in further depth compared to prior work\n- Paper is relatively well written, though missing some important details", "weaknesses": "- Content: Surfacing similar images is a not a meaningful evaluation of attribution. It is a good sanity check, but doesn't say anything about surrogacy. For example, finding similar images using CLIP similarity would also show similar images, though CLIP is in no way a \"surrogate\" to the model being studied\n\n- More broadly, I'd be more careful about making any claims of \"data attribution\" (which has a specific meaning as used in recent ML) as the paper does carry out any counterfactual evaluations.\n\n- Overall, the contributions seems somewhat marginal. Also, the fast approximate versions implemented primarily rely on prior work (Park et al.)'s implementation, so not sure there is much to claim as contribution there (since Park et al. also used it for faster approximations to eNTK).\n\n- Writing: is hard to follow at times and doesn't provide the relevant details (see Questions).\nOn one hand, the paper goes into more detail than necessary in defining rank correlation / R2, etc from scratch,\nand at the same time, doesn't actually provide details about what those measures are computed over exactly.\nIt's possible I missed it, but at least doesn't seem very clearly written based on my multiple attempts to parse this information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tey6gwytxC", "reviewer": "Reviewer_e28a", "summary": "This paper proposes to use the approximate empirical neural tangent kernel (eNTK) as a faithful surrogate model for neural networks. Focusing on NNs for classification, the authors define the trace neural tangent kernel (trNTK), which is the cosine similarity between the concatenated gradients of all logits with respect to the parameters for a trained NN. The trNTK is then plugged into a kernel general linear model (kGLM) to obtain a surrogate model for the NN, which can be used to attribute the prediction of the NN to the training data points. To evaluate the faithfulness of such surrogate models, the authors argue that a preferred way is to measure the rank correlation between the softmax probabilities of the surrogate model and the NN for the correct class. A random projection variant of the trNTK is also proposed to reduce the computational cost. The experiments show that the proposed surrogate model is generally more faithful than other kernel-based surrogate models.", "questions": "- Is the non-sparsity of the attribution a general phenomenon or just a property of trNTK? Is it a consequence of the statement \"It has been suggested that this normalization helps smooth out kernel mass over the entire training dataset\"?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written and easy to follow.\n- The proposed surrogate model is simple and easy to implement. trNTK performs consistently better than other neural kernels.\n- The rank correlation seems to be a better metric than existing alternatives for evaluating the faithfulness of surrogate models for classification NNs. It takes into account the global structure of the predictions.\n- Based on the proposed surrogate model and data attribution method, the authors observe that the attribution is NOT dominated by a few data points. This is an interesting observation and has practical implications.", "weaknesses": "- Only the rank correlation of the softmax probabilities for the **correct** class is considered. However, to be faithful enough, the surrogate model should also behave similarly to the NN for the **incorrect** classes. An important application of data attribution is to explain why a NN makes a wrong prediction. This is not considered in the paper.\n- Eq. (4) is confusing. In the denominator, the $\\cdot ^ {\\frac{1}{2}}$ is applied to the inner product. However, according to Appendix C and the definition of cosine similarity, the $\\cdot ^ {\\frac{1}{2}}$ should be applied to the sum, not the inner product.\n- The quality of Figure 2 could be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LuyctMK6pE", "reviewer": "Reviewer_dXvz", "summary": "This paper contributes to the growing literature of approximating NNs with simpler, more interpretable, models. A common approach is to approximate NNs with the empirical Neural Tangent Kernel (eNTK). However, computing the eNTK can be computationally unfeasible so simpler approximations have been proposed in the literature. The authors focus on this problem by studying the empirical properties of one such approximation, the trace NTK, and adapt random projection methods to make it more computationally attractive. Furthermore, the authors propose the Kendall rank correlation as a new measure to assess the faithfulness of the surrogate kernel method to the NN. \n\nThe main contribution of the paper is to show that the trace NTK and projected trace NTK can be used to generate faithful surrogate models of the underlying NNs. The authors show this through a variety of empirical exercises across different benchmark datasets (MNIST, CIFAR) and models (CNNs, ResNet18, BERT). They compare how good different NTK approximations are with respect to the underlying NN in terms of prediction error and rank correlation and find that trace NTK has good performance. Additionally, the authors compare the different models representations through a data attribution exercise and a poisoned data attribution exercise and identify which surrogate models perform better in each case.  Finally, the authors show the practicality of the projected NTK methods by analyzing computational complexity of each method. The most relevant finding is that the trace NTK and projected trace NTK  perform similarly in settings in which the projected trace NTK required an order of magnitude less computation time.", "questions": "Besides the questions raised in the weakness section regarding the key contributions. I also have some additional questions:\n\n* Which Chen et al. paper is the main reference for trace NTK, I was confused by the reference. \n* Is it true that when rank correlation <1 there exists not invertible mapping? \n* In the case in which the rank correlation is 1 is the invertible mapping unique? How does this result translate to the exercises and neural net behavior? Should we expect the same data attributions as the NN? Expanding on the implications of a good rank correlation vs test accuracy seems key to show the usefulness of the paper for researchers.\n* Given the data attribution with kernels theory in page 3, wouldn’t you be able to test directly whether a kGLM is an “ideal surrogate” (according to eq 2) by comparing across all data points the NN confidence in each class with the data attribution for each class? Is there a way beyond fit/correlation measures to more systematically compare how well the kernel performs in the data attribution exercise besides evaluating individual examples?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper tackles a common problem in an active area of research: that of finding computationally attractive NTK approximations to NNs. While the main proposed methods in the paper are drawn from the literature, the authors are original in adapting random projection methods to the trace NTK and pseudo NTK and considering the rank correlation as a sensible alternative to prediction error for assessing faithfulness.\n\nThe main strength of the paper lies in the extensive set of empirical evidence comparing various NTK approximations with different NNs architectures across tasks and datasets. Furthermore, the authors provide an extensive appendix with additional exercises and comparisons. Overall, researchers looking to use NTK approximations may find these exercises useful in choosing which method to use depending on their task and their computational constraints.", "weaknesses": "While the paper offers a wealth of empirical evidence for the methods it investigates, its central weakness is that it is unclear what the main findings and contributions are. The paper does a lot of things and it would benefit from more succinctly explaining what it is trying to achieve and how each exercise demonstrates it.  \n\n* The paper should be more clear about its relative contribution to the literature (and what its main contribution is). The paper gives confusing statements about what is new and what is taken from the literature. The authors state that the 3 main contributions are (1) new kernel functions for NTK approximation, (2) first to show that eNTK models are consistently correlated with NNs across experiments, (3) first to compare NNs decisions through NTKs through a data attribution and explain by example strategy. \n\n\n* For point (1) however the authors also state that the tr NTK was introduced in Chen et al. 2021 (end of page 2) and that the random projections approach is based on Park et al. 2023 (end of page 1).  Is the main contribution of the paper proposing a new NTK method or evaluating empirical exercises? \n\n* The authors consider different alternative NTKs to compare to the trNTK, but never compare in the main text the methods to the eNTK or the pNTK. Given that the motivation of the paper in the abstract, introduction etc is to approximate the eNTK it is odd that this not done in the main text of the paper. While computational constraints are important, maybe it could be done a simpler dataset (MNIST)? \n\n* For point (2) if using the rank correlation is new it should be clearly stated as a major contribution. The paper repeatedly expresses that other measures are flawed and while the authors give some reasons why, without a proper theoretical statement the authors should at least relate these notions more directly to the findings of the empirical examples. For example, what is a clear case in which using fit or pearson correlation would be misleading in the sense that two NTK models give you very different attributions despite having the same fit to the NN, but rank correlation is not misleading.\n\n* For point (3) the paper should explain more carefully why these are carried out. If the goal is to assess how good a NTK approximation is by considering whether it performs similarly to the NN in a data attribution task then this should be the focus of the results. It seems that the authors do these exercises in the appendix, but in the main write up they just give an instance of this and its unclear how much we can learn from it. If well addressed I would be inclined to raise my score.\n\n* The paper could benefit from better exposition and more clear presentation. For example, the choice of what is defined in the main text vs appendix and when it is defined is sometimes odd. The eNTK, while being referenced to extensively, is never properly defined in the main text. The, trNTK0 is introduced in Additional Kernel Functions after the trNTK without motivation, despite featuring prominently in the appendix when the different methods are compared.\n\n* The paper may also suffers from typos and plots are sometimes misleading (squished axis in Figure 1). Some typos include pseudo vs psuedo, missing points, figure labels that overlap, subscripts in mathematical notation etc. Some references are also repeated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9fVryfqT9J", "reviewer": "Reviewer_LB8S", "summary": "This is an empirical paper on Neural Tangent Kernel (NTK) surrogate models. The content of the paper is two-folded:\n\n1. Several approximations of NTK are introduced and evaluated quantitatively by various metrics through different experiments, showing how theses approximations capture the decision mechanism of Neural Networks (NN) on classification problem.\n2. Then the paper argues how the NTK surrogate models give explanation for NN decision and states its limitation on SVM and adversarial attacks.\n\nThe paper includes a detailed and motivated introduction to Trace NTK and pseudo NTK, and experimental results on various data sets and  NN models. Its appendix contains a detailed section explaining the relationships between different kernels introduced in the paper, and a detailed result of the experiments together with visualisations.", "questions": "I have only one question:\nIn section 5, You mentioned: \"...an interesting follow-on work would investigate using kernel functions in K-Nearest Neighbors surrogate models.\" How much argument of this paper can transfer to KNN or generally any other surrogate models on explaining NN decision?\n\nAlso, there are some of the minor typos in the paper:\n\nSection 2 PRELIMINARIES Neural Networks for Classification third line: it should be \\mathcal{Y} instead of Y.\n\nAppendix F FORMAL DEFINITION OF EVALUATION METRICS last equation: it should be SS_res instead of SS_ret.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality: The paper is innovative to use the Kendall-$\\tau$ rank correlation to evaluate the approximation, such as TrNTK, pNTK, CK...,  on the empirical NTK (eNTK). The angle to experiment on explaining NN by surrogate NTK is also novel. \n\nQuality: The paper is written nicely with rigorous definitions and detailed descriptions on the experiments. \n\nClarity: The paper clearly states the problem and presents their experiments. Also the motivation of the paper is clearly elaborated. \n\nSignificance: The paper is important in the area of explainable AI through the lens of surrogate NTK. This paper could lead to more research on related topics.", "weaknesses": "There is barely any flaws in the paper, and the limitation of the experiments is clearly stated in the limitations subsection in section 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)