GPEX, A Framework For Interpreting Artificial Neural Networks
Authors: Amir Akbarnejad
Gilbert Bigras
Nilanjan Ray
Venue: NeurIPS 2023 poster (2023)
Keywords: Gaussian processes
Explainable AI

Abstract: The analogy between Gaussian processes (GPs) and deep artificial neural networks (ANNs) has received a lot of interest, and has shown promise to unbox the blackbox of deep ANNs. Existing theoretical works put strict assumptions on the ANN (e.g. requiring all intermediate layers to be wide, or using specific activation functions). Accommodating those theoretical assumptions is hard in recent deep architectures, and those theoretical conditions need refinement as new deep architectures emerge. In this paper we derive an evidence lower-bound that encourages the GP's posterior to match the ANN's output without any requirement on the ANN. Using our method we find out that on 5 datasets, only a subset of those theoretical assumptions are sufficient. Indeed, in our experiments we used  a normal ResNet-18 or feed-forward backbone with a single wide layer in the end. One limitation of training GPs is the lack of scalability with respect to the number of inducing points. We use novel computational techniques that allow us to train GPs with hundreds of thousands of inducing points and with GPU acceleration. As shown in our experiments, doing so has been essential to get a close match between the GPs and the ANNs on 5 datasets. We implement our method as a publicly available tool called GPEX: https://github.com/amirakbarnejad/gpex. On 5 datasets (4 image datasets, and 1 biological dataset) and ANNs with 2 types of functionality (classifier or attention-mechanism) we were able to find GPs whose outputs closely match those of the corresponding ANNs. After matching the GPs to the ANNs, we used the GPs' kernel functions to explain the ANNs' decisions. We provide more than 200 explanations (around 30 in the paper and the rest in the supplementary) which are highly interpretable by humans and show the ability of the obtained GPs to unbox the ANNs' decisions.

GPEX, A Framework For Interpreting Artificial Neural Networks
Amir Akbarnejad Department of Computing Science
Gilbert Bigras
Department of Laboratory Medicine and Pathology
University of Alberta Edmonton, AB, Canada ah8@ualberta.co
University of Alberta
Edmonton, AB, Canada
Gilbert. allllerrBiggaaaallrrrrppeessssss
Nilanjan Ray
Department of Computing Science
University Alberta Edmonton, AB, Canada nray1@ualberte
Abstract
The analogy between Gaussian processes (GPs) and deep artificial neural networks (ANNs) has received a lot interest, and has shown promise to unbox the blackbox of deep ANNs. Existing theoretical works put strict assumptions on the ANN (e.g. requiring all intermediate layers to be wide, or using specific activation functions). Accommodating those theoretical assumptions is hard in recent deep architectures, and those theoretical conditions need refinement new deep architectures emerge. In this paper we derive an evidence lower-bound that encourages the GP's posterior to match the ANN's output without any requirement on the ANN. Using our method we find out that on 5 datasets, only a subset of those theoretical assumptions are sufficient. Indeed, in our experiments we used normal ResNet-18 18 or feed-forwan backbone with a single wide layer in the end. One limitation of training GPs is the lack of scalability with respect to the number of inducing points. We use novel computational techniques that allow us to train GPs with hundreds thousands inducing points and with GPU acceleration. As shown in our experiments, doing so has been essential to get close match between the GPs and the ANNs on datasets. We implement https://github..omt wwaiittuuu.oo/mmmmmi. aamirak bummirarbrnnjaad/ppes On 5 datasets (4 image datasets, and biological dataset) and ANNs with 2 types of functionality (classifier or attention- mechanism) we were able to find GPs whose outputs closely match those the corresponding ANNs. After matching the GPs to the AÑNs, used the GPs' kernel functions to explain the ANNs' decisions. We provide more than 200 explanations (around 30 explanations in the paper and the rest in the supplementary) which are highly interpretable by humans and show the ability of the obtained GPs unbox the ANNs decisions.
method publicly available tool called GPEX:
Introduction
Artificial neural networks (ANNs) are widely adopted in machine learning. Despite their benefits, ANNs are known be black-box to humans, meaning that their inner mechanism for making predictions is not necessarily interpretableexxplinabb. to humans. ANN's black-box property impedes its deployment in safety-cen -eittcciiiiiall applications like medical imaging or autonomous driving, and makes them hard-to-troublesshod for machine learning researchers.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Attribution- based explanation methods like LIME[3I), SHAP[2I and most gradient- expla- nation methods like DeepLIFT [5] presume linear surrogate model. Given a test instance Itest- this simpler surrogate model is encouraged to have the same output "locally" around Ftest. Because of this "local assumptions", explanations from these methods might be unreliable, and can be easily manipulated by an adversary model [1] [28). Moreover, these models may produce discordant
explanations for a fixed model and test instance [66.
Considering Gaussian processes (GPs) 126 as the explainer model is beneficial, because: 1. Gaussian processes are highly interpretable. 2. Researchers have long known that GP's posterior has the potential to match an ANN's output "globally". More precisely, given an ANN and some requirements on [24] (8, there might exist GP whose posterior matches the ANN's s output all over the input- space X (as opposed to the local explanation models for which the match happens only locally around a test instance Ptest E X). Not many explainer models can globally match the ANN's output. Among madientt-based methods, with the best of our knowledge only Integrated Gradients [R has a weak sense of ANN's global behaviour over the input space. Having some conditions on an ANN, representer point selection (37) finds a "globally faithfull" explainer model that, similar to GPs, works with a kernel function. As we will elaborate upon in Sec. 4.3 and Sec. S6 of the supplementary. the GP's kernel that we find in this paper is superior due to technical point in the formulation of representer point selection B7. All in all, using GPs to explain ANNs is quite promissing and has advantages over other approaches to explain ANNs.
The contributions of this paper are as follows:
Theoretical results ANN-GP analogy impose some restrictions ANNs under which the ANN will be equivalent to a GP. These conditions are too restrictive for recently used deep architectures. Moreover, those theoretical conditions need refinement architectures emerge. In this paper we derive an ELBO for training GPs which encourages GP's posterior to match ANN's output. Our formulation and method doesn't 't impose any restriction on the ANN and the method used to train it.
deep
Using our method, we empirically show that on 5 datasets (4 image datasets, and 1 biological dataset) and ANNs with 2 types of functionality (classifier or arention-mechanisn the ANN needs to fulfill only subset of those theoretical conditions. Indeed, in our experiments
we used a normal ResNet-18 or feed-forward backbone with a single wide layer in the end.
Scalability is a major issue in training GPs. To address this issue, we adopted computational techniques recently used for fast spectral clustering [13 as well as novel method to learn the GPs using aani-batches of inducing points and training instances. These computational techniques allow us to train GPs with hundreds of thousands of inducing points. According to our analysis, increasing the inducing points has been essential to get a good match between the trained GPs and ANNs. Indeed, without many inducing points GPs posterior cannot be a complex function (a function with many ups and downs (36) and fails to match ANNs' output.
With the best of our knowledge, our work is the first method that performs knowledge distillation between GPs and ANNs.
We implement our method as a public python library called GPEX (Gaussian Processes for EXplaining ANNs). GPEX takes in an arbitrary Py PoTorch module, and replaces any ANN submodule of choice by GPs. Our package makes use of GPU-accelaration, and enables effortless application of GPs without getting users involved in details of the inference procedure. GPEX can be used by machine learning researchers to interpre//toubllshoot their artificial neural networks. Moreover, GPEX can be used by researchers working on the theoretical side of ANN-GP analogy to empirically test their hypotheses.
2 Proposed Method
2.1 Notation
In this article the function g(.) always denotes an ANN, The kernel of Gaussian process is denoted by the double-input function K(...). We assume the kernel similarity between two instances æ; and æÿ is equal to f(::)² [xx;) where f()) maps the input-space to the kernel space. this paper u (resp. v) denotes vector in the kernel-space (resp. the posterior mean) of a GP. In some sense and denote

---

general feed-forward module
ANN
GPsmesn
GPsmean covariance
Figure 1: a) A general feed-forward pipeline, with an ANN sub-module to be explained by GPEX. b) Typical behaviour of Guassian process posterior given set of observed values.
the input and the output of a GP, respectively. We have that: K(æ;, xz) = f(x;;* f(tz)) u, uj. uz-
The number of GPs is equal to the number of the outputs from the ANN. other words, we consider one GP per scalar output-head from the ANN. We use index & to specify the é-th GP as follows: Kee m.)) = fe( fe
fe( = ( ( uy(²), We parameterize the ε-th GP by set of M inducing
points { { den ) J The tilde in (ā em)) indicates that ú is one of the M inducing points in the kernel space. However, u (without tilde) can be an arbitrary point in the continuous kernel-space.
2.2 The Proposed Framework
To make our framework as general as possible, we consider general feed-forward pipeline that contains an ANN as a submodule. In Fig. [ajthe bigger square illustrates the general module. The input-output of the general pipeline are denoted in Fig. Ia by X and V. The general pipeline has at least one ANN submodule to explained by GPEX. Fig. Ta illustrates this ANN by the small blue rectangle within the general pipeline. The onput-output of the ANN denoted in Fig. [ by and v. Note that X' and y can be anything, including without any limitation, a set of vectors, labels, and inntrminfommm.on.. However, input-output of the ANN (i.e. and y) are required to be in tensor format. The exact requirements are provided in the online documentation for GPEX. Moreover, the general module can have other arbitrary submodules, which are depicted by the blue clouds. The relations between the submodules, as illustrated by the dotted-lines in Fig. Tal can also be quite general. Our probabilistic formulation only needs access the conditional distributions p(x).X) and p())), X). Similarly, the proposed GPEX is completely agnostic about the general pipeline and only requires the ANN's input-output to be in the tensor format. Given a PyTorch module, the proposed GPEX tool automatically grabs the distributions p X and p() ax, X.) from the main module given.
The inducing points (a( em JMM m= parameterize the E-th GP. Note that S fm A feature point like x is first mapped the kernel-space ( fe(x). Note that the kernel functions (fr()) are implemented as separate neural networks, or for the sake of efficiency as a single neural network backbone with L different heads. Afterwards, the GP's posterior on depends the
kernel similarities between (2) and the inducing points { 1.M J-==1- More precisely, the posterior of
the f-th GP on æ is random variable (f) whose distribution is as follows 26
((() = N (N(()) (h (u. ū 11:M7 (1:M), cov, ú I.M (1:M ((u(0) (+), ū 11:
(E)
((E)
(1)
where po and c...(..,,.) are the mean and covariance of GP's posterior computed as: (()
(t) 1:M v K( a (K((
1:b0 u1:M) + I
[(0)
U.M
(2)
and
co ū uiM 0
(.rr =K(ul0), (0)) (0))
(9 a( x [K(a( (
o2 I
K(u ()) (3)
gp
(5) As the variables ( 1 and are latent hidden, train the model parameters by optimizing variational lower-bound. We consider the following variational distributions:
qa (v(0) (z) = (v((() ge(x), oz), q2 (((() = =N -N(((( 1410) .
3

---

In Eq. 4 the function ge(.) is the E-th output from the ANN. Note that as the set of hidden variables (bm)) mmm is finite, we have parameterized their variational distribution by a finite set of numbers (pm MME However, as the variables a can vary arbitrarily in the feature space, the variable u()) varies arbitrarily in the kernel space. Therefore, the set of values e(() may be infinite. Accordingly, the variational distribution for v.() is conditioned on æ and is parameterized by the ANN g(-).
2.3 The Derived Evidence Lower-Bound (ELBO)
Due to space limitation, the derivation of the lower-bound is moved to Sec. SI of the supplementary material. In this section we only introduce the derived ELBO and discuss how it relates the GP, the ANN and the training cost of the main module in intuitive way. The ELBO terms containing the GP parameters (i.e. the parameters of the kernel function f()) is denoted by Cyp. According to Eq. S9 of the supplementary material Lgp is as follows:
(0)
L. (IM (e( + o²
cap
Enng
(oll( uj:Mi (
log( Cowo (2ll( (1) (1) IIM: V1MM )] + (const.),
E
(5)
2
og
8=11
where q(.. is the variational distribution that factorizes to the qi (.) and g2(.) distributions defined Eq. 4 the first term Eq. [5] the numerator encourages the GP and the ANN to have the same output. More precisely, for a feature point æ we can compute the corresponding point in the kernel space as u = fax) and then compute the GP's posterior mean based on kernel similarities between u and the inducing points to get the GP's mean Mlg. In Eq. 5he GP's mean μ4 is encouraged to match the ANN's s output (z(x). In Eq. 5 because of the denominator of the first term, the ANN-GP similarity is not encouraged uniformly over the feature-space. Wherever the GP's uncertainty is
low, the term cou, (v, (f) ..(8) in the denominator becomes small. Therefore, the GP's mean U1:as
highly encouraged to match the ANN's output. On the other hand, in regions where the GP's uncertainty is high, the GP-ANN analogy is less encouraged. This formulation is quite intuitive according to the behaviour of Gaussian processes. Fig. [Tb] illustratee the posterior of GP with radial-basis kernel for a given set of observations. In regions like [3, oo) and (-x,, -4] there are no nearby observed data. Therefore, these regions the GP is highly uncertain and the blue uncertainty margin is thick in such regions. Intuitively, our derived ELBO in Eq. 5 Jnncoragess the GP-ANN analogy only when GP's uncertainty is low and gives less importance to regions similar to [3, 2o) and (-00, -4] in Fig. lb Note that this formulation makes no difference for the ANN ANNs are known to be global approximators. However, this formulation makes a difference when training the GP, because the GP is not required to match the ANN in regions where there are similar training instances. The ELBO terms containing the ANN parameters is denoted by Lann. According to Sec.
S1.2 of the supplementary material, Lann is follows:
(8) E. Σ (po(al)) u (I. 9((2)) + Erng p(),(,)))))
1
Cann
(6)
coo,(ul), i 1
(0) [0]
In the above objective the first term encourages the ANN have the same output as the GP. Similar
to the objective of Eq. [ the denominator of the first term gives more weight to ANN-GP analogy
when GP's uncertainty is low. In the rinht-hand-side of Eq. (6] the second term is the likelihood of the
pipeline's ' output(s), i.e. y in Fig. la This term can be, e.g., the cross-entropy loss when ) contains class scores in a classification problem, or the mean-squared error when y is the predicted value for regression problem, or a combination of those costs in a multi-task setting.
3 Algorithm
We consider a separate Gaussian process for each output head of an ANN. other words, given an ANN we have as many GPs as the number of the ANN's output heads. To explain ANN, we find the explainer GPs by optimizing the objective in Eq. 5 w.r.t. to the kernel mappings (fe(-)) To do so, we need to have Pe which in turn means we need to have all cerncl-space representations

---

Algorithm Method Optim_Kern
Input: Input instance æ and inducing instance ã. list of matrices U, list of vectors V. Output: Kernel-space mappings (J1())...,Jz) (2)))
Initialisation loss +- O.
μl, cav forward GP(x, U, V) //feed x to GPs, "forward GP" is Alg.SI in supplementary.
2: Monn <- g(x) //feed x to ANN.
3: for to I do
4: loss <- loss + Mal0-mann(0)) (1) + log(coo((). 1/Eq.
could
5: end for
6:
a params (1 (1f.()....ff //the gradient of loss.
loss
7: params([fi ... ftl) + params
Ir x 8 //update the parameters.
8: Ir e updated learning rate
9: return [f(().... ... ft(.)
Algorithm 2 Method Explain ANN
Input: Training dataset ds train, and the inducing dataset ds inducing. Output: Updated kernel-space mappings (fr(.)...., and the other GP parameters U and V. Initialisation U, V +- Init GPparams(ds, (i //Alg.S3 in supplementary. 1: for iter = 1 to max sete_jterro randsclect (ds (ts_rri)......
2:
3: i f randselect (ss (id_snnducing)
4:
(r.,,..,..(.... Φ < randselect (dd (ds_inducing).
Optim KemMapinggsa;, a, U, V).
5:
6:
for l =-11 to do
7:
//update kernel-space representations. f((
8:
9: end for
end for
11: return (2()),,,(((,,. U,
( JM However, it is computationally prohibitive to feed thousands of inducing instances to the kernel mappings T!! = (m for {1, ..., M) in each gradient-descent iteration.
On the other hand, as the kernel-space mappings {fe(-)]E keep changing during training, we 1
need to somehow track how the inducing points {ū(0) mm=1 change during training. To this end, we put the kernel-space representations of the inducing points matrices denoted by U. During training. these matrices are repeatedly updated by feeding mini-batches of inducing instances to the caamel-mappings.
Alg. optimizes the objective of Eq. 5 w.r.t. the kernel mappings (fe(-))/ First, single training instance æ and single inducing point x are selected (line 2-3). Afterwards, the procedure of Alg. is called to update the kernel mappings (line 4 of Alg. (2.. To update the kernel-mappings. the GP posterior is computed via the matrices U (line 1 of Al The "forward GP" procedure (called in line 1 of Alg- [ is provided in Alg. S1 of the supplementary, and uses the matrices U to compute GP's posterior. Only the rows of U that correspond to the selected inducing point क are computed using the kernel-mappings, so that the gradient w.r.t. the kernel-mappings can be computed in the backward pass (lines 5-6 of Alg. SI in the supplementary). Finally, the matrices U are updated (lines 6-8 of Alg. 2). Due to the lack of space, the routines "forward "P" and "Init, maait_GPpaaamm's and more details are moved to Sec. S2 of the supplementary. Of course instead of a single training/inducing instance, used mini-batch of multiple training/inducing instances.
One difficulty of training GPs is the matrix inversion of Eqs 2 and 3 which has ((M3) complexity using standard matrix inversion methods. To address this issue, we adopted computational techniques recently used for fast spectral clustering [3] Let A be an arbitrary M x matrix where M >> D. Moreover, let b be M-dimensional vector and let be scalar. The computational techniques [3 allow us to efficiently compute: (A + d² Mx as) -1 (Note the similar terms the right hand side of Eqs. and The idea is that AA T is of rank D. Therefore, from linear algebra it follows
5

---

that (AA) +o²IMxx)"" has M - D aigen-values all of which are equal to o Therefore, in the space of those eigen-vectors, the transformation on b is simply scaling by o2, The details and more computational techniques are provided in Sec. S2. 1 of the supplementary. These computational techniques allow us to efficiently compute the GP-posterior for hundreds of thousands of inducing points in each gradient descent iteration.
note on the used datasets in Alg. 2 According to our analysis of Sec. S7 in the supplementary material, "ds jos_inducing" should be as large as possible so the GP posteriors can be flexible enough to match the ANNs. Therefore a good practice is to include all training instances (without data augmentation) in "ds sss_indduinng.. By doing so, the following issue arises. An instance from "ds ss_ttaiin" like x is an augmented version of an inducing instance ă. Because æ and ã are close, their kernel-space representations f(x) and f(z) also become close regardless of parameters of f()). Consequently, regardless of f()), GP's posterior mean will be roughly equal for both x and æ. Indeed, in this case Alg. 2 fails to find the kernel mappings (fe(-))E=1 To avoid this issue, we sample æ in line 2 of Alg. 2 as follows: æ1 and 22 are randomly selected from "ds "ds_train",, and uni form(-1, 2), and 0 + (1 - a) 22 The rest of Alg. aater line 2 is run as before.
4 Experiments
We conducted several experiments on four publicly available datasets: MNIST [9] Cifarl0 [19].
Kather [13), and DogsWolves (34]. For MNIST 9 and Cifarl0 [9] we used the standard split to training and test sets provided by the datasets. For Kather [5] and Dogs Wolves [34]]] we randomly
selected 70% and 80% of instances as our training set. The exact parameter settings for running Alg.
aare elaborated upon in Sec. S5 of the supplementary. We trained the ANNs as usual rather than using Eq. because our proposed GPEX should be applicable to ANNs which are trained as usual.
4.1 Measuring Faithfulness of GPs to ANNS
We trained a separate convolutional neural network (CNN) on each dataset to perform the classification task. For MNIST [9). Ci [1), and Kather [15] used a ResNet-18 [12] backbone followed by some fully connected layers. Dogs Wolves [4] is a relatively small dataset, and very deep architectures like ResNet [12 overfit to training set. Therefore,
used convolutional backbone which is suggested in the dataset website [34]. For all datasets, we set the width (i.e. the number of neurons) of the second last fully-connected layer to 1024. Because according to theoretical results on GP-ANN analogy, the second last layer of ANN should be wide. We used an implementation of ResNet [12 which is publicly available online 2 We trained the pipelines for 20, 200, 20, and 20 epochs on MNIST (9) Cifar [9]. Kather 15 and DogsWolves 3. respectively. For C0 [9. we used the exact optimizer suggested by 22. For other datasets we used an Adam [17] optimizer with a learning-rate of 0.0001. The test accuracies of the models are equal to 99.56%, 95.43%, 96.80% and 80.50% on MNIST 99. Cifar10 [9). Kather (15). and DogsWolver (34), respectively. We also applied our proposed GPEX to a state-of-the-a art cell-embedding method called scArches 200) We ran a tutorial notebook 32 and applied GPEX to the decoder whose job is to predict expression of some genes given scArches 20 cell embeddings. More details are provided in our public github repository (repository link is provided in page 1).
We explained each classifier ANN using our proposed GPEX framework (i.e. Alg D. As discussed in Sec. 3 given an ANN we have as many kernel-spaces (and as many GPs) as the number of ANN's output heads. The exact parameter settings and practical considerations for training the GPs elaborated upon in Sec. S5 of the supplementary material. To measure the faithfulness of GPs to ANNs, we compute the Pearson correlation coefficient for each ANN head and the mean of the corresponding GP posterior on unseen test instances. The results are provided in Fig. In Fig. 3 the first five groups of bars (i.e. the groups labeled as Cifar10 (classifier), MNIST (classifier), Kather (classifier). Dogs (classifier), and scArches (classifier)) correspond to applying the proposed GPEX to the five classifier ANNs trained on the four datasets and scArches embeddings. According to Fig. 3 our trained GPs almost perfectly match the corresponding ANNs. Only for Dogs Wovles 34 as illustrated by the 4-th bar group in Fig. 3 the correlation coefficients are lower compared
to other datasets. We hypothesize that this is because the DolsWslves dataset [4 has very few images. GP posterior mean can be changed only by moving the inducing points in the kernel-space.

---

/
/
/
/
/
/
/
/
3
2
2
2
2
2
2
2
a
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
1
4-
44
4
4
4
4
4
4
4
4
/
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4/
4
4
3
5
5
'
5
5
T
5
5
5 3
3
5
5
5
5
5
5
5
5
5
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7 8
7 3
3
3
3
<
8
8
&
6
P
8
6
8
Y
8
6
5
9
9
9
9
9
9
9
9
9 9
9
9
9
9
9
9
9
9
9
9.
9
9
9
9
(b)
(d)
Figure 2: (a,c,d) Sample explanations for MNIST, Cifarl0, and DogsWolves In each row a test instance is shown the first column, and the 10 nearest neighbours (in the kernel-space of the GP that corresponds to the output-head with maximum value at the test instance) is shown in columns 2-11. (b) Evaluating our proposed method, representer point selection 37], and influence functions [18 in dataset debugging task.
Therefore, when very few inducing points are available GP posterior mean is less flexible (36 This is consistent with our parameter analysis in Sec. S7 of supplementary material.
In Fig. Ta we discussed that GPEX not only able explain a classifier ANN, but it can explain
any ANN which is a subcomponent of any feed-forward pipeline. To evaluate this ability, we trained
three classifiers with attention mechanism (22. Each classifier has two ResNet-18 12] backbones: one extracts a volumetric map containing deep features, and the other produces a spatial attention mask. For each attention backbone, we the width of the second last layer 1024, followed by linear layer and sigmoid activation. We applied our proposed GPEX (i.c. Alg 2) to each classifier, but this time the ANN to be explained (i.e. the box called "ANN" in Fig, Tap set to be the attention submodule. Note that each attention backbone produces a spatial attention mask of size h by w. We think of each attention backbone as an ANN which has h x w output heads. We trained three classifier pipelines with attention mechanism on Cifar10 [1). MNIST (9), and Kather [5]] with the same training procedure as previous part. In Fig. 3 6-th, 7-th, and 8-th bar groups show the correlation coefficients between the attention backbones and the corresponding GPs on unseen test instances. According to Fig- 3 our proposed GPEX able find GPs which are faithful to attention
subcomponents of the classifier pipelines. Note that didn 't include all attention heads, because
7

---

Corretation coelitcien: between and ANN
0.95
0.55
3.8
Cifrr0 (dassflir)
MNISI (clasillie)
Kather (dasstier)
Uogeto less ScArches (claseifier) (clssdierroateeniiin))
Giar10
MIIIST (attention)
Kather (attention)
Figure 3: Faithfulness of GPs to ANNs measured by the Pearson correlation coefficient.
some pixels in attention masks are always off. In Sec. S3 of the supplementary material we have included more information and insights about the faithfulness of GPs to ANNs.
4.2 Explaining ANNs' Decisions
In Sec. 4.1 we trained four CNN classifiers on Cifarl0 [19]. MNIST [9], Kather [15], and DogsWolves Wolves [4]. respectively. Afterwards, we applied our proposed explanation method each CNN classifier. In this section, we are going to explain the decisions made by the classifiers via the obtained GPs found by Alg. 2 We explain the decision made for a test instance like Etest as follows. We consider the GP and the kernel-space that correspond to the ANN's head with maximum value (i.e. the ANN's head that relates the predicted label). Consequently, among the instances in the inducing dataset, we find the 10 closest instances to Brest, like { 222 ..., 2010). J. Intuitively the ANN has labeled Frest in that way because it has found Ptest to be similar to (a+11 Z222 2210)
For MNIST digit classification, some test instances and nearest neighbours in training set are shown test instance. The first column depicts the test instance itself and columns 2 to 11 depict the 10 nearest neighbours. For example, in Fig. 2a the image in row3-coll depicts a test instance Fiest and the images in row3, cols2-11 depict the nearest neighbours {.:,1 Izz...... Øs10): }. According to rows 1 and 2 of Fig. 2a the classifier has labeled the two images as digit because it has found digits with similar inclinations in the training set (in Fig. 2a in row 1 all digits are vertical but in row 2 all digits are inclined). We see the model has also taken the inclination into account for the test instances of rows 7, 8, 15, 16, and 17 of Fig. 2a Fig. 2a according to rows 3, 4, and 5 the test instances are classified as digit 2 because 2 digits with similar styles are found the training set. We see the model has also taken the style into account for the test instances of rows 6, 7, 8, 9, 10. 12, 13, 14, 15, 16, and 17 of Fig. 2a For instance, the test instance in row 6 of Fig. 2a is a 4 digit with a short stand and the two nearest neighbours are alike. Or for the test instances in rows 13, 14, and 15 of Fig. 2a the test instances have incomplete circles in the same way as their nearest neighbours. More explanations are provided in the supplementary material Sec. S4.
in Fig. 2a In this figure each row corresponds
Fig. 2c illustrates some sample explanations for Cifar10 19]. Like before, each row corresponds to a test instance, the first column depicts the test instance itself and columns neighbours. In Fig. 2c] the test instances of rows 1, 2, 3, 4, and 5 are captured from horses' heads from closeby, and the nearest neighbours are alike. However, in rows 6, 7, 8, 9, 10, and 11 of Fig. 2c the test images are taken from faraway and the found similar training images are also taken from faraway. Intuitively, as the classifier not aware of 3D geometry, it finds training images which are captured from the same distance. In rows 9, 10, and 11 of Fig. 2c we see that the testing images contain riders. Similarly, the nearest neighbours also tend to have riders. Therefore, in rows 9, 10, and 11 of Fig. 2c the model has made use of the riders or other context information to classify the test instances as horse. More explanations are provided in the supplementary material in Sec. S4.
11 depict the 10 nearest
Besides finding the nearest neighbours, we provide CAM-like [38] explanations as to why Drest and an instance like æij, S j < 10 are considered similar by the model (according to the procedure of Sec. $2.2 in the supplementary material). Fig illustrates some sample explanations for DogsWolves 34 dataset. In row of Fig. 2d]] the first column depicts the test instance itself and columns 2 to 11 depict the 10 nearest neighbours. The second and third rows highlight the pixels
8

---

that contribute the most to the similarities. The second and third rows highlight the pixels of Etest and {æ:11 F22 .... F.1o) respectively. According to row 3 of Fig. 2d the pink object next to the dog's leg has contributed the most to the similarities. According to row 2 of Fig. regiios like the baby in column 3, the dog colar or costume in columns 4, 5, and 6, human finger in column 9, and the background in columns 10 and 11 have contributed the most to their similarity to the test instance. These are patterns that usually happen for dogs images. Indeed, since the training set has been small (1600 images), to detect dogs the model is making use of patterns that normally exist in indoor scenes and do not normally appear in wolves images. We see a similar pattern for the test instance in row of Fig. 2d and also several explanations in Sec. of the supplementary.
4.3 Comparing GPEX to Representer-Point Selection and Influence Functions
In Sec. S6 of the supplementary material we qualitatively compare GPEX explanations to those of representer point selection (37). According to the experiments and detailed discussions of Sec. S6 in the supplementary, the GP's kernel that we find in this paper is superior due to a technical point in the formulation of representer point selection [37.. Besides the analysis of Sec. S6, we compared our proposed GPEX with representer point selection [37] and influence functions [88 in
dataset debugging task. In these experiments we only selected images from Cifar 19 that are labeled as either automobile or horse. To corrupt the labels, we randomly selected 45% of training instances and changed their labels. Afterwards, we trained a classifier CNN with ResNet18 [22 backbone with the same training procedure explained in Sec. 4.1 In dataset debugging task, training instances are shown to user in some order. After seeing an instance, the user checks the label of the instance and corrects it if needed. One can use explanation methods to bring the corrupted labels to the user's attention more quickly. Given an explanation method, we repeatedly select test instance which is misclassified by the model. Afterwards, we show to the user the closest training instance (of course among the training instances which are not yet shown to the user). We repeat this process for test instances in turn until all training instances are shown to the user. We compared our proposed GPEX to representer point selection [37] and influence functions 18] in dataset debugging task. We used an implementation of influence functions
(18] based on LiSSA 4 with 10 steps for
each instance. The implementation is publicly available 1 For representer point selection (37] we used the implementation by authors which is publicly available [3] The result is shown in Fig. 2b
According to the plot on the left in Fig. 2b when correcting the dataset by GPEX, the model accuracy becomes close to 90% after showing about 4000 instances to user. But when using representer point selection 37 or influence functions [8) this happens when the user has seen about 7000 training instances. With noisy labels model training becomes unstable. Therefore, in the plot on the left of Fig. 2b/we we repeat the training 5 times and we report the standard errors by the lines in top of the bars. According to the plot on the right of Fig. 2bb after showing a fixed number of training instances the user, when using the proposed GPEX more corrupted labels are shown to the user. Indeed, GPEX brings the corrupted labels to the user's attention quicker than representer point selection X does. Interestingly, according to the plot in the right hand side of Fig. [26 influence functions [18] is quicker at spotting incorrect labels, but the instances found by our proposed method are more effective in increasing the accuracy quicker.
5 Related Work
The first theoretical connection between ANNs and GPs was that under some conditions, a random single-layer neural network converges to the mean of a Gaussian process (33 as the width of that single layer goes to infinity. This connection was later proven for ANNs with many layers (8) and for ANNs trained with gradient descent [14]. The theoretical requirements are usually too restrictive. For example, [8] requires all intermediate layers to be wide and also requires the dataset to be countable (so augmamtattnntnnnnnn like volor-jitter are not allowed). Or (44 requires the ANN to be trained with MSE loss and requires all intermediate layers to be wide. In this paper we do not presume any conditions on the ANN and simply distill knowledge from a neural network to some GPs. Of note, those theoretical conditions may facilitate knowledge distillation and improve the Pearson correlation coefficient between the ANNs and the GPs obtained by our method.
Scalability is a major issue when training GPs, and including few inducing points may limit the flexibility of GP's posterior [36) Here we review some previous methods to tackle the computational challenges of training GPs. SV-DKL [27] derives a lower-bound for training GP with deep
9

---

kernel. In this method, a grid of inducing points are considered in the kernel-space (like the vectors ((u!) on ) with the notation of this paper). Afterwards, each input instance firstly mapped to the kernel-space and the output is computed based on similarities to the grid points in the kernel-space.
Since the GP posterior is computed via the grid points, SV-DKL [27 scalable. But unfortunately the number of grid points cannot be increased to above 1000 even for Cifar10 [19] and with RTX 3090 GPU. Therefore, this may limit the flexiblity of the GP's posterior [36].
more recent framework called GPytorch (29] provides GPU acceleration. However, its computa- tional complexity is quadratie in number of inducing points. Other approaches to improve scalability of GPs include: considering structured kernel matrices [7]. kernel interpolation [35], and imposing grid-structure on including points (27]. Stack of Gaussian processes are shown to be connected to ANNs (IO) 30 [24]. By stacking kernels, GP kernels work on intermediate representations and therefore are not necessarily interpretable to humans. But in our method the GPs' kernels work directly on the input-space itself. Knowledge distillation (KD) is closely related to this work. With the best of our knowledge and according to the authors, [6] is the first work that applies KD to GPs. But the distinction of our work is that we distill knowledge from ANN to GP, as opposed to the self-distillation of [6] that distills knowledge from GP to another GP.
Limitations and Outlook: In this work we used Eq. to distill knowledge from ANN to GP. One may use Eq. to distill knowledge from GP ANN in order to, e.g., transfer GP's good generalization to the ANN. Our method scales very well, and Alg. runs without memory/computtaional issues even on imagenet with more than 1M inducing points (i.e. images) and Resnet-18 18 [12] when a few output-heads are selected, but on imagenet we failed to match the GPs to ANN in a 2-3 day runtime. The issue is that the U matrices have be updated very often (the update of line 8 of Alg. 2 that GPs' kernels are updated according to an accurate estimate of kerne--spac representations. Otherwise the convergence may not happen especially for millions of inducing points and a small batch-size (as required for, e.g., CNNs). We used control
(25), but one may use more ad advanced heuristics [35 to achieve convergence for datasets like imagenet and with a reasonable computation time. In this paper we analyzed the effect of number of inducing points, the width of the second last layer, and the number of epochs for which the ANN trained. One can use the proposed tool to answer other questions, like, is the GP kernel required to have more parameters than the ANN itself? May it so happen that a test instance is equally close to hundreds of training instances thereby limiting a human's 's ability to understand ANNs decision? Is the uncertainty provided by the GP correlated to the understandability of the explanations to humans or to the ANN's failures?
6
Acknowledgements
The experiments of this paper were enabled in part by the Digital Research Alliance of Canada. This work was supported in part by the NSERC Discovery Grant.
References
[1] A simple PoTorch implementation of influence functions. http:://gihhub.cc//aaattttoo torch- crrch-influence [Online; accessed 19-Oct-2023).
[2] An implementation for ResNet in httppp///ggtttiooooo/////uuuuuta [Online; accessed 19-1e.-2021).
[3] Implementaiton of represnter point selection. SelnctSelection [Online; accessed 19-Oct-2023).
[4] N. Agarwal, B. Bullins, and E. Hazan. Second-order stochastic optimization for machine learning in linear time. The Journal of Machine Learning Research, 18(1):4148 4187, 2017.
[5] Avanti Shrikumar et al. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine I Learning Research, pages 3 3155-3153. PMLR, 06-11 Aug 2017.
[6] K. Borup and L. N. Andersen. Self Self-distillation for gaussian process regression and classification. arXiv preprint arXiv:2304.026111 2023. 2023.
10

---

[7] M. K. Cohen, S. Daulton, and M. A. Osborne. Log-linear-tingg time gaussian processes using binary tree kemelss. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.
[8] A. G. de G. Matthews, J. Hron, M. Rowland, R. F. Turner, and 7. Ghahramani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations, 2018
L. Deng. The mnist database of handwritten digit images for machine learning rescarch. IEEE Signal Processing Magazine, (9(6):141111) 2012.
[10] Y. Gal and Z. Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059, PMIR, 2016.
[11] A. Ghorbani, Abid, and Zou. Interpretation of neural networks fragile. Proceedings of AAAI Conference on Artificial Intelligence, 53(00)):68113666 Jul. 2019.
[12] K. He, Zhang, Ren, and Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Compuder Vision and Puttern Recognition (CVPR), June 2016.
[13] L. He, Ray, Y. Guan, and H. Zhang. Fast large-scale spectral clustering via explicit feature mapping. IEEE Transactions on Cybernetics, 49)3):0058-1077 2019.
[14] Jachoon Lee et al. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
[15] J. N. Kather, C.-A. Weis, F. Bianconi, S. M. Melchers, L. R. Schad. T. Gaiser, A. Marx, and F. G. Zöllner. Multi-class texture analysis in colorectal cancer histology. Scientific Reports. 6, 2016.
[16] A. Khakzar, P. Khorsandi, R. Nobahari, and N. Navab. Do explanations explain? model knows best. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10244 10253, June 2022.
[17] D. P. Kingma and J. Ba. Adam: method for stochastic optimization, 2017.
[18] P. W. Koh and P. Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pages 1885-1894. PMLR, 06-11 Aug 2017.
[19] A. Krizhevsky. Learning multiple layers of features from tiny images. pages 32-33, 2009.
[20] Lotfollahi, M. Naghipourfar, M. D. Luecken, M. Khajavi, M. Büttner, M. Wagenstetter, 2. Avsec, A. Gayoso, N. Yosef, M. Interlandi, et al. Mapping single-cell data to reference atlases by transfer learning. Nature Biotechnology, pages 1-10, 2021.
[21] S. M. Lundberg and S.-L Lec. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems 30, pages 1765-47744 Curran Associates, Inc., 2017.
[22] Meng-Hau Hao Guo et al. Attention mechanisms in computer vision: A survey. CoRR, abs/2111.00624. 2021.
[23] R. Neal, Bayesian Learning for Neural Networks. Lecture Notes in Statistics. Springer New York, 2012.
[24] R. Novak, Xiao, Hron, Lee, A. Alemi, Sohl-dickstein, and Schoenholz. Neural tangents: Fast and easy infinite neural networks in python. 2020.
[25] Paisley, Blei, and M. Jordan. Variational bayesian inference with stochastic search. arXiv preprint arxiv: NPV61206.44 6430, 2012.
[26] C. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005.
[27] A. Wilson et al. Stochastic variational deep kemel learning. Advances Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
[28] D. Slack et al. Fooling lime and shap: Adversarial attacks on post hoc explanation methods. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, AIES '20, page 180-186, New York, NY, USA, 2020. Association for Computing Machinery.
[29] J. Gardner et al. Gpytorch: Blackbox matrix matrixx gaussian process inference with gpu acceleration. Advances in Neural Information Processing Systems, 2018 December::776 7586, 2018.
11

---

[30] V. Dutordoir et al. Deep neural networks as point estimates for deep gaussian processes. Advances in Neural Information Processing Systems, 34:9443-9455, 2021.
[31] M. T. Ribeiro, S. Singh, and C. Guestrin. "why should I trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, Francisco, CA, USA, August 13-17, 2016, pages 1133-1144, 2016.
[32] scArches CVAB notebook.
pipeline_basic.hne
[33] M. Sundararajan, Taly, and Q. Yan. Axiomatic attribution for deep networks, In Proceedings the 34th International Conference on Machine Learning, volume 70 Proceedings of Machine Learning Research. pages 3319-3328. PMLR, 06-11 Aug 2017.
[34] Vutukuri. Dogs vs Wolves Classification Dogs and Wolves. https://ww.kagg// com/ harishvutukuri//em dogss - wolves vs-ylvess
2019. [Online; accessed 19-Dec-2021J.
[35] Wilson H. Nickisch. Kernel interpolation for scalable structured gaussian processes (kiss-gp). In International conference on machine learning, pages 1775-1784. PMLR, 2015.
[36] Y. Bengio's post on gp ann httph::///// /ttp:::///qraaa/vq22a
[37] C.-K. Yeh, J. Kim, I. E.-H. Yen, and P. K. Ravikumar, Representer point selection for explaining deep neural networks. Advances neural information processing systems, 31, 2018.
[38] B. Zhou, Khosla, A. Lapedriza, Oliva, and A. Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR). June 2016.
12


Reviews:
[{"review_id": "VMeX9cOYBy", "reviewer": "Reviewer_nub9", "summary": "The paper derives an ELBO to end-to-end distill a deep neural\nnetwork into a set of Gaussian processes (one per output) with deep neural\nnetwork kernels and the full training set used as inducing points.\nBy using a low output-dimensionality for the feature map neural network,\nthe computational cost of the inversion/decomposition of the kernel matrix\nis kept constant while arbitrarily scaling the number of inducing points.\nThe paper empirically validates the discrepency between the teacher neural\nnetwork and the student DNN-GP on 5 datasets with 2 different models (5 on\nResNet-18, 2 on some custom attention-based architecture).\nTwo approaches are introduced to attempt to explain the prediction decisions of\nthe distilled neural networks based on the DNN-GP: through similarity in the\nDNN-kernel's feature space, and a pixel-wise contribution based on the\nsimilarity to the nearest neighbors in the DNN's kernel space.\nAn accompanying framework is provided, which claims to distill any DNN in Pytorch.", "questions": "- The paper could be improved by being more verbose about using DNN-kernels.\n\n- It might be better for the paper to instead of stretching the\n  scalability-claim, openly discuss how a small, explicit feature space leads\n  to a constant rank of the noise-free kernel matrix, creating a situation\n  where GPs do not scale beyond the explicit feature space's size, thus arguing\n  for low-dimensional DNN-kernel-spaces.\n\n- Given the increased complexity of the kernel's DNN to the teacher model, a\n  discussion on the complexity-explainability trade-off would be very interesting.\n  Further experiments with smaller kernel DNN's in the distillation-process\n  could also bring insight in this regard.\n\n- For the experiments in Figure 3, comparing to some baseline might help in\n  understanding the PCC values better. While a PCC of 0.9 sounds good, it would\n  help to also highlight the differences between student and teacher, although\n  I do not have a concrete idea how this could be done.\n\n- To put the gained explainability into context, especially in the qualitative\n  experiments, it would really help to compare to some baselines.\n\n- The quantitative experiment comparing to Representer-Point selection is very\n  valuable, but would be even more valuable with multiple tries, and on\n  different data sets. It feels somewhat incomplete.\n\n- The software claim in the beginning of the paper suggests a well-maintained\n  software framework for the GP model distillation, which does not seem to be true.\n  I would suggest to soften this claim (i.e., simply provide a footnote stating\n  \"Code for methods/experiments provided at...\")\n\n\n### Minor\n\n- Alg. 2 title is misleading, a better name may be: training loop for GP\n  distillation.\n\n- Alg. 1 can be moved to the supplement, as it is only a step of\n  gradient descent for the DNN-kernels and does not add much to the manuscript\n\n- l.126 nominator -> numerator\n\n- figures should be described more in detail, especially the qualitative one", "limitations": "The paper discusses some limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The ELBO and distillation process from GP to DNN-GP is a novel contribution.\n\n- Code is provided for all experiments, which is a major strength.\n\n- Important derivations are provided in the supplement.", "weaknesses": "- The paper reports on GPs, but is not very verbose in the fact that\n  DNN-kernels are used, although they seem to be strictly necessary for the\n  distillation (e.g., the low-dimensional, explicit features space). Prior work\n  on DNN-GPs is only partly discussed (e.g. [23]).\n\n- The paper claims to introduce a novel approach to scale the number of inducing points.\n  However, the computational trick comes down to using a low-dimensional, explicit\n  feature space, leading to a low-rank, high-dimensional noise-free kernel\n  matrix, which is only useful for low-dimensional explicit feature spaces as\n  in DNN-GP.\n\n- The DNN-kernels are significantly more complex (resnet50) than the teacher\n  DNN (ResNet18). This explainability-complexity trade-off is not explicitly\n  discussed, and seems like a significant limitation.\n\n- The experiments are not presented very well: Figure 2 (in particular a,c,d) is\n  quite hard to understand without jumping to the respective text passage.\n  Figures descriptions in general are lacking in detail (i.e., which are\n  nearest neighbours, and which is the analyzed sample in Figure 2).\n  Fig 2(b) has very small captions.\n\n- In the experiment in Figure 3: while the classifiers on MNIST/CIFAR10/Kather\n  are almost completely above 0.95, a PCC of 0.90 and slightly below for the\n  other datasets and models are not very insignificant, especially given that\n  these models are distilled in order to be explained. This should be discussed\n  in more detail.\n\n- The qualitative analysis of the explanations, both based on the\n  kernel-similarities, which show similarities and a CAM-based approach, is\n  difficult to interprete. While the samples generally seem to show other\n  samples from the same class somehow semantically related, these results are\n  not directly compared to any approach, making it hard to argue in favor of\n  supporting the explainability claim. The focus should be more on the\n  experiment in Fig 2b, ideally with more trials and different data sets.\n\n- The explainability approach is only compared to a single baseline, which is\n  too few.\n\n- While the experiment in Fig 2b presents a quantitative analysis of the\n  explainability of the method compared to one other baseline, it is only\n  conducted a single time, on only a single data set. Since the paper claims\n  explainability of the models, there should be more experiments in order to\n  support this claim.\n\n- While the paper claims to provide a tool, the linked software framework has\n  close to no docstrings, provides no documention, features no tests, and does\n  not provide a plug-and-play package (no setuptools).\n  It is good practice to publish accompanying code, and the paper goes further\n  by also including code for the corresponding experimental results.\n  However, at the reviewed state, the *tool* simply presents code to the\n  paper, rather than providing the software framework suggested in the\n  introduction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please see the attached global pdf.\n\n\n\n\\textcolor{blue}{\nThe paper reports on GPs, but is not very verbose in the fact that DNN-kernels are used, ....\n}\n\\\\In Sec. 2.2. we added the following sentence\n\\\\\\textit{\n.Note that the kernel functions $\\lbrace f_\\ell(.)\\rbrace_{\\ell=1}^L$ are implemented as separate neural network, or for the sake of efficiency as a single neural network backbone with $L$ different heads.\n\n}\n\n\n\\\\\\\\\\textcolor{blue}{\nPrior work on DNN-GPs is only partly discussed (e.g. [23]).\n}\n\\\\In the related work section, we added the following details about the SV-DKL method\n\\\\\\textit{\"\n... SV-DKL [23] derives a lower-bound for training a GP with a deep kernel. In that method, a grid of inducing points are considered in the kernel-space (like the vectors $\\lbrace(\\tilde{\\akvec{u}}^{(\\ell)}_{m}, \\tilde{v}^{(\\ell)}_m) \\rbrace_{m=1}^{M}$ with the notation of this paper).\nAfterwards, each input inst acne is firstly mapped to the kernel-space and the output is computed based on similarities to the grid points in the kernel-space.  ...\"\n}\n\\\\\\\\Besides SV-DKL, in the paper we have cited and discussed GPytorch [25], KISGP [31], and binary-tree kernels [4]. We also added the work of Wilson et. al. (ref. [35] at the end of this document) which actually is based on KISS-GP [31].\n\n\n\\\\\\\\\\textcolor{blue}{\nThe paper claims to introduce a novel approach to scale the number of inducing points. However, the computational trick comes down to ...\n}\n\\\\\\begin{itemize}\n    \\item The core idea of our method is storing the kernel-space representaitons in the matrix $\\mathbf{U}$ and involving only one row of $\\mathbf{U}$ in the computation graph (as done in line 6 of Alg. S1). This way of handling GP kernel is completely novel.\n    \\item Also doing knowledge-distillation between ANN and GP is novel.\n    \\item Although the low-rank approximation is a simple idea, it hasn't been applied to GP training before.\n\\end{itemize}\n\n\n\\textcolor{blue}{\nThe DNN-kernels are significantly more complex (resnet50)....\n}\n\\\\It might be the case that the GP kernel has to have more parameters than the ANN, but this is not a limitation at all. Instead, it is a question to be answered.\nPlease note that we cannot answer all questions in one paper, and that's why we have implemented GPEX as a publicly available tool with a simple API .  \n\\\\We added the following part to the paper. \n\\\\\\textit{\n... In this paper we analyzed the effect of number of inducing points, .... One can use the proposed tool to answer other questions, like, is the GP kernel required to have more parameters than the ANN itself, as observed in the experiments of this paper? Is the uncertainty provided by the GP correlated with the understandability of the explanations to humans or the ANN's failures? ...\n}\n\n\n\\\\\\\\\\textcolor{blue}{\nThe experiments are not presented very well: Figure 2 (in particular a,c,d) is quite hard to understand without jumping to the respective text passage. Figures descriptions in general are lacking in detail (i.e., which are nearest neighbours, and which is the analyzed sample in Figure 2). Fig 2(b) has very small captions.\n}\n\\\\Thank you. We added caption to all figures.\n\n\n\\\\\\\\\\textcolor{blue}{\nThe explainability approach is only compared to a single baseline, which is too few.\n}\n\\\\We added another baseline \"influence-function\". For details please refer to Sec. 1 of this document. Note that unlike attribution-based methods, there are not many similarity-based explanation methods. So we compared against representer point selection [33] and influence function [36].\n\n\n\\\\\\\\\\textcolor{blue}{\nWhile the paper claims to provide a tool, the linked software framework has close to no docstrings, provides no documention, features no tests, and does not provide a plug-and-play package (no setuptools). It is good practice to publish accompanying code, and the paper goes further by also including code for the corresponding experimental results. However, at the reviewed state, the tool simply presents code to the paper, rather than providing the software framework suggested in the introduction.\n}\n\\\\We added documentation via \"readthedocs\" as well as sample notebooks in the github repository. Please refer to the the anonymous github repo (link available in the submitted paper). \n\n\n\\\\\\\\\\textcolor{blue}{\nGiven the increased complexity of the kernel's DNN to the teacher model, a discussion on the complexity-explainability trade-off would be very interesting. Further experiments with smaller kernel DNN's in the distillation-process could also bring insight in this regard.\n}\n\\\\Great suggestion. We have done some parameter analysis on the effect of width of the last layer, number of inducing points, and the number of epochs for which the ANN is trained. Unfortunately we cannot answer all questions in one paper (the paper is already 80 pages long including the supplementary). We added this to future directions\n\\\\\\textit{\n... In this paper we analyzed the effect of number of inducing points, the width of the second last layer, and the number of epochs for which the ANN is trained. One can use the proposed tool to answer other questions, like, is the GP kernel required to have more parameters than the ANN itself, as observed in the experiments of this paper? Is the uncertainty provided by the GP correlated with the understandability of the explanations to humans or the ANN's failures? ...\n}\n\n\n\\\\\\\\\\textcolor{blue}{\nTo put the gained explainability into context, especially in the qualitative experiments, it would really help to compare to some baselines.\n}\n\\\\In Sec. S6 of the supplementary we have qualitatively compared our explanations to those of representer point selection [33]. Please refer to Sec. S6 for more details.\n\n\n\\\\\\\\\\textcolor{blue}{\nl.126 nominator -> numerator\n}\n\\\\corrected.\n\n\n\\\\\\\\\\textcolor{blue}{\nfigures should be described more in detail, especially the qualitative one\n}\n\\\\Thank you. We added captions to all figures."}, {"review_id": "XE3Ww9EsD5", "reviewer": "Reviewer_6BV2", "summary": "This paper derive an evidence lower-bound that encourages GP's posterior to match ANN's output without any requirement on ANN. And the uses the GPs' kernel functions to explain the ANNs' decisions.", "questions": "None", "limitations": "None", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1) this paper provides a theoretical way for the algorithm \n\n2) implementation is publically available\n\n3) it is good to see that GP output matches ANN well", "weaknesses": "1) The main point of this paper is to explain ANN. However, Figure 2 is the only experiment results that are about explanation, which only contains some sample explanations. There is no comparison to existing explanation methods and there is not quantitative results about the evaluation performance.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please see the attached global pdf.\n\n\n\\textcolor{blue}{\nThe main point of this paper is to explain ANN. However, Figure 2 is the only experiment results that are about explanation, which only contains some sample explanations. There is no comparison to existing explanation methods and there is not quantitative results about the evaluation performance.\n}\n\n\n\\\\In Sec. 4.3 of the original submission, we compare against reprenter point selection[33] both quantitatively and qualitatively. Moreover, in the global pdf we added comparison to another method called influence function [36]."}, {"review_id": "ttsQLwnzcy", "reviewer": "Reviewer_efq3", "summary": "This article proposes a new training approach for Gaussian Processes (GPs) which encourages their posterior to match a given Artificial Neural Network (ANN) output. Their approach adopt a scheme that permits a scalable training, which is an issue when it usually comes to GPs, and that is motivated by the derivation of an evidence lower bound (ELBO) encouraging the faithfulness of the GP to the target ANN. The authors then  explain the decisions made by an ANN with the help of a matching GP with two different methods : explanations by similar examples (similar in the kernel space of the GP), and using saliency map to determine why some examples where similar. ", "questions": "1. Aren’t those two sentences (« The analogy between Gaussian processes (GPs) and deep artificial neural networks (ANNs) has received a lot of interest, and has shown promise to unbox the blackbox of deep ANNs » (lines 1-3) and the third contribution, « With the best of our knowledge, our work is the first method that performs knowledge distillation between GPs and ANNs. ») contradictory?\n\n2. Can some details be provided concerning the kernel choice $f(\\cdot)$?\n\n3. Why choosing Pearson correlation coefficient for comparing GPs and ANNs? Is it what’s often seen in the literature? Why not comparing the coherence of the predictions, for two predictors can have similar prediction function but predict differently most of the time?", "limitations": "The authors did not discuss the limitations of their explainability approach. Nowadays, many problems with (for example) saliency maps are well-known (not robust to adversarial perturbations [1], simply unreliable [2], etc.), thus one would expect a few words on those matters.\n\n\n\n[1] Ghorbani, Amirata, Abubakar Abid, and James Zou. “Interpretation of neural networks is fragile.” Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.\n\n[2 ]Kindermans, Pieter-Jan, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. “The (un) reliability of saliency methods.” In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pp. 267-280. Springer, Cham (2019).", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. It is, to the knowledge of the authors, truly original to use a GP in order to explain a matching ANN. \n\n2. The approach for training GPs tackles a main issue of traditional approaches : scalability with regards to the number of inducing points.", "weaknesses": "_Major_\n\n1. My first concern, which could probably summarise many flags I have to raise, for it leads to several problems : the article is too dense. One consequence is that there are too many things putted in Supplementary material that are necessary to understand the article in its whole. For example, the way the CAM-like explanations (line 282) are computed is not even slightly described in the main article; or, « According to the experiments and detailed discussions of Sec. S6 in the supplementary, the GP’s kernel that we find in this paper is superior due to a technical point in the formulation of representer point selection » (lines 299-300); or the fact that the derived  ELBO is a contribution in itself, and isn’t even presented in the main paper, only pieces of it. The Supplementary material should not contain information that are vital to understanding the methods, the idea and the approaches, but further readings.\n\n2. When manipulating concepts such as explainability and interpretability (which seem, in the article, to be used interchangeably; lines 20-21 :  « After matching the GPs to the ANNs, we used the GPs’ kernel functions to explain the ANNs’ decisions. »; the title says « interpreting artificial neural networks », and on lines 75-76 : « GPEX can be used by machine learning researchers to interpret [...] their artificial neural networks. »), since they do not have a clear and universal definition, it is important to state what these terms refer to in the article, just like, in Section 2.1, some notations are defined. This lack of clearness makes it harder to grasp the goal of the article.\n\n3. If I understand correctly, as for the explainability approach, the way the nearest neighbours are found is by looking at the nearest neighbours, but in the kernel space. How does that differ from only computing the nearest neighbours in the output space of the network? How does that makes the neighbours that are found more relevant? Both the kernel and the network can be seen as an embedding. Plus, the GP's job is to imitate the network. It seems to me like this approach combines two explainability approaches : finding a « more interpretable » predictor imitating the predicting behaviour of the former, and finding examples that are similar to one another. Why is combining those two approaches, both having their flaws, would lead to any better results when it comes to explainations?\n\n4.1. I find the title misleading; explaining (interpreting, as the title says) ANNs would seem to be the main point of the article, but in reality, explaining ANNs concerns only page 8 and half of the last page, whereas training a GPs faithfully to a trained ANN really is the main point of the article. Plus, both the saliency map and looking at the most similar examples (which corresponds to approximate the decision function of the network by a more interpretable predictor : a k-nearest neighbours predictor) from the training set are not novel ideas.\n\n4.2. As for the related works, explaining using GPs is only discussed on ~3 lines, citing a single work.This might be the first time an explanation is based on a bridge between GPs and ANNs, but the authors stated that « Gaussian processes are highly interpretable » (lines 37-38) and that « The analogy between Gaussian processes (GPs) and deep artificial neural networks (ANNs) has received a lot of interest, and has shown promise to unbox the blackbox of deep ANNs » (lines 1-3); it is therefore expected to see « what kind of interest they have received », especially in the discussion / related work sections of the article.\n\n5. One of the contribution is that the proposed approach is more scalable when compared to previous approaches. Without any empirical comparison with state-of-the-art for training GPs, both in terms in training time and performances, it is hard to state whether the proposed approach works particularly well or if it is just have effective as what is found in the literature. That is also why the experiments ran in Section 4.1 could be misleading. Since no benchmark is used to compare the proposed approach to, how can one know if the obtained results are good or not? This lack of information makes it hard to state whether or not the results are positive.\n\n6. I found some claims to be vague, doubtful or at least are highly debatable, but most importantly sometimes not supported at all; lines 37-38 : « Gaussian processes are highly interpretable. » How are they more interpretable? In what way? Doesn’t that depend on the kernel that is used? The same goes for lines 207-208 : « Because according to theoretical results on GP-ANN analogy, the second last layer of ANN should be wide. » This might be true, but there is a need to cite some works here. Also : « Our method scales very well, and Alg. 2 runs fine even on imagenet with more than 1M inducing points » (lines 343-344); what’s « fine » mean? (See also Weaknesses – Major – 5)\n\n7. It is said that experiments were conducted on « 5 datasets (4 image datasets, and 1 biological dataset) » (lines 17-18), but nothing in the article is discussed about the biological dataset.\n\n_Typos / Minor_\n\n1. Line 85 : « The number of GPs is equal to the number of $\\underline{the}$ outputs from the ANN. »\n\n2. Some nomenclature and terminology is unclear. For example, line 91-92 : « we consider a general feed-forward pipeline that contains an ANN as a submodule »; what’s a feed-forward pipeline? What is a submodule? Figure 1a) somewhat doesn’t help understanding. (See also Weaknesses – Major - 2)\n\n3. Equation 5 : Is there a missing term?: what is the expectation computed with regards to? (?~q)\n\n4. Line 164 : « procecdure »\n\n5. Line 167 : « matrcies »\n\n6. Line 179 : « from linear algebra if follows that »\n\n7. Line 327 : requries\n\n8. Having the Related Work at the end of the article is peculiar; it seems to me like precious information for better understanding the relationship between ANNs and GPs during the reading of the article (and better understanding the relevance of the work) are put at the end of the article.\n\n9. It should be explicitely written that the kernel function $\\mathcal{K}(\\cdot, \\cdot)$ outputs a matrix whose dimensions correspond respectively to the cardinality of the first and the second input, for it is implicit, but not explicit yet important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please see the attached global pdf.\n\n\\textcolor{blue}{\nThe article is too dense.\n}\n\\\\The paper is dense, because it contains two main ideas. 1. Scalability and 2. Knowledge distillation. Without scalability, successful knowledge distillation wouldn't have happened  (as underlined in the analysis of Fig. S66). So we had to include both scalability and knowledge distillation in one paper.\n\n\n\\\\\\textcolor{blue}{\n When manipulating concepts such as explainability and interpretability ... This lack of clearness ...\n}\\\\\nWe added the following paragraph to the paper, that describes how the GP can explain the ANN.\n\\\\\\textit{\n    ... After successful distillation, both the ANN and the GP correspond to a single function which is parameterized in two different ways. One is the parameterization by ANN weights, which is not necessarily understandable to humans. The other is the parameterization by GP's posterior which is understandable to humans. ... \n}\n\\\\As you mentioned, \"interpretability\" and \"explainability\" are used to convey different meanings in the literature. But in this paper we use the terms interchangeably. We added the following to the paper\n\\\\\\textit{\n... In this paper we used the terms interpretability and explainability interchangeably. ...\n}\n\n\n\\\\\\textcolor{blue}{\n  How does that differ from only computing the nearest neighbours in the output space of the network?\n}\\\\\nIf the ANN is a classifier with $C$ classes, in the final layer instances from different classes are supposed to be close to one another. For example all instances of class 0 are near the vector $[1.0, 0.0, ..., 0.0]$,  all instances of class 1 are near the vector $[0.0, 1.0, ..., 0.0]$, ..., all instances of class $C$ are near the vector $[0.0, 0.0, ..., 1.0]$. So the final layer is not a good candidate to compute the similarities. But an interesting question is, what is someone computes the similarity using intermediate layers of the neural network?\\\\\nWe cannot answer all questions in one paper (the paper is already 80 pages long with supplementary), so we implemented GPEX as a tool that researchers can use to answer these questions. In the paper we added the below paragraph about future directions to answer more questions. \n\\\\\\textit{\n... In this paper we analyzed the effect of number of inducing points, ... One can use the proposed tool to answer other questions, like, is ... . How the similarities provided by GP kernel compares to the similarities obtained from intermediate layers of the ANN? ...\n}\n\n\n\\\\\\textcolor{blue}{\n   Explaining ANNs concerns only page 8 and half of the last page, whereas training a GPs faithfully to a trained ANN really is the main point of the article. \n}\\\\\nThis is because Gaussian process is a white-box model with well-known behaviour. Therefore, the hard part is finding a GP which is equivalent to the ANN and the rest (obtaining explanations by looking at the GP) is straightforward.\\\\\n\n\n\\\\\\textcolor{blue}{\n   As for the related works, explaining using GPs is only discussed on $\\approx$3 lines, citing a single work.This m\n}\\\\\nThank you. We extended the related works section by providing more details about SV-DKL [23]. Moreover, we added another method (ref [35] below) to the paper.\n\\textit{\"\n... SV-DKL [23] derives a lower-bound for training a GP with a deep kernel. In that method, a grid of inducing points are considered in the kernel-space (like the vectors $\\lbrace(\\tilde{\\akvec{u}}^{(\\ell)}_{m}, \\tilde{v}^{(\\ell)}_m) \\rbrace_{m=1}^{M}$ with the notation of this paper).\nAfterwards, each input inst acne is firstly mapped to the kernel-space and the output is computed based on similarities to the grid points in the kernel-space. Since the GP posterior is computed via the grid points, SV-DKL [23] is scalable. But unfortunately the number of grid points cannot be increased to above 1000 even for Cifar10 \\cite{ds_cifar10} and with a RTX 3090 GPU. Therefore, this may limit the flexiblity of the GP's posterior [32]. ...\"\n}\n\n\n\\\\\\textcolor{blue}{\n   Gaussian processes are highly interpretable. How are they more interpretable? \n}\\\\\nGaussian process is definitely a white-box model. \n\n\n\\\\\\textcolor{blue}{\n   Aren’t those two sentences (...) contradictory? \n}\nIn summary, \n - sentence 1: GP-ANN analogy has shown promise before.\n  - sentence 2: we are the first method applying knowledge distillation to achieve GP-ANN anology.\n\nThe promise in sentence 1 has been in other ways than knowledge distillation. So there is no contradiction between the two sentences. For example, in neural tangents [20] the neural network is transformed into a GP by putting each and every single layer wide and training on a specific loss (not knowledge distillation). \n\n\n\\\\\\textcolor{blue}{\n   Can some details be provided concerning the kernel choice $f(.)$?\n}\nThank you. We added the following sentence to the paper to clarify\n\\\\\\textit{\n... A feature point like $\\akvec{x}$ is first mapped to the kernel-space as $\\akvec{u}^{(\\ell)} = f_{\\ell}(\\akvec{x})$. Note that the kernel functions $\\lbrace f_\\ell(.)\\rbrace_{\\ell=1}^L$ are implemented as separate neural network, or for the sake of efficiency as a single neural network backbone with $L$ different heads.\nAfterwards, the GP's posterior on $\\akvec{x}$ depends on the kernel similarities between. ...\n}\n\n\\\\\\textcolor{blue}{\n   The authors did not discuss the limitations of their explainability approach.\n}\nWe added the following point to highlight the limitations of the proposed method.\n\\\\\\textit{\n... . Although the obtained GP-kernel is faithfull globally, the number of nearest neighbours that equally affect the predictions may prevent a human from understanding the expalanations. Moreover, the CAM-like explanations that we obtain might be prone to the known problems of attribution-based methods. ...  \n}."}, {"review_id": "u8sWdCXbkP", "reviewer": "Reviewer_ycEM", "summary": "This paper uses Gaussian Processes trained to match Neural Networks to subsequently \"explain\" the NN outputs by finding the nearest neighbors to any given test point in the training samples. An evidence lower-bound is derived that encourages the GP’s posterior to match the NN’s output. Scalability is obviously an issue for GPs. The authors use GPU acceleration techniques to enable training GPs with O(100k) inducing points. Various example experiments are shown where the 10 nearest-neighbors in training illuminate why the NN responds the way it does to a test example. \n", "questions": "Have you looked at examples in regions where the GP error band is large? \n\nAssuming it's true that explanations there are less reliable, can you return info using the size of the GP error at x_test that flags this for the user? \n\nIt's nice that you can interrogate the NN using your GP to explain specific examples, but it's not obvious (to me) how one would systematically study a NN in this way, since a human can only visually inspect a small number of examples as is done in the paper. Any comments / thoughts on this?\n", "limitations": "The Limitations paragraph is super concise (presumably due to space constraints), but it'd be good to expand it a bit and make it more clear. E.g., it talks about how the method runs well with O(1M) points, but that the GP fails to match the NN. As written this is hard to interpret. \n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method seems straightforward and intuitive. The GP matching is less aggressive where the uncertainty is large (far from any training points). The examples shown seem to demonstrate that the method appears to work on these examples. \n", "weaknesses": "I didn't find any mention about using the GP error bands in the explanations. Presumably the explanations are not as good in regions where the GP and NN are less closely matched, which should be the case where GP error bands are large since the method doesn't weight these regions as much in the loss. I.e., far from any training points, it's not obvious that the nearest neighbors chosen by the GP kernels are really driving the NN response. This isn't discussed at all as far as I can see. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please see the attached global pdf."}]

Author rebuttal:
the global response

Meta-review: The results of this paper should be of interest to the NeurIPS community. Having read the reviews, rebuttal and extensive discussion, there are a number novel and technically significant contributions, from distilling an NN to GPs, to interpretability applications and the inclusion of comparisons to influence functions in the rebuttal. The experiments + code contribution will also be a useful resource for the community to build on.
Decision: Accept (poster)