Title: GPEX, A Framework For Interpreting Artificial Neural Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Gaussian processes

Explainable AI
Abstract: The analogy between Gaussian processes (GPs) and deep artificial neural networks (ANNs) has received a lot of interest, and has shown promise to unbox the blackbox of deep ANNs. Existing theoretical works put strict assumptions on the ANN (e.g. requiring all intermediate layers to be wide, or using specific activation functions). Accommodating those theoretical assumptions is hard in recent deep architectures, and those theoretical conditions need refinement as new deep architectures emerge. In this paper we derive an evidence lower-bound that encourages the GP's posterior to match the ANN's output without any requirement on the ANN. Using our method we find out that on 5 datasets, only a subset of those theoretical assumptions are sufficient. Indeed, in our experiments we used  a normal ResNet-18 or feed-forward backbone with a single wide layer in the end. One limitation of training GPs is the lack of scalability with respect to the number of inducing points. We use novel computational techniques that allow us to train GPs with hundreds of thousands of inducing points and with GPU acceleration. As shown in our experiments, doing so has been essential to get a close match between the GPs and the ANNs on 5 datasets. We implement our method as a publicly available tool called GPEX: https://github.com/amirakbarnejad/gpex. On 5 datasets (4 image datasets, and 1 biological dataset) and ANNs with 2 types of functionality (classifier or attention-mechanism) we were able to find GPs whose outputs closely match those of the corresponding ANNs. After matching the GPs to the ANNs, we used the GPs' kernel functions to explain the ANNs' decisions. We provide more than 200 explanations (around 30 in the paper and the rest in the supplementary) which are highly interpretable by humans and show the ability of the obtained GPs to unbox the ANNs' decisions.

GPEX, A Framework For Interpreting Artificial Neural Networks

Abstract

The analogy between Gaussian processes (GPs) and deep artificial neural networks (ANNs) has received a lot of interest, and has shown promise to unbox the blackbox of deep ANNs. Existing theoretical works put strict assumptions on the ANN (e.g. requiring all intermediate layers to be wide, or using specific activation functions). Accommodating those theoretical assumptions is hard in recent deep architectures, and those theoretical conditions need refinement as new deep architectures emerge. In this paper we derive an evidence lower-bound that encourages the GP's posterior to match the ANN's output without any requirement on the ANN. Using our method we find out that on 5 datasets, only a subset of those theoretical assumptions are sufficient. Indeed, in our experiments we used a normal ResNet-18 or feed-forward backbone with a single wide layer in the end. One limitation of training GPs is the lack of scalability with respect to the number of inducing points. We use novel computational techniques that allow us to train GPs with hundreds of thousands of inducing points and with GPU acceleration. As shown in our experiments, doing so has been essential to get a close match between the GPs and the ANNs on 5 datasets. We implement our method as a publicly available tool called GPEX: https://github.com/amirakbarnejad/gpex. On 5 datasets (4 image datasets, and 1 biological dataset) and ANNs with 2 types of functionality (classifier or attentionmechanism) we were able to find GPs whose outputs closely match those of the corresponding ANNs. After matching the GPs to the ANNs, we used the GPs' kernel functions to explain the ANNs' decisions. We provide more than 200 explanations (around 30 explanations in the paper and the rest in the supplementary) which are highly interpretable by humans and show the ability of the obtained GPs to unbox the ANNs' decisions.

1 Introduction

Artificial neural networks (ANNs) are widely adopted in machine learning. Despite their benefits, ANNs are known to be black-box to humans, meaning that their inner mechanism for making predictions is not necessarily interpretable/explainable to humans. ANN's black-box property impedes its deployment in safety-critical applications like medical imaging or autonomous driving, and makes them hard-to-troubleshoot for machine learning researchers.

Attribution-based explanation methods like LIME[31], SHAP[21] and most gradient-based explanation methods like DeepLIFT [5] presume a linear surrogate model. Given a test instance x test , this simpler surrogate model is encouraged to have the same output "locally" around x test . Because of this "local assumptions", explanations from these methods might be unreliable, and can be easily manipulated by an adversary model [11][28]. Moreover, these models may produce discordant explanations for a fixed model and test instance [16].

Considering Gaussian processes (GPs) [26] as the explainer model is beneficial, because: 1. Gaussian processes are highly interpretable. 2. Researchers have long known that GP's posterior has the potential to match an ANN's output "globally". More precisely, given an ANN and some requirements on it [24][8], there might exist a GP whose posterior matches the ANN's output all over the inputspace X (as opposed to the local explanation models for which the match happens only locally around a test instance x test ∈ X ). Not many explainer models can globally match the ANN's output. Among gradient-based methods, with the best of our knowledge only Integrated Gradients [33] has a weak sense of ANN's global behaviour over the input space. Having some conditions on an ANN, representer point selection [37] finds a "globally faithfull" explainer model that, similar to GPs, works with a kernel function. As we will elaborate upon in Sec. 4.3 and Sec. S6 of the supplementary, the GP's kernel that we find in this paper is superior due to a technical point in the formulation of representer point selection [37]. All in all, using GPs to explain ANNs is quite promissing and has advantages over other approaches to explain ANNs.

The contributions of this paper are as follows:

Theoretical results on ANN-GP analogy impose some restrictions on ANNs under which the ANN will be equivalent to a GP. These conditions are too restrictive for recently used deep architectures. Moreover, those theoretical conditions need refinement as new deep architectures emerge. In this paper we derive an ELBO for training GPs which encourages GP's posterior to match ANN's output. Our formulation and method doesn't impose any restriction on the ANN and the method used to train it.

Using our method, we empirically show that on 5 datasets (4 image datasets, and 1 biological dataset) and ANNs with 2 types of functionality (classifier or attention-mechanism) the ANNneeds to fulfill only a subset of those theoretical conditions. Indeed, in our experiments we used a normal ResNet-18 or feed-forward backbone with a single wide layer in the end.

Scalability is a major issue in training GPs. To address this issue, we adopted computational techniques recently used for fast spectral clustering [13] as well as a novel method to learn the GPs using mini-batches of inducing points and training instances. These computational techniques allow us to train GPs with hundreds of thousands of inducing points. According to our analysis, increasing the inducing points has been essential to get a good match between the trained GPs and ANNs. Indeed, without many inducing points GPs posterior cannot be a complex function (a function with many ups and downs [36]) and fails to match ANNs' output.

With the best of our knowledge, our work is the first method that performs knowledge distillation between GPs and ANNs.

We implement our method as a public python library called GPEX (Gaussian Processes for EXplaining ANNs). GPEX takes in an arbitrary PyTorch module, and replaces any ANN submodule of choice by GPs. Our package makes use of GPU-accelaration, and enables effortless application of GPs without getting users involved in details of the inference procedure. GPEX can be used by machine learning researchers to interpret/troubleshoot their artificial neural networks. Moreover, GPEX can be used by researchers working on the theoretical side of ANN-GP analogy to empirically test their hypotheses.

2 Proposed Method

2.1 Notation

In this article the function g ( . ) always denotes an ANN. The kernel of a Gaussian process is denoted by the double-input function K ( ., . ) . We assume the kernel similarity between two instances x i and x j is equal to f ( x i ) T f ( x j ) , where f ( . ) maps the input-space to the kernel space. In this paper u (resp. v ) denotes a vector in the kernel-space (resp. the posterior mean) of a GP. In some sense u and v denote

a

-5.0

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

-5.0

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

Observed Data

GP's mean

GP's mean ± covariance

b

the input and the output of a GP, respectively. We have that: K ( x i , x j ) = f ( x i ) T f ( x j ) = u i T u j . The number of GPs is equal to the number of the outputs from the ANN. In other words, we consider one GP per scalar output-head from the ANN. We use index ℓ to specify the ℓ -th GP as follows: K ℓ ( x i , x j ) = f ℓ ( x i ) T f ℓ ( x j ) = u i ( ℓ ) T u j ( ℓ ) . We parameterize the ℓ -th GP by a set of M inducing points { (˜ u ( ℓ ) m , ˜ v ( ℓ ) m ) } M m =1 . The tilde in (˜ u ( ℓ ) m , ˜ v ( ℓ ) m ) indicates that ˜ u is one of the M inducing points in the kernel space. However, u (without tilde) can be an arbitrary point in the continuous kernel-space.

2.2 The Proposed Framework

To make our framework as general as possible, we consider a general feed-forward pipeline that contains an ANN as a submodule. In Fig. 1a the bigger square illustrates the general module. The input-output of the general pipeline are denoted in Fig. 1a by X and Y . The general pipeline has at least one ANN submodule to be explained by GPEX. Fig. 1a illustrates this ANN by the small blue rectangle within the general pipeline. The input-output of the ANN are denoted in Fig. 1a by x and v . Note that X and Y can be anything, including without any limitation, a set of vectors, labels, and meta-information. However, input-output of the ANN (i.e. x and y ) are required to be in tensor format. The exact requirements are provided in the online documentation for GPEX. Moreover, the general module can have other arbitrary submodules, which are depicted by the blue clouds. The relations between the submodules, as illustrated by the dotted-lines in Fig. 1a, can also be quite general. Our probabilistic formulation only needs access to the conditional distributions p ( x |X ) and p ( Y| x , X ) . Similarly, the proposed GPEX is completely agnostic about the general pipeline and it only requires the ANN's input-output to be in the tensor format. Given a PyTorch module, the proposed GPEX tool automatically grabs the distributions p ( x |X ) and p ( Y| x , X ) from the main module it is given.

The inducing points { ˜ u ( ℓ ) m , ˜ v ℓ m } M m =1 parameterize the ℓ -th GP. Note that ˜ u ( ℓ ) m = f ℓ (˜ x m ) . A feature point like x is first mapped to the kernel-space as u ( ℓ ) = f ℓ ( x ) . Note that the kernel functions { f ℓ ( . ) } L ℓ =1 are implemented as separate neural networks, or for the sake of efficiency as a single neural network backbone with L different heads. Afterwards, the GP's posterior on x depends on the kernel similarities between u ( ℓ ) and the inducing points { ˜ u ( ℓ ) m } M m =1 . More precisely, the posterior of the ℓ -th GP on x is a random variable v ( ℓ ) whose distribution is as follows [26]:

where µ v ( ., ., . ) and cov v ( ., ., . ) are the mean and covariance of a GP's posterior computed as:

and

As the variables { v ( ℓ ) m } M m =1 and v are latent or hidden, we train the model parameters by optimizing a variational lower-bound. We consider the following variational distributions:

In Eq. 4, the function g ℓ ( . ) is the ℓ -th output from the ANN. Note that as the set of hidden variables { ˜ v ( ℓ ) m } M m =1 is finite, we have parameterized their variational distribution by a finite set of numbers { φ ( ℓ ) m } M m =1 . However, as the variables x can vary arbitrarily in the feature space, the variable u ( ℓ ) varies arbitrarily in the kernel space. Therefore, the set of values v ( ℓ ) may be infinite. Accordingly, the variational distribution for v ( ℓ ) is conditioned on x and is parameterized by the ANN g ( . ) .

2.3 The Derived Evidence Lower-Bound (ELBO)

Due to space limitation, the derivation of the lower-bound is moved to Sec. S1 of the supplementary material. In this section we only introduce the derived ELBO and discuss how it relates the GP, the ANN and the training cost of the main module in an intuitive way. The ELBO terms containing the GP parameters (i.e. the parameters of the kernel function f ( . ) ) is denoted by L gp . According to Eq. S9 of the supplementary material L gp is as follows:

where q ( . ) is the variational distribution that factorizes to the q 1 ( . ) and q 2 ( . ) distributions defined in Eq. 4. In the first term of Eq. 5, the numerator encourages the GP and the ANN to have the same output. More precisely, for a feature point x we can compute the corresponding point in the kernel space as u ( ℓ ) = f ℓ ( x ) and then compute the GP's posterior mean based on kernel similarities between u and the inducing points to get the GP's mean µ v . In Eq. 5 the GP's mean µ v is encouraged to match the ANN's output g ℓ ( x ) . In Eq. 5, because of the denominator of the first term, the ANN-GP similarity is not encouraged uniformly over the feature-space. Wherever the GP's uncertainty is low, the term cov v ( u ( ℓ ) , ˜ u ( ℓ ) 1: M , ˜ v ( ℓ ) 1: M ) in the denominator becomes small. Therefore, the GP's mean is highly encouraged to match the ANN's output. On the other hand, in regions where the GP's uncertainty is high, the GP-ANN analogy is less encouraged. This formulation is quite intuitive according to the behaviour of Gaussian processes. Fig. 1b illustrates the posterior of a GP with radial-basis kernel for a given set of observations. In regions like [3 , ∞ ) and ( -∞ , -4] there are no nearby observed data. Therefore, in these regions the GP is highly uncertain and the blue uncertainty margin is thick in such regions. Intuitively, our derived ELBO in Eq. 5 encourages the GP-ANN analogy only when GP's uncertainty is low and gives less importance to regions similar to [3 , ∞ ) and ( -∞ , -4] in Fig. 1b. Note that this formulation makes no difference for the ANN as ANNs are known to be global approximators. However, this formulation makes a difference when training the GP, because the GP is not required to match the ANN in regions where there are no similar training instances. The ELBO terms containing the ANN parameters is denoted by L ann . According to Sec. S1.2 of the supplementary material, L ann is as follows:

In the above objective the first term encourages the ANN to have the same output as the GP. Similar to the objective of Eq. 5, the denominator of the first term gives more weight to ANN-GP analogy when GP's uncertainty is low. In the right-hand-side of Eq. 6, the second term is the likelihood of the pipeline's output(s), i.e. Y in Fig. 1a. This term can be, e.g., the cross-entropy loss when Y contains class scores in a classification problem, or the mean-squared error when Y is the predicted value for a regression problem, or a combination of those costs in a multi-task setting.

3 Algorithm

We consider a separate Gaussian process for each output head of an ANN. In other words, given an ANN we have as many GPs as the number of the ANN's output heads. To explain an ANN, we find the explainer GPs by optimizing the objective in Eq. 5 w.r.t. to the kernel mappings { f ℓ ( . ) } L ℓ =1 . To do so, we need to have µ v which in turn means we need to have all kernel-space representations

Algorithm 1 Method Optim_KernMappings

Input: Input instance x and inducing instance ˜ x , list of matrices U , list of vectors V . Output: Kernel-space mappings [ f 1 ( . ) , ..., f L ( . )] . Initialisation : loss ← 0 .

1: µ , cov ← forward_GP( x , ˜ x , U , V ) //feed x to GPs, "forward_GP" is Alg.S1 in supplementary.

2: µ ann ← g ( x ) //feed x to ANN.

3: for ℓ = 1 to L do

4: loss ← loss + ( µ [ ℓ ] -µ ann [ ℓ ]) 2 + σ 2 g cov [ ℓ ] +log( cov [ ℓ ]) . //Eq.5.

5: end for

6: δ ← ∂ loss ∂ params ( [ f 1 ( . ) ,...,f L ( . )] ) .//the gradient of loss.

7: params ( [ f 1 , ..., f L ] ) ← params ( [ f 1 , ..., f L ] ) -lr × δ //update the parameters.

8: lr ← updated learning rate

9: return [ f 1 ( . ) , ..., f L ( . )]

Algorithm 2 Method Explain_ANN

Input: Training dataset ds _ train , and the inducing dataset ds _ inducing . Output: Updated kernel-space mappings [ f 1 ( . ) , ..., f L ( . )] , and the other GP parameters U and V . Initialisation : U , V ← Init_GPparams(ds_inducing) //Alg.S3 in supplementary.

4: [ f ( . ) , ..., f ( . )] ← Optim_KernMapings( x , ˜ x , U , V ).

1: for iter = 1 to max _ iter do 2: x ← randselect ( ds _ train ) . 3: ˜ x ← randselect ( ds _ inducing ) 1 L 5: ˜ x ← randselect ( ds _ inducing ) . 6: for ℓ = 1 to L do 7: //update kernel-space representations. 8: U [ ℓ ][ ˜ x .index ] ← f ℓ ( ˜ x ) 9: end for 10: end for 11: return [ f 1 ( . ) , ..., f L ( . )] , U , V

{ ˜ u ( ℓ ) m } M m =1 . However, it is computationally prohibitive to feed thousands of inducing instances to the kernel mappings as ˜ u ( ℓ ) m = f ℓ ( ˜ x m ) for m ∈ { 1 , 2 , ..., M } in each gradient-descent iteration. On the other hand, as the kernel-space mappings { f ℓ ( . ) } L ℓ =1 keep changing during training, we need to somehow track how the inducing points { ˜ u ( ℓ ) m } M m =1 change during training. To this end, we put the kernel-space representations of the inducing points in matrices denoted by U . During training, these matrices are repeatedly updated by feeding mini-batches of inducing instances to the kernel-mappings.

Alg. 2 optimizes the objective of Eq. 5 w.r.t. the kernel mappings { f ℓ ( . ) } L ℓ =1 . First, a single training instance x and a single inducing point ˜ x are selected (line 2-3). Afterwards, the procedure of Alg. 1 is called to update the kernel mappings (line 4 of Alg. 2). To update the kernel-mappings, the GP posterior is computed via the matrices U (line 1 of Alg.1). The "forward_GP" procedure (called in line 1 of Alg. 1) is provided in Alg. S1 of the supplementary, and uses the matrices U to compute GP's posterior. Only the rows of U that correspond to the selected inducing point ˜ x are computed using the kernel-mappings, so that the gradient w.r.t. the kernel-mappings can be computed in the backward pass (lines 5-6 of Alg. S1 in the supplementary). Finally, the matrices U are updated (lines 6-8 of Alg. 2). Due to the lack of space, the routines "forward_GP" and "Init_GPparams" and more details are moved to Sec. S2 of the supplementary. Of course instead of a single training/inducing instance, we used a mini-batch of multiple training/inducing instances.

One difficulty of training GPs is the matrix inversion of Eqs.2 and 3, which has O ( M 3 ) complexity using standard matrix inversion methods. To address this issue, we adopted computational techniques recently used for fast spectral clustering [13]. Let A be an arbitrary M × D matrix where M >> D . Moreover, let b be a M -dimensional vector and let σ be a scalar. The computational techniques [13] allow us to efficiently compute: ( AA T + σ 2 I M × M ) -1 b . (Note the similar terms in the right hand side of Eqs. 2 and 3.) The idea is that AA T is of rank D . Therefore, from linear algebra it follows

that ( AA T + σ 2 I M × M ) -1 has M -D eigen-values all of which are equal to σ -2 . Therefore, in the space of those eigen-vectors, the transformation on b is simply a scaling by σ -2 . The details and more computational techniques are provided in Sec. S2.1 of the supplementary. These computational techniques allow us to efficiently compute the GP-posterior for hundreds of thousands of inducing points in each gradient descent iteration.

A note on the used datasets in Alg. 2: According to our analysis of Sec. S7 in the supplementary material, "ds_inducing" should be as large as possible so the GP posteriors can be flexible enough to match the ANNs. Therefore a good practice is to include all training instances (without data augmentation) in "ds_inducing". By doing so, the following issue arises. An instance from "ds_train" like x is an augmented version of an inducing instance ˜ x . Because x and ˜ x are close, their kernel-space representations f ( x ) and f (˜ x ) also become close regardless of parameters of f ( . ) . Consequently, regardless of f ( . ) , GP's posterior mean will be roughly equal for both x and ˜ x . Indeed, in this case Alg. 2 fails to find the kernel mappings { f ℓ ( . ) } L ℓ =1 . To avoid this issue, we sample x in line 2 of Alg. 2 as follows: x 1 and x 2 are randomly selected from "ds_train", and α ∼ uniform ( -1 , 2) , and x = α x 1 +(1 -α ) x 2 . The rest of Alg. 2 after line 2 is run as before.

4 Experiments

We conducted several experiments on four publicly available datasets: MNIST [9], Cifar10 [19], Kather [15], and DogsWolves [34]. For MNIST [9] and Cifar10 [19] we used the standard split to training and test sets provided by the datasets. For Kather [15] and DogsWolves [34] we randomly selected 70% and 80% of instances as our training set. The exact parameter settings for running Alg. 2 are elaborated upon in Sec. S5 of the supplementary. We trained the ANNs as usual rather than using Eq. 6, because our proposed GPEX should be applicable to ANNs which are trained as usual.

4.1 Measuring Faithfulness of GPs to ANNs

Wetrained a separate convolutional neural network (CNN) on each dataset to perform the classification task. For MNIST [9], Cifar10 [19], and Kather [15] we used a ResNet-18 [12] backbone followed by some fully connected layers. DogsWolves [34] is a relatively small dataset, and very deep architectures like ResNet [12] overfit to training set. Therefore, we used a convolutional backbone which is suggested in the dataset website [34]. For all datasets, we set the width (i.e. the number of neurons) of the second last fully-connected layer to 1024. Because according to theoretical results on GP-ANN analogy, the second last layer of ANN should be wide. We used an implementation of ResNet [12] which is publicly available online [2]. We trained the pipelines for 20, 200, 20, and 20 epochs on MNIST [9], Cifar10 [19], Kather [15], and DogsWolves [34], respectively. For Cifar10 [19], we used the exact optimizer suggested by [2]. For other datasets we used an Adam [17] optimizer with a learning-rate of 0 . 0001 . The test accuracies of the models are equal to 99.56%, 95.43%, 96.80%, and 80.50% on MNIST [9], Cifar10 [19], Kather [15], and DogsWolves [34], respectively. We also applied our proposed GPEX to a state-of-the-art cell-embedding method called scArches [20]. We ran a tutorial notebook [32] and applied GPEX to the decoder whose job is to predict expression of some genes given scArches [20] cell embeddings. More details are provided in our public github repository (repository link is provided in page 1).

We explained each classifier ANN using our proposed GPEX framework (i.e. Alg.2). As discussed in Sec. 3, given an ANN we have as many kernel-spaces (and as many GPs) as the number of ANN's output heads. The exact parameter settings and practical considerations for training the GPs is elaborated upon in Sec. S5 of the supplementary material. To measure the faithfulness of GPs to ANNs, we compute the Pearson correlation coefficient for each ANN head and the mean of the corresponding GP posterior on unseen test instances. The results are provided in Fig. 3. In Fig. 3, the first five groups of bars (i.e. the groups labeled as Cifar10 (classifier), MNIST (classifier), Kather (classifier), DogsWolves (classifier), and scArches (classifier)) correspond to applying the proposed GPEX to the five classifier ANNs trained on the four datasets and scArches embeddings. According to Fig. 3, our trained GPs almost perfectly match the corresponding ANNs. Only for DogsWovles [34], as illustrated by the 4-th bar group in Fig. 3, the correlation coefficients are lower compared to other datasets. We hypothesize that this is because the DogsWolves dataset [34] has very few images. GP posterior mean can be changed only by moving the inducing points in the kernel-space.

(a)

(b)

(c)

(d)

力

4

H

4

y

4

4

4

中

D

9

GPEX

Therefore, when very few inducing points are available GP posterior mean is less flexible [36]. This is consistent with our parameter analysis in Sec. S7 of supplementary material.

In Fig. 1a we discussed that GPEX is not only able to explain a classifier ANN, but it can explain any ANN which is a subcomponent of any feed-forward pipeline. To evaluate this ability, we trained three classifiers with an attention mechanism [22]. Each classifier has two ResNet-18 [12] backbones: one extracts a volumetric map containing deep features, and the other produces a spatial attention mask. For each attention backbone, we set the width of the second last layer to 1024, followed by a linear layer and sigmoid activation. We applied our proposed GPEX (i.e. Alg.2) to each classifier, but this time the ANN to be explained (i.e. the box called "ANN" in Fig. 1a) is set to be the attention submodule. Note that each attention backbone produces a spatial attention mask of size h by w . We think of each attention backbone as an ANN which has h × w output heads. We trained three classifier pipelines with attention mechanism on Cifar10 [19], MNIST [9], and Kather [15] with the same training procedure as previous part. In Fig. 3, 6-th, 7-th, and 8-th bar groups show the correlation coefficients between the attention backbones and the corresponding GPs on unseen test instances. According to Fig. 3, our proposed GPEX is able find GPs which are faithful to attention subcomponents of the classifier pipelines. Note that we didn't include all attention heads, because

Correlation coefficient between GP and ANN

1.0

pea

pea

peay

01 peaq

peau

Z peau

peau

pea

Ci peay

9 pea

+S peau

11 pe9q

peat

16 peay

Z peaq

001 peaq

0.95

S peaq

0.9

0.85

0.8

Cifar10

MNIST

Kather

DogsWolvesscArches

Cifar10

MNIST

Kather

(classifier)

(classifier)

(classifier)

(classifier)(classifier)(attention)

(attention)

(attention)

some pixels in attention masks are always off. In Sec. S3 of the supplementary material we have included more information and insights about the faithfulness of GPs to ANNs.

4.2 Explaining ANNs' Decisions

In Sec. 4.1 we trained four CNN classifiers on Cifar10 [19], MNIST [9], Kather [15], and DogsWolves [34], respectively. Afterwards, we applied our proposed explanation method to each CNN classifier. In this section, we are going to explain the decisions made by the classifiers via the obtained GPs found by Alg. 2. We explain the decision made for a test instance like x test as follows. We consider the GP and the kernel-space that correspond to the ANN's head with maximum value (i.e. the ANN's head that relates to the predicted label). Consequently, among the instances in the inducing dataset, we find the 10 closest instances to x test , like { x i 1 , x i 2 , ..., x i 10 } . Intuitively the ANN has labeled x test in that way because it has found x test to be similar to { x i 1 , x i 2 , ..., x i 10 } .

For MNIST digit classification, some test instances and nearest neighbours in training set are shown in Fig. 2a. In this figure each row corresponds to a test instance. The first column depicts the test instance itself and columns 2 to 11 depict the 10 nearest neighbours. For example, in Fig. 2a the image in row3-col1 depicts a test instance x test and the images in row3, cols2-11 depict the nearest neighbours { x i 1 , x i 2 , ..., x i 10 } . According to rows 1 and 2 of Fig. 2a, the classifier has labeled the two images as digit 1 because it has found 1 digits with similar inclinations in the training set (in Fig. 2a in row 1 all digits are vertical but in row 2 all digits are inclined). We see the model has also taken the inclination into account for the test instances of rows 7, 8, 15, 16, and 17 of Fig. 2a. In Fig. 2a, according to rows 3, 4, and 5 the test instances are classified as digit 2 because 2 digits with similar styles are found in the training set. We see the model has also taken the style into account for the test instances of rows 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, and 17 of Fig. 2a. For instance, the test instance in row 6 of Fig. 2a is a 4 digit with a short stand and the two nearest neighbours are alike. Or for the test instances in rows 13, 14, and 15 of Fig. 2a the test instances have incomplete circles in the same way as their nearest neighbours. More explanations are provided in the supplementary material in Sec. S4.

Fig. 2c illustrates some sample explanations for Cifar10 [19]. Like before, each row corresponds to a test instance, the first column depicts the test instance itself and columns 2 to 11 depict the 10 nearest neighbours. In Fig. 2c, the test instances of rows 1, 2, 3, 4, and 5 are captured from horses' heads from closeby, and the nearest neighbours are alike. However, in rows 6, 7, 8, 9, 10, and 11 of Fig. 2c the test images are taken from faraway and the found similar training images are also taken from faraway. Intuitively, as the classifier is not aware of 3D geometry, it finds training images which are captured from the same distance. In rows 9, 10, and 11 of Fig. 2c, we see that the testing images contain riders. Similarly, the nearest neighbours also tend to have riders. Therefore, in rows 9, 10, and 11 of Fig. 2c the model has made use of the riders or other context information to classify the test instances as horse. More explanations are provided in the supplementary material in Sec. S4.

Besides finding the nearest neighbours, we provide CAM-like [38] explanations as to why x test and an instance like x ij , 1 ≤ j ≤ 10 are considered similar by the model (according to the procedure of Sec. S2.2 in the supplementary material). Fig.2d illustrates some sample explanations for DogsWolves [34] dataset. In row 1 of Fig. 2d, the first column depicts the test instance itself and columns 2 to 11 depict the 10 nearest neighbours. The second and third rows highlight the pixels

that contribute the most to the similarities. The second and third rows highlight the pixels of x test and { x i 1 , x i 2 , ..., x i 10 } respectively. According to row 3 of Fig. 2d, the pink object next to the dog's leg has contributed the most to the similarities. According to row 2 of Fig. 2d regions like the baby in column 3, the dog colar or costume in columns 4, 5, and 6, human finger in column 9, and the background in columns 10 and 11 have contributed the most to their similarity to the test instance. These are patterns that usually happen for dogs images. Indeed, since the training set has been small (1600 images), to detect dogs the model is making use of patterns that normally exist in indoor scenes and do not normally appear in wolves images. We see a similar pattern for the test instance in row 6 of Fig. 2d and also several explanations in Sec.S4 of the supplementary.

4.3 Comparing GPEX to Representer-Point Selection and Influence Functions

In Sec. S6 of the supplementary material we qualitatively compare GPEX explanations to those of representer point selection [37]. According to the experiments and detailed discussions of Sec. S6 in the supplementary, the GP's kernel that we find in this paper is superior due to a technical point in the formulation of representer point selection [37]. Besides the analysis of Sec. S6, we compared our proposed GPEX with representer point selection [37] and influence functions [18] in dataset debugging task. In these experiments we only selected images from Cifar10 [19] that are labeled as either automobile or horse. To corrupt the labels, we randomly selected 45% of training instances and changed their labels. Afterwards, we trained a classifier CNN with ResNet18 [12] backbone with the same training procedure explained in Sec. 4.1. In dataset debugging task, training instances are shown to a user in some order. After seeing an instance, the user checks the label of the instance and corrects it if needed. One can use explanation methods to bring the corrupted labels to the user's attention more quickly. Given an explanation method, we repeatedly select a test instance which is misclassified by the model. Afterwards, we show to the user the closest training instance (of course among the training instances which are not yet shown to the user). We repeat this process for test instances in turn until all training instances are shown to the user. We compared our proposed GPEX to representer point selection [37] and influence functions [18] in dataset debugging task. We used an implementation of influence functions [18] based on LiSSA [4] with 10 steps for each instance. The implementation is publicly available [1]. For representer point selection [37] we used the implementation by authors which is publicly available [3]. The result is shown in Fig. 2b. According to the plot on the left in Fig. 2b, when correcting the dataset by GPEX, the model accuracy becomes close to 90% after showing about 4000 instances to user. But when using representer point selection [37] or influence functions [18], this happens when the user has seen about 7000 training instances. With noisy labels model training becomes unstable. Therefore, in the plot on the left of Fig. 2b we repeat the training 5 times and we report the standard errors by the lines in top of the bars. According to the plot on the right of Fig. 2b, after showing a fixed number of training instances to the user, when using the proposed GPEX more corrupted labels are shown to the user. Indeed, GPEX brings the corrupted labels to the user's attention quicker than representer point selection [37] does. Interestingly, according to the plot in the right hand side of Fig. 2b influence functions [18] is quicker at spotting incorrect labels, but the instances found by our proposed method are more effective in increasing the accuracy quicker.

5 Related Work

The first theoretical connection between ANNs and GPs was that under some conditions, a random single-layer neural network converges to the mean of a Gaussian process [23] as the width of that single layer goes to infinity. This connection was later proven for ANNs with many layers [8], and for ANNs trained with gradient descent [14]. The theoretical requirements are usually too restrictive. For example, [8] requires all intermediate layers to be wide and also requires the dataset to be countable (so data-augmentations like color-jitter are not allowed). Or [24] requires the ANN to be trained with MSE loss and requires all intermediate layers to be wide. In this paper we do not presume any conditions on the ANN and simply distill knowledge from a neural network to some GPs. Of note, those theoretical conditions may facilitate knowledge distillation and improve the Pearson correlation coefficient between the ANNs and the GPs obtained by our method.

Scalability is a major issue when training GPs, and including a few inducing points may limit the flexibility of GP's posterior [36]. Here we review some previous methods to tackle the computational challenges of training GPs. SV-DKL [27] derives a lower-bound for training a GP with a deep

kernel. In this method, a grid of inducing points are considered in the kernel-space (like the vectors { (˜ u ( ℓ ) m , ˜ v ( ℓ ) m ) } M m =1 with the notation of this paper). Afterwards, each input instance is firstly mapped to the kernel-space and the output is computed based on similarities to the grid points in the kernel-space. Since the GP posterior is computed via the grid points, SV-DKL [27] is scalable. But unfortunately the number of grid points cannot be increased to above 1000 even for Cifar10 [19] and with a RTX 3090 GPU. Therefore, this may limit the flexiblity of the GP's posterior [36].

A more recent framework called GPytorch [29] provides GPU acceleration. However, its computational complexity is quadratic in number of inducing points. Other approaches to improve scalability of GPs include: considering structured kernel matrices [7], kernel interpolation [35], and imposing grid-structure on including points [27]. Stack of Gaussian processes are shown to be connected to ANNs [10][30][24]. By stacking kernels, GP kernels work on intermediate representations and therefore are not necessarily interpretable to humans. But in our method the GPs' kernels work directly on the input-space itself. Knowledge distillation (KD) is closely related to this work. With the best of our knowledge and according to the authors, [6] is the first work that applies KD to GPs. But the distinction of our work is that we distill knowledge from ANN to GP, as opposed to the self-distillation of [6] that distills knowledge from a GP to another GP.

Limitations and Outlook : In this work we used Eq. 5 to distill knowledge from ANN to GP. One may use Eq. 6 to distill knowledge from GP to ANN in order to, e.g., transfer GP's good generalization to the ANN. Our method scales very well, and Alg. 2 runs without memory/computational issues even on imagenet with more than 1M inducing points (i.e. images) and Resnet-18 [12] when a few output-heads are selected, but on imagenet we failed to match the GPs to ANN in a 2-3 day runtime. The issue is that the U matrices have to be updated very often (the update of line 8 of Alg. 2) so that GPs' kernels are updated according to an accurate estimate of kernel-space representations. Otherwise the convergence may not happen especially for millions of inducing points and a small batch-size (as required for, e.g., CNNs). We used control-variate [25], but one may use more advanced heuristics [35] to achieve convergence for datasets like imagenet and with a reasonable computation time. In this paper we analyzed the effect of number of inducing points, the width of the second last layer, and the number of epochs for which the ANN is trained. One can use the proposed tool to answer other questions, like, is the GP kernel required to have more parameters than the ANN itself? May it so happen that a test instance is equally close to hundreds of training instances thereby limiting a human's ability to understand ANNs decision? Is the uncertainty provided by the GP correlated to the understandability of the explanations to humans or to the ANN's failures?

References

A simple PyTorch implementation of influence functions. https://github.com/alstonlo/ torch-influence . [Online; accessed 19-Oct-2023].

An implementation for ResNet in pytorch. https://github.com/kuangliu/pytorch-cifar . [Online; accessed 19-Dec-2021].

Implementaiton of represnter point selection. https://github.com/chihkuanyeh/Representer_ Point_Selection . [Online; accessed 19-Oct-2023].

N. Agarwal, B. Bullins, and E. Hazan. Second-order stochastic optimization for machine learning in linear time. The Journal of Machine Learning Research , 18(1):4148-4187, 2017.

Avanti Shrikumar et al. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 3145-3153. PMLR, 06-11 Aug 2017.

K. Borup and L. N. Andersen. Self-distillation for gaussian process regression and classification. arXiv preprint arXiv:2304.02641 , 2023.

M. K. Cohen, S. Daulton, and M. A. Osborne. Log-linear-time gaussian processes using binary tree kernels. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022.

A. G. de G. Matthews, J. Hron, M. Rowland, R. E. Turner, and Z. Ghahramani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations , 2018.

L. Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

Y. Gal and Z. Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning , pages 1050-1059. PMLR, 2016.

A. Ghorbani, A. Abid, and J. Zou. Interpretation of neural networks is fragile. Proceedings of the AAAI Conference on Artificial Intelligence , 33(01):3681-3688, Jul. 2019.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016.

L. He, N. Ray, Y. Guan, and H. Zhang. Fast large-scale spectral clustering via explicit feature mapping. IEEE Transactions on Cybernetics , 49(3):1058-1071, 2019.

Jaehoon Lee et al. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

J. N. Kather, C.-A. Weis, F. Bianconi, S. M. Melchers, L. R. Schad, T. Gaiser, A. Marx, and F. G. Zöllner. Multi-class texture analysis in colorectal cancer histology. Scientific Reports , 6, 2016.

A. Khakzar, P. Khorsandi, R. Nobahari, and N. Navab. Do explanations explain? model knows best. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10244-10253, June 2022.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization, 2017.

P. W. Koh and P. Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 1885-1894. PMLR, 06-11 Aug 2017.

A. Krizhevsky. Learning multiple layers of features from tiny images. pages 32-33, 2009.

M. Lotfollahi, M. Naghipourfar, M. D. Luecken, M. Khajavi, M. Büttner, M. Wagenstetter, Ž. Avsec, A. Gayoso, N. Yosef, M. Interlandi, et al. Mapping single-cell data to reference atlases by transfer learning. Nature Biotechnology , pages 1-10, 2021.

S. M. Lundberg and S.-I. Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems 30 , pages 4765-4774. Curran Associates, Inc., 2017.

Meng-Hao Guo et al. Attention mechanisms in computer vision: A survey. CoRR , abs/2111.07624, 2021.

R. Neal. Bayesian Learning for Neural Networks . Lecture Notes in Statistics. Springer New York, 2012.

R. Novak, L. Xiao, J. Hron, J. Lee, A. Alemi, J. Sohl-dickstein, and S. Schoenholz. Neural tangents: Fast and easy infinite neural networks in python. 2020.

J. Paisley, D. Blei, and M. Jordan. Variational bayesian inference with stochastic search. arXiv preprint arXiv:1206.6430 , 2012.

C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning) . The MIT Press, 2005.

A. Wilson et al. Stochastic variational deep kernel learning. In Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016.

D. Slack et al. Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , AIES '20, page 180-186, New York, NY, USA, 2020. Association for Computing Machinery.

J. Gardner et al. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. Advances in Neural Information Processing Systems , 2018-December:7576-7586, 2018.

V. Dutordoir et al. Deep neural networks as point estimates for deep gaussian processes. Advances in Neural Information Processing Systems , 34:9443-9455, 2021.

M. T. Ribeiro, S. Singh, and C. Guestrin. "why should I trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016 , pages 1135-1144, 2016.

scArches CVAE notebook. https://docs.scarches.org/en/latest/expimap_surgery_ pipeline_basic.html .

M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 3319-3328. PMLR, 06-11 Aug 2017.

H. Vutukuri. Dogs vs Wolves Classification of Dogs and Wolves. https://www.kaggle.com/ harishvutukuri/dogs-vs-wolves , 2019. [Online; accessed 19-Dec-2021].

A. Wilson and H. Nickisch. Kernel interpolation for scalable structured gaussian processes (kiss-gp). In International conference on machine learning , pages 1775-1784. PMLR, 2015.

Y. Bengio's post on gp vs ann https://qr.ae/pvqZn7 .

C.-K. Yeh, J. Kim, I. E.-H. Yen, and P. K. Ravikumar. Representer point selection for explaining deep neural networks. Advances in neural information processing systems , 31, 2018.

B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses Gaussian Processes trained to match Neural Networks to subsequently \"explain\" the NN outputs by finding the nearest neighbors to any given test point in the training samples. An evidence lower-bound is derived that encourages the GP’s posterior to match the NN’s output. Scalability is obviously an issue for GPs. The authors use GPU acceleration techniques to enable training GPs with O(100k) inducing points. Various example experiments are shown where the 10 nearest-neighbors in training illuminate why the NN responds the way it does to a test example.\n\nSTRENGTHS:\nThe method seems straightforward and intuitive. The GP matching is less aggressive where the uncertainty is large (far from any training points). The examples shown seem to demonstrate that the method appears to work on these examples.\n\nWEAKNESSES:\nI didn't find any mention about using the GP error bands in the explanations. Presumably the explanations are not as good in regions where the GP and NN are less closely matched, which should be the case where GP error bands are large since the method doesn't weight these regions as much in the loss. I.e., far from any training points, it's not obvious that the nearest neighbors chosen by the GP kernels are really driving the NN response. This isn't discussed at all as far as I can see.\n\nQUESTIONS:\nHave you looked at examples in regions where the GP error band is large? \n\nAssuming it's true that explanations there are less reliable, can you return info using the size of the GP error at x_test that flags this for the user? \n\nIt's nice that you can interrogate the NN using your GP to explain specific examples, but it's not obvious (to me) how one would systematically study a NN in this way, since a human can only visually inspect a small number of examples as is done in the paper. Any comments / thoughts on this?\n\nLIMITATIONS:\nThe Limitations paragraph is super concise (presumably due to space constraints), but it'd be good to expand it a bit and make it more clear. E.g., it talks about how the method runs well with O(1M) points, but that the GP fails to match the NN. As written this is hard to interpret.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article proposes a new training approach for Gaussian Processes (GPs) which encourages their posterior to match a given Artificial Neural Network (ANN) output. Their approach adopt a scheme that permits a scalable training, which is an issue when it usually comes to GPs, and that is motivated by the derivation of an evidence lower bound (ELBO) encouraging the faithfulness of the GP to the target ANN. The authors then  explain the decisions made by an ANN with the help of a matching GP with two different methods : explanations by similar examples (similar in the kernel space of the GP), and using saliency map to determine why some examples where similar.\n\nSTRENGTHS:\n1. It is, to the knowledge of the authors, truly original to use a GP in order to explain a matching ANN. \n\n2. The approach for training GPs tackles a main issue of traditional approaches : scalability with regards to the number of inducing points.\n\nWEAKNESSES:\n_Major_\n\n1. My first concern, which could probably summarise many flags I have to raise, for it leads to several problems : the article is too dense. One consequence is that there are too many things putted in Supplementary material that are necessary to understand the article in its whole. For example, the way the CAM-like explanations (line 282) are computed is not even slightly described in the main article; or, « According to the experiments and detailed discussions of Sec. S6 in the supplementary, the GP’s kernel that we find in this paper is superior due to a technical point in the formulation of representer point selection » (lines 299-300); or the fact that the derived  ELBO is a contribution in itself, and isn’t even presented in the main paper, only pieces of it. The Supplementary material should not contain information that are vital to understanding the methods, the idea and the approaches, but further readings.\n\n2. When manipulating concepts such as explainability and interpretability (which seem, in the article, to be used interchangeably; lines 20-21 :  « After matching the GPs to the ANNs, we used the GPs’ kernel functions to explain the ANNs’ decisions. »; the title says « interpreting artificial neural networks », and on lines 75-76 : « GPEX can be used by machine learning researchers to interpret [...] their artificial neural networks. »), since they do not have a clear and universal definition, it is important to state what these terms refer to in the article, just like, in Section 2.1, some notations are defined. This lack of clearness makes it harder to grasp the goal of the article.\n\n3. If I understand correctly, as for the explainability approach, the way the nearest neighbours are found is by looking at the nearest neighbours, but in the kernel space. How does that differ from only computing the nearest neighbours in the output space of the network? How does that makes the neighbours that are found more relevant? Both the kernel and the network can be seen as an embedding. Plus, the GP's job is to imitate the network. It seems to me like this approach combines two explainability approaches : finding a « more interpretable » predictor imitating the predicting behaviour of the former, and finding examples that are similar to one another. Why is combining those two approaches, both having their flaws, would lead to any better results when it comes to explainations?\n\n4.1. I find the title misleading; explaining (interpreting, as the title says) ANNs would seem to be the main point of the article, but in reality, explaining ANNs concerns only page 8 and half of the last page, whereas training a GPs faithfully to a trained ANN really is the main point of the article. Plus, both the saliency map and looking at the most similar examples (which corresponds to approximate the decision function of the network by a more interpretable predictor : a k-nearest neighbours predictor) from the training set are not novel ideas.\n\n4.2. As for the related works, explaining using GPs is only discussed on ~3 lines, citing a single work.This might be the first time an explanation is based on a bridge between GPs and ANNs, but the authors stated that « Gaussian processes are highly interpretable » (lines 37-38) and that « The analogy between Gaussian processes (GPs) and deep artificial neural networks (ANNs) has received a lot of interest, and has shown promise to unbox the blackbox of deep ANNs » (lines 1-3); it is therefore expected to see « what kind of interest they have received », especially in the discussion / related work sections of the article.\n\n5. One of the contribution is that the proposed approach is more scalable when compared to previous approaches. Without any empirical comparison with state-of-the-art for training GPs, both in terms in training time and performances, it is hard to state whether the proposed approach works particularly well or if it is just have effective as what is found in the literature. That is also why the experiments ran in Section 4.1 could be misleading. Since no benchmark is used to compare the proposed approach to, how can one know if the obtained results are good or not? This lack of information makes it hard to state whether or not the results are positive.\n\n6. I found some claims to be vague, doubtful or at least are highly debatable, but most importantly sometimes not supported at all; lines 37-38 : « Gaussian processes are highly interpretable. » How are they more interpretable? In what way? Doesn’t that depend on the kernel that is used? The same goes for lines 207-208 : « Because according to theoretical results on GP-ANN analogy, the second last layer of ANN should be wide. » This might be true, but there is a need to cite some works here. Also : « Our method scales very well, and Alg. 2 runs fine even on imagenet with more than 1M inducing points » (lines 343-344); what’s « fine » mean? (See also Weaknesses – Major – 5)\n\n7. It is said that experiments were conducted on « 5 datasets (4 image datasets, and 1 biological dataset) » (lines 17-18), but nothing in the article is discussed about the biological dataset.\n\n_Typos / Minor_\n\n1. Line 85 : « The number of GPs is equal to the number of $\\underline{the}$ outputs from the ANN. »\n\n2. Some nomenclature and terminology is unclear. For example, line 91-92 : « we consider a general feed-forward pipeline that contains an ANN as a submodule »; what’s a feed-forward pipeline? What is a submodule? Figure 1a) somewhat doesn’t help understanding. (See also Weaknesses – Major - 2)\n\n3. Equation 5 : Is there a missing term?: what is the expectation computed with regards to? (?~q)\n\n4. Line 164 : « procecdure »\n\n5. Line 167 : « matrcies »\n\n6. Line 179 : « from linear algebra if follows that »\n\n7. Line 327 : requries\n\n8. Having the Related Work at the end of the article is peculiar; it seems to me like precious information for better understanding the relationship between ANNs and GPs during the reading of the article (and better understanding the relevance of the work) are put at the end of the article.\n\n9. It should be explicitely written that the kernel function $\\mathcal{K}(\\cdot, \\cdot)$ outputs a matrix whose dimensions correspond respectively to the cardinality of the first and the second input, for it is implicit, but not explicit yet important.\n\nQUESTIONS:\n1. Aren’t those two sentences (« The analogy between Gaussian processes (GPs) and deep artificial neural networks (ANNs) has received a lot of interest, and has shown promise to unbox the blackbox of deep ANNs » (lines 1-3) and the third contribution, « With the best of our knowledge, our work is the first method that performs knowledge distillation between GPs and ANNs. ») contradictory?\n\n2. Can some details be provided concerning the kernel choice $f(\\cdot)$?\n\n3. Why choosing Pearson correlation coefficient for comparing GPs and ANNs? Is it what’s often seen in the literature? Why not comparing the coherence of the predictions, for two predictors can have similar prediction function but predict differently most of the time?\n\nLIMITATIONS:\nThe authors did not discuss the limitations of their explainability approach. Nowadays, many problems with (for example) saliency maps are well-known (not robust to adversarial perturbations [1], simply unreliable [2], etc.), thus one would expect a few words on those matters.\n\n\n\n[1] Ghorbani, Amirata, Abubakar Abid, and James Zou. “Interpretation of neural networks is fragile.” Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.\n\n[2 ]Kindermans, Pieter-Jan, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. “The (un) reliability of saliency methods.” In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pp. 267-280. Springer, Cham (2019).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

comments: SUMMARY:
This paper derive an evidence lower-bound that encourages GP's posterior to match ANN's output without any requirement on ANN. And the uses the GPs' kernel functions to explain the ANNs' decisions.

STRENGTHS:
1) this paper provides a theoretical way for the algorithm 

2) implementation is publically available

3) it is good to see that GP output matches ANN well

WEAKNESSES:
1) The main point of this paper is to explain ANN. However, Figure 2 is the only experiment results that are about explanation, which only contains some sample explanations. There is no comparison to existing explanation methods and there is not quantitative results about the evaluation performance.

QUESTIONS:
None

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper derives an ELBO to end-to-end distill a deep neural\nnetwork into a set of Gaussian processes (one per output) with deep neural\nnetwork kernels and the full training set used as inducing points.\nBy using a low output-dimensionality for the feature map neural network,\nthe computational cost of the inversion/decomposition of the kernel matrix\nis kept constant while arbitrarily scaling the number of inducing points.\nThe paper empirically validates the discrepency between the teacher neural\nnetwork and the student DNN-GP on 5 datasets with 2 different models (5 on\nResNet-18, 2 on some custom attention-based architecture).\nTwo approaches are introduced to attempt to explain the prediction decisions of\nthe distilled neural networks based on the DNN-GP: through similarity in the\nDNN-kernel's feature space, and a pixel-wise contribution based on the\nsimilarity to the nearest neighbors in the DNN's kernel space.\nAn accompanying framework is provided, which claims to distill any DNN in Pytorch.\n\nSTRENGTHS:\n- The ELBO and distillation process from GP to DNN-GP is a novel contribution.\n\n- Code is provided for all experiments, which is a major strength.\n\n- Important derivations are provided in the supplement.\n\nWEAKNESSES:\n- The paper reports on GPs, but is not very verbose in the fact that\n  DNN-kernels are used, although they seem to be strictly necessary for the\n  distillation (e.g., the low-dimensional, explicit features space). Prior work\n  on DNN-GPs is only partly discussed (e.g. [23]).\n\n- The paper claims to introduce a novel approach to scale the number of inducing points.\n  However, the computational trick comes down to using a low-dimensional, explicit\n  feature space, leading to a low-rank, high-dimensional noise-free kernel\n  matrix, which is only useful for low-dimensional explicit feature spaces as\n  in DNN-GP.\n\n- The DNN-kernels are significantly more complex (resnet50) than the teacher\n  DNN (ResNet18). This explainability-complexity trade-off is not explicitly\n  discussed, and seems like a significant limitation.\n\n- The experiments are not presented very well: Figure 2 (in particular a,c,d) is\n  quite hard to understand without jumping to the respective text passage.\n  Figures descriptions in general are lacking in detail (i.e., which are\n  nearest neighbours, and which is the analyzed sample in Figure 2).\n  Fig 2(b) has very small captions.\n\n- In the experiment in Figure 3: while the classifiers on MNIST/CIFAR10/Kather\n  are almost completely above 0.95, a PCC of 0.90 and slightly below for the\n  other datasets and models are not very insignificant, especially given that\n  these models are distilled in order to be explained. This should be discussed\n  in more detail.\n\n- The qualitative analysis of the explanations, both based on the\n  kernel-similarities, which show similarities and a CAM-based approach, is\n  difficult to interprete. While the samples generally seem to show other\n  samples from the same class somehow semantically related, these results are\n  not directly compared to any approach, making it hard to argue in favor of\n  supporting the explainability claim. The focus should be more on the\n  experiment in Fig 2b, ideally with more trials and different data sets.\n\n- The explainability approach is only compared to a single baseline, which is\n  too few.\n\n- While the experiment in Fig 2b presents a quantitative analysis of the\n  explainability of the method compared to one other baseline, it is only\n  conducted a single time, on only a single data set. Since the paper claims\n  explainability of the models, there should be more experiments in order to\n  support this claim.\n\n- While the paper claims to provide a tool, the linked software framework has\n  close to no docstrings, provides no documention, features no tests, and does\n  not provide a plug-and-play package (no setuptools).\n  It is good practice to publish accompanying code, and the paper goes further\n  by also including code for the corresponding experimental results.\n  However, at the reviewed state, the *tool* simply presents code to the\n  paper, rather than providing the software framework suggested in the\n  introduction.\n\nQUESTIONS:\n- The paper could be improved by being more verbose about using DNN-kernels.\n\n- It might be better for the paper to instead of stretching the\n  scalability-claim, openly discuss how a small, explicit feature space leads\n  to a constant rank of the noise-free kernel matrix, creating a situation\n  where GPs do not scale beyond the explicit feature space's size, thus arguing\n  for low-dimensional DNN-kernel-spaces.\n\n- Given the increased complexity of the kernel's DNN to the teacher model, a\n  discussion on the complexity-explainability trade-off would be very interesting.\n  Further experiments with smaller kernel DNN's in the distillation-process\n  could also bring insight in this regard.\n\n- For the experiments in Figure 3, comparing to some baseline might help in\n  understanding the PCC values better. While a PCC of 0.9 sounds good, it would\n  help to also highlight the differences between student and teacher, although\n  I do not have a concrete idea how this could be done.\n\n- To put the gained explainability into context, especially in the qualitative\n  experiments, it would really help to compare to some baselines.\n\n- The quantitative experiment comparing to Representer-Point selection is very\n  valuable, but would be even more valuable with multiple tries, and on\n  different data sets. It feels somewhat incomplete.\n\n- The software claim in the beginning of the paper suggests a well-maintained\n  software framework for the GP model distillation, which does not seem to be true.\n  I would suggest to soften this claim (i.e., simply provide a footnote stating\n  \"Code for methods/experiments provided at...\")\n\n\n### Minor\n\n- Alg. 2 title is misleading, a better name may be: training loop for GP\n  distillation.\n\n- Alg. 1 can be moved to the supplement, as it is only a step of\n  gradient descent for the DNN-kernels and does not add much to the manuscript\n\n- l.126 nominator -> numerator\n\n- figures should be described more in detail, especially the qualitative one\n\nLIMITATIONS:\nThe paper discusses some limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}