A Flexible Nadaraya-Watson Head Can Offer Explainable and Calibrated Classification
Authors: Alan Q. Wang
Mert R. Sabuncu
Venue: Accepted by TMLR ()

Abstract: In this paper, we empirically analyze a simple, non-learnable, and nonparametric Nadaraya-Watson (NW) prediction head that can be used with any neural network architecture. In the NW head, the prediction is a weighted average of labels from a support set. The weights are computed from distances between the query feature and support features. This is in contrast to the dominant approach of using a learnable classification head (e.g., a fully-connected layer) on the features, which can be challenging to interpret and can yield poorly calibrated predictions. Our empirical results on an array of computer vision tasks demonstrate that the NW head can yield better calibration with comparable accuracy compared to its parametric counterpart, particularly in data-limited settings. To further increase inference-time efficiency, we propose a simple approach that involves a clustering step run on the training set to create a relatively small distilled support set. Furthermore, we explore two means of interpretability/explainability that fall naturally from the NW head. The first is the label weights, and the second is our novel concept of the ``support influence function,'' which is an easy-to-compute metric that quantifies the influence of a support element on the prediction for a given query. As we demonstrate in our experiments, the influence function can allow the user to debug a trained model. We believe that the NW head is a flexible, interpretable, and highly useful building block that can be used in a range of applications.

Published in Transactions on Machine Learning Research (02/2023)
A Flexible Nadaray--Watson Head Can Offer Explainable and Calibrated Classification
Alan Q. Wang
aw8470cormell.edu
School of Electrical and Computer Engineering Cornell University
Mert R. Sabuncu
msahuncu&cornnlll..o edu
School of Electrical and Computer Engineering Cornell University
Reviewed on OpenReview; https: openreview. net/ per//foum? id= 1Eg6lhG403
Abstract
In this paper, we empirically analyze simple, non-learnable, and nonparametric Nadaraya- Watson (NW) prodiction head that can be used with any neural network architecture. the NW head, the prediction is a weighted average labels from a support set. The weights are computed from distances between the query feature and support features. This in contrast to the dominant approach of using a learnable classification head (e.g., a fully-connected layer) on the features, which can be challenging to interpret and can yield poorly calibrated predictions. Our empirical results an array of computer vision tasks demonstrate that the NW head can yield better calibration with comparable accuracy compared to its para- metric counterpart, particularly in data-limited settings. To further increase inference-lime efficiency, we propose a simple approach that involves a clustering step run on the training set to create a relatively small distilled support set. Furthermore, we explore two means of interpretabilityy httpetaabblit/eeellaaaaiiiiii
that fall naturally from the NW head. The first is the label weights, and the second is novel concept of the "support influence function," " which is masy-to-compute metric that quantifies the influence of a support element on the prediction for a given query. As we demonstrate in our experiments, the influence function can allow the user to debug a trained model. We believe that the NW head is a flexible, interpretable, and highly useful building block that can be used a range of applications.
1 Introduction
Many state-of-the-art classification models are parametric deep neural networks, which can be viewed a combination of a deep feature extractor followed by one or more fully-connected (FC) classification lay- ers (He et al.) 2015 Huang et al.] 2016 Dosovitskiy et al. 2020 Two common problems with parametric deep learning models are that they can be hard to interpret (Li et al. 2021 and often suffer from poor calibration (uu al. 2017).
Nonparametric (and, similarly, attention-based modeling strategics have been explored extensively in deep learning (Wilson al. 2015 Papernot & McDaniel 2018 Chen et al. 2018 Kossen et al. 2021 Laenen & Bertinctto 2020 Iscen et al. 2022 Frosst et al. 2019). In this approach, the prediction for given test input (which we will also refer to as a query) is computed as an explicit function of training datapoints. These models can be more interpretable, since the dependence on other datapoints gives an indication about what is driving the prediction Hanawa computational overhead, can take a hit performance versus parametric approaches, and/ or require complex approximate inference schemes (Bui et al. 2016 Salimbeni & Deisenroth 2017).
al.] (2021); Tacsiri et al. (2022). Yet, they can suffer from significant

---

Published in Transactions on Machine Learning Research (02/2023)
Query
90
NW Head
Support
211 X
90
Softmax
+W2 X
233 X
Figure 1: The NW head. Query image and support images are fed through feature extractor go. Pairwise similarities -) (e.g. negative Euclidean distance) are computed between the query and each support fea- ture. These similarities are normalized and used as weights for computing the weighted sum of corresponding support labels.
In this work, we present an analysis of a Nadaraya-Watson (NW) head, a simple, non-learnable, and non- parametric prediction layer based on the Nadaraya-Watson model (Nadaraya) 1964 Watson 1964 Bishop 2006). For a given query, pre-defined similarity is computed between the query feature and each feature in a "support set', which can be made up of samples from the training data. Those similaritics are normalized to weights, which are lurn used compute a weighted average of the labels in the support set used as the final prediction.
As discuss in this paper, the choice of the support set for the NW head affords the user additional degree of freedom in implementation. We demonstrate several ways of constructing the support set, including simple support set distillation approach based on clustering, and discuss the relative advantages and trade- offs. In our experiments, we show that the NW head can achieve comparable (and sometimes better) performance than the FC head on a variety of computer vision datasets. In particular, we find that the NW head exhibits better calibrated predictions. Improvements are particularly pronounced small medium-sized fino-grained classification tasks, where there is little sacrifice computation time.
In addition, explore two means of The first is interpreting the support set weights, which directly correspond to the degree of contribution specific training datapoint has on the prodiction. The second is our novel concept of "support influence', which highlights helpful and harmful support examples and can be used as diagnostic tool to understand and explain model behavior. In our experiments, we demonstrate the utility of these methods for purposes of interpreting, explaining, and potentially intervening in model predictions.
which fall naturally from the NW head.
2
Related Work
2.1 Nonparametric Deep Learning and Attention
Nonparametric models in deep learning have received much attention in previous work. Deep Gaussian Processes (Damianou & Lawrence 2013)), Deep Kernel Learning (Wilson et al. 2015), and Neural Pro- cesses (Kossen et al. 2021) build upon Gaussian Processes and extend them to representation learning. Other works have generalized k-nearest. neighbors (Papernot & McDaniel 2018 Taesiri et al. 2022 de- cision trees (Zhang et al.] 2018), density estimation (Fakoor et al. (2200). and more general kernel-baser basod methods (gguyen et al., [2021] ZZhang al. 2021 Ghosh & Nag] 2001) to deep networks and have explored the interpretability that these frameworks provide.
Closcly-related but orthogonal to nonparametric models are attention-based models, most notably self-attention mechanisms popularized in Transformer-based architectures in natural language process- ing Vaswani et al.) 2017) and, more recently, computer vision (Dosovitskiy et al.) 2020 Jaegle et al.
2021 Parmar al.] 20118 Retrieval models NLP learn to look up relevant training instances for pre-
diction (Sachan al. 2021 Borgeaud et al. (202)) in a nonparametric or semiparametric manner. Many
works have recognized the inherent interpretable nature of attention and have leveraged it in vision (Chen
2

---

Published in Transactions on Machine Learning Research (02/2023)
al. 2018 Guan al. 2018 Wang et al. 2017 Xu et al. 2015 More recently, nonparametric trans- formers (NPT) Kossen et al. 2021 applied attention in a nonparametric setting. NPT leverages self- and cross-attention between both attributes and datapoints, which are stacked up successively in an alternating fashion to yield a deep, non-lincar architecture.
Building upon and contrast to these prior works, we present a comparatively simple nonparametric head which compptationnlll--fffceet and performant, and which additionally lends itself naturally to being interpretable.
2.2 Metric Learning
Metric learning seeks to quantify the similarity between datapoints for use in downstream tasks (e-g-, un- supervised pretraining (Wu et al. 2018 or retrieval (Chen et al., 2021 ). Of particular relevance to our method is metric learning in the context of low-shot classification (Snell et al.) 2017 Vinyals et al. 2016 Sung et al. 2017 Santoro al. 2016). particular, Matching Networks (Vinyals et al.) 2016) for low-shot and meta-lcarning are most similar predictions, where both the query image and support set are composed of examples from wnseen classes. In contrast, we analyze the NW head from a nonparametrie perspective in the typical, many-shot image classification setting.
model in the sense that it uses NW-style model to compute
2.3 Influence Functions
Following classical work by Cook Weisberg (1982). Koh Liang 2017) propose the influence function means of explaining black-box neural network predictions. Let denote a function parameterized by A.
D = (zi, 3))² is set of training examples and L(h)(z)) indicates the loss for a training example 0* is the set of optimal paramcters which minimizes the empirical risk. Similarly, the parameter set associated with up-weighting a training example by is found by solving:
(z
n
arg min
L(ho(zi)) e((h(z))).
(1)
T2
Koh & Liang (2017) define influence as the change in the loss value for particular test point ž when a training point is up-weighted. Using a first-order Taylor expansion and the chain rule, influence can be approximated by a closed-form expression:
= ((tloo-(x))))))))) LLo- (z)),
(2)
where Ho- represents the Hessian with respect to 0" I(zz, 2)/n I is approximately the change in the loss for the test-sample 2, when training sample z is removed from the training set. However, this result is based on the assumption that the underlying loss function is strictly convex in and that the Hessian matrix is positive-defiite. Follow-up work has questioned the efficacy of influence functions in practical scenarios where these assumptions are violated (Basu et al. 2020
In this work, we define the support inffuence function which follows naturally from the NW head; it computes the change in the loss directly on the prediction vector itself (and not indirectly through the model parameters 0). Furthermore, it is aas-to-compute, requires approximations, and does not require any assumptions convexity or differentiability.
3 Methods
3.1 Nadaraya-Wateee Head
Consider a "support sot" of N, examples and their associated labels, S {4+ : (a,, 2 This support set can be a subset, the whole, of the training dataset. In this paper, we will focus on the image classification setting, where x's images and y's integer class labels from label set C (1,..., C). We will use to
3

---

N. == exp (-1 Published in Transactions on Machine Learning Research (02/2023)
denote the one-hot encoded version of y. We note that the NW head can be used with any input data type, not just images, and can be readily extended to the regression setting.
For query image the Aadaraya-Watson prediction is computed as the weighted sum of support labels Ts where the weights quantify the similarity of the query image with each support image 24:
N.
(x, Σ w(r, 1y )
The weights can be thought of attention kernels (Vinyals et al. 2016) and are similar to attention mechanisms in deep learning (Vaswani et al., 2017: Dosovitskiy et al., 2020 Kossen et al. 2021 Classically, these weights are defined function of handcrafted kernel functions
((x, 2.)
w(x, x4) = Σnnn k(2, my))
(4)
where .) quantifics the non-negative similarity between its two arguments (Bishop. 2006). larity values correspond to more similar pairs of datapoints.
Higher simi-
In the NW head, we define the weights as:
exp (- 2o (0(x)))l2///
wolx, xx))
(5)
Here, Do is a feature extractor with parameters 0, which we implement as neural network. t is a tem- perature hyperparameter; unless otherwise stated, we set == 1. || 1/2 denotes Euclidean distance. Since E (z xx) = 1 and J. 's are one-hot encoded vectors, the output f of the estimator can be interpreted as conditional probability for the class label given the image and support set.
3.2 Loss Function and Training
Let D be training set of image and label pairs. Our objective to minimize is:
arg min
Σ ESnD ((fo(z,)).))
(6)
0
(a.y)ED
where L is the cross-entropy loss. This indirectly encourages similar-looking images to have higher similarity score, since more similar images will tend to have the same labels.
We optimize this objective using stochastic gradient descent (SGD) by sampling a mini-batch size No of query-support pairs and computing gradients with respect to the mini-batel Note that our model assumes dependency between support set samples; indeed, we are free to sample any arbitrary S and query 2, as long as the class of is within the support classes. Therefore, the support set size N, = |S is a hyperparameter analogous and orthogonal to the batch size N₁-
3.3 Inference
The ability to select and manipulate the support set at inference time unlocks degree of flexibility not possible with parametric classification models. In particular, the label set C of the support set directly determines the label set of the resulting prediction. For example, if the user has prior knowledge which classes a particular query is most likely to be, the support set can be restricted to only contain those classes.
Note that we assune the model has been trained with random support sets drawn from the training data for each mini-batch. To characterize the effect of the support set inference, in our experiments implement the following "inference modes":
1. Full. Use the entire training set: S D.
4

---

Published in Transactions on Machine Learning Research (02/2023)
2. Random. Sample uniformly at random over the dataset, such that each class is represented & times: S ~ D and S| = CC
Cluster. Given the trained model, we first compute the features for all the training datapoints. Then, perform k-means clustering the features of the training datapoints for each class. These l cluster centroids are then used as the support features for each class. Note that in cluster mode, the support set does not correspond to observed datapoints.
4. Closest cluster (CC). Same Cluster, except the support features correspond to real training datapoints that closest to the cluster centroids in terms of Euclidean distance.
Each mode except Full assumes that cach class is represented & times in S. One can, obviously, modify this assumption to reflect any class imbalance that might exist in the training data.
3.4 Support Influence
Koh & Liang (017) define influence as the change in the loss as a result of upweighting a particular training point. With an NW head, quantifying the influence of a support image to the given query is straightforward and can be computed in flosed-form. Indeed, the proposed support influence computes the change in the loss directly on the prediction vector itself (and not. indirectly through the model parameters f), and furthermore does not require approximations,
assumptions on convexity differentiability.
Denote by S-z, = S 2g, i.e. the support set with an element removed. For fixed query
f(x,S=.,) - w(z,
w(1, x.)
We define the support influence of z, on z as the change the cross-entropy loss incurred from removing a support clement 2g:
- fu 2.)) ) 2(2,2)) S - = log fv- w(I, x. (1p
(8)
(1=0a))
where superscript denotes indexing into vector and 1 is the indicator function. The derivation is provided in the Appendix.
4 Experiments
4.1 Experimental Setup
Datasets. We experiment with an array of computer vision datasets with different class diversity and training set size. For general image classification, experiment with Cifar-100 (Krizhevsky 2009). For fine-grained image classification with small to medium training set size (less than 12k training samples), we experiment with CUB-200 200-2011 (Bird-200) (Wah et al.) 2011), Stanford Dogs (Dog-120) (Khosla et al. 2011), Oxford Flowers (Flower- 102) Nilssack & Zisserman 2008), and FGVC-Aíreraft (Aircraft-100) (Maji et al. 2013). For large-scale fino-grained image classification task (500k training samples), we experiment with NNtturalist-OOk (Grant Van Horn 2021). Additional dataset details are provided in Appendix A.1
Network Architecture and Hyperparameters. For the feature extractor ge, neural network (CNN) followed by a lincar projection to an embedding space of dimension d. For Cifar-100, Bird-102, Dog-120, Flowers- 102, and Aircraft-100 dataset, we experiment with the ResNet-18 (He el al. 2015) and DenseNet- 121 (Huang et al.) 2016) architectures. For iNaturalist- 10k, we experiment with the ResNet-50 architecture, following prior work (Zhou al.) 2019). For all feature extractors, we set d = 128 and We train each model 3 times with different random initializations. Additional model and hyperparameter details provided in Appendix A.2
convolutional
5

---

Published in Transactions on Machine Learning Research (02/2023)
Cifar-100
Bird-200
Dog-120
Flower-102
Aircraft-100
0.5350
B:5325
0.49
0.248
0.44
0.34
0.4450
0.43
8:29
0.240
0.32
0.4425
0.42
17.75
11
26.5
24.
12
10
17:25
17:00
10.75
1.50
3.400
2.375
2.35
1.45
1.40
1.55
1.80
1.40
1.35
2,525
1.75
1.50
1.00
0.95
1.30
#Support Images
4 Support Images
8 Support Images
# Suppport Images
Support images
FC
NW. Random
NW. Cluster
NW. Closest cluster
NW. Full
Figure 2: Model performance across different support set sizes for DenseNet-121. Full mode and parametric baseline are constant across support set size and are depicted as horizontal lines.
Figure 3: Corresponding images of embeddings closest to cluster centroids, for h: and h: 9.
Baseline. We compare the NW head to standard parametric model with a single-layer fully-connected (FC) head, and refer to it throughout this work as FC. For FC, we use the same feature extractor go and keep all relevant hyperparameters identical to NW.
Metrics. We measure performance using error rale (the fraction of incorrectly classified test cases) and expected calibration error (ECF) (Guo al. 2017 Naeini et. al. 2015). F.E approximates the difference in expectation between confidence and accuracy, computed by partitioning the model predictions into bins and taking a weighted average of each bins' difference between confidence and accuracy.
4.2 Model Performance
Tables 2 and show error rate and ECE performance for all models and datasets, and Fig. furthrr shows reliability diagrams. We generally observe that the NW head achieves an error rate that is comparable and times better than its parametric counterpart. Calibration particularly improved in small medium-sized datasets with relatively fine-grained differences between classes.

---

Published in Transactions on Machine Learning Research (02/2023)
Table
Error rate + (%). Bold best and underline is second best. k for NW models.
Model
Cifar-
Bird-200
Dog-120
Flower- Aircraft-100
ResNet-18
FC
26.862075
49.77 29..721.20
37.4221.1
47.00=0.27
49.4120.8
NW, Random
28.41+1.00
51.45
45.33+1.0s
43.86=0.22
41.76=1.0s
NW, Cluster
27.6611:15
46.60 10.6001..s0
38.10 18..00
42.87 10.32
39.94
NW, CC
27.82+0090
47.76=1.07
39.00
43.70=0.a8
39.99
NW, Full
27.58
46.98
38.07 t0.91
42.97 =0.30
39.93+t...
DenseNet- 121
FC
24.21 +0.96
45.68+0.
32.62+a0 +0.00
53.54+0.a2
49. 19.774+0.73
NW, Random
25.0321.14
44.59
36. 96.1440..8
44.9820.37
45.3321.06
NW, Cluster
24.63
43.23
+..0
44.36=0.34
42.42
NW, CC
24.8420.97
43.6821.08
33.961o.se
44.56
43.0220.s6
NW, Full
24.76+0.59
43.33+1.8
33.07
44.28 +0.30
42.27 +0.84
Table 2: ECE + (%). Bold is best and underline is second best. A = for NW models.
Model
Cifar-100
Bird-200
Dog-120
Flower-102
hireraft-100
ResNet-18
FC
15.40
20.2321.10
16.0221.07
22.26
24.55 40.96
NW, Random
18.84+1.1
7.762+1.10
5.583+0.s0
19.59=0.4s
12.16
NW, Cluster
70
6.051
4.025+0.00
19.83
12.12
NW, CC
19.47+0.44
6.22440.87
4.757
20.09
11.66 +0.75
NW, Full
17.68
3.327 10.65
4.25621.7
19.10
9.235
DenseNet-121
FC
10.74.0.77
17.67
12.741o.2
26.792o.41
25.03
NW, Random
17.19
10.74
10.2640.00
21.8920.4
12.40 11.05
NW, Cluster
17.62+0.96
10.78+0.07
8.709
22.27
11.23+1.14
NW, CC
17.73+1.08
10.23+1.19
9.535+0.00
22.11=0.0
11.39
NW, Full
16.95 95
7.997 t:..s
4.892
21.39 #0.38
8.248 :.11
Table shows inference times Bird-200 for both parametric and the NW-head models, where assume that all support emboddings, cluster centroids, and closest cluster embeddings have been precomputed. We observe that. inference limes are nearly identical to the FC head, except Full. On small dataset like Bird-200, Full corresponds to a two-fold increase in computation time.
In Fig. 2 we plot various metrics across support set size k. Generally, performance improves as support set size increases. We altribute this to the fact that more support images leads to more robust predictions, as more support examples per class can average out potential errors. At the limit, Full mode typically performs the best, at the cost of inference time.
Cluster mode often improves over Random mode and is par with Full mode (at larger k) while having favorable inference time. However, this comes at the cost of loss of interpretability, since the cluster centroids do not correspond to actual images in the support set. Closest cluster addresses this deficiency, often with minimal sacrifice to performance compared to Cluster.
Fig. visualizes the support images used in Closest cluster for two classos, with = and summary compression of
These
images are the prototypical examples of the class and can be interpreted the class and dataset. We observe that for k = the support image is prototypical example of the class. For k 9, the images have variety of poses and backgrounds, allowing the model to have a higher chance of computing similarities with a wide range of class prototypes. Further research might explore leveraging these prototypes.
Overall, our experiments indicate that the NW head has advantages accuracy and calibration the datasets we tested. The NW head pulls ahead in performance particularly for fine-grained classification fasks, and pulls ahead even further for meall-medium sized datasets. Computational load is higher than FC,

---

Published in Transactions on Machine Learning Research (02/2023)
Table 3: Error rate / ECF 4 (%) for label smoothing (T.S) and temperature scaling (TS) for ResNet-18.
Bold best. Cluster and k for NW models.
Model
Bird-200
Dog-120
Flower- 102
Aircraft-100
FC-LS
45.36+1.03
15.83+0.98
37.58=114
10.79=0.a3
46.35
/ 3.84-6.a8
48.72
/ 7.16+0.07
NW-LS
45.10
2.099 =0.86
38.29
2.24
43.23
1.99
45.76=1.01
/ 13.61.o.s
FC-TS
49.77 +1.20
22.711 +0.92
37.42+1.1
2.208 +0.07
47.00+0.2
/ 4.426+o.s1
49.41+a.ss / /4423 11000
NW-TS
46.6010.s
22.460=o..
38.1010.80
1.91 10.77
42.87 2.693
39.94
/ 5.191 21.10
Model
Time (10-3 sec)
Model
Error Rate
ECE
FC
4.00
FC
43.720.08
15.020
NW, k
4..5
NW, k=1
43.1+1.70
16.7
NW, & 10
4.78+0.00
NW, k-10
42.5
13.420.8
NW, Full
7.76
Table 5: iNaturalist-10k results on ResNet-50. Cluster mode for NW.
Table 4: GPU inference times for Bird-200 and ResNet-18.
but is only two times higher for our most expensive inforence mode on smaller datascts with less than 10k training samples.
4.2.1 Improving Calibration
In this section, we explore whether existing calibration techniques originally proposed for parametric models further improve calibration for NW models. Note that improved calibration is not specifically enforced in our architecture training procedure; thus, many effective calibration techniques proposed in the literature can be readily applied to the NW head.
We explore two popular and effective techniques: label smoothing (LS) and temperature scaling (TS). Label smoothing (Szegody et al.) 2015) replaces onc-hot labels during training with soft labels which are a weighted average the one-hot labels and the uniform distribution. Temperature scaling (Guo el al. 2017) is a post- training technique which optimizes the temperature of the softmax with respect the loss on the validation set (and freezing all trained parameters). For the NW head, the temperature is analogous to in Eq. Additional details are provided in A.3
We show the results in Table 3 In general, we find that the NW head outperforms the FC head under both LS and TS techniques. In particular, the simple, post-training technique of TS is a simple strategy that further improves our calibration performance.
4.3
Interpretability and Explainability
In the following section, we restrict our attention to Full mode and explore the interpretability and explain- ability aspects of the NW head.
4.3.1 Interpreting the Weights
Visualizing the support images ranked by weight w can give insight to users about how the model is making prediction. For cases where the model is unsure, visualizing the support images along with confidence scores can uncover where the model unsure and potentially allow for user intervention.
In Fig. G we plot three examples of a query image and the top support set images ranked by weight w, along with the top 3 softmax predictions. We notice significant visual similarity between the query image and the top support set images. Notably, the second query of "cardinal' is incorrectly classified by the model. We
8

---

Published in Transactions on Machine Learning Research (02/2023)
C0f
Bird-200
Dog-125
Aircraft-100
INaturalist-
NwV rours)
Confidence
Confilince
conficerce
confidnce
confioence
Conficence
Figure 4: Reliability diagrams. NW generally has an advantage in calibration performance on small to medium-sized datasets.
notice that the model is confused between the top classes in the support set. The model thinks the third query of "barbeton daisy' is most similar to "sunflowers", possibly due to the similarity in color and shape. A user can look at the support set and potentially intervene, either by correcting the model's prediction or removing confounding elements from the support set. An automated method for flagging conspicuous predictions could be based on confidence scores. Additional visualizations are provided in Appendix A.7
Comparing against FC features. In Ap- pendix A.8] we compare against ranking images by Euclidean distance in the feature space for the base- line FC model, and show that the degree of semantic similarity between the query and top most similar images is lower than the NW head. In Fig. 5 we quantify this by computing the percentage of top support set labels which match the query label. The NW model has a higher degree of semantic similar- ity with its top support images. We attribute this to the fact that semantic similarity is explicitly en- forced in the NW head (i.e. features from the same class are encouraged to be close), while this is not enforced in the FC head.
60
Percent
40
20
NW
Cifar-100
Bird-200
Dog-120
Flower 102
Aircraft-102
#aturallist104 10kk
Figure 5: Percentage of top support set labels which match the query label. Top labels are found by rank- ing support images by negative Euclidean distance and taking the top 10 images. The NW model has higher degree of semantic similarity with its top support im- ages.
4.3.2 Explaining
Model Behavior with Support Influence
Support influence can be used to debug and diagnose issues otherwise black-box neural networks. In this section, we explore the use of support influence in helping to determine helpful and harmful training examples. Note that the support influence requires knowledge of the ground truth label for the query image and thus cannot be used during inference/test time.
By ranking and visualizing the support set by support influence, we can see the support images which are most helpful (highest Z(z,z,) values) and most harmful (lowest I(2, 2.) values) in predicting the given query z. In Fig. 7 we visualize three query images and their top 5 most helpful and top 5 most harmful support images. The top images are "helpful" the sense that they belong to the same class as the query and look visually similar. The bottom images are "harmful" since, despite the fact that they are visually similar, they are a different class. For the first dog example, we see that "redbone" and "walker hound' breeds are harmful examples for the "beagle" query image.
The bird and flower queries are the same query images from Fig. 6 this time ranked by influence. For the bird, we see that helpful and harmful support images separates the two classes that had the highest weight and were confusing the model in Fig. 6 For the flower, we notice that the most helpful support images are from the correct class but have different colors, whereas the images which the model thinks are most similar the query are in fact the most harmful. From this analysis, it is evident that the model is using the shape and color to reason about the query image, which are confounding features for this query.
9

---

Published in Transactions on Machine Learning Research (02/2023)
Query
Support by Weight
Tin soore
1.0
D.S.
0.2
dinment
camoorure
OU
mananger
thnsger
masgerr
0.5 Snoger
tanager
nandin
tachsper
Carta
Panager
Las
1.6
0.6
02
Sobeen
0.0
Figure 6: Examples of correctly (dog) and incorrectly (bird and flower) classified query images and their top support images ranked by w. Ordering is left to right, lop to bottom.
5
Discussion
In this study, find that the NW head has comparable to improved accuracy on all datasets tested on. For calibration, NW head had the most improvement on data-starved scenarios with relatively fine-grained classification tasks. Indeed, on the small to medium-sized datasets (Bird-200, Dog-102, Flower-102, and Aircraft-100), we find the most dramatic improvement calibration on all model variants tested. This calibration improverent is less pronounced in large dataset. like iNaturalist-10k, and the improvement disappears (although is still comparable) for more diverse datasets like Cifar-1od]] Intuitively, this could be attributed to the fact that classification tasks with lower inter-class variability will tend to have higher similarities across classes, thus leading to better-calibrated uncertainties in the model predictions. On the other hand, tasks with high inter-class variability might be more suited for learning separating hyperplanes. as is the case with FC. We believe that an NW model is well-suited for applications where data is not
RRceent work shown that large datasets are as prone poor calibration Minderer al. 2021
10

---

Published in Transactions on Machine Learning Research (02/2023)
Query
boagn
beant
boggee
beig
beage
Hormuu Infucrco
Query
Heppt Infucrco
Hall
candmal
CarObnal
Query
Bamagerr
tanager
nelind Infuunnc
Sarc Sar
Betton
Horman Pomuen
MurT Coune
Tr.17 COOOM
burflowwr
Figure 7: Query images and support images separated by most helpful influence (top row) and most harmful influence (bottom row).
abundant and differences between classes are subtle, like in medical imaging. Due to its simplicity, future work may explore combining the FC and NW heads in some fashion to further improve model performance.
The computational load of the NW head for Full mode higher than the FC head, although this load is less for smaller datasets. However, flexibility at inference-time and interpretability head. In this study, we explore several choices of inference mode which each have advantages and trade-offs. In particular, we find that Cluster mode can be sufficient replacement to Full mode (Fig. -. while being computationally cheaper (Table 4 Nevertheless, Full mode allows user practitioners to interpret and debug a model's behavior (Section 4.3) Thus, one can imagine a workflow for the practitioner which would involve using Cluster mode perform efficient inference, and then turning to Full mode on select few (potentially problematic) test queries understand model behavior. We stress that this flexibility at inference time is not possible with FC baseline models, which have the clear advantage in terms of computational efficiency.
advantages of the NW
11

---

Published in Transactions on Machine Learning Research (02/2023)
6 Conclusion
We presented the Nadaraya- Kadaraya-Waason Watson (NW) head as a gencral-purpose, flexible, and interpretable building block for deep learning. The NW head predicts the label for a given query by laking the weighted sum of labels in a support set. We show that the NW head can be an efficient replacement for the fully-connected layer and offer good accuracy with excellent calibration, particularly in data-starved and fine- grained classification tasks. Existing calibration techniques proposed for parametric models can lead further calibration gains. Additionally, the NW head interpretable in the sense that the weights correspond to the contribution of each support image the query image. Finally, we define support influence, and show that this can be used tool to explain model behavior by highlighting helpful and harmful support images.
7 Acknowledgements
This work supported by NIH grants RO1LM012719, ROLAG053949, and RF1 MH123232; the NSF NeuroNex grant 1707312, and the NSF CAREER 1748377 grant.
12

---

Published in Transactions on Machine Learning Research (02/2023)
References
Samyadeep Basu, Philip Pope, and Soheil Feizi. Inftuence functions in deep learning are fragile, 2020.
Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics).
Vernnger-Verlgg, Berlin, Heidelberg, 2006. ISBN 0387310738.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of lokens, 2021. URL
04426
Thang D. Bui, Daniel Hemándee-TLobtto, Yingzhen Li, José Miguel Heránndez-oobato and Richard E. Turner. Deep gaussian processes for regression using approximate expectation propagation, 2016. URL
https://arxiv httpss////axiv.orr/abbbass/ 1603301133 04133
Chaofan Chen, Oscar Li, Chaofan Tao, Alina Jade Barnett, Jonathan Su, and Cynthia Rudin. This looks like that: Deep learning for interpretable image recognition, 2018. URL https://arxii http:/ivvwrgraaaa//reeeeea aas///b/////// /006. 10574
Wei Chen, Yu Liu, Weiping Wang, Erwin Bakker, Theodoros Georgiou, Paul Fieguth, Li Liu, and Michael S. Lew. Deep learning for instance retrieval: A survey, 2021. URL
wttp:/wwww..rgg//////....1/ 11282
R. D. Cook and S Weisberg. Residuals and influence in regression, 1982.
Andreas Damianou and Neil D. Lawrence. Deep gaussian processes. In Carlos M. Carvalho and Pradeep Ravikumar (ods.), Proceedings of the Sinteenth International Conference Artificial Intelligence and Slalistics, volume 31 of Proceedings of Machine Learning Research, pp. 207-215, Scottsdale, Arizona, USA, 29 Apr-O1 01 May 2013. PMLR. URL httpps:/pproceedings mlr pprsss/311ddmiinouu html
A Aexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2020. URL
https://arxivv prgg/bs//000 11929
Rasool Fakoor, Pratik Chaudhari, Jonas Mueller, and Alexander J. Smola. Trade: Transformers for density estimation, 2020. URL
024441
Nicholas Frosst, Nicolas Papernot, and Geoffrey Hinton. Analyzing and improving representations with the 01889
soft nearest neighbor loss, 2019. URL
J. Ghosh and A. Nag. An Overview of Radial Basis Function Networks, pp. 1 36. Physica-Verlag HD, Heidelberg, 2001. ISBN 978-3-T8 978-3-7908-1826 1826-1 doi: 1110007977---7900-111 10. 10.11077978----908-1838
URL https://doi.org/
macaodha Grant Van Horn. inat challenge 2021 fgvc8, 2021. htttss//kagglee.comooopttttt
Inaturalist-2021
Qingji Guan, Yaping Huang, Zhun Zhong, Zhedong Zhong, Liang Zheng, and Yi Yang. Diagnose like a
radiologist: Attention guided convolutional neural worw for thorax disease classification, 2018. URL
https://arxiv. org/abs/ aataaaasa.1.... 1////10000.0999 09927
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks, 2017. URL http::::/aaxxv..rr/aabb.ss/ 04599
Kazuaki Hanawa, Sho Yokoi, Satoshi Hara, and Kentaro Inui. Evaluation of similarity-based explanations. In International Conference on Learning Representations, 2021. URL httpss//opearevvve.nntt/ooo id=9uvhpy@wzM
13

---

Published in Transactions on Machine Learning Research (02/2023)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks, 2016. URL
05027
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolu- tional networks, 2016. URL
06993
Ahmet Iscen, Jack Valmadre, Amurag Arnab, and Cordelia Schmid. Learning with neighbor consistency for noisy labels, 2022. URL htttps//aaxxii http:://..wvvoorr/..a/// http://bs/222...2222
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Per- ceiver: General perception with iterative attention, 2021. URL
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine-grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions, 2017.
Jannik Kossen, Neil Band, Clare Lyle, Aidan N. Gomez, Tom Rainforth, and Yarin Gal. Self-attention between datapoints: Going beyond individual input-output pairs in deep learning, 2021. URL https:
wwwpiii.rrgaabb////eeee 2200b/028880881111 02584
Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
Steinar Laenen and Luca Bertinetto. On episodes, prototypical networks, and few-shot learning, 2020. URL
https://arxin: http://axvvvv.. arg/abs///t abt//a//..22222/////2222222e222/ 2012.00122.0 09831
Xuhong Li, Haoyi Xiong, Xingjian Li, Xuanyu Wu, Xiao Zhang, Ji Liu, Jiang Bian, and Dejing Dou. Interpretable deep learning: Interpretation, interpretability, trustworthiness, and beyond, 2021. URL www.aaiww..aaaaaaeeeaaeeaoooeeee org/abg/////oooooooooooooooooeot 210.//001101
S. Maji, Kannala, Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft, 2013.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks, 2021. URL htt.hhttss://a// org/abs/2106 07998
E. Nadaraya. On estimating regression. Theory of Probubility & Its Applications, 9(1):11 142, 1964. doi: 10.1137/1109020. URL https://doi. org/10. 10. 1137/1109020
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Ninth Ninth AAAT Conference on Artificial Intelligence, AAAI15, pp. 2901-2907. AAAI Press, 2015. ISBN 0262511290.
Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression, 2021.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In Indian Conference Computer Vision, Graphics and Image Processing, Dec 2008.
Nicolas Papernot and Patrick McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning, 2018. URL
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer, 2018. URL http:p///aax...orr/////////////
14

---

Published in Transactions on Machine Learning Research (02/2023)
Devendra Singh Sachan, Siva Reddy, William Hamilton, Chris Dyer, and Dani Yogatama. End-to-end training of multi-document reader and retriever for open-domain question answering, 2021. URL https:
//arxiv.org/abb/2106.coo
Hugh Salimbeni and Mare Deisenroth. Doubly stochastic variational inference for deep gaussian processes, 08933
2017. URL
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. One-shot learning with memory-angmented neural networks, 2016. URL https://arxiv.. wtter//w..rr/aa///1.....6
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning, 2017. URL
https///arravv.or///b////110 05175
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H. S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning, 2017. URL httpp/////aaiiioo///////////////
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision, 2015. URL hatpp:://arxxiv.og///b///222
Mohammad Reza Tacsiri, Giang Nguycn, and Anh Nguyen. Visual correspondence--asse explanations improve Al robustness and human-Al team accuracy. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL. https: //openreview //openteeiee..ne//fforrrmr2
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017. URL http:///axxvvo.or/bbsee 03762
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning, 2016.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongic. The caltoch-ucsd birds-200-2011 dataset, 2011.
Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang. Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang. Residual attention network for image classification, 2017. URL http:///aaxxvv.o//ab/////// 06904
G. S. Watson. Smooth regression analysis. Sankhya: The Indian Journal of Statistics, Series A(26):359 372, 1964.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning, 2015.
URL https://aaxxi httpe///axxvvvoorrabbba wttp:/r//////a/./.aa 02222
Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non-parametric
instance-level discrimination, 2018. URI. https: wwwaawwwwoo//b///1/...5...1
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhuldinov, Richard Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention, 2015. URL https://aaxii: www.aiii.../.......e..ee
wwtg/abs/Iie 1502/1102..00 03044
Sukmin Yun, Jongjin Park, Kimin Lee, and Jinwoo Shin. Regularizing class-wise predictions via self- knowledge distillation, 2020. URL https://arxiv orgg/as///00///oo/ 13964
Dequan Zhang, Ning Zhang, Nan Ye, Jianguang Fang, and Xu Han. Hybrid learning algorithm of radial basis function networks for reliability analysis. IEEE Transactions Reliability, 70(3):887 900, 2021. doi: wttp:/wwwwwrgg///////..... 00121
Quanshi Zhang, Yu Yang, Haotian Ma, and Ying Nian Wu. Interpreting cnns via decision trees, 2018. URL
Boyan Zhou, Quan Cui, Xhu-Shen Wei, and Zhao-Min Chen. Bbu: Bilatcral-branch network with cumulative learning for long-tailed visual recognition, 2019. URL htttp:://axxxw.rrr//bb///111
15

---

Published in Transactions on Machine Learning Research (02/2023)
Table 6: Summary of Datasets
Dataset
Description
# Troin
# Test
# Classes
Gifar-100
Krizhevsky
2009
Image classification
50,000
10,000
100
Bird-200
Wah et al.; 2011
Bird species classification
5,794
5,994
200
Dog- 120
Khosla et al. 2011
Dog species classification
12,000
8,580
120
Flower- 102
NNllsbac & Zisserman
2008
Flower species classification
2,040
6,149
102
Aircraft-100 100
[Maji et al. 2013
Aircraft classification
3,334
3,333
100
NNtuuralst
Grant Van Horn
2021
Species classification
500,000
100,000
10000
A
Appendix
A.1 Datasets
In this section, we give an overview of the datasets used in this study. Table gives a summary of all datasets, including description, train/tees sizes, and number classes. Cifar-100 has input image sizes of 32x32, while all other datasets are resized to 256x256 and cropped to 224x224. For Cifar-100, Flower-102, Aircraft-100, and iNaturalist-0Ok we use the implementation in the torchvision package. For iNatralist- the "mini" training dataset from the 2021 competition, which has balanced images per class, and test the validation set (the test set is not provided). For Bird-200 and Dog-120$) pull the train/test splits from the dataset websites.
we use
A.2 Models and Training Details
For all proposed models and baselines, use models implemented in the torchvision package, with randomly initialized weights. Following the setup of Yun et al. (2020), we use standard ResNet-18 with 64 filters and DenseNet-122 121 with a growth rate of 32 for image size 224 x 224. For Cifar- 100, we use PreAct ResNet-18 (He et al., 2016), which modifies the first convolutional layer4 with kernel size x 3, strides 1 and padding 1, instead of the kernel size 7 x 7, strides 2 and padding 3, for image size 32x32. We use DenseNet-BCi structure (Huang et al., 2016), and the first convolution layer of the network is also modified in the same way as in PreAct ResNet-18 18 for image size 32 32. We use the standard data augmentation technique of random flipping and cropping for all experiments.
For NW training, use SGD with momentum 0.9, weight decay le-4, and an initial learning rale of 1e-3. The learming rate is divided by 10 after 20K and 30K gradient steps, and training stops after 40K gradient steps. For all datasets except NNaturalist-00k, we set the randomly sampled support size to be N, = 10 for all datasets, and the mini-batch size to be N,, = 32 Cifar-100 and N,, = for Bird-200, Dog-120, Flower-102, and Aircraft-100. That is, we sample a unique support 10k, we use a mini-batch size N₂ 128 and support size N, 250. We sample single support set for each mini-batch (instead of each query) for computational efficiency. Effectively, this makes the total number of images per mini-batch to be N, + N₂, instead of N, Note that there is very little ellect of this change on training because the number of unique pairwise similarities is preserved.
for each query, following Eq. (6] For iNaturalist-
For FC training, we train for 200 epochs, use an initial learning rate of 0.1, and divide the learning rate by after epochs 100 and 150. We found that this setting yielded better results for the baseline models. All other hyperparameters are identical to the NW setup.
All training and inference is done an Nvidia A6000 GPU and all code is written in Pytorci
2011/
http://visiin. stanford wtt.a//owt.......i.iieaaeeeeeeea
Our code available al http:////tthabbbccccaaaaagnngnnn
16

---

Published in Transactions on Machine Learning Research (02/2023)
A.3 Details Calibration Experiments
To produce results in Section 4.2.1 we follow methods and hyperparameters from the original papers
temperature scaling (TS) (uuo et a.. 2017) and label smoothing (LS) (Szegedy et al. 2015).
For LS, we modify all ground truth labels by taking the weighted average between the onc-hot label and the uniform distributtin:
l = (1 - +) ب
where C is the number of classes and is vector of all ones. We choose the weight used in the original paper, = 0.1.
For TS, we sweep т over 100 values linearly spaced between 0.5 and 3. We select the optimal which minimizes loss on the validation set.
A.4 Derivation of Support Influence
We derive the expression in Eq. For brevity, let f = f(x, and f- Sl.))_..))
I(z, = L(S-1)) L(f.v)
Σ of log l
Σ o log l
log fd log f
y = yss then w(x,x,) x.) must be subtracted from fu and re-normalized valid probability. Otherwise, f is simply re-normalized:
if = y/
f
1 m
else.
Thus,
f -
xee (()
I(2,2,) = loo - -log
- w(2
f fow(s,
= llog
fd w It (y=u,)
A.5 Connection between Support Influence and Weights
We prove that ranking by helpful/harmful examples via the support influence is equivalent to ranking by the weights for correct and incorrect classes. Let us start from the definition of support influence in Eq.
S
= log ( (fu w0-w(xx (
f - ff (
I(2, = -
where 0 S fM < 1. Note that S fa because they are related by Eq. 0
Separate into two cases:
log
1- 1-w(xx,.)) (-w(x,,,)/1) log(1 w xz )),
if y = Us elsc.
I(z, =
In the first case where y = Us. I(z, 2x) incrcases w incrcases, with the constraints on w and f".
In the second case where + ys. I(2, 2x) decreases as w increases.
In words, the support influence increases as the weight increases for the correct class, and decreases as the weight increases for the incorrect class. Thus, the support influence can be interpreted as simply ranking by the weights for correct and incorrect classes.
17

---

Published in Transactions on Machine Learning Research (02/2023)
Table 7: Error rate for various settings of (Nb; N₂), assuming fixed NgN, Cluster and A for NW models.
NG)
(1, 40)
(2, 20)
(4, 10)
(5, 8)
(8, 5)
(10, 4)
(20, 2)
Error rate
54.7
51.2
46.6
46.8
47.1
51.0
58.2
A.6 Analysis of Batch Size and Support Set Size
In this section, we provide hyperparameter sweep of N, and N₂ and present error rate results on the Bird- 200 dataset and ResNet-18. As discussed in A.2 the total number of images per mini-batch is defined by N,Ng, Common practice in machine learning dictates that the total number of images should be maximized in order to obtain best convergence, Thus, in this analysis, we keep No to fixed to 40 (same as our reported results), and sweep the values with this constraint.
We find that there is sweet spot around (4,10) and (5,8). We hypothesize that low batch size No hurts convergence, as is typical for SGD optimization. On the other hand, low N, may be non-ideal because the model does not have sufficient number of "negative" samples to compare against. With that said, the performance is not particularly sensitive with respect to reasonable values of No. We believe that balancing this tradcoff is important to maximizing performance on a given dataset.
18

---

Published in Transactions on Machine Learning Research (02/2023)
A.7
More NW Visualizations
Query
Top Support Weight
Back
bark moten
ock meted
Bocc
loted
Incted
lnnted
1.0
0.6
0,2
Soaey
janger
1000ed
pamarinc
footed alcatross
Pooted HOWTTOSS
booted
Jreger
FDS
rod
red
PCd
No pecherr
peoddgedker
woodpecker
wwoddeecker
woodseecue
1.0
foeded
0.6
helded
headed
hea
Reedded
Readed
monda
Honda
Ronda
Jay
by
bbv
Ferida
0.8
0.4
Row
0.0
Q-1
gaucEus (Ml
0.2
Calfoma
cut
TAI
quil
Quil
Figure 8: Query image, top 3 predicted probabilities, and lop support ranked by weight for Bird dataset.
19

---

Published in Transactions on Machine Learning Research (02/2023)
Query
Influsece
Back boted
Back albairous
blckk abatroa
Back
Bbok Wbb
aak
Elattoas
long gangee
Query
e Maatioon
Jangj
breasted gratheals
bostated
breasted
breagted
helded
headed woodpeceer
headed
headed
Query
weodpeeber
Help
Eonda
Query
AR0
1ng 06
biled sull
Hinged
winged gull
Ga Forts
gull
Figure Query image, top helpful support images, and top harmful support images rankod via support influence for Bird dataset.
20

---

Published in Transactions on Machine Learning Research (02/2023)
Query
Top Support by Weight
molon cote
Rion
mroon archud
mo00 KEchid
occlud
0.8
canterturry
bul
onchiid
anchtid
orchis
archid
chid
0.0
CC Dom
CITOOIN
CROCT
1.0
0.6
0.2
boadded
mellows
meliow
pantian
1.0
0.6
Star Hocd
CHEEP
anemoon
ember
aster
0.8
0.4
TRICK CASTE
mesisar
Ispanise
5.0
Figure 10: Query image, top predicted probabilities, and top support ranked by weight for Flower dataset.
21

---

Published in Transactions on Machine Learning Research (02/2023)
Query
archid
Query
calla Lilly
lamoon
lenton
250e
Query
Halp'
Stam
Harmfal
Query
byac HA
Tracincte
byac nt
eecinth
medicaa
asanese
mesican
CAnT
anamoon
Figure 11: Query image, top helpful support images, and top harmful support images ranked via support influence for Flower dataset.
22

---

Published in Transactions on Machine Learning Research (02/2023)
Query
Top Support by Weight
minoninc
mintuurc
minirure
0.8
Jcrkptie
Cardo
cchnauper
schnoupor
ochAa..00 201
0.0
Obctan
siondan
aeetan
1.0
0.6
dardaa
62.
Penner
schno Der
henter
obetam
berr
actm
Schn muuzer
1.0
0.6
Sbatan
Stendaand
Marddand
124
0.8
0.6
6.0
Figure 12: Query image, top 3 predicted probabilities, and top support ranked by weight for Dog dataset.
23

---

Published in Transactions on Machine Learning Research (02/2023)
Query
Query
7011 Am
Berrin
obscan
obetan
obcaank
obecan
Handaad
zandard
standard
Query
slandand
Mandard
Mandand
Standare
Query
burrier
batr
Helpfial
Hamerful
Figure 13: Query image, top helpful support images, and top harmful support images ranked via support influence for Dog dataset.
24

---

Published in Transactions on Machine Learning Research (02/2023)
25

---

Published in Transactions on Machine Learning Research (02/2023)
A.8 Ranking FC Images By Euclidean Distance in Feature Space
Query
Images Ranked by Euclidean Distance in Feature Space
Hert dbsttoos
Roleed Mootoooo
Robed abbetroos
looted elbatross
Mlbatrac
peger
Back albatroos
alta
Bigatebiru
commoranlt
auk
raly nummingsstd
Sum rooE
Mingband
Pumminggso
Grehwar
nee
Grandh
rantnem
Baraal
blidd
conuleaa con
cing
bunbing
Master
Calforaa Qu1
looged kithwoke
pu
Vul
ter
bem
barked
Calforin
Ingged In Elioaa
cul
Quil
ONJ
Figure 14: Query image and top images ranked by Euclidean distance in feature space of FC model for Bird dataset. The degree of semantic similarity between the query and the top images lower than the NW model (see differences in class labels).
26

---

Published in Transactions on Machine Learning Research (02/2023)
Query
Images Ranked by Euclidean Distance in Feature Space
thorn apple
Bioon
Bhor bpol
Shorn
thomn
thom
beed
RELM
anbaan
rose
bouge melico
Wibot OwN
S'apdragon
oemban
Bower
Beemad
clemanin
DOK
maxican
THOK
ngo 1e50
MeNIIOn
Figure 15: Query image and top images ranked by Euclidean distance in feature space of FC model for Flower dataset. The degree of semantic similarity between the query and the top images is lower than the NW model (see differences in class labels).
27

---

Published in Transactions on Machine Learning Research (02/2023)
Query
Images Ranked by Euclidean Distance in Feature Space
yonisaro
porishne
Tinattuee
yereshin
manaature
Dcntam
Srandard
503 Saalyiam am Zommer
SCTNIN DET
poodle
hemrer
000000
Standrd
oerrier
nd Tattet..oo
Zandard SchnS.cc 201
bemrer
bep
bemier
Speniet
hhirdese
barless
boy
mallase
Figure 16: Query image and top images ranked by Euclidean distance in feature space of FC model for Dog dataset. The degree of semantic similarity between the query and the top images is lower than the NW model (see differences in class labels).
28

---

Published in Transactions on Machine Learning Research (02/2023)
A.9
Performance support set size ResNet- 18
Ctfar-100
Bird-200
Dog-120
Flower-102
Aircraft-100
0.285
0.52
0.470
0.453
0.462
0.44D
0.45
0.800
0.50
0.425
0.435
0.41
0.48
0.215
0.430
0.40
0.375
0.265
18
22.0
24
19
ECE
18
2.50
2.0
1.5
2.35
1.40
1.8
1.4
10
9100
Suppprt Images
Support Imagees
7 Support Images
Support Images
Support Images
FC
NW. Random
NW. Cluster
NW. Closest cluster
NW. Full
A.10
Reliability diagram on DenseNet- 121
Cifar-100
Bird-200
Dog-120
Aircraft-100
Flower-102
NW (ours)
Accuracy
0.5
0.5
0.5 Confidence
0.5
0
0.5
0.5 Confidence
Confiiienc
Confidence
Conferece
29


Reviews:
[{"review_id": "VfbjxXIDt7", "reviewer": "Reviewer_dLd3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Although it is not a very original topic, the paper achieves to show very clearly and in a relatively convincing manner the benefit and relevance of a non-parametric classification layer. I found the explanations provided to be quite clear, the level of details is appropriate (but the proposed approach is relatively simple by itself, which is good), and the relation to other related work is well made and does a proper overview (although a little short) of other related approaches.\n\nI think this paper can have an impact in establishing a way for implementing a non-parametric classification output layer, which is of interest for a relatively broad part of the deep learning community. It can reveal to have a good citation level in a few years from now.\n\nRegarding the weaknesses of the paper, I think it falls short in terms of experiments. It is currently limited to image classification over four datasets (bird, dog, flower, aircraft), which appear of similar limited size (10000s to 20000s instances) and similar number of classes (100s-200s classes per dataset). That’s quite a restrictive setting and the capacity of the approach to properly perform on different datasets, especially larger ones, is not clear to me. I would in fact guess that more classical models with fully connected output layers will behave much better with larger training sets and possibly outperform the proposed approach.\n\nAnother issue is the scaling of computation with the proposed approach, with larger datasets. Results in Table 3 show a slowdown by a factor of 50% with the full approach on the bird dataset, which is only of ~5000 training instances. I think performance may be hurt much more with larger datasets, as the algorithm will not scale very well, having an O(n^2) complexity regarding the dataset size, if implemented relatively naively.\n\nAlso, I note that the calibration results appear quite good in Table 1 when compared to FC output layer results, but are much closer when calibration with temperature scaling is done in Table 2. In fact, I consider that the temperature constant used for the proposed NW approach presented in Eq. 5 (temperature is the $\\tau$) already leads to some form of temperature scaling for the proposed NW approach. Given this, I consider the evaluation of NW with non-calibration FC is somehow not fair and should not be presented in Table 1 and Figure 4. Even more, I am making the hypothesis that the good results in Table 2 results from a much finer calibration done with NW (a double calibration in fact, one in Eq. 5 and a subsequent one with temperature scaling).\n\nExperiments settings are not well detailed on several aspects and I am not convinced that they can easily be reproduced only from the paper. Details given in page 6 are not enough, in my opinion. Also, given the lack of details on the hyperparameters used, it gets difficult to spot if any of them have great influence on the performances. More analysis and possibly some ablation studies would allow to increase confidence that the configuration used for the proposed approach is well optimized.\n\nArguments on explainability are mostly based on examples. That remains relatively weak, in my opinion, I would have liked to get stronger arguments than that. One may argue that the examples were cherry-picked and the general behavior may not be as nice as what is presented. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper is presenting a new output layer for deep networks in the context of supervised learning. The proposed layer is a non-parametric model that achieves classification of given query instances regarding other support instances associated with classes, relying on distances in the feature space obtained with the deep network. Although such a model is not of outstanding originality – the authors are making a good overview of other works proposing similar non-parametric approaches for classification with deep networks – it makes the proposal very clear and well developed. In particular, the authors argue that such an approach allows some form of (partial) interpretation, similarly to attention mechanisms according to the classification weights of the supports, while achieving better calibrated output. Results are presented on some datasets to support the claims.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Experiments with other datasets, especially larger datasets, should be conducted to validate that the approach proposed is not good only with the type of datasets used for evaluation (small-mid size image classification datasets with 100s-200s classes), and can work in other settings.\n\nFull details on datasets (size, number of features, number of classes, other characteristics) and full details experimental setting should be provided. This can be in the appendix. It should be made to both apprehend better the results reported – I need to check on the Web for getting the details on the datasets and figure out how similar they were – and allow reproducibility of the results.\n\nMore explanations on the choice of hyperparameters used, showing that the choices for the values used have been properly studied, with possibly some ablation studies over some design choices, to validate that they are the right ones. \n\nExperiments on calibration are not fair on some aspects, as the approach is already doing some form of calibration through the $\\tau$ parameter of Eq. 5. Adjustment of this parameter is not detailed, but I am quite sure adjusting it would explain a big share of the proper calibration obtained. I think this part of the results should be revised completely to allow a better comparison with FC.\n", "broader_impact_concerns": "No broader impact concerns with this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "h9wjSnkzOa", "reviewer": "Reviewer_5nr3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n  * Non-parametric approaches for calibration have been overlooked in recent studies of neural network calibration.\n  * The Taylor expansion of the influence function (equation 2) can be computed exactly for this new 'head.'\n  * Classically, the main pain point of non-parametric approaches is 'how to find the support set?' and 'what size support set to store?'. The paper addresses these questions finely with the comparisons of support sets in section 3.3.\n\n\nWeaknesses:\n\n * The main weakness of this paper is the experimental evaluation. All datasets are roughly ten years old or older (2008, 2011, 2013) and do not reflect modern problems. If I counted correctly, all datasets are smaller than 15k images (CUB (6k), S Dogs (12k), Flowers (1k), Aircraft (10k)). Recent works show that larger datasets and larger models already yield 'good calibration' [1].\n* --> It should be noted that small experimental results need not be a negative point in this review. However, suppose one chooses to use small datasets and small models, I expect better experimental conduct. For example, it seems quite easy to calculate error bars or quantiles for the results in Table 1. Also, the diagrams in Figure 4 could easily be plotted with error bars or quantiles for such small models and datasets. \n* A small comment on Table 1, all major studies on neural network calibration have recommended tempering the predictions[1, 2], which has not been used here?\n * This new 'head' will require more memory and compute compared to fully connected layers. And on this note: Table 1 shows that 'NW, full' is best or second best on almost all datasets, and Table 3 shows that it's 50% more expensive in terms of compute. What applications do the authors have in mind for this method that takes so much more compute?\nLikewise, all visualization studies in section 4.3 are made with the 'full support set', which is 50% more expensive in terms of compute. What are the conclusions from such visualization study when the model is compared with another model that uses the same amount of compute?\n\n[1] Minderer et al. \"Revisiting the calibration of modern neural networks.\" NeurIPS 2021\n[2] Guo et al. \"On calibration of modern neural networks.\" ICML 2017\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of calibration in deep learning. Given the output on a query, one desires the probability of error to be reflected by that number. The proposed method attaches to a neural network like a 'head', makes non-parametric decisions for classification, and obtains the outcome 'score' from a similar non-parametric approach. \n\nThis method could be a major contribution if the experiments were run on recent and practical datasets. However, most datasets in this study are ten years old and of small size. For these experiments, no error bars or quantiles are reported. Moreover, the visualization studies are run with a method that uses 50% more compute than other methods. \n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Adjustments:\n  * Either report Table 1 and Figure 4 with error bars/quantiles, or evaluate on larger datasets, e.g. ImageNet. \n\nMinor points:\n * Table 1 is confusing to interpret, because accuracy is better when higher, and ECE is better when lower. Why not report error rate instead of accuracy? Then both metrics are better when lower (or report 1-ECE so both metrics are better when higher).\n\nOpen question that is unrelated to this review:\n* Could a special case of the non-parametric classifier be proven to be a proper scoring rule? In that case, there might be a bound on the ECE.\n", "broader_impact_concerns": "N.A. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jJYCiLVgdR", "reviewer": "Reviewer_Su24", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed NW head is simple and easy to implement and integrate into existing convolutional neural networks.\n- The paper is clearly written and easy to follow.\n\nWeaknesses:\n- The idea is not novel. In fact, there have been more sophisticated deep prototype-based models that learn prototypical examples/prototypical parts of various image classes, and classify an input image by comparing the image with those prototypical examples/prototypical parts in the feature space (e.g., ProtoPNet by Chen et al., NeurIPS 2019). Those learned prototypical examples/prototypical parts can be understood as \"support images.\" There are also deep k-NN based models that classify an input image by comparing the image with (patches of) training images in the feature space (e.g., visual correspondence model by Nguyen et al., NeurIPS 2022). In this case, the entire training set can be understood as \"support images.\" All these models classify an input image by comparing it with \"support images\" in the feature space.\n- The accuracy of the NW models (and also the baseline models) is low. Did you use ImageNet-pretrained models?\n- There is a lack of comparison with other state-of-the-art models in terms of accuracy, and there is a lack of comparison with other interpretable models in terms of interpretability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors proposed a Nadaraya-Watson (NW) classification head that can be attached to any deep convolutional feature extractor for image classification. In particular, the proposed NW head compares an input image (\"query\") with a set of support images in the feature space, and classifies the input image into a class whose support images have the highest total similarity with the input image in the feature space. A model with an NW head can be trained using a cross-entropy loss.\n\nDuring inference, a support set can be the entire training set, a random subset of the training set, the cluster centers (in the feature space), or the closest training images to the cluster centers (in the feature space). The authors also proposed the concept of support influence, which quantifies the change in the cross-entropy loss on a test instance when a support element is removed.\n\nIn their experiments, the authors trained ResNet-18- and DenseNet-121-based NW models on the CUB-200-2011, Stanford Dogs, Oxford Flowers, and FGVC-Aircraft datasets, and found that their NW models achieved comparable performance with the baseline models with a fully-connected last layer. They also studied the top support images to given query images, as well as support images that are most helpful and most harmful to query images (defined by the highest/lowest support influences).\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Based on the above assessment, I recommend rejecting this paper. Since the idea is not novel, I encourage the authors to more carefully study the existing literature and to develop more novel ideas that can improve both the accuracy and interpretability over the current state-of-the-art.", "broader_impact_concerns": "There are no ethical concerns associated with this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission investigates a nonparametric alternative to the fully-connected output layer for deep neural network classifiers. The Nadaraya-Watson (NW) prediction head, as it is called, makes a prediction for a query example in the form of a weighted average of the support set labels. The weights themselves are expressed as a function of the Euclidean distance (in the space of embeddings output by the neural network) between the query example and individual support examples, and are normalized to sum up to 1.

The NW prediction head's properties are analyzed empirically through experiments on CUB-200-2011, Stanford Dogs, Oxford Flowers, and FGVC-Aircraft and are compared against those of a conventional fully-connected prediction head.

Reviewers find the proposal very clear and well developed (dLd3, Su24), and relevant to the community (dLd3), especially given that non-parametric approaches for calibration have been overlooked in recent studies on neural network calibration (5nr3). Concerns on consistency of the presentation in the tables (5nr3), error bars (5nr3), details of the experimental setup (dLd3), and missing connections to related works on deep prototype-based models (Su24) are adequately addressed by the authors in their response.

The main outstanding reviewer concerns are:

1. The submission's experiments are limited to image classification with relatively old datasets of similar limited size and similar numbers of classes (dLd3, 5nr3). In the context of recent works showing that larger datasets and larger models already yield "good calibration", Reviewer 5nr3 would like to see experiments at a larger scale to determine whether the NW head's calibration benefits are still significant in that regime.
2. The NW head has poor scaling properties with respect to dataset size (dLd3, 5nr3).

In their response, the authors present results on two new, larger datasets (CIFAR-100, iNaturalist-10k). They acknowledge the computational shortcomings of the NW head, but point to the Cluster mode as a cheap and effective replacement for the Full mode. They also argue that high-risk, safety-critical domains like medical imaging put less emphasis on high inference throughput and more emphasis on performance, interpretability, and explainability.

Opinions remain split on acceptance. While it is true that investigating even larger data regimes would strengthen the submission, the submission as it stands is well-supported given the scope of the claims made, and reviewers agree that the topic is of interest to the TMLR community.

I therefore recommend acceptance.
Decision: Accept (poster)