Worst-case Few-shot Evaluation: Are Neural Networks Robust Few-shot Learners?
Authors: Yudong Wang
Ma Chang
Qingxiu Dong
Lingpeng Kong
Zhifang Sui
Jingjing Xu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Distributional Robustness
few-shot evaluation

Abstract: Neural networks have achieved remarkable performance on various few-shot tasks. However, recent studies reveal that existing few-shot models often exploit the spurious correlations between training and test sets, achieving a high performance that is hard to generalize. Motivated that a robust few-shot learner should accurately classify data given any valid training set, we consider a worst-case few-shot evaluation that computes worst-case generalization errors by constructing a challenging few-shot set. Specifically, we search for the label-balanced subset of a full-size training set that results in the largest expected risks. Since the search space is enormous, we propose an efficient method NMMD-attack to optimize the target by maximizing NMMD distance (maximum mean discrepancy based on neural tangent kernel). Experiments show that NMMD-attack can successfully attack various architectures. The large gap between average performance and worst-case performance shows that neural networks still suffer from poor robustness. We appeal to more worst-case benchmarks for better robust few-shot evaluation.

Under review as a conference paper at ICLR 2023
WORST-CASE FEW-SHOT EVALUATION: ARE NEURAL NETWORKS ROBUST FEW-SHOT LEARNERS?
Anonymous authors Paper under double-blind review
ABSTRACT
Neural networks have achieved remarkable performance on various few-shot tasks. However, recent studies reveal that existing few-shot models often exploit the spurious correlations between training and test sets, achieving high performance that is hard to generalize. Motivated by fact that robust few-shot learner should accurately classify data given any valid training set, we consider a worst-case few- shot evaluation that computes worst-case generalization errors by constructing a challenging few-shot set. Specifically, we search for the label-balanced subset a full-size training set that results in the largest expected risks. Since the search space is enormous, we propose an efficient method NMMD-attack to optimize the target by maximizing NMMD distance (maximum mean discrepancy based on neural tangent kernel.. Experiments show that NMMD-attnn MMMD-attack can successfully attack various architectures. The large gap between average performance and worst-case performance shows that neural networks still suffer from poor robustness. We appeal more worst-case benchmarks for better robust few-shot evaluation.
INTRODUCTION
Given limited number of supervised samples, few-shot learning aims to achieve high generalization
performance on unseen test data (Wang al., 2020 Yue al.
2020). Recent years have witnessed
rapid advancement of few-shot learning. particularly with neural networks pre-trained with self-
supervised data (Brown et al., 2020; Dosovitskiy et al., 2021; Tan Le 2021). Neural networks
gradually become the dominant solution to few-shot learning. Some networks (He et al.) 2021 Xu
ctt al.
2022) even surpass humans standard benchmarks (Wang et 2019 Mukherjee et 2021).
Despite promising results on fixed sets, recent work (Sagawa et al. 2020 Taori et al. 2020a Koh
ct al. 2021 Tang et al. 2022) reveals that neural networks as few-shot learners exacerbate spurious correlations and easily fail on distribution shifts. Spurious correlation is typical representation learning problem in which models learn to classify based on superficial features. For example, when learning the class label pigs from a few pig images, neural network models sometimes learn to guess based on superficial features (e.g., background with farm fences) rather than learn to generalize base on essential features (e.g., the facial characteristics of pigs), shown in Figure] [
Over-fitting to spurious attributes brings performance increase hallucination but does not guarantee better robustness, which explains over-optimistic performance on existing benchmarks (Mutton et al.) 2007 Vinyals et al. 2016 Oreshkin et al. 2018) SSchick & Schützz 2021 Alayrac et al. 2022aa). The performance models are assessed according the averaged test accuracy given fixed training set (1-fold evaluation) or several random subsets of the training set (k-fold evaluation). In that procedure, is easy for the superficial features be carried by the few-shot sets and eventually exploited by the neural networks since training and test data usually come from the same data distribution in the construction of the benchmark (Sagawa al. 2020).
Motivated by fact that robust few-shot learner should accurately classify data given any valid training set, we propose a worst-case evaluation for few-shot learners in this work. Worst-case evaluation targets evaluate generalization error bounds. Instead of randomly sampling few-shot sets, we search for the worst-case few-shot set from a full-size training set with balanced labels. An illustration of worst-case few-shot evaluation is shown in Figure I Inspired by the notion that spurious correlations often arise from common statistical features (Sagawa et al. 2019 Tang et al. 2022), the distribution of unbiased samples generally has large divergence with the full-size training

---

Under review as a conference paper at ICLR 2023
On couch cat the grass sheep fence fence-ppigs
Worst-case Few-ehot Set Distribution
Random
Model
Aoc: 92.1%
Test Set
Worst-case
Acc: 76.3%
Model
Real-world Data Distribution Pij
Figure I: An illustration of worst-case evaluation. Spurious correlations affect few-shot evaluation. Since training and test data usually come from the same distribution, over-fitting to spurious features brings performance increase but does not bring better robustness. For example, models learn to classify pig based on superficial features (e.g.. in a fence), rather than the shape. In this work, we consider a worst case evaluation for few-shot robustness evaluation by extracting challenging and label-balanced subset from full-size training set with the largest expected risk.
set. Therefore, we adopt a distribution divergence maximization approach NMMD-attack to find the challenging few-shot set. This approach also theoretically guaranteed as the generalization error of a few-shot set is bounded by the maximum mean discrepancy (MMD) distance (Gretton al. 20T2) between the few-shot distribution and the original training distribution. The goal of maximizing generalization етrоr can then be simplified to maximizing the MMD distance between the few-shot set and the full-size training Following the MMD maximization principle, use the MMD distance in the hypothesis space to define group (or set) distance. Since the MMD distance still intractable, we borrow neural tangent kernel, an approximation over sere-prametteized neural networks, to estimate MMD distance without optimization. Given the searched subset, we train models and report test accuracy.
Experiments show that NMMD-attack challenges high few-shot performance in randomly-sampled cases. It can successfully attack various model architectures with large performance drops. For example, the performance of DenseNet-122 drops by 10.02% on the generated few-shot set. Also, our case study on ImageNet-IK 1K and CIFAR-10 demonstrates that the generated few-shot sets show much fewer spurious attributes than mandomly-samples few-shot training sets.
This work re-examine the actual ability of representative neural networks on few-shot cases. The large performance drop indicates large improvement space in future work. Actually, the bias to spurious attributes can be a severe system bug and loophole for all few-shot learners. The attacker manipulate sets with unseen correlations destroy model, which is hard detect. How to avoid learning spurious features is a hopeful direction to improve few-shot robustness. Furthermore, compared with existing few-shot benchmarks, worst-case evaluation can provide new view demon- strating how worse a model can be such that we can prepare backup plans in case of accidents in real-word applications.
this work, we provide feasible solution estimate generalization error bounds for fhot cases, we appeal to more worst-case benchmarks for better few-shot evaluation future.
2 RELATED WORK
In this work, we review related topics, including adversarial attack, distribution shift, and distribution robustness optimization.
Our work comes as form attack inspired by adversarial attack literature (Szegedy et al. 2013 Goodfellow et al.] 2014). Adversarial attacks aim to fool neural networks while keeping innocuous to humans. This form of attack, though effective, alters each sample independently and ignores group correlations. Performing attacks mainly concerns making slight variations, e.g. adding noise to the sample. Since few-shot cases usually have serious spurious correlations between groups, the target of worst-case few-shot evaluation is evaluate robustness to distribution shifts.
2

---

Under review as a conference paper at ICLR 2023
Distribution shift naturally arises from real-world applications and has been widely used in robust- ness (Sagawa et al. 2019,2021,, Also, distribution discrepancy metrics, similar to the NMMD metric in our work, have been extensively used (Peng et al. 2019; Chen et al. 2020; Zhao et al. 2021 in predicting generalization performance under data shifts. Distributional robustness (Sinha et al. 2017). though inspired by distribution shift, is distinct direction. Distribution robustness requires
models to minimize empirical error under any valid training set. Unlike attack methods (Madry et al. 2017: Cohen et al. 2019) that define their perturbation set with Ly-ball, distributional robustness analyzes set robustness.
Other works have also explored distribution shifts and adopted samples with spurious correlations as benchmarks. However, most studies (Sagawa et al. 2019) Taori et al., [220bb] Sagawa et al. 2021) rely on prior knowledge of spurious correlations, which limits its application to real-world tasks with complicated and unseen correlations. Existing benchmarks for few-shot learning, like Mini-Imagenet (Dhillon et al. 2019), CIFAR-FS (Bertinetto al.) 2018), liered-ImageNNt (Ren) et al. 2018), 8), mainly focus cross-class query and label shifts. A more practical setting is based on the idea of "in context" learning few-shot examples for large pre-trained models (Alayrac et al. 2022b Yang et al. 2022; Tsimpoukelli et al., 2021) and is the field that suffers over-optimistic performances. Our sho- robustness evaluation also follows this setting. We are the first to appeal to worst-case benchmarks.
3 NOTATIONS
For a better description, we introduce the notations used in this paper. Let be the data space, Y' be the label space, and H be the hypothesis space in our investigation H. X' - V. We consider classification tasks for simplification. The training set X consists
samples (z,, y:) ~ P(X, Y). The model fe E H. maps input to prediction Y E R4, where d denotes the dimensions of the output and represents the parameters of the model. To give probabilistic view of the problem, the training distribution is denoted as P(X.Y) = 1 Σ/= ((m..))) where denotes the Dirac delta distribution n. concentrated on (I,, v.). Consequently, P(X) = Ei oe, and P(Y) 1. + 8gz-
4 FEW-SHOT ROBUSTNESS: THE WORST-CASE GENERALIZATION ERROR
In this work, we discuss the problem of few-shot robustness. In traditional few-shot learning, construct a few-shot set and measure the performance given the fixed training set and test set. Here are interested in if different few-shot training sets lead to large model performance degradation. Inspired by the notion of robustness in distributional robust optimization (Sagawa et al., 2019 Kuhn al. 2019), we aim to find a challenging few-shot training set Ptew that maximizes the expected risk on the test set. For convenience, we focus on a data-driven few-shot distribution, where Piew is centered around empirical examples extracted from the original training set distribution.
We consider a few shot set with & examples. The example index is defined as 4s := (11,
C [n). and therefore Prew (X) R D m= ōx... For short, also write Prew (X) as Pı, (X). Let
if be the model hypothesis and Q as the test distribution, the empirical risk of f w.r.t training (()))(
P(y]=)], and the expected risk of / w.r.t. Q is denoted as (f( ((=) ((f(=)) (zylz)). We assume the original training sets and test sets are sampled i.i.d from a natural distribution. Then we can safely make Assumption. 4.1 that the full-size training set can teach model with acceptable generalization errors.
set P is ef()
Assumption 4.1. (Generalization) Let H. X + Y be the hypothesis space, then there exists a model f satisfies f E HH and cf(f) S (f( < B, where a, B are constants approaching zero.
In real-world setting, a few-shot dataset is usually constructed by choosing a few representative samples (Parnami & Lee| 2022] Gao et al.) 2020). Data imbalance is preferably avoided so as not to inject bias (Ochal et al. [221] Wang et al. 20207. It is thus logical to assume that the few-shot subset should match the balanced label distribution of the original training set, as stated in Assumption. 4.2
PI, A Assumption 4.2. (Label Alignment) Let Pl, (X,Y) Y) be the few-shot set selected from P(X,Y) Y), then
T should satisfy P(Y)= Ptew (Y), i.e. 1 n Em_ dy; Di== :Y
3

---

Under review as a conference paper at ICLR 2023
Few-shot Robustness Under these assumptions, the few-shot robustness problem address asks to solve the worst-case problem denoted as,
max (Q(x))
(1)
where fI, is hypothesis that satisfies EP, (fi.) S and 14 satisfies 1 Ei= dy; = ト S
[n]] m Ig, Is [n] {Pr.} is the uncertainty set for constraining this distributional robustness problem. Different few-shot training sets X1, lead to varied generalization errors.
Traditional few-shot evaluation usually adopts a fixed training set. Correlations between the training set and the test set largely decide the generalization errors on the test set. Higher performance does not represent better robustness due to the missing evaluation of the worst-case performance. To evaluate few-shot robustness in Eq. [1 the target is to find subset Xr, to prevent models from learning biased features and increase the difficulty for generalization performance.
NMMD-ATTACK FOR WORST-CASE FEW-SHOT EVALUATION
This section introduces our method for evaluating the worst-case generalization error by constructing a challenging few-shot set based on Eq. [T] The intuition of NMMD-ATTACK is based on eliminating the spurious correlation between few-shot training set and test set. Since most samples hold spurious attributes (Sagawa et al., 2019) and samples with the same spurious attributes tend to be more similar, it is more likely to find less biased subsets based on distribution discrepancy. Therefore, propose a NMMD-atten approach to find the worst-case few-shot subset with the largest MMD discrepancy from the original training distribution, while maintaining the balance of labels.
Here, We first proves the intuition that the worst-case can be achieved by maximizing the MMD distance between few-shot set and the original training set, and then introduce how to quantitatively evaluate the MMD distance using neural tangent kernel. Last, a practical solution is provided to generate the worst few-shot set. All proofs are available in the Appendix A
5.1 FEW-SHOT ROBUSTNESS EVALUATION WITH MMD MAXIMIZATION
To find the worst-case few-shot set, our goal is to solve the optimization problem stated in Eq. [ Intuitively, the generalization errors increase when there are fewer correlations and lower distribution similarity between the few-shot set and test set. In many benchmarks and contests, the test set is not directly available, but we can use a well-rounded training set as a resource of natural data distribution. In this case, our optimization goal becomes maximizing the distribution distance between the few-shot set and original training set. Here we use MMD distance following Gretton et al. (2012).
Definition 5.1. (Distribution Distance) Let H be the family of mappings from X' Rd. and let P and Q be two distribution, then the MMD distance can be defined as,
MMD H, P. Q) == sup Pf -Ep--l/ fEH
(2)
We theoretically show that the MMD distance is an upperbound for the few-shot robustness metric. Theorem 5.1. (Few-shot Robustness Measured by MMD Discrepancy) Let H be the hypothesis space X > R fia is the empirical risk CP₁, (f) minimizer, and f is the hypothesis that minimizes
expected risk (.) Then
<Q(Si.) S (Q(() + MMD( P) + + + +L+
where tcet and Eh are small constants describing the error occurred in training, the sampling behay- ior of training distribution and the hypothesis space complexity. Details are shown in Appendix A.
This thcorem gives the intuition that a high discrepancy between the few-shot set and a training set results in high expected risk. Following statistics literature (Cheng & Xiel 2021), biased empirical estimate of the MMD obtained by replacing the population expectations with the empirical average computed on the samples. Then we have.
1 k
1 Ef(x) JEH n
MMD(H, P.PI.. sup
Σ f(zim)) m m=l
(4)
1=1
4

---

Under review as a conference paper at ICLR 2023
For wer-parameterized non-linear models, caleulating MMD in the hypothesis space is intractable. Neural Tangent Kernel (NTK), however, is a simple approximation to understand neural networks as kernel regression that can be optimized in its Reproducing Kernll Hilbert Space. Following the convention in NTK literature, we consider the NTK kernel function defined for optimization step
t 2 0 as:
K" (x, = ( x), Vo. (o(t,)))=)))
(5)
Then the MMD distance could be deducted following kernel conventions (Gretton et al. 2012).
Proposition 5.1. (NMMD Distance) The MMD static with NTK at time t is,
M- Se K" (x,,)))) K (1) (dd
(6)
where X and X' denotes the support sel ser P and Q.
When the neural network approaches its infinite width limit, the NTK matrix stays constant during training (Arora et al. 2019 ; Du et al., 2019), i.c., K equals to K This property enables us to accurately estimate the MMD distance without costly training. Together with the approximate ability of NTK for neural network training (shown Proposition. A.I we show that NMMD can be used approximation the MMD) distance.
Theorem 5.2. (NMMD Approximation) Let ,MD, be the MMD distance calculated with model H and let MMDx k the MMD static w.r.. the NTK kernel K. Assume has taken t steps of training at learning rate n, and f is Ly--ipschitz Continuous. Denote NTK at initialization then the two MMD statistics satisfies:
n, K(2, x¹) = (Vof(0(0), x), (0(O), x))- If S n
2
MMD,(P. PI.)) Otn MMD Pr,))
This theorem formally justifies using NTK kernel to calculate MMD distances. Compared with estimating MMD distance with neural networks, this method requires no training, and its calculation can be easily decomposed into kernel similarity K (Ix, 2z)) with the help of the kernel trick. The simplified distance would be,
2 nk
2
Σ K(T, 1
+
K K(I,, [t.)
Σ K(z,,
22
12
nk RETL-E%
8,6
intra-set kernel similarity
inter-set kernel similarity
(8)
This distance would achieve its maximum while maximizing the intra-set kernel similarity of the few-shot set and minimizing the inter-set kernel similarity.
5.2 IMPLEMENTATION
We have simplified the task of finding the worst few-shot set to maximizing the NMMD distance between few-shot distribution and the full-size training distribution, as shown Eq. However, there are several challenges in implementation: (1) The computational complexity for the NTK matrix on all training examples O(n²). making hard scale large-scale sets; (2) Optimizing the MMD distance requires solving a combinatorial problem of O(²)) In our work, we address these problems by taking advantage of the sparsity of NTK matrix.
Efficient Optimization Eq. g In experiments, we discovered the NTK is usually a sparse, diagonally dominant matrix. As shown in Figure 2 the diagonal terms K (I 22) are more than 10x larger than cross terms K ((x,,,,) m.). In this way, we can discard cross terms for simplification in Eq. 8. K ( x:)) is actually gradient norm (Vof vz), (o, x., a.))) Note that the gradient of model f : + Id (ff()())) Rpxd= p is the number of parameters f. The calculation of K (z .T)) can be written as,
P
K(x,, 24) = (Vof(0.))
Σ>
Σ
(x)
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Details of SaMMDDAtttck
Input: Full-size training set X (zi 9))3 few- shot set size k. model f with random parameters initial parameters on-
Output: The few-shot attack set.
1: initialize fo with du
2: for = 1 1:: do
3: G (VoJo(a.), Volo(z.))
4: end for
5: Sort examples in X based on G3
6: For each label, select top &s examples into the final few-shot set iew to optimize Eq. 10 Figure 2: Visualization of NTK matrix with 50
7: return Xiew
examples randomly sampled from CIFAR-10.
After reducing the sparse terms in the NTK matrix, our goal is to maximize the intra-set kernel similarity using the gradient norm terms:
max
K ((r 2s,), E Is
(10)
Implementation Algorithm Given Eq. 10 the target of finding the worst-case few-shot set is simplified into finding the few-shot set with the maximum sum gradient norms. The algorithm details shown in Algorithm First, We rank all training samples w.r.t. their gradient norms, i.e., K ( ) > ( (ja) ) > K T;,). then we choose the top-k samples (Tj, Vi. [{[M] our attack few-shot set. Last, we train models on the searched few-shot and report its test accuracy.
6 EXPERIMENTS
Datasets We explore the few-shot robustness on image classification datasets, namely the CIFAR- 10 dataset (Krizhevsky al. 2009), the CIFAR-100 dataset (Krizhevsky al., 2009). the MNIST dataset (LeCun et al. 1998) and the ILSVRC-2012 ImageNet dataset with 1K classes (Deng et al. attack set. To build natural few-shot and
2009p. We select subset of the full-size training guarantee the label alignment constraints, each label keeps k corresponding instances in few-shot training set.
Models and syper--prrmettrs To evaluate the robustness of wver-parameterized neural networks, we consider the following models: 1) FFN, a feed-forward neural networks; 2) VGG (Simonyan
Zisserman 2014), classical convolutional neural network; 3) ResNet (He et al. (016)) residual neural network; 4) ResNeXt (Xie et al., 2017); 5) DenseNet (Huang et al.) 2017). Besides, to verify the attack ability pre-trained models,
also re-implement two pre- pre-trrined models: 1) EfficientNetV2 (Tan) & Le, 2021). Appendix DopendixBpprovdeee provides detailed model deseription and hyper-parameter settings. For each result, we conduct m experiments and report the mean and variance. m = 5 for MNIST, CIFAR-10, and CIFAR-100, and 3 for ImageNet.
Transformer-bast
ViT (Dosovisskiy et al., 2021) and 2) Convolutional
Baseline and Comparison NMMD-attack finds the subset from the full-size training set to attack models. We implement an "average-case" baseline that randomly samples few-shot sets from the full-size training set as a comparison. After training models on the few-shot group, we report the accuracy of the test sets for both methods. make a fair comparison, NMMD-attack and the baseline have the same dataset size and label distribution. To be specific, training set as a few-shot set. Each label has 500 examples in MNIST, 500 examples in CIFAR- 10, 50 examples in CIFAR- 100, 100 examples in ImageNet- IK. Since NMMD-atter -ttacck requires gradient norms, we implement various networks to calculate gradient norms. needs to notice that such gradients do not require trained network. Only architecture and initialization required.
extract 10% data from the

---

Under review as a conference paper at ICLR 2023
Table The comparison between average performance and attack (FFN-attack) performance. "Acc." represents accuracy. "Test Acc. Gap" represents the gap between average performance and attack performance. "Abs." represents absolute gap and "Rel." represents relative gap. As we can see, NMMD-attack can successfully attack models with large performance drop.
Datasets
Models
Average-case
NMMD-attack
Test Acc. Gap
Train Acc.
Test Acc.
Train Acc.
Test Acc.
Abs.
Rel.
PFN
100.00 +0000
97.63
#0.12
99.99 :0001
93.87
+:073
3.76
3.84
VGG- 16
100.00 +:0.00
98.73
+0.13
100.00 +0000
76,47
+0.65
22.26
22.55
MNIST
ResNet-18
100.00 1:0.0
98.65
#0.05
100.00
1:0.0
75.41
1:0.4
23.24
23.55
ResNeXt-29
100.00 1:0.00
98.42
10.10
100.00
=00.0
70.64
1:0.7
27.78
28.23
DenseNet-121
100.00 +-0.0
99.14
+.0.6
100.00
+00.00
77.20
+:0.66
21.94
22.13
FFN
100,00 1:0000
49.38
+0.47
100,00
00
42.29
+:0.73
7.09
14.36
VGG-16
98.85 +-0.31
66,20
+00.1
98.82
+:0.84
51.43
+:107
14.77
22.31
CIFAR-I0
ResNet-18
::0.02
63.11
1.44
#:0.02
48.72
1:089
14.52
23.01
ResNeXt-29
99.52 ::0.48
61.75
+0.58
99.66 16
48.59
+0062
13,16
21.31
DenseNet-121
99.53 ::0.45
71.19
0:0.8
99. #:0.18
$4.76
1.56
16.43
23.08
FFN
99.99 1:000
14,46
#0.51
99.98 ::0.00
11.91
+:0.51
2.55
17.63
VGGi-1
100.00 4:0.00
27.15
++0.00
14.35
CIFAR-100
ResNet- 18
100.00 1:0.00
24.91
#0020
100.00 #:0.00
14.27
+:0.29
10.64
42.71
ResNeXt-29
100.00 1:000
23.88
#0.42
100.00
1:0000
14.54 1:0.40
9.34
39.11
DenseNet-121
100.00 ::0.00
32.74
100.00 ::00000
17.18
1:0.17
15.56
47.53
99.94
++0.00
2.08
VGG-16
98.11 1:0.02
14.22
00.24
97.97 1:0.55
9.74
+:050
2.54
40.71
ImageNet-IK
ResNet-18
99.96 1:000
30.06
+0.29
99.72 10000
23.66
+:0.2
6.93
38.01
ResNeXt-50
99.94 4:0.01
37.97
+:0.05
9.72
37.28
DenseNet-121
99.84 1:0.01
37.76
00.73
99.61 #:0.02
28.49 +:036
10.02
37.63
6.1 WORST-CASE EVALUATION ON
ARCHITECTURES
NMMD-attack successfully attack various models with large performance drop. As Table illustrates, WMMM-attack can successfully attack neural networks with worse generalization errors. For example, DenseNet- trained on the randomly sampling few-shot set achieves high test results with 71. 19% CIFAR-10, 10, 99.14% on MNIST. When DenseNet-122 121 is trained on the adversarial few-shot sets generated by NMMD-attack the performance drops significantly with 16.43% on CIFAR-10, 10, 21.94% on MNIST, respectively.
Neural Networks suffer from poor robustness. First, Table I shows that the neural networks do perform well on complicated few-shot datasets, including CIFAR-I00 and ImageNet-1K with over 100 labels. It proves that few-shot generalization to complicated sets is still a big challenge. Second, even perl-perfonming models on CIFAR- 10 and MNIST easily learn spurious correlations and fail on adversarial sets. The large performance gap between NMMD-attac and average-case baseline implies that neural networks still suffer from the robustness problem.
FFN-attack and ResNext-attack, two NMMD-attack variants, show better attack performance. Since NTT requires gradient norm during the few-shot set search, we implement different networks variants to observe how backbones affect attack performance. Due the steady of NTK, do not require a trained network during the search. The results are shown in Figure 3 FNN-attack and ResNeXt-attack achieve better attack performance. It indicates that gradients of FNN and ResNeXt give valuable information on unbiased examples. also shows that worse models are more suitable for attack implementation. The over-fitting to spurious attributes empowers models with solid ability to distinguish spurious attributes.
6.2 WORST-CASE EVALUATION ON PRE-TRAINED MODELS
We further verify the effectiveness of the NMMD- on pre-trained models. Recently, pre-training and fine-tuning have become a new paradigm of neural networks with strong learning ability (He) et al. 2019p. We conduct FFN-based NMMD
two pre-trained models, transformer- based Vit- B/16 (Dosovitskiy et al. 202T) and convolution-bar EfficiemtNetV2-S (Tan & Le 2021) on
100 These models both pre-trained ImageNet- K before downstream fine-tuning
7

---

Under review as a conference paper at ICLR 2023
Restext-attack ResNat-attack
Voc-attack
FRN-attack
DenseNkt-attack
20
%6 Accurayy
Fo
%
ITN
VGC
Demsntet
Dashet
Deraebet
CIFARIO
CIFR100
Figure 3: Results of NMMI-attack variants. Since NTK requires gradient norm during few-shot set search, we implement different networks. FNN-attack and ResNeXt-attack show good attack performance. Due to its simplicity, We adopt FNN-attack by default in the following experiments.
Table 2:
The attack results (FFN-attack) pre-trained models. The abbreviations follow Table As we can see, pre-trained models have better defense performance on 500-shot cases but also suffer from poor robustness on 50-shot cases.
Average-case
NMMD-attack
Test Acc. Gap
Datasets
Models
Train Acc.
Test Acc.
Train Acc.
Test Acc.
Abs.
Rel.
EfffentNerV2-8
99.72 +-0.03
91.71
40.29
99.75
+0.04
91.50
=0.43
0.21
0.23
CIFAR-I0 (500-shot)
VIT-B/16
100.00 +0.00
97.14
10.14
100.00
10.00
96.97
20.13
0.17
0.18
CIFAR-I0 (50-shot)
EffientNerV2-S
99.88 1:010
68.79
1:148
100.00
1:000
65.36
2266
3.43
VIT-B/16
100.00 +0000
88.97 +:041
100.00
+0.00
80.57
+1.7
8.40
9.44
Pre-trained networks outperform randomly-initialized networks under the same worst-case few-shot. The first column Table CIFAR-IO 10 (500-shot) shows performance of pre-trained networks on the few-shot set searched by FNN-attack. As we can see, pre-trained networks achieve much better robustness than aunomly-initializee networks on 500-shot cases where the FFN- -Fsed NMMD-atee -ttack slightly deduces the few-shot learning ability of the pre-trained models. ViT-B/16 has 0.73% relative accuracy drop on CIFAR-10 10. It proves that large-scale pre-training can reduce the over-fitting to biased attributes and help robustness.
Pre-trained networks suffer from poor robustness on smaller few-shot sets. In particular, pre-trained models have solid performance on 500-shot cases. We are wondering if high performance can be generalized to smaller few-shot sets. We reduce the number of few-shot sets to 50-shot, where each label has 50 samples. As Table iillustraees, the FFN-based NMMD-atter few-shot performance on smaller few-shot sets. EfficientNetV2 has 4.97% % relative accuracy drop and VIT-B/16 has 9.44% relative accuracy drop, respectively. In summary, NMMD-attack leads to performance deduction for various models, not only over- wwer-parameterized networks but also large pre-trained models. Moreover, we report our experiments for one-shot models Appendix D.I Our worst-case evaluation poses higher challenge to defense such attacks for few-shot learning.
consistently deduces
7 DISCUSSION
7.1 How IS THE QUALITY OF THE ADVERSARIAL FEW-SHOT SETS?
We first explore the instance similarity in the worst-case few-shot sets generated by NMMD-attack.
To this end, we analyze the instance similarity via three metrics: 1) Structural Similarity (SSIM), widely recognized method for measuring the similarity between two images (Wang et al. 2004): 2) Mutual Information (MI), a measure between two variables in information theory; Peak signal- to-noise ratio (PSNR), a measure between super resolved image and the original one. Details of similarity measures can be found at Appendix
The searched few-shot sets are natural and diverse. Table 3 shows the instance similarity in the worst-case few-shot sets and random few-shot sets. As can see, the searched sets are natural and diverse, and close to the instance similarity in the randomly sampled few-shot set. This indicates that approach cannot be replaced by trivial implementations. such aggregating a group of similar instances or picking out noise instances. Appendix G shows sampled instances from the searched few-shot set.
8

---

Under review as a conference paper at ICLR 2023
Table The similarity analysis on the searched few-shot sets and randomly-sampled few-shot sets. For SSIM and MI, lower scores represent better diversity. For PSNR, higher scores represent better diversity. The searched few-shot sets show competitive diversity scores compared with randomly- sampled sets.
Diversity Score
FEN-attack
VGG-attack
ResNet-attack
ResNeXt-attack
DenseNet-attack
Average-case
SSIM
0.06
0.12
0.18
0.20
0.12
0.12
MIJ
0.44
0.44
0.37
0.35
0.46
0.44
PSNR
54.57
57.25
56.18
58.04
56.03
57.43
Label
Trained Random Choose
Model Trained on MAND attack Chosen
Not plana
Samples
Samples
ane (
Not
Aippaane
Airpbre
Spurious attr butbbutt
Sky 6S Rechegooun
Sky Sockgrund
Brrigmundd Beckgmund
AST(A)
P(S(A)
PS5AAI- 6-402
JUSSA)
Other (8) Backkgrurrd
Other Oxikgroudd
PNSIA)
Other
POSIAO -85H
Figure 4: Case study of spurious correlation: Sky Background - airplane. Specifically, P(S|A) refers to the number of airplane images with a sky background correctly judged, and so on. This spurious correlation is witnessed in the classification results of the model trained on candom-chosed few-shot set. The NMMD-attack results barely show this correlation.
7.2 EXPLAINING THE EFFECTIVENESS OF NMMD-ATTACK WITH SPURIOUS CORRELATION
Spurious correlation occurs statistical phenomenon, whereas confounders in data be used to perform inference. These confounding attributes are often superficial features like background or lightning and easy capture. Here we compute the conditional probability of the spurious attribute to labels. All probabilities are statistically estimated by counting the figures with or without the hand-annotated spurious attribute. As shown in Figure. 4 airplanes in the CIFAR-10 dataset appear more common in a sky background than other classes, and thus models are likely to learn the spurious correlation "figure of airplanes -> sky background -> planes". These spurious correlations between training and test sets thus bring high performance that is yet hard to generalize.
However, our worst-case few-shot set on CIFAR- 10 picks several samples for each class as a few-shot set, all with an almost plain white background. This forbids models from learning to classify airplanes based on the sky background. The essential features are much harder to learn, resulting in much lower accuracy. Similar cases are common in various datasets. Other cases on Imagenet-1K include Bamboo Leaves > Small Panda, Water Background > Drake, and Human Activity - Paddle, as shown in Appendix I All spurious correlation experiments are based on ResNeXt-attack set and finetuned with VGG model.
8 CONCLUSION
This paper proposes a worst-case evaluation to re-examine neural networks as few-shot learners by constructing a label-balanced subset from a full-size training set that results in the largest expected risks. An efficient method NMMD-attack is proposed to optimize the target in this work. Experiments show that NMMD-attack can find natural and diverse few-shot sets that successfully attack various architectures, even pre-trained models. The quantitative analysis gives several case studies to understand how spurious correlations between training and test sets affect few-shot evaluation. We find that the searched few-shot sets have fewer spurious attributes than -aaoolee few-shot sets, which can explain why the searched few-shot set is more challenging. The worst-case evaluation re-examines the actual ability of neural networks on few-shot cases and also brings new problems to defense such attack for better robustness. Our work is still limited in the computer vision domain and we intend to apply to natural language tasks in the future. We also intend to further explore the negative impact of spurious attribute on general benchmarking and extend our theoretical analysis to the impact of distribution shift on spurious correlation.

---

Under review as a conference paper at ICLR 2023
9 REPRODUCIBILITY STATEMENT
Experiment Reproducibility: We have included the code, data, and instructions repro- duce the experimental results. The codes and searched few-shot sets are available in the supplementary materials. We also record the gradient norms in the materials, which enables a quick re-implementation. Models and hyper- parameter settings are described in 6 For each result, we run experiments with different random seeds and report the average accuracy with standard deviation.
Theory Reproducibility: The proofs of all propositions, lemmas, and theorems are included in A A few theorems and lemmas referenced from other works provided with careful citation.
REFERENCES
Can--Bptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katic Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. CoRR, 2022a.
Jean-Baptiste Alayrac Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022b.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Proc. of NeurlPS, 2019.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 2010.
Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differen- tiable closed-form solvers. arXiv preprint mrXXiv:8003 08036, 2018.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen. Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proc. of NeurlPS, 2020.
Minghao Chen, Shuai Zhao, Haifeng Liu, and Deng Cai. Adversarial-learned loss for domain adaptation. In Proc. of AAAI, 2020.
Xiuyuan Cheng and Yao Xie. Neural tangent kernel maximum mean discrepancy. Proc. of NeurIPS, 2021.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. Proc. of ICMI,, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. of CVPR, 2009.
Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. arXiv preprint 1xXiv:190.002720 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In Proc. of ICLR,, 2021.
Simon Du Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In Proc. of ICML, 2019.
10

---

Under review as a conference paper at ICLR 2023
Tianyu Gao, Adam Fisch, and Dangi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723, 2020.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6577 2014.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 2012.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. of CVPR, 2016.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proc. of ICCV, 2019.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: decoding-enhanced bert with disentangled attention. In Proc. of ICLR, 2021.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks, In Proc. of CVPR, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proc. of ICML, 2015.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran S. Haque, Sara M. Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In Proc. of ICML, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's 's thesis, Department of Computer Science, University of Toronto, 2009.
Daniel Kuhn, Peyman Mohajerin Esfahani, Viet Anh Nguyen, and Soroosh Shafieezadeh-Abee Abadeh. Wasserstein distributionally robust optimization: Theory and applications in machine learning. In Operations research and management science in the age of analytics. 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. of the IEFF, 1998.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.0333 03983, 2016.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 1rxivv1706.06000 2017.
Subhabrata Mukherjee, Xiaodong Liu, Guoqing Zheng, Saghar Hosseini, Hao Cheng, Greg Yang, Christopher Meek, Ahmed Hassan Awadallah, and Jianfeng Gao. CLUES: few-shot learning evaluation in natural language understanding. CoRR, 2021.
Andrew Mutton, Mark Dras, Stephen Wan, and Robert Dale. GLEU: automatic evaluation of sentence-level fluency. In Proc. ACL, 2007.
Mateusz Ochal, Massimiliano Patacchiola, Amos Storkey, Jose Vazquez, and Sen Wang. Few-shot learning with class imbalance. arXiv preprint arXiv:2101.02525 2021.
Boris N. Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. TADAM: task dependent adaptive metric for improved few-shot learning. In Proc. of NeurlPS, 2018.
Archit Parnami and Minwoo Lee. Learning from few examples: A summary of approaches to few-shot learning. arXiv preprint 202449200,.0429), 2022.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proc. of ICCV, 2019.
11

---

Under review as a conference paper at ICLR 2023
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard Zemel. Meta-learning for semi semi-supervised few-shot classification. arXiv preprint
2018.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. 2019.
arXiv preprini
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. In Proc. of ICML, 2020.
Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao. Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga. Henrik Marklund, et al. Extending the wilds benchmark for unsupervised adaptation. arXiv preprint arXiv: 1oXiv:212200590 2021.
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In Proc. of NAACI, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Aman Sinha, Hongseok Namkoong. Riccardo Volpi, and John Duchi. Certifying some distributional robustness with principled adversarial training. arXiv preprint 1rXiv:1710.10571, 2017.
Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using
large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications, 2019.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Proc. of NeurlPS, 2017.
Colin Studholme, Derek LG Hill, and David J Hawkes. An overlap invariant entropy measure of 3d medical image aligmment. Pattern recognition, 1999.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 13 2013.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proc. of CVPR, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proc. of CVPR, 2016.
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception- resnet and the impact of residual connections on learning. Proc. of AAAI, 2017.
Mingxing Tan and Quoe Le. Efficientnetv2: Smaller models and faster training. Proc. of ICML, 2021.
Kaihua Tang, Mingyuan Tao, Jiaxin Qi, Zhenguang Liu, and Hanwang Zhang. Invariant feature learning for generalized long-tailed classification. arXiv preprint 22ZV:2207700504 2022.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robusiness to natural distribution shifts in image classification. In Proc. of NeurIPS, 2020a.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. Proc. of NeurIPS, 2020b.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Proc. of NeurlPS, 2021.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Proc. of NeurIPS, 2016.
12

---

Under review as a conference paper at ICLR 2023
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In Proc. of NeurlPS,
Yaqing Wang, Quanming Yao. James T. Kwok, and Lionel M. Ni. Generalizing from few examples: A survey on few-shot learning. ACM Comput. Surv., 2020.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing. 2004.
Saining Xic, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proc. of CVPR, 2017.
Yichong Xu, Chenguang Zhu, Shuohang Wang. Siqi Sun, Hao Cheng, Xiaodong Liu, Jianfeng Gao, Pengcheng He. Michael Zeng, and Xuedong Huang. Human parity on commonsenseqa: Augmenting self-attention with external attention. Proc. of IJCAI, 2022.
Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang.
An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proc. of AAAI, 2022.
Zhongqi Yue, Hanwang Zhang, Qianru Sun, and Xian-Sheng Hua. Interventional few-shot learning. In Proc. of NeurlPS, 2020.
Haiteng Zhao, Chang Ma Qinyu Chen, and Zhihong Deng. Domain adaptation via maximizing surrogate mutual information. arXiv preprint arXiv:2110112184, 2021.
13

---

Under review as a conference paper at ICLR 2023
A PROOF
A.1 PROOF FOR THEOREM 1.1
Lemma 1. Define (s E - 8(f())) For any hypothesis h, H/' E H, there exists
cH > O which satisfies,
EPI (h,11) - Ep(h, MMD(H, P)+
EH (1I)
2
éн is constant for the complexity of hypothesis space.
Proof.
|ern (h,h') er (n,h')
CPis (h, cp h')
sup A
sup N.
Pæ-Pr [8(h(x)) # ((h((x))) Pawp (0(h(x)) + 6(h((z))]
Pa-Ply [h(x)) + h [xx)]] Pa- (h(xx) h'(x))
sup A
/ (e J Int0)
sup A.KE
(12)
Inspired by proof in Peng et al. 2019), we use continuous function to approximate the indicator function (1()))((())) Formally, for any h, h', (x(=) is a L function, thus there exists f e (x)) that satisfies.
sup I In(a))Arra)dUP.. S (nt ANNEH
/ x
f(x)dupi, - - -J f(z))up
EH
<
Sup(EEmpPF) -EyrPr (f())) fell
+ 2
EH
IMMD(H. Plz P)] + CH
2
(13)
where C.(X) is a subset of the hypothesis space with compact support and e a constant for the complexity of hypothesis space which measures the closeness of the approximation.
Lemma 2. Let fI, be the trained classifier on the few-shot distribution Pt,, and f be the trained classifier on distribution P. Since Pte is formed by subset of the training examples, when training
error (f( - and EPi (fr.) + 0, EPi, (fiv. S Ea where ea is constant approaching zero.
Following Ben-David et al. (010)), we use Lemma 1 and 2 to prove Theorem 5.1
Proof
( + f)
=c((f) + en, (fh,
EP,, (fr. f)) (cr(ff -
Sea(f) + ep, (r +1 (fr,,) -p(sie
sca(f) +c. +1 MMD
(14)
In which,
14

---

Under review as a conference paper at ICLR 2023
1 f1 (1)
J fi, (o)+
[ep(Sue (eq(ee
(15)
= 1 (1x()))(((1)) 11 (()))(()
Here suppose test set Q matches the distribution of data for this classification task, and P is con- structed by sampling n ii.d samples from the distribution Q. Using Hoeffding inequality we have,
P((cr(fiu, f) cQ(ff, ( f)l > t) s 2e 2nt
(16)
Therefore, with a probability over - 22 2en2 (in practice n is very large and the probability approaches 1), we have
<Q(sr)) S <c(s)) + MMD P) ca cH
(17)
A.2 PROPERTY OF NTK
Proposition A.1. (Neural Tangent Kernel, Arora et al. (2019) Consider minimizing the loss l(0) by gradient descent with infinitesimally small learning rate: di
a9(t) -V(((tt) Let u(t)
(f(((t), [e))e(() Rnd be the network outputs all x; time t, and Y = (uk)setn) be the ground
truth outputs, loss is I[ (0, Y). Then u(t) follows the following evolution, where Kt is the NTK matrix, (i.))
=f( (8(()))
09
00
nl) - u(1 1) K al(O(t))
Ou(t) + O((0, 02-1)3)
(18)
The proof follows Arora et al. 2019
A.3 ON PROPOSITION 5.1
This proposition is adopted from Lemma.4 in Gretton et al. 2012)
Lemma (Restatement of Lemma.4 in Gretton et al 2012 Let H be the Reproducing Ker- nel Hilbert Space for kernel K(--)) If K(.,)) -) is measurable and Ex~pK (x, x) >> 0, V. E X, EzzoK(r x) > 0, V. X', then mean embeddings pr, paq exists such that Ezf (f uP) and satisfies,
MMD(P,Q) ||up - Pollæ
(19)
The NTK kernel a positive definite matrix and satisfies constraints in this Lemma. Also based on the definition of distance in RKHS, can rewrite Equation, into the form in Proposition. [5.]
A.4 PROOF FOR THEOREM 5.2
Proof
Using the MMD expression in integral form (Cheng Xie 2021), we can simplify the expression in Definition 5.1
MMD,((.Q) sup / f(0,x)) (x)dr
(20)
where و 1 12,, Hy P. and 1 8y. 3t ~ Q. 5, the indicator function that equals to when n input equals to otherwise the output is O.
m
Similarly we can write NTK-MMD initialized with f(0) as.
2MD? K. (P,Q) (000)=AA))
-f
Ko (a, x') ( (z) (2') dodx
(21)
15

---

Under review as a conference paper at ICLR 2023
We that MMD/ involves solving optimization problem. Here design a loss achieves the maximum at the last time step during fine-tuning. At the same time, we require the model f to learn the distribution of the task.
((((Ot,)) x)) = - - ( [ Erlf(x) - ( (r)
(22)
Unalignment loss
Classification loss
During training, take two steps approach, first aligning the distributions and then minimizing the classification error (in practice, this can be done by training the representation encoder and then the classification head). Here suppose at initialization, the loss is quite large such that the two distributions are aligned.
The NTK at time step is Ke(x,x')= { (1)(a.0()) 0f(a' (itt) Our P is the training set distribution,
0(1) 0()
and Pr, the distribution Q (here write Q for convenience). Using the LI loss, combined with Proposition 3.1, we have:
u(x, 1) - u(x, - Vo.l (o,((x;0,) (0.- - 0.-1) + (10, 0--11p)
/ Ve. Vo.f (x;0,) Vo.f (z):0.) (- (-1ø ql + (2) +0 (100: - 01-111))
(23)
Recalling that, the NTK matrix remains stable during training, thus write it as,
J Venf (z; 00) Vo. (z':0) (-10 - al + (2))) + +0 +0(10, - -1--11?))
t( - u(xxt - 1) =
(24)
Here X' is the support set of and Q. Also,
|10 - 10,-0-111 // V L(S(O()),s) mi1
! Vo, ((((() S nL
(25)
Therefore, combining t = en steps (c epochs) training we have:
u(z,t) - u(z,0) = 17 / V (on (1)) (T)('' Σo
n²
=n [. Va,f (m) Vo. (z': (F) + (1) 0
(26)
Then combined with the definition of MMD. in 2k:
/ ((xxt)] ((c,,))i--ll) (x))d
MMD)(P.Q)
/ [u(xz, 0) + J. (2 ()( dt] (l(x)dr + O(tn² L3)
S 2n J.Jx 1 Kd (z. x')
(x)dr ((t)
= (P.
(27)
16

---

Under review as a conference paper at ICLR 2023
Also,
/ u(x,,)lū - (a)dd
(M,)(
J Iu(E 0) +1 J. / K (x,x')( al + p) (2') d al(x)dr + O(tr)
x
" J. / K (x, 2')
atlp (x))dd + 13)
(PMMD)x(PP.))+OO + ((
(28)
17

---

Under review as a conference paper at ICLR 2023
Table 4: Hyperparameters settings. These settings are reported in their official repository for best practice. CA refers to cosine annealing scheduler (Loshchilov & Hutter 2016). The Linear refers
LinearLR scheduler in Pytorch. OneCycle refers 1-yycle learning rate policy (Smith & TTopin 2019).
Models
Datasets
Batch Size
Training Epochs
Optimizer
Learning Rate
MNIST
128
50/200
SGD
0.01 (CA)
CIFAR-10
128
50/200
SGD
0.01 (CA)
FFN
CIFAR-100
128
50/200
SGD
0.01 (CA)
ImageNet-IK I
32
40/90
SGD
[0.01, 0.001, 0.0001]
MNIST
128
50/200
SGD
0.01 (CA)
CIFAR-10
128
50/200
SGD
0.01 (CA)
VGG
CIFAR- 100
128
50/200
SGD
0.01 (CA)
ImageNet-I IK
40/90
SGD
[0.01, 0.001. 0.0001]
MNIST
128
50/200
SGD
0.01 (CA)
CIFAR-10
128
50/200
SGD
0.01 (CA)
ResNet
CIFAR- 100
128
50/200
SGD
0.01 (CA)
ImageNet-1K
40/90
SGD
[0.1, 0.01, 0.001]
MNIST
128
50/200
SGD
0.01 (CA)
CIFAR-JU
128
50/200
SGD
0.01 (CA)
ResNeXt
CTFAR- 100
128
50/200
SGD
0.01 (CA)
ImageNet-IK
100
40/90
SGD
[0.1, 0.01, 0.001]
MNIST
128
50/200
SGD
0.01 (CA)
CIFAR-10
128
50/200
SGD
0.01 (CA)
DenseNet
CIFAR-100
128
50/200
SGD
0.01 (CA)
ImageNet- I
32
40/90
SGD
10.1. 0.01, 0.001]
CIFAR-10
Adam
5e-5 (Linear)
VIT-B/16
CIFAR-100 100
32
4
Adam
5e-5 (Linear)
CIFAR-10
32
4
Adam
1c-3 (OneCycle)
EfficientNerV2-S
CIFAR-100
Adam
1e-3 (OneCycle)
B MODELS AND HYPERPARAMETERS
To evaluate the robustness of over-parameterized neural networks, we consider the following models. FFN, a feed-forward neural network with two convolution and pooling layers and three feed- forward layers. 2) VGG (Simonyan & Zisserman, 2014). classical convolutional neural network. We use the VGG- 16 with 13 convolution layers and three fully connected layers as implementation. 3) ResNet (He et al. 2016). a residual neural network. We use the ResNet- 18 with 16 residual blocks. one convolution layer, and one fully connected layer as implementation; 4) ResNeXt (Xie et al. 2017) incorporating the advantages of ResNet and Inception (Szegedy et al. 2015 2016 2017: loffe Szegedy 2015). We use the ResNeXt-29 (2x64d) for MNIST, CIFAR-10 0, and CIFAR-TON 100, and ResNeXt-50 (32x4d) for ImageNet. 5) DenseNet (Huang et al. 2017). We use DenseNet- 121 with 121 layers, one convolution layer, and one fully connected layer as re-implementation !] Besides, to verify the attack ability NMMD-attack
the pre-trained models, pre-trained models: 1) Transformer based VIT (Dosovitskiy et al. 2021 and 2) Convolutional- based Fl For FFN, VGG, ResNet, ResNeXt, and DenseNet on ImageNet, we resize all the images into 256 x 256 and then center-crop them into 224 x 224. For ViT on CIFAR, we resize all the images into 224 224, while 384 x 384 for EfficientNetV2.
also re-implement two
EfficientNet22 (Tan & Le. 2021
We list hyper-paramter settings in Table 4 All the SGD optimizers are with a momentum of 0.9. For Adam/Adam AdamW, we set (0.9, 0.999) ). For the learning rate selected Imagenet, the milestones are [15, 30J, while [30, 60] for the full-sized. We conduct all the experiments on a single A100 GPU.
For VGG, ResNet, ResNeXt, and DenseNet on CIFAR and MNIST, we use the implementation from https: //githb...tt...../mo thub. com/ http://aaggguuytttooooeo htppytocch-ct ci far As for ImageNet, we use the implementation
1 i
from torch.models.
We
use
implementation
from
vit aarge- parge-ptcch 224
We use the implementation from norch.models.
18

---

Under review as a conference paper at ICLR 2023
C DETAILS OF SIMILARITY MEASURE METHODS
1) Structural SIMilarity (SSIM), which is widely recognized method for measuring the similarity between two images (Wang et al. 2004). The SSIM index considers three measurements the two images I and y, including luminance, contrast, and structure. Following the settings of Wang et al. (004). we formulate SSIM by:
SSIM(x, v) = 8
(29)
We set the weights a, 9, and 7 all to 1.
2) Mutual Information (MI), which is a measure the mutual dependence between the two variables in information theory. Following (Studholme al. 1999). we calculate MI for image matching. Given the signal intensity in an image, it evaluates how well the signal in the other image will be predicted. Compared with other measurements, MI has fewer restrictions on image modality and alignment. We implement MI by applying the normalized mutual information function in scikit-learn toolkit and normalizing to the interval
1.
3) Peak Signal-to-Noise Ratio (PSNR) is frequently used metric for image quality comparison between two images, especially in the area of image compression. PSNR computes the mean-square error of the compressed and the original image and further calculates the peak error by:
R
PSNR 10 10g10
MSE(I.))
(30)
where R is the maximum fluctuation in the image data type. In our experiment, we normalized the images, took the MSE of the images in the worst-case group one-to-one, and set the 1.
httpss///ciiitt-leemm.co org
19

---

Under review as a conference paper at ICLR 2023
1.0
FFN
1.000
-0.014
0.123
0.423
0.162
0.8
VGG
-0.014
1.000
0.025
0.098
0.6
ResNet
0.123
0.025
1.000
0.487
0.319
0.4
ResNeXt
0.423
0.098
0.487
1.000
0.400
0.2
0.162
0.021
0.319
0.400
1.000
-0.0
FFN
VGG
ResNet
ResNeXt
DenseNet
Figure 5: Spearman correlation of gradient norms calculated by different models. All results have r 0.005 except for the VGG correlation, whose r 0.05.
D INTER-MODEL CORRELATION ANALYSIS
For different models, the gradient norms are different and thus various models can generate different adversarial few-shot sn this section, explore whether there is intrinsic gradient ranking correlation across different models.
We report the correlation analysis between different models in Figure As can be seen from the graph, the VGG is not correlating with any of the other models' rankings. The correlation result also aligns with our main results that the VGG attack is almost invalid. Beyond that, we can see that ResNet, DenseNet, and ResNeXt, which have similar residual structures, maintain some correlation. Apart from that, the FFN, which also has a strong attack capability, only maintains high correlation with ResNeXt, showing that influential ranking is model-independent.
D.1 WORST-CASE EVALUATION ON FEW-SHOT LEARNING MODELS
We also explored the ability of the NMMD-Attack on the FEW-SHOT learning models. We selected the ProtoNet (Snell et al. 2017 for experiments CIFAR-100 with FFN-attack. From the results, we can see that even though the model is designed for few-shot learning, it still suffers from our attack.
Table 5: The attack results on few-shot models on CIFAR-100. The abbreviations follow Table
Average-case
NMMD-attack
Test Acc.
Gap
Models
Train Acc.
Test Acc.
Train Acc.
Test Acc.
Abs.
Rel.
ProtoNet
99.97
54.88
99.96
52.46
2.42
4.41
'We
use
the
for- ing ht:p::/giithub.commrrrii
implementation
from
Prototypical Networks fortoorks-or-Feeeee
20

---

Under review as a conference paper at ICLR 2023
Table The comparison between average performance and attack performance of different size on CIFAR-10. The abbreviations follow Table
Average-case
NMMD-attack
Test Acc. Gap
k-shot
Models
Train Acc.
Test Acc.
Train Acc.
Test Acc.
abs.
rel.
FFN
52.16 110.93
30.13
1160
84.52
19.82
24.90
11.00
5.23
17.36
VGG-16
100.00 10.00
40.00
10.41
100.00
10.00
22.14
+2.02
17.89
44.65
50-shot
ResNet-18
100.00 +0.00
35.06
=0.46
100.00
+0.00
19.01
+1.17
10.83
45.78
ResNeXt-29
100.00 =0.00
34.95
=111
100.00
10.00
24.12
+0.59
10.83
30.99
DenseNet-121
100.00 =0.00
40.57
11.5
100.00 10.00
24.29
11.88
16.28
40.13
FFN
100.00 =0.00
43.08
#1.29
99.99
=0.01
36,12
#1.68
6.06
16.16
VGG-16
100.00 =0.00
56.93 =0.41
100.00
=0.00
34.93
#1.24
22.00
38.64
200-shot
ResNet-18
100.00 10.00
53.65
10.88
100.00
34.42
11.69
19.23
35.84
ResNeXt-29
100.00 =0.00
52.07
=1.10
100.00
=0.00
34.66
+1.19
17.41
33.44
DenseNet-
100.00 10.00
60.85
0.882
100.00
10.00
36.18
11.42
24.67
40.54
PFN
100.00 +0.00
59.10
00669
99.99
+0.01
55.84
+0090
3.26
5.52
VGG-16
100.00 =0.00
82.66
-0.15
100.00
78.22
=0.29
4.44
5.37
2000-shot
ResNet- 18
100.00 =0.00
80.68
-.25
100.00
=0.00
76.49
-0.77
4.19
5.19
ResNeXt-229
100.00 10.00
79.10 0.35
100.00
10.00
76.00
10.37
3.10
3.92
MenseNet-121
100.00 =0.00
85.96
=0.45
100.00 +0.00
82.64
#0.34
3.32
3.86
E ATTACK ON THE FEW-SHOT SETS WITH DIFFERENT SIZES
conducted attack experiments on few-shot sets with different sizes. The results show that with the increase of training data, the robustness gradually becomes better. There is considerable overlap between the attack and random datasets at larger data sizes. The performance drop on 2000-shot cases also can indicate the effectiveness of our methods.
30
25
20
15
10
5
0
50
200
500
2000
DenseNet
FFN
ResNet
ResNeXt
VGG
Figure 6: Attack performances on different scale subsets of CIFAR-I0. The bars represent the gap between NMMD-attack and average-case.
21

---

Under review as a conference paper at ICLR 2023
F SPURIOUS CORRELATION ANALYSIS
We case study the correlation between several spurious attributes and labels on the Imagenet-1KG The results in Figures 6,7&8 shows that spurious correlations are common. When trained randomly selected few-shot set, all three cases suffer from spurious correlations at different levels. However, the spurious correlations problem becomes less severe after training on our NMMD set in Bamboo Leaves VS. Small Panda and Water Background VS. Drake cases. Compared to results on CIFAR- 10, the spurious correlation alleviation less precise, possibly because the model has not learned the spurious attributes on Imagenet well. Furthermore, the spurious attributes on complicated figures may also be more complicated and hard to gain.
Model Trained Random Chaten Modal Trained NSAMD attack Feae-hort Samples
Feue-thet Samplees
Prd
(P)
Padcle
Not Podd
(P)
POWIZ)
00000
-0.77
(.91
0.145
Hormen 02 PNOWWAS
PVR by
PPP 0.501
PEW (P)
PVW 15)
-.1194
0.00
-8.555
Figure 7: Case study of spurious correlation: Human Involvement Paddle.
dea- Samples
Samalees
Drakec 305
A0R10) 0082
PTW
0.496
0.188
066
Wotee Backgrsnt
210
Wator
Figure 8: Case study of spurious correlation: Water Background + Drake.
Model Trained on Random Chesen Samphee
Model on MateeD--rtss artuc Chosse
Small Fando
Not Small Patce
Handa
[P]
Sumboo
Lowe Dt
Banboo
LEWeS RR
PrR1P)
NA Sarbace
LeOwa: (P)
P3 0.377
Leawen (8)
PRIP:
P:BIP)
Figure Case study of spurious correlation: Bamboo Leaves - Small Panda.
22

---

Under review as a conference paper at ICLR 2023
G VISUALIZATION OF THE FEW-SHOT SET SEARCHED BY NMMD-ATTACK
Figure 10: Visualization of the sho- set searched by NMMD-attack for CIFAR-10. We randomly
choose 50 examples for each label.
23


Reviews:
[{"review_id": "_ZGwgXW620", "reviewer": "Reviewer_Hiwt", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to evaluate the robustness of recognition systems to adversarially sampled training sets, and thus get an estimate of the worst case performance. The main contribution is an approach to identify the worst possible training set; this approach is based on maximizing the MMD between the training set and the test set, and introduces approximations based on the neural tangent kernel for tractably working with the MMD. Results are shown on four standard datasets and a few different model architectures.", "summary_of_the_review": "This paper is a step towards an important problem: understanding the robustness of neural networks to different training sets. However, in its current form it offers no useful insights and has minimal experiments that are not relevant to the stated title or thesis. Therefore it does not meet the bar.\n\nThe issues in this paper are too significant to be addressed in the rebuttal. I recommend that the authors address the weaknesses above and resubmit to a future conference.", "strength_and_weaknesses": "Strengths:\n1. The problem of finding worst case performance is an important one. There is not much analysis of how training sets impact downstream accuracy, so this is a step in the right direction.\n2. The idea of using NTK to get around the intractability of MMD is clever, and leads to a practical algorithm.\n\nWeaknesses:\n1. This paper positions itself as few-shot evaluation. Yet, experiments are with 50-500 examples per class, which is actually a moderately large training set. This is not few-shot evaluation in any shape or form.\n2. In addition, while the paper talks of evaluating few-shot models, *not even a single few-shot learning system is evaluated*. Instead, the paper chooses to evaluate standard classification approaches that are explicitly designed to work with large training sets. As such, the fact that these networks lose performance in the worst case evaluation is unsurprising and yields no useful insight. Few-shot learning techniques are intended to work well with small training sets, and may in fact be more resilient in this worst case evaluation. Comparing different few-shot learning techniques would yield some insight re:possible benefits of few-shot learning.\n3. At the very least, with standard classification approaches, there are knobs such as data augmentation and weight decay that one can change. There are also ways of training the network with adversarial data augmentation. I would expect at least some analysis along these lines to get some insight into what affects worst case performance. No such analysis is present.\n4. The fact that the worst case performance is going to be low is obvious just from the average case numbers: with the exception of MNIST, all models are clearly overfitting dramatically on the training set (see large gap between average case test and train performance in Table 1). It is therefore not surprising at all that the test-train gap will be even larger when the training set is adversarially chosen. This further underscores the need to evaluate regularization/augmentation.\n5. The one novel contribution of this paper is an approach to produce adversarially sampled training sets. But this is not evaluated well. The only baseline presented is average case performance. Here is a simple baseline that I would like to see: sample N different training sets, and report the *minimum* accuracy over all N. \n6. In practical scenarios, one usually has some control over what images to get labeled. I was hoping for some insight on this, and some general advice to downstream applications on how to intelligently sample a good training set. This is once again missing.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "FuVBDFl8s0", "reviewer": "Reviewer_X5Fg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims to evaluate the few-shot capabilities of a neural network on the worst-case subset of a dataset. The main motivation is to showcase the fragile ability of neural networks to memorize spurious statistical cues in the dataset, leading to poor generalization. To find the worst-case subset, the authors propose an algorithm that picks the subset with the highest gradient norms at initialization. With extensive experiments, they showcase the failure of well-known models in fewshot setting on datasets like CIFAR-10, CIFAR-100, and Imagenet-1k.", "summary_of_the_review": "Overall, my scores are borderline. The paper proposes a novel attack to understand the relation between few-shot evaluation and spurious correlations between training and test sets. However, there are issues with the theoretical motivation underlying the proposed algorithm. Hence, I would like the authors to clarify the questions raised above.", "strength_and_weaknesses": "Strengths:\n\nThe paper conducts multiple experiments to show that for a randomly initialized or a  pretrained model we can form a small subset of examples with the largest gradients at initialization (not necessarily of the same model), on which the model's performance drops drastically, compared to a random subset of examples. This clearly shows the fragility of different models in the few-shot setting. Furthermore, with a small ablation study, the authors show that the \"adversarial\" subset discovered is as diverse as a random subset of examples, but doesn't contain much spurious correlation that the model can latch on. This further shows the fragility of the real-world models to latch on to spurious correlations to perform well in few-shot settings.\n\n\nI have the following questions/concerns:\n\na) In eq. (2), the difference between $\\epsilon_Q( f_{I_k} ) $ and $\\epsilon_Q (f)$ is given by the MMD( P_{I_k}, P ). However, in theorem 5.2, we approximate $ MMD_f ( P, P_{I_k} ) $ with the NTK w.r.t. the function $f$. How do the authors relate the difference between $\\epsilon_Q( f_{I_k} ) $ and $\\epsilon_Q (f)$ with NTK w.r.t. $f$? \n\nb) Does the NTK assumption (the kernel doesn't change much during training) hold true for the models in the few-shot training in Table 1? \n\nc) What is the empirical gap observed in eq. (3) using the NTK of the models in Table 1? Does the NTK estimate give a non-trivial empirical bound for $\\epsilon_Q( f_{I_k} ) - \\epsilon_Q(f)$ in equation (3)?\n\nd) How do VGG-attack and ResNeXt-attack perform in table 2? Will we observe a similar drop in performance?\n\nFurthermore, how do subsets using gradients of ViT-B/16  and EfficientNetV2 perform in Table 2? Should one expect the drop in performance to be higher than the ones with the FFN attack?\n\ne) If the NTK assumption holds true, then we can also use the NTK of a few-shot finetuned model to gather the \"adversarial\" subset.\n\nFor the models used in Table 1, how do the models perform against the NMMD attack from their finetuned versions? That is if we finetune a model on a small random subset and use the model to create the adversarial subset, will we observe a similar drop in performance?  What about an NMMD attack with respect to the gradients of a model that has been finetuned on the entire dataset?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BgbT1HhJNz", "reviewer": "Reviewer_di2U", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper seeks to investigate the worst-case performance of neural network models in the low-data regime with respect to data sample index choice. Studying this problem is motivated by wanting to investigate neural network’s tendency to rely on spurious correlations for classification. The authors propose a method called NMMD attack that specifically selects the dataset subset that results in the worst performance. Experiments on several popular vision datasets (MNIST, CIFAR, ImageNet) and a variety of neural network architectures show that model performance does indeed suffer from picking the worst-case subset of the dataset.", "summary_of_the_review": "Studying the worst case performance of models as a function of which subset of the data is selected is an interesting problem, and not one I’m aware of much prior work on. W2 and W4 are my main concerns. With the approximations that the authors made in order to achieve tractibiliy, the authors aren’t truly finding the “worst” case datasets, as is claimed. Even if they are though, it’s not clear to me what the practical use of the proposed method is: as an adversarial attack, it would fail to be particularly sneaky, and as a warning to ML practitioners, it would seem that the likelihood of picking such a bad dataset subset are relatively unlikely. If the goal is just to show that neural network performance drops if we try pick the training set giving the worst-case accuracy, then the findings are perhaps somewhat unsurprising.", "strength_and_weaknesses": "Strengths:\n- S1. This paper’s topic is certainly interesting. Given only a few examples to learn from, the risk of overfitting to non-robust features is understandable. It’s well-known that few-shot accuracy on meta-test sets can vary wildly depending on which images/classes happen to be sampled, which is why the FSL community usually reports results averaged over thousands of meta-test sets. I’m not aware of any prior works that have specifically studied this problem.\n- S2. Experiments do show a significant drop-off in performance with the worst case datasets produced by the proposed NMMD attack. This drop-off is reflected across several models and datasets, so this appears to be a general phenomenon rather than a quirk of a particular architecture or dataset. This is perhaps unsurprising, but Table 1 does clearly quantify how bad this drop can be.\n- S3. The writing is fairly understandable. That said, the draft could use another round of proofreading, as there are a number of grammatical and idiomatic errors. See Miscellaneous below for a non-exhaustive list.\n\nWeaknesses:\n- W1. Terminology: I’m not confident that the framing of the methodology is quite right. I can see the connections and how the authors may have been inspired, but there are some subtle differences here that require care, as using terminology from other common problems can lead people to make assumptions about the setting and the proposed solution.\n  - Adversarial attacks: Adversarial attacks typically refer to a scenario where an adversary trying to sabotage the model with inputs with imperceptible changes. There is no adversary here, nor are the changes “imperceptible,” as there are no modifications to individual images here. Rather, we’re specifically selecting the worst images for training.\n  - Distribution shift: While the distribution here technically is indeed different and therefore a “shift,” it’s due to a specific sampling strategy, not because of factors that people usually refer to as distribution shifts (e.g. domain changes, temporal evolution). \n  - Robustness: I would also take caution when talking about neural networks having poor “robustness”, which often refers to individual samples during inference, rather than training datasets.\n  - Few-shot: The experimental evaluation is more akin to standard supervised training and evaluation in the low data regime, as opposed to the meta-dataset set-up common in FSL: FSL typically has both far fewer examples per class (e.g. 5, as opposed to 500 in this paper), and evaluation of transfer from base classes to novel classes (train and eval here are done on the same classes). These differences do not necessarily invalidate the results, but it may be a bit confusing to people expecting more typical FSL benchmarks like MiniImageNet/TieredImageNet when calling the method few-shot. NMMD seems like it should apply to such setting, so the authors may consider more standard FSL benchmark as well.\n\n- W2. Methodology: If my understanding is correct, the method is just identifying the examples in each class that result in the largest magnitude gradients when the model is randomly initialized. This doesn’t strike me as necessarily resulting in the worst case few-shot sets. In particular, tossing the off-diagonal terms means inter-sample terms are discarded. This seems incongruous with the authors’ claim in the Introduction that lack of robustness can occur due to spurious correlations between samples. Furthermore, when I look at the resulting few-shot sets in Appendix G, these don’t strike me as being adversarially challenging few-shot sets, nor are spurious correlations visible. While it was the original goal, I don’t know if “worst” case is the right term here.\n\n- W3. Notation: While I appreciate the desire to be precise, I do find some of mathematical notation somewhat unnecessarily verbose. For example, Section 4 uses several paragraphs largely to say that we assume there’s a model that generalizes from the support set to the query set (an underlying assumption in ML), that the support and query sets have the same label distribution (a standard property of classification benchmarks), and that we’re trying to find data samples that result in the highest error (already stated in the Introduction and Related Work). \n\n- W4. Practicality: While the results are somewhat interesting, what’s the significance? When collecting a dataset, it’s highly unlikely that one would be unlucky enough to assemble such a worst case dataset, and one can mitigate the effects by collecting more data. As an adversarial attack perpetuated by a malicious actor, NMMD is not practical either, because the adversary would have to replace the entire dataset (in contrast to, say, data poisoning attacks), and the choice of including the hardest image samples would likely be highly noticeable to the model owner. \n\n\nQuestions:\n\n- Q1. Given that popular datasets are known to have errors or ambiguity [1,2], I wonder if the worst case results in support sets with mislabeled support sets [3]? How do we ensure that worst case is still representative or useful?\n- Q2. What were the pre-trained models pre-trained on? ImageNet? If so, there is a lot of overlap with the CIFAR-10 classes, so this isn’t truly a few-shot setting. These models would similarly be much less significantly impacted by a handful of training samples, which would explain the much smaller drop in performance from NMMD.\n- Q3. How was the test accuracy evaluated? Was it the accuracy at the end of training, or did you use early stopping? Or was the best test accuracy during training reported? Asking because the train accuracies show strong evidence of overfitting (mostly 99, 100%) and 200 epochs as reported in Table 4 is a lot. When the test accuracy is reported may have a strong impact on the results.\n- Q4. In few-shot learning evaluations, we often randomly sample not only the K shots per class, but also the N-way classification task. Have you considered how specific combinations of classes may result in extra poor performance?\n- Q5. Have you considered looking at the reverse scenario? Which samples give the best performance? How are they different from the worst case samples? Can the best case samples be used a compact, more portable version of the dataset?\n\n\n[1] Yun et al. Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels. CVPR 2021.\n\n[2] Northcutt et al. Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks. NeurIPS D&B 2021.\n\n[3] Liang et al. Few-shot Learning with Noisy Labels. CVPR 2022.\n\nMiscellaneous:\n- “few-shot” vs “few shot” as a hyphenated adjective\n- P_few isn’t really used and therefor doesn’t seem like a necessary notation to introduce.\n- Section 4: “Assumption. 4.1”, “Assumption. 4.2” <= extra period\n- Section 5: “Here, We first proves” <= “Here, we first prove”\n- Section 5: “Last” <= “Lastly”\n- Definition 5.1: notation collision. y is already used as the labels in Sec. 3 + 4.\n- Theorem 5.1: “occurred” <= “incurred”?\n- Equation 4: Should 1/m be 1/k and x_i_m be just x_m?\n- Section 6: “a feed-forward neural networks” <= “a feed-forward neural network”. Also, all the models listed are feed-forward neural networks. The description of 1) relative to the other models needs to be clearer. \n- Section 6: The distinction between models like ResNet and DenseNet with “pretrained models” ViT and EfficientNet is confusing. All of the listed models can be pre-trained.\n- Section 6: notation collision for m, which is used as the number of trials and as an index in Eq 4.\n- Section 7.1: Missing space after “3)”\n- Table 3: “Average-case” is confusing. I initially thought this column was the average across the previous columns. Better call it “Random” to better match the text.\n- Reproducibility Statement: What’s 6? “A” is “Appendix A”?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject