Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper aims to evaluate the few-shot capabilities of a neural network on the worst-case subset of a dataset. The main motivation is to showcase the fragile ability of neural networks to memorize spurious statistical cues in the dataset, leading to poor generalization. To find the worst-case subset, the authors propose an algorithm that picks the subset with the highest gradient norms at initialization. With extensive experiments, they showcase the failure of well-known models in fewshot setting on datasets like CIFAR-10, CIFAR-100, and Imagenet-1k.
strength_and_weaknesses: Strengths:

The paper conducts multiple experiments to show that for a randomly initialized or a  pretrained model we can form a small subset of examples with the largest gradients at initialization (not necessarily of the same model), on which the model's performance drops drastically, compared to a random subset of examples. This clearly shows the fragility of different models in the few-shot setting. Furthermore, with a small ablation study, the authors show that the "adversarial" subset discovered is as diverse as a random subset of examples, but doesn't contain much spurious correlation that the model can latch on. This further shows the fragility of the real-world models to latch on to spurious correlations to perform well in few-shot settings.


I have the following questions/concerns:

a) In eq. (2), the difference between $\epsilon_Q( f_{I_k} ) $ and $\epsilon_Q (f)$ is given by the MMD( P_{I_k}, P ). However, in theorem 5.2, we approximate $ MMD_f ( P, P_{I_k} ) $ with the NTK w.r.t. the function $f$. How do the authors relate the difference between $\epsilon_Q( f_{I_k} ) $ and $\epsilon_Q (f)$ with NTK w.r.t. $f$? 

b) Does the NTK assumption (the kernel doesn't change much during training) hold true for the models in the few-shot training in Table 1? 

c) What is the empirical gap observed in eq. (3) using the NTK of the models in Table 1? Does the NTK estimate give a non-trivial empirical bound for $\epsilon_Q( f_{I_k} ) - \epsilon_Q(f)$ in equation (3)?

d) How do VGG-attack and ResNeXt-attack perform in table 2? Will we observe a similar drop in performance?

Furthermore, how do subsets using gradients of ViT-B/16  and EfficientNetV2 perform in Table 2? Should one expect the drop in performance to be higher than the ones with the FFN attack?

e) If the NTK assumption holds true, then we can also use the NTK of a few-shot finetuned model to gather the "adversarial" subset.

For the models used in Table 1, how do the models perform against the NMMD attack from their finetuned versions? That is if we finetune a model on a small random subset and use the model to create the adversarial subset, will we observe a similar drop in performance?  What about an NMMD attack with respect to the gradients of a model that has been finetuned on the entire dataset?


clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. The theoretical statements have some issues, which I have pointed out above. However, the experiments are novel and clearly show the fragility of models in few-shot settings.
summary_of_the_review: Overall, my scores are borderline. The paper proposes a novel attack to understand the relation between few-shot evaluation and spurious correlations between training and test sets. However, there are issues with the theoretical motivation underlying the proposed algorithm. Hence, I would like the authors to clarify the questions raised above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper seeks to investigate the worst-case performance of neural network models in the low-data regime with respect to data sample index choice. Studying this problem is motivated by wanting to investigate neural network’s tendency to rely on spurious correlations for classification. The authors propose a method called NMMD attack that specifically selects the dataset subset that results in the worst performance. Experiments on several popular vision datasets (MNIST, CIFAR, ImageNet) and a variety of neural network architectures show that model performance does indeed suffer from picking the worst-case subset of the dataset.
strength_and_weaknesses: Strengths:
- S1. This paper’s topic is certainly interesting. Given only a few examples to learn from, the risk of overfitting to non-robust features is understandable. It’s well-known that few-shot accuracy on meta-test sets can vary wildly depending on which images/classes happen to be sampled, which is why the FSL community usually reports results averaged over thousands of meta-test sets. I’m not aware of any prior works that have specifically studied this problem.
- S2. Experiments do show a significant drop-off in performance with the worst case datasets produced by the proposed NMMD attack. This drop-off is reflected across several models and datasets, so this appears to be a general phenomenon rather than a quirk of a particular architecture or dataset. This is perhaps unsurprising, but Table 1 does clearly quantify how bad this drop can be.
- S3. The writing is fairly understandable. That said, the draft could use another round of proofreading, as there are a number of grammatical and idiomatic errors. See Miscellaneous below for a non-exhaustive list.

Weaknesses:
- W1. Terminology: I’m not confident that the framing of the methodology is quite right. I can see the connections and how the authors may have been inspired, but there are some subtle differences here that require care, as using terminology from other common problems can lead people to make assumptions about the setting and the proposed solution.
  - Adversarial attacks: Adversarial attacks typically refer to a scenario where an adversary trying to sabotage the model with inputs with imperceptible changes. There is no adversary here, nor are the changes “imperceptible,” as there are no modifications to individual images here. Rather, we’re specifically selecting the worst images for training.
  - Distribution shift: While the distribution here technically is indeed different and therefore a “shift,” it’s due to a specific sampling strategy, not because of factors that people usually refer to as distribution shifts (e.g. domain changes, temporal evolution). 
  - Robustness: I would also take caution when talking about neural networks having poor “robustness”, which often refers to individual samples during inference, rather than training datasets.
  - Few-shot: The experimental evaluation is more akin to standard supervised training and evaluation in the low data regime, as opposed to the meta-dataset set-up common in FSL: FSL typically has both far fewer examples per class (e.g. 5, as opposed to 500 in this paper), and evaluation of transfer from base classes to novel classes (train and eval here are done on the same classes). These differences do not necessarily invalidate the results, but it may be a bit confusing to people expecting more typical FSL benchmarks like MiniImageNet/TieredImageNet when calling the method few-shot. NMMD seems like it should apply to such setting, so the authors may consider more standard FSL benchmark as well.

- W2. Methodology: If my understanding is correct, the method is just identifying the examples in each class that result in the largest magnitude gradients when the model is randomly initialized. This doesn’t strike me as necessarily resulting in the worst case few-shot sets. In particular, tossing the off-diagonal terms means inter-sample terms are discarded. This seems incongruous with the authors’ claim in the Introduction that lack of robustness can occur due to spurious correlations between samples. Furthermore, when I look at the resulting few-shot sets in Appendix G, these don’t strike me as being adversarially challenging few-shot sets, nor are spurious correlations visible. While it was the original goal, I don’t know if “worst” case is the right term here.

- W3. Notation: While I appreciate the desire to be precise, I do find some of mathematical notation somewhat unnecessarily verbose. For example, Section 4 uses several paragraphs largely to say that we assume there’s a model that generalizes from the support set to the query set (an underlying assumption in ML), that the support and query sets have the same label distribution (a standard property of classification benchmarks), and that we’re trying to find data samples that result in the highest error (already stated in the Introduction and Related Work). 

- W4. Practicality: While the results are somewhat interesting, what’s the significance? When collecting a dataset, it’s highly unlikely that one would be unlucky enough to assemble such a worst case dataset, and one can mitigate the effects by collecting more data. As an adversarial attack perpetuated by a malicious actor, NMMD is not practical either, because the adversary would have to replace the entire dataset (in contrast to, say, data poisoning attacks), and the choice of including the hardest image samples would likely be highly noticeable to the model owner. 


Questions:

- Q1. Given that popular datasets are known to have errors or ambiguity [1,2], I wonder if the worst case results in support sets with mislabeled support sets [3]? How do we ensure that worst case is still representative or useful?
- Q2. What were the pre-trained models pre-trained on? ImageNet? If so, there is a lot of overlap with the CIFAR-10 classes, so this isn’t truly a few-shot setting. These models would similarly be much less significantly impacted by a handful of training samples, which would explain the much smaller drop in performance from NMMD.
- Q3. How was the test accuracy evaluated? Was it the accuracy at the end of training, or did you use early stopping? Or was the best test accuracy during training reported? Asking because the train accuracies show strong evidence of overfitting (mostly 99, 100%) and 200 epochs as reported in Table 4 is a lot. When the test accuracy is reported may have a strong impact on the results.
- Q4. In few-shot learning evaluations, we often randomly sample not only the K shots per class, but also the N-way classification task. Have you considered how specific combinations of classes may result in extra poor performance?
- Q5. Have you considered looking at the reverse scenario? Which samples give the best performance? How are they different from the worst case samples? Can the best case samples be used a compact, more portable version of the dataset?


[1] Yun et al. Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels. CVPR 2021.

[2] Northcutt et al. Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks. NeurIPS D&B 2021.

[3] Liang et al. Few-shot Learning with Noisy Labels. CVPR 2022.

Miscellaneous:
- “few-shot” vs “few shot” as a hyphenated adjective
- P_few isn’t really used and therefor doesn’t seem like a necessary notation to introduce.
- Section 4: “Assumption. 4.1”, “Assumption. 4.2” <= extra period
- Section 5: “Here, We first proves” <= “Here, we first prove”
- Section 5: “Last” <= “Lastly”
- Definition 5.1: notation collision. y is already used as the labels in Sec. 3 + 4.
- Theorem 5.1: “occurred” <= “incurred”?
- Equation 4: Should 1/m be 1/k and x_i_m be just x_m?
- Section 6: “a feed-forward neural networks” <= “a feed-forward neural network”. Also, all the models listed are feed-forward neural networks. The description of 1) relative to the other models needs to be clearer. 
- Section 6: The distinction between models like ResNet and DenseNet with “pretrained models” ViT and EfficientNet is confusing. All of the listed models can be pre-trained.
- Section 6: notation collision for m, which is used as the number of trials and as an index in Eq 4.
- Section 7.1: Missing space after “3)”
- Table 3: “Average-case” is confusing. I initially thought this column was the average across the previous columns. Better call it “Random” to better match the text.
- Reproducibility Statement: What’s 6? “A” is “Appendix A”?

clarity,_quality,_novelty_and_reproducibility: As I noted above, the method is presented in a way that’s perhaps more formal and verbose than is actually necessary, especially since the diagonal assumption this paper makes dramatically simplifies the algorithm being proposed. Regardless, the paper is still fairly clear and understandable. I’m not aware of other previous works studying this topic, so in that sense the paper is original, though I don’t find the key findings to be all that surprising.

A Reproducibility Statement is included, and reproducing code is provided.

summary_of_the_review: Studying the worst case performance of models as a function of which subset of the data is selected is an interesting problem, and not one I’m aware of much prior work on. W2 and W4 are my main concerns. With the approximations that the authors made in order to achieve tractibiliy, the authors aren’t truly finding the “worst” case datasets, as is claimed. Even if they are though, it’s not clear to me what the practical use of the proposed method is: as an adversarial attack, it would fail to be particularly sneaky, and as a warning to ML practitioners, it would seem that the likelihood of picking such a bad dataset subset are relatively unlikely. If the goal is just to show that neural network performance drops if we try pick the training set giving the worst-case accuracy, then the findings are perhaps somewhat unsurprising.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper aims to evaluate the robustness of recognition systems to adversarially sampled training sets, and thus get an estimate of the worst case performance. The main contribution is an approach to identify the worst possible training set; this approach is based on maximizing the MMD between the training set and the test set, and introduces approximations based on the neural tangent kernel for tractably working with the MMD. Results are shown on four standard datasets and a few different model architectures.
strength_and_weaknesses: Strengths:
1. The problem of finding worst case performance is an important one. There is not much analysis of how training sets impact downstream accuracy, so this is a step in the right direction.
2. The idea of using NTK to get around the intractability of MMD is clever, and leads to a practical algorithm.

Weaknesses:
1. This paper positions itself as few-shot evaluation. Yet, experiments are with 50-500 examples per class, which is actually a moderately large training set. This is not few-shot evaluation in any shape or form.
2. In addition, while the paper talks of evaluating few-shot models, *not even a single few-shot learning system is evaluated*. Instead, the paper chooses to evaluate standard classification approaches that are explicitly designed to work with large training sets. As such, the fact that these networks lose performance in the worst case evaluation is unsurprising and yields no useful insight. Few-shot learning techniques are intended to work well with small training sets, and may in fact be more resilient in this worst case evaluation. Comparing different few-shot learning techniques would yield some insight re:possible benefits of few-shot learning.
3. At the very least, with standard classification approaches, there are knobs such as data augmentation and weight decay that one can change. There are also ways of training the network with adversarial data augmentation. I would expect at least some analysis along these lines to get some insight into what affects worst case performance. No such analysis is present.
4. The fact that the worst case performance is going to be low is obvious just from the average case numbers: with the exception of MNIST, all models are clearly overfitting dramatically on the training set (see large gap between average case test and train performance in Table 1). It is therefore not surprising at all that the test-train gap will be even larger when the training set is adversarially chosen. This further underscores the need to evaluate regularization/augmentation.
5. The one novel contribution of this paper is an approach to produce adversarially sampled training sets. But this is not evaluated well. The only baseline presented is average case performance. Here is a simple baseline that I would like to see: sample N different training sets, and report the *minimum* accuracy over all N. 
6. In practical scenarios, one usually has some control over what images to get labeled. I was hoping for some insight on this, and some general advice to downstream applications on how to intelligently sample a good training set. This is once again missing.


clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is relatively clear.
Reproducibility: The paper is reproducible to the extent I see.
Quality: I am not sure what is meant by quality. If this is an overall rating of the work, then I think this work does not meet the bar for ICLR.
Novelty: The idea of maximizing MMD is novel, but none of the experimental findings are novel.
summary_of_the_review: This paper is a step towards an important problem: understanding the robustness of neural networks to different training sets. However, in its current form it offers no useful insights and has minimal experiments that are not relevant to the stated title or thesis. Therefore it does not meet the bar.

The issues in this paper are too significant to be addressed in the rebuttal. I recommend that the authors address the weaknesses above and resubmit to a future conference.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper studies the robustness of few-shot learners to adversarially selected training sets. The paper introduces an algorithm to approximate worst set selection by searching for the label-balacned subset of images with largest (approximate) MMD w.r.t. the full training set. This attack reduces the performance of a variety of few-shot learning methods.

Strengths:
* Interesting novel problem
* Good analysis of spurious features in selected set

Weaknesses:
* Practical utility is unclear
* The approximation to MMD makes strong assumptions and it's unclear if it is a good approximation
* The framing of the paper could use work; in particular, the community might not consider this to be few-shot (50-500 shots), and the paper mostly does not evaluate algorithms specifically designed to be few-shot learners (the proposal to reframe as "limited data" may help)
* Generally, the reviewers did not find the results to yield clear and significant insights


justification_for_why_not_higher_score: All reviewers found that this paper did not meet the bar for ICLR. Two revewers engaged in extensive discussion, raising issues with the framing of the paper, the thoroughness of the experiments, and the utility of the method. The authors rebuttal to these points was considered but in the end the AC agrees the paper needs more work. It could help to add more experiments on methods specifically designed for few-shot / limited data settings, and it also might help to demonstrate practical uses, such as using the discovered worst case sets to improve model robustness.
Decision: Reject