Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to address the problem of adversarial attack for low shot image classification. This work is motivated by the challenging scenario where there is a need of significant amount of data to train an adversarial robust classifier and there is not much data under few-shot setting. This work proposed and demonstrated a simple approach for robust few shot classifier. A model is first adversarially trained on the base classes and produce a robust base model. The feature extractor of the robust base model is then frozen. With the frozen feature extractor, 2  different manners are used for training a novel / few shot classifier, including (1) training a linear classifier and (2) computing category centroid for each novel class and perform nearest neighbor classification using the centroid. The experiment demonstrates that this simple baseline can outperform prior baselines on 3 datasets. The ablation study includes 1 vs 5 shot and different adversarial training methods. However, the core contribution of this paper is weakened due to the lack of theoretical analysis.
main_review: Strength:
1. The paper is simple to follow and clear for the reader to understand.
2. The experiment is valid and cover various ablation study.

Weakness
1. This paper demonstrates a simple approach for learning a robust few shot classifier. Despite its simplicity, the contribution of this paper remains weak and vague. To my best knowledge, this paper has minor technical novelty, because the few shot learning method used in this work have already existed in the few shot literature. The adversarial training used in this work also follow the standard adversarial training manner. As a result, the technical contribution of this work remains unclear.
2. While this paper demonstrates a simple baseline for learning an adversarial robust few shot classifier, there is no theoretical analysis of why such simple manner is working. From my best understanding, the only support of this paper is the experiment result, which is only evaluated on 2 different few shot approach. It is unclear whether the proposed training scheme can generalize to other few shot learning method.   Moreover, the main experiments are conducted using PGD and IBP attack/adversarial training, indicating the generalization of this framework to other attack method is questionable.

summary_of_the_review: While this paper demonstrates good results with the proposed simple framework, there is no theoretical analysis of why such simple method is working. Moreover, it is also unclear to the reader whether the proposed framework generalize to other attack methods and other few shot approaches.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to train adversarially robust few-shot classifiers. The main idea is to apply adversarial training (PGD) to pre-train the feature extractor and apply Nearest Centroid to update the classifier with novel data for few-shot learning.
main_review: (+) The paper is well written and organized and easy to follow.
(+) The proposed approach is simple and technically sound.
(+) Multiple benchmarks have been employed to validate the performance.

(-) The proposed method is a straightforward combination of well-known knowledge and techniques. Although it is technically sound, it is mainly empirically designed without significant contribution to the field.
(-) The experiments are limited as:
1. all the three benchmarks are toy-level datasets; 
2. only two baselines are compared that can hardly represent the current state of the arts; 
3. A single metric of accuracy is used for all the experiments which failed to justify the proposed method from a different perspective. 
4. The performance increase is either negative or very subtle in general.

summary_of_the_review: To sum up, this is a mediocre submission and I would rank it below the acceptance bar of ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper 1) proposes a simple transfer learning approach to enable train adversarially robust few-shot classifiers for few-shot image classification, and 2) present a method for novel classification task based on calibrating the centroid of the few-shot category towards the base classes. Results show good performance has been achieved on three benchmarks i.e., Mini-ImageNet, CIFAR-FS, and CUB datasets.

My main concern of this work is that the improved performance can be mainly resulted from the pretrained stage (base training) using the base dataset $X_b$. Normally, pretraining can improve the performance on small tasks.
main_review: 1) What is the abundantly-labeled base dataset $X_b$ used in this work? How do you guarantee that there is no overlap information between $X_b$ and $X_n$. 

2) It is important to under standard each stage i.e., base training and novel training performance. However, I have not seen ablation uses base training only and novel training only.

summary_of_the_review: My main concern of this paper is the comparison maybe not fair i.e., this work uses a pretrained stage. It is also difficult for us to characterize if any overlap information between $X_b$ and $X_n$. Overall, I think it should compare all methods in the same condition. The ablation of base training and novel training is necessary to characterize the importance of each stage for robust few-shot learning.
review_title: null

Meta-review: This paper finally received divergent and borderline reviews with one positive (6) and two negative (5) rates. Based on the reviews, authors’ responses and updated manuscript, we would like to decide to reject this work at this time even though this submission has a lot of potentials such as simplicity and efficiency.
Positively, all the reviews agree that the proposed approach is simple but effective to improve the robustness of few-shot classifiers. However, there is some room for improvement to be a stronger submission: (i) the technical novelty may need to be better presented, and (ii) the improved performance may need to be better justified (e.g., the effect of the pretrained stage).
Decision: Reject