Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary

This paper investigates the adversarial robustness of model agnostic meta-learning (MAML). Adversarial robustness can be added to MAML in two places, meta-update stage and and fine-tune stage. It shows that robustifying the meta-update stage via fast attack generation method is sufficient to achieve fast robustness adaptation without losing generalization and computation efficiency in general. The paper also demonstrates that unlabeled data can help using contrastive representation learning to improve generalization and robustness. 

Strengths

It shows:
1. adversarial training with the projected gradient descent (PGD) attack generation method is only needed at meta-update stage for MAML.

2. computationally efficient adversarial training methods such as  almost-no-inner-loop (ANIL) fine-tuning strategy and FGSM are enough.

3. training with unlabeled data using contrastive learning further improves generalization and robustness. 

Weaknesses

Adversarial robustness of MAML has been studied in (Goldblum et al., 2019). This paper provides further investigation. The study seems to be straightforward and incremental. 

All results are based on miniImageNet. MAML has also been applied in reinforcement learning. It is not clear the results in this paper carries out to the RL setting.

Decision

The paper improves MAML with adversarial training at meta-update stage and unlabeled data through contrastive learning. However, the studies and techniques seem to be incremental compared with related work. 


=====POST-REBUTTAL COMMENTS======== 

The authors provided additional experiments on CIFAR-FS and Omniglot. The results show that their methods outperform the baseline method adversarial querying (AQ). 

It is still not clear whether the methods work in the reinforcement learning setting. As the original MAML paper shows that MAML works for RL problems, it is important to address this question. Otherwise, it could potentially limits the applicability of the proposed methods in the paper.

I still have concerns over their novelty and the significance of their contributions. 

Overall, I applaud their effort to address my comments. I am more positive on the paper than before. My rating is a solid 6. The paper in the current stage does not warrant a higher rating for ICLR in my opinion.
review_title: explores adversarial robustness and contrastive learning of MAML

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper explores a way to promote the adversarial robustness in Model-agnostic meta-learning (MAML). It conducts extensive experiments to show regularizing adversarial robustness at meta-update level is sufficient to offer fast and effective robustness adaptation on few-shot test tasks. However, it lacks the theoretical analysis for this conclusion. Also, the experiments only are conducted on few-shot image classification task on miniImageNet dataset. This makes this conclusion lack sufficient credibility.

This paper also shows the use of unlabeled data augmentation can provide additional benefits on adversarial robustness of MAML. Due to the low-data regime in few-shot classification, it is reasonable to introduce contrastive learning task to few-shot learning, which may facilitate the development of following research.

It is of great importance to investigate the adversarial robustness in meta-learning framework, such as MAML in this paper. It might give us insights on designing network structure. However, it would be better to give more insights on the way to integrate the adversarial robustness rather than list the experimental results.

Novelty of this paper seems to be technically marginal. While the aggregation of adversarial robustness on meta-learning framework is new, detailed architecture explanation with novelty is lacked.
review_title: clear motivation and reasonable method, but marginal novelty

final_score: rating: 6: Marginally above acceptance threshold
review_content: It is an interesting paper empirically addressing adversarial robustness of model agnostic meta learning (MAML). The paper investigates where to incorporate robust regularization in MAML in order to improve adversarial robustness, and based on that *efficient* robust MAML methods are proposed. Interestingly, contrastive learning is incorporated and derive a more robust MAML model. 

My concerns are listed as below:

- The paper is highly empirical, whearas only one dataset is employed. The claims would be more convincing with more datasets.
- The key point is not properly emphasized. For example, Section 4 and Section 5.1 (TRADES) do not provide very important insights compared with Section 5.2 (CL) — maybe Section 5.2  could be extended furthur with some other contents defered in the Appendix.
review_title: Clear and interesting paper but may not be enough convincing

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes to integrate meta-learning (MAML specifically) and adversarial training (AT) to improve adversarial robustness of meta-learning in terms of fast and effective adaptation on few-shot test tasks. To achieve this, the authors provide extensive investigation and solid answers on when, how and why their method works.

Overall I vote for accepting. The authors found that it seems to be natural to combine MAML and AT together to enhance the adversarial robustness of meta-learning, however, due to computational complexity and bi-level optimization in MAML and AT, they further pose the above three questions when, how, and why step by step, to which they also provide affirmative answers in the paper. Some suggestions are given in the cons part below.

Pros:

1. Overall the paper is well-written. In particular, its motivation and contribution are clearly explained and summarized, which is quite fluent when reading.

2. The paper presents visual evidence of when to incorporate robustness regularization in MAML by leveraging input attribution maps of neurons, which is reasonable and interesting for me.

3. The paper leverages contrastive learning into MAML to help the model’s adversarial robustness, which is novel and interesting to me.

Cons:

1. For the experiment part, more other few-shot learning datasets (such as Omniglot) or other few-shot settings (such as 5-way or 20-way) can be utilized to evaluate the proposed method as well as to present more data points.
review_title: submission 2301 review

Meta-review: The reviewers’ main concern was a lack of experiments, and additional experiments were provided by the authors.  While the rebuttal was not addressed by the reviewers, the AC feels that the rebuttal did address a number of experimental concerns well enough to justify accepting this paper.
Decision: Accept (Poster)