Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces self-supervised adversarial learning into adversarial meta-learning, and proposes a transferable robust meta-learning via bilevel attack (TROBA) method by using a bilevel attack scheme. Specifically, TROBA is built on BOIL and TRADES by adding a self-supervised loss. Extensive experiments effectively show the effectiveness of the proposed method.  
strength_and_weaknesses: - **Strength**: 
1. This paper is well organized and easy to follow.
2. The core parts of the proposed TROBA are clear.
3. The experimental results seem to be good. 

- **Weakness**: 
Overall, the key contributions especially the technique contribution are not clear. This is because self-supervised adversarial learning and TRADES have been widely used in general adversarial training, and BOIL is one existing meta-learning method extended from MAML. If the bilevel parameter augmentation and attack are the main technique contribution, the difference and effectiveness of this strategy compared to the normal data augmentation used in general self-supervised adversarial learning should be clearly claimed and demonstrated. If the entire framework is the main contribution, it would be interesting to see the effectiveness of this framework of using different adversarial training methods and different meta-learning methods.     

- Several main concerns are as follows:
1. What’s the main difference between the proposed bilevel parameter augmentation and the normal data augmentation? Is it different only because of the usage of Siamese network with different parameters? Could you verify the effectiveness of this operation by experiments?
2. What’s the main difference between the bilevel attack and the normal self-supervised adversarial learning? Is it different only because of the swapping operation of two views? Could you verify the effectiveness of this operation by experiments?
3. It is not clear if the adversarial example l_n^adv used in Eq. (7) is generated by class-wise attack or instance-wise attack. 
4. It is not clear how to update the backbone and classifier in Eq. (2) for readers. Note that the classifier is fixed in BOIL. 
5. I am not sure if all the comparison methods employ the same backbone and the same epsilon of attack. Does this paper re-implement all these methods in the same framework?
6. It is doubtful for the experiment in Table 7. Are the results still for 5-shot classification? Note that the splits used in CIFAR-FS and CIFAR-100 are different. Why not directly perform other models on CIFAR-FS? How many samples used for training for TROBA and how many samples used for other models? This part should be clearly explained, otherwise the results are doubtful. 
7. Do all the comparison methods employ the same input image size of 32 * 32?

clarity,_quality,_novelty_and_reproducibility: - **Clarity**: Good.
- **Quality**: Good.
- **Novelty**: Incremental.
- **Reproducibility**: Great.
summary_of_the_review: The overall of this work is great. However, there are still multiple parts that are not clear, which need to be further clarified.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to solve the problem of adversarial meta-learning, where the difficulty lies in the few-shot learning of each inner loop task. The proposed method consists of 1) generating adversarial query examples by attacking a contrastive loss between clean and perturbed images; 2) using two independent augmentations in both inner loop optimization and attack; 3) adversarial meta learning with TRADES loss and a contrastive loss. Experimental results show that the proposed method achieves improved robustness performance compared with several baselines.
strength_and_weaknesses: Strength:

The studied problem is interesting and practical as learning with few examples per task is important in some applications.

Experimental results show the effectiveness of the proposed loss function in terms of robustness. Ablation studies also verify that each individual design is effective.


Weaknesses:

The major weakness of this work is that the proposed loss function seems to be a combination of the adversarial meta-learning method AQ and the objective function of contrastive adversarial learning. The proposed loss function thus inherited the advantages from both domains. The technique is not novel.

The title of the paper suggests learning transferable representations for adversarial few-shot learning. My feeling is that the word transferable seems to be redundant since meta-learning for few-shot learning itself already has the implication of improving the transferability of representations. This work mainly uses self-supervised learning methods, specifically contrastive learning loss function, to learn better representations that can transfer to different tasks or domains without explicit transfer methods (e.g. things like task augmentation or domain alignment) beside meta-learning. Therefore, in my opinion it is perhaps better to claim it as few-shot self-supervised robust representation learning.

clarity,_quality,_novelty_and_reproducibility: Most explanations of the proposed method are clear and are presented with high quality. The technique is not novel and some implementation details are not given.
summary_of_the_review: The proposed method targets the problem of adversarial meta-learning using techniques from contrastive learning. The incorporation of existing loss functions is not novel, and performance gain in experiments are expected due to the effectiveness of contrastive learning loss functions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: It proposes a adversarial meta-learning framework with bilevel attacks, which allows the model to learn generalizable robust representations across tasks and domains. The framework obtains higher robustness in few-shot tasks both in the seen domain and the unseen domains.
strength_and_weaknesses: \+ The visualization of the representations of the instance-wise adversarial samples from the unseen domains in Figure 2 shows that the model is able to obtain well-separated feature space for attacked samples on this domain (CIFAR-10) even before adapting to it, while
the previous adversarial meta-learning framework learns a feature space with large overlaps across the adversarial instances belonging
to different classes.

\+ TROBA shows smoother loss surface to adversarial examples in Figure 4 which is directly associated with better robustness and generalization.

\+ The ablation study checks the performance of each component, such as it compares the performance with or without augmentation, and with instance-wise or class-wise attack. It also checks the performance of the loss function compared with CE + selfsup loss or TRADES loss only. 

\- The technical contribution may be limited. It adopts many techniques in this paper. Most of techniques are proposed in prior works such as 
TRADE loss,  data augmentation with random crop/flip/color distortion, and instance-wise classification loss in adversarial self-supervised learning. It combines these techniques but the technical contribution may not be significant. 
clarity,_quality,_novelty_and_reproducibility: see the strength and weaknesses. It provides detailed instructions about how to reproduce the results in figures and tables in the appendix. It  It provides visualization results to demonstrate why it performs better, which is easy to follow. The technical contribution may be limited as it adopts many existing techniques. 
summary_of_the_review: It combines a few existing techniques to achieve good performance in adversarial meta-learning. It provides visualization results to demonstrate why it performs better, which is easy to follow. My main concern is that the technical contribution may be limited as most of the techniques are adopted from prior works.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper develops a new algorithm for adversarially robust few-shot MAML.  The interesting part is that the adversarial robustness transfers across domains.  Experiments are conducted on multiple datasets and against multiple baseline methods.
strength_and_weaknesses: In my opinion, the strength here is the transferability of robustness across domains.  The empirical results otherwise do not seem very strong.

It would be good to test on higher performing meta-learning algorithms such as R2D2 and MetaOptNet, as in the AQ paper, at least as baselines for AQ since they may not be applicable to your method instead of only testing on MAML.  The numbers in the AQ paper look higher even for directly comparable experiments such as seen domain 5-shot mini-ImageNet.  It seems like a major flaw that your method can only be used with MAML despite MAML not being a very good few-shot learning algorithm.  This is my central critique of this paper.

Algorithm 1 is impossible to understand, especially the arguments of the loss function in the gradient computation.

Some of the tables are a bit weird and contain numbers which are very hard to compare.  For example, Table 7 contains the proposed method as well as other methods trained against other stronger attacks.  In many cases, the competitors outperform the proposed method.

clarity,_quality,_novelty_and_reproducibility: The writing is generally clear, although like I indicated above, the notation in Algorithm 1 is hard to understand.
summary_of_the_review: This area of work is valuable, but the proposed method seems only applicable to MAML even though the baseline methods work with better quality few-shot learning algorithms.  This might be a major performance limitation of this paper.  In order to convince me that it is not a limitation, I would suggest either adapting the method to a better few-shot learner than MAML or demonstrate that it can outcompete competitors even when they use very strong meta-learning algorithms.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper develops a new algorithm for adversarially robust few-shot learning with bilevel attacks, which allows the model to learn generalizable robust representations across tasks and domains. 

Initially, this paper received scores of 3566. After rebuttal, the scores have increased to 5666. The AC therefore held an online discussion meeting, where all of the 4 reviewers have joined the discussion. 

On one hand, the reviewers agree that this paper is well written, and the presentation is clear. The results also look reasonably good. On the other hand, after the virtual discussion meeting, several concerns still remain. (1) The proposed method is not novel enough. Though the method contains some new design, it mostly combines different pieces of existing techniques together. (2) Results are not impressive enough. One reviewer mentioned that the results are strong, and the improvements are significant. On the other hand, other reviewers commented that the authors should probably push the experiments to a larger scale to make results more convincing. 

Overall, this paper is a boderline case, and during the virtual discussion meeting, the reviewers had a thorough discussion on the pros and cons of the paper. On balance, the cons slightly outweigh the pros, and by the end, the AC decides to recommend rejection of the paper, and encouages the authors to further refine the paper and submit to a future conference. 
justification_for_why_not_higher_score: This paper is a borderline case. During the virtual discussion meeting, the reviewers have commented that the novelty of the proposed method is not enough, writing can be further improved, and results can be made more convincing via performing experiments at a larger scale. By the end, the reviewers reached the agreement to reject the paper at its current form.
Decision: Reject