Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes meta dropout, which leverages adaptive dropout training for regularizing gradient based meta learning models, e.g., MAML and MetaSGD. Experiments on few shot learning show that meta dropout achieves better performance.

Overally, I think this paper is well motivated and experiments on few shot learning are impressive. I have only two major concerns.

1. Sec 3.2. According to my understanding, Meta dropout introduces a learnable prior for latent $z$, but the training objective does not require posterior inference and thus no variational inference is needed. I think it is ok to say that meta dropout tries to optimize a lower bound of log p(Y|X;\theta,\phi^*), but meta dropout does not regularize the variational framework because there is no variational inference framework.

2. Experiments on adversarial robustness can be further improved. (1) the settings and the analysis of adversarial robustness experiment can be discussed in details. For example, how to build ''adversarial learning baseline'' in meta learning settings and why the result implies the perturbation directions for generalization and robustness relates to each other; (2) how other regularization methods (e.g., Mixup, VIB and Information dropout) perform on adversarial robustness? Does Meta dropout performs better than them? (3) FGSM is a quite weak adversarial attack method, which makes evaluating adversarial robustness on FGSM may be misleading. I suggest trying some other STOA attack methods (e.g., iterative methods).

Some typos: 
Page 3, Regularization methods, 3rd line, ````wwwdiscuss
Page 7, 2nd line from the bottom, FSGM->FGSM

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The authors propose to meta-learn, using MAML, the mean of an elementwise, input-dependent, multiplicative noise to improve generalization in few-shot learning.
The motivation is that meta-learning the noise allows to learn how to best perturb examples in order to improve generlization.  This claim is supported by ample experimental evidence and comparisons against many baselines, as well as additional ablation studies w.r.t design choices of the algorithm itself. The paper is well written and easy to read. Consequently, I think this is a nice paper and should be accepted. 

Edit (leaving everything else unchanged for now): After reading R3's assessment, I agree with them that it's worrying that the Deterministic Meta-Dropout performs better than baseline MAML - maybe it's an effect of a larger number of parameters in the model? 

Edit:
Thank you for your response.

I will leave my score as is.
 I would strongly encourage the authors to incorporate the baseline "(1)" as proposed by R3 in a future version of the paper as I agree with them that this is a relevant baseline.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper proposes learning to add input-dependent noise to improve the generalization of MAML-style meta-learning algorithm. The proposed method is evaluated on OmniGlot and miniImageNet. The paper reports improvements upon MAML, MAML with meta-learned parameter-wise learning rates, as well as a few regularization methods that are based on input/hidden state perturbations (Mixup, Variational Information Bottleneck). An ablation study also compares the proposed meta-dropout algorithm with a number of modifications, such as a fixed noise, input-independent noise, etc. It is furthermore shown that meta-dropout somewhat improves the model’s robustness against an adversarial attack. 

The paper is somewhat incremental considering that Li et al, (2017) and Balaji et al, (2018) have already proposed meta-learning parameter-wise learning rates and parameter-wise regularization coefficient respectively. One difference from the methods above is that in the proposed method noise is controlled by the input. The ablation however shows that in 5-shot classification case simply adding non-trainable noise works quite well. 

It seems like the choice of the particular method for adding the noise was performed using the test set. If it’s true, this is methodologically wrong: model selection should be performed on a development set (or meta-development) set. Futhermore, Table 2 contains some results named “Add.”, which I guess stands for additive noise. I did not find an explanation of what is the specific method for adding noise used in this case. Such additive noise is also missing from ablation experiments. 

Overall, it seems that paper falls short of clearly proving that back-propagating through MAML to the noise parameters is helpful. The “Deterministic Meta-Dropout” performs better than baseline MAML, and arguably, meaning that some part of the improvement upon MAML can be due to the architectural differences and not due to noise. “Independent Gaussian” and “Weight Gaussian” baselines perform worse than non-trainable noise (“Fixed Gaussian”). Learning the variance for the noise is shown to be detrimental. There is just too much confusion in the results, the improvements are not very robust. 

The paper writing is okay, but there are serious issues. I am not sure I understand the argument in Section 3.2 that meta-dropout performs variational inference. It seems like Equation 7 is wrong because  y_i is missing from the second argument of the KL divergence term. The transition to Equation 8 is therefore also wrong, and as far as I can understand, the whole argument breaks down. Line 7 in Algorithm 1 in Appendix A (which by the way should really be in the main text) does not make sense.

Other issues: 
- the second sentence of the abstract is not implied by the first, the usage of “thus” does not seem appropriate
- the intro should probably mention L1 and L2 regularization as well
- in Section 3.1 there is a forward reference to Equation 5, makes understanding the text quite hard
- “meta-droput”, “robustenss”: typos in many places
- Figure 4 visualization is not clear. 
- the architectural change required to add noise is not explained in the paper (i.e. what is \phi and how it’s used) 
- no comparison to meta-learned L1 regularization 
- a baseline is missing in which \phi is treated as a part of \theta and trained with vanilla MAML

review_title: Official Blind Review #3

Meta-review: This paper proposes a type of adaptive dropout to regularize gradient based meta-learning models. The reviewers found the idea interesting and it is supported by improvements on standard benchmarks. The authors addressed several concerns of the reviewers during the rebutal phase. In particular, revisions added results against other regularization mthods. We recommend that further attention is given to ablations, in particular the baseline proposed by Reviewer 1.
Decision: Accept (Poster)