Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: Summary
========
This paper introduces a mechanism for gradient-based meta-learning models for few-shot classification to be able to adapt to diverse tasks that are imbalanced and heterogeneous. In particular, each encountered task may have varying numbers of shots (task imbalance) and even within each task, different classes may have different numbers of shots (class imbalance). Further, test tasks might come from a different distribution than the training tasks. They propose to handle this scenario by introducing three new types of variables which control different facets of the degree and type of task adaptation, allowing to decide how much to reuse meta-learned knowledge versus new knowledge acquired from the training set of the given task.

Specifically, their newly introduced variables are: 1) the factor for learning rate decay (a scalar) for the inner-loop task adaptation optimization which allows to not deviate too much from the global initialization when insufficient data is available, 2) a class-specific learning rate (one scalar per class) that allows to tune more for under-represented classes of the training set, 3) a set of weights on the global initialization (one scalar per dimension) that can down-weigh each component if it’s not useful for the task at hand (e.g. if test tasks have significantly different statistics than training tasks did). 

The values of these variables are predicted based on the training set of the task: the support set is encoded via a hierarchical variant of a set encoding (where pooling is done using higher order statistics too instead of simply averaging). The resulting encoded support set is the input to the network that produces the values for the three sets of variables discussed above. Each new variable is treated in a Bayesian fashion: a prior is defined over it (Normal(0,1)), which is updated by conditioning on the training set to form a posterior for each given task. Specifically, each of the above variables is represented by a Gaussian whose mean and variance are the learnable parameters that are produced by the network described above. 

Experimentally, this method outperforms others on a setting of imbalanced tasks (the shot is sampled uniformly at random from a designated range). The gain over other methods is large in particular when evaluated on out-of-distribution tasks (coming from a different dataset) and when the imbalance is large.

Comments (in decreasing order of importance)
========================================
A) The Bayesian framework helps because it offers an elegant way to use a prior. In the deterministic version, was any effort made to resemble the effect of that prior? For example, one can define a regularizer that penalizes behaviors that ignore the meta-knowledge too much (e.g. too large values for \gamma, or for the class-specific learning rates etc). Albeit more ‘hacky’, if these regularization coefficients are tuned properly, they might result in a similar effect to that of having a prior. A fair comparison to the deterministic variant should include this.

B) I think that \gamma and z can be merged into a single set of parameters? In particular, imagine a per-dimension-of-\theta learning rate. This would then be large for a dimension when there is a larger need for adapting that dimension of \theta. In the case of large training sets, this can be large for all dimensions, recovering the behavior of a large \gamma. For the case of diverse datasets, this would behave as the current z (updates a lot the dimensions of \theta that are irrelevant for the given task due to the dataset shift).

C) Meta-Dataset (https://arxiv.org/abs/1903.03096) is a recent benchmark for few-shot classification that introduces both of what is referred to here as task imbalance and class imbalance and also is comprised of heterogeneous datasets and evaluates performance on some held-out datasets too. The current state-of-the-art on it (as far as I know) is CNAPs [1] which employs a flexible adaptation mechanism on a per-task basis but is fully amortized (performs no gradient-based adaptation to each task) and makes no explicit effort to tackle imbalanced tasks as is done here. I’m curious how this method would compete in that setup. It definitely seems to be a strong candidate for that benchmark!

Less important
=============
D) which dataset is used in Tables 4 and 5? I assume it’s Omniglot (due to the numbers being in the 90s) but it would be good to say this explicitly.
E) In section 5.2, expressions such as x5 and x15 are used to characterize the degree of imbalance of a task. How exactly are these computed? Does x5 mean that the largest shot is 5 times larger than the smallest shot? It would be good to explicitly state this.
F) In the Related Work section, in the Meta-learning paragraph there is a sentence that’s not accurate: “Metric-based approaches learn a shared metric space [...] such that the instances are closer to their correct prototypes than to others”. This sentence does not describe all metric based approaches. It describes Prototypical Networks (Snell et al) but not, for example, Matching Networks (Vinyals et al) nor many others that like Matching Networks perform example-based comparisons and don’t aggregate a class’ examples into a prototype.

In a nutshell
===========
I think this work is a useful contribution for moving towards a more realistic setting in few-shot classification. It captures some desiderata of models that can operate in more realistic settings and outperforms previous models in those scenarios. My comments above are mostly suggesting improvements and clarifications but I am inclined to recommend acceptance of this paper.

References
=========
[1] Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes. Requeima et al. NeurIPS 2019.

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The paper proposes a Bayesian approach for meta learning in settings were the tasks might be OOD or have imbalanced class distribution. The proposed approach has 3 task-specific balancing variables with a prior and an inference network. Using an amortized inference scheme, the model unifies the meta-learning objective loss with the lower bound of probabilistic model marginal likelihood. 

The paper is well-written and well-motivated. I only have some minor comments and questions:
- Can you add the standard errors for the results in Section 5.2 (maybe at least in the Appendix)? 
- Specifically it would be interesting to see if the results for analyzing the class imbalance variable are statistically significant; specially in light of the recent work on the effects of importance weighting in DL (see “What is the Effect of Importance Weighting in Deep Learning?” by Byrd and Lipton) which essentially question the value of importance weighting for handling class imbalance in various DL settings. 
- For the experiments in Section 5.1 can you also report the values for the three task-specific balancing variables? (Maybe in the Appendix).

Minor:
“obtains significantly improves over” -> "significantly improves over"

Overall, I found the paper interesting and practically useful, although I believe some additions to the empirical evaluation can improve the impact of the paper. 
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: Summary
-------------
This paper proposed to improve existing meta learning algorithms in the presence of task imbalance, class imbalance, and out-of-distribution tasks. Starting from the model-agnostic meta-learning (MAML) algorithm (Finn et al. 2017), to tackle task imbalance, where the number of training examples of varies across different tasks, a task-dependent learning rate decaying factor was learned to be large for large tasks and small for small tasks. In this way, the small task can benefit more from the meta-knowledge and the large task can benefit more from task-specific training. To tackle class imbalance, a class-specific scaling factor was applied to the class-specific gradient. The scaling factor was large for small class and small for large class so that different classes can be treated equally. To tackle the out-of-distribution tasks, a task-dependent variables was learned to emphasize meta-knowledge for the test task similar to training tasks. Additional model parameters are learned through variational inference. Experimental results on benchmark datasets demonstrate the proposed approach outperformed its competing alternatives. Analysis of each component confirm they work as expected.

Comments
---------------
This paper is well motivated and clearly written. The empirical evaluation also support major claims in the paper.

Can the author provide more details on the inference of the model? In the likelihood term in Eq. (7), the task specific parameters \theta^{\tau} was parameterized by Eq. (3), which contains K iterative gradient updates. How was the gradient w.r.t. \theta was computed in this setting?

The task-specific learning rate decaying factor was constrained to be between 0 and 1 using the function f(). The class-specific scaling factor made use of the SoftPlus() function, for the same purpose of scaling learning rate, why do these two different options of functions were applied?

For the scaling vector of the initial parameters g(z^{\tau}), for its zero entries, the initialization of the corresponding entries in task-specific parameter \theta would be zero. Would it be better to apply a linear interpolation between \theta and a randomly-initialized vector in Eq (2)?

Edits after reading the author's rebuttal
==================================
The author's reply well addressed my questions. After reading other reviewers' positive comments and the author's thorough reply, I decide to increase my rating to 8: Accept. 
review_title: Official Blind Review #2

Meta-review: The reviewers generally agreed that the paper presents a compelling method that addresses an important problem. This paper should clearly be accepted, and I would suggest for it to be considered for an oral presentation.

I would encourage the authors to take into account the reviewers' suggestions (many of which were already addressed in the rebuttal period) and my own suggestion.

The main suggestion I would have in regard to improving the paper is to position it a bit more carefully in regard to prior work on Bayesian meta-learning. This is an active research field, with quite a number of papers. There are two that are especially close to the VI method that the authors are proposing: Gordon et al. and Finn et al. (2018). For example, the graphical model in Figure 2 looks nearly identical to the ones presented in these two prior papers, as does the variational inference procedure. There is nothing wrong with that, but it would be appropriate for the authors to discuss this prior work a bit more diligently -- currently the relationship to these prior works is not at all apparent from their discussion in the related work section. A more appropriate way to present this would be to begin Section 3.2 by stating that this framework follows prior work -- there is nothing wrong with building on prior work, and the significant and important contribution of this paper is no way diminished by being up-front about which parts are inspired by previous papers.
Decision: Accept (Talk)