Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The authors argue that existing metric learning approaches, for few-shot learning, may cause overfitting to the feature distributions encoded only from the seen domains and thus fail to generalize to unseen domains. This is a domain generaliation task under a few-shot learning setting. The authors proposed a so-called feature-wise transformation layer and integrate it into some existing metric-based few-shot learning methods, which helps learn more diverse feature distributions for better generalization ability.

I think this is a pioneer work on few-shot learning for domain generalization, which is quite interesting.

In terms of experiments, good performance has been shown by using three existing metric-based few-shot learning methods on a number of settings using five image datasets.

Below list my major concerns:
1. The authors mentioned that it is intuitive to have feature-wise transformation layers produce more diverse feature distributions, which would lead to better generaliztion ability for domain generalization. However, it is not obvious to me. Why the generalization ability would be improved with more diverse features?

2. In the experiments, feature-wise transformation layers with hyper-parameters empirically (\theta_\gamma, \theta_\beta) set as (0.3, 0.5) are compared against learning-to-learn version of the proposed method. Why (0.3, 0.5) was chosen? Is there any chance that another set of (\theta_\gamma, \theta_\beta) may lead to better performance than the learning-to-learn version? It would be much clearer if the authors can provide detailed parameter analysis (e.g. grid search).

3. Algorithm 1 should be quite expensive in time. It is not clear how the algorithm will be terminated. And in general, how many iterations in the algorithm will be until it achieves some convergence?
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors propose a feature-wise transformation layer for the cross-domain few-shot classification task. Besides, they apply the learning-to-learn procedure to tune the hyperparameters automatically. The primary motivation behind this method does make a lot of sense for me. The reduction of significant shifts in the feature norm could be crucial for successfully transferring the source domain model to the target domain. Also, such a method is method-agnostic, which makes the proposed layer more accessible for all kinds of metric-based query&support pipeline. The extensive experiments for three baselines, MatchingNet, RelationNet, and GNN, indicate the generalizable effectiveness of the proposed layer. Specifically, the visualization and analysis of learned feature space and layer parameters look interesting to me. Remember that the proposed method is designed to minimize the large discrepancy of the feature distribution. Such a study could be beneficial for the reader to understand this paper in depth.

Meanwhile, I have some questions and suggestions for the authors:
1. I am suggesting the authors include more recent state-of-the-art as baselines and comparisons, which could make such submission much stronger.
2. This paper delivers an extensive study of the classification problem, how about the other tasks, which heavily rely on classification head, like detection or segmentation. I think it could be more attractive if the author could also show some improvement for these tasks in either a qualitative or quantitive way.
3. This paper mainly focuses on metric-based few-shot frameworks. At the same time, there are also other two groups, recurrent-based and optimization-based schemes. It could be much better if the authors could also mention the potential of the proposed layer.
4. This paper leverages the learning-to-learn mechanism to tune the hyper-parameters. How about considering the adaptive or dynamic approach. In other words, build the connection between the theta parameter and image feature.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a feature-wise transformation layer to augment the image features, which is a new regularization of neural networks and leads to better generalization ability of the features. And the proposed method performs well in the few-shot classification problem. Furthermore, the paper develops a learning-to-learn model in the cross-domain setting to choose optimal hyper-parameters of the feature-wise transformation layers. Which leads to consistent improvement in the cross-domain leave-one-out setting.

Although I vote weak accept for this paper, I still have several concerns:
* In equation $(7)$, how to calculate the gradient of $L^{pu}$ w.r.t $\theta_f^t$ in the condition that the $L^{pu}$ is calculated after removing the feature-wise transformation layers from the model? In my understanding, The $L^{pu}$ is not related to $\theta_f^t$ after the removal of the feature-wise transformation layers.
* In Section 4.2, why not choose the Prototypical networks? According to the results in (Chen et al., 2019a), it performs much better in the mini-ImageNet to CUB setting.
* In Section 4.3, what's the initial value of $\theta_{\gamma}$ and $\theta_{\beta}$? Is the initial value sensitive to the performance?
* In Section 4.2, actually, you can learn the $\theta_{\gamma}$ and $\theta_{\beta}$ automatically even in a single domain. For example, use a different batch to update $\theta_{\gamma}$ and $\theta_{\beta}$. What's the performance under this setting?
* There is no direct comparison to the state-of-the-art methods.

Overall, the paper is well written and the figures are well illustrated. The experiments show the effectiveness of the proposed feature-wise transformation layers and the learning-to-learning approach. But the above concerns should be addressed.
review_title: Official Blind Review #2

Meta-review: This submission addresses the problem of few-shot classification. The proposed solution centers around metric-based models with a core argument that prior work may lead to learned embeddings which are overfit to the few labeled examples available during learning. Thus, when measuring cross-domain performance, the specialization of the original classifier to the initial domain will be apparent through degraded test time (new domain) performance. The authors therefore, study the problem of domain generalization in the few-shot learning scenario. The main algorithmic contribution is the introduction of a feature-wise transformation layer. 

All reviewers suggest to accept this paper. Reviewer 3 says this problem statement is especially novel. Reviewer 1 and 2 had concerns over lack of comparisons with recent state-of-the-art methods. The authors responded with some additional results during the rebuttal phase, which should be included in the final draft. 

Overall the AC recommends acceptance, based on the positive comments and the fact that this paper addresses a sufficiently new problem statement.

Decision: Accept (Spotlight)