Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: A new task is suggested, similarly to FSL the test is done in an episodic manner of k-shot 5-way, but the number of samples for base classes is also limited. The model is potentially pre-trained on a large scale dataset from another domain. The suggested method is applying spatial attention according to entropy criteria (or certainty) of the original classifier (from a different domain).


I think the suggested task is important and more realistic than the usual FSL benchmarks. I would modify it so instead of discarding mini-imagenet classes that are overlapping with Places I would discard the problematic Places classes. This way it will be easier to compare to standard FSL. Also, I don’t understand why for CUB the benchmarks includes k={0,1,5} while for mini-imagenet it is k={0,20,50}, obviously k={0,1,5} are more interesting.

As for the suggested method, I find it hard to judge since there are no strong baselines to compare against. Also, the ablation study of removing the attention and/or adaptation doesn’t result in a definitive conclusion. 


Update:
While your comments do weaken some of my concerns, I'm afraid it is not enough for changing my previous rating. I think being more careful about the benchmark definition with regards to train/test overlap and comparing to stronger baselines will help improve the paper for future submissions.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposed a new realistic setting for few-shot learning that we can obtain representations from a pre-trained model trained on a large-scale dataset, but cannot access its training details. Also, there may be a large domain shift between the dataset of the pre-trained model and our dataset. For the pre-trained model, they will not only use its weights but also use it to generate a spatial attention map and help the model focuses on objects of images. Back to the standard few-shot classification problem, they will first adapt the model with base class samples and then adapt to novel classes.

The proposed new setting is very meaningful since we already have many powerful pre-trained models and why not exploit its usage for few-shot learning problems. However, I doubt the novelty and effectiveness of the attention way used in the paper. The attention module helps the model focuses on the objects not the background, which is absolutely correct. But there are already some relevant studies in the missing reference Large-Scale Long-Tailed Recognition in an Open World, CVPR2019. Also, from the results, the significant improvements come from the weights of the pre-trained model but not the attention used. Is the attention way used in the paper a good way to exploit the pre-trained model for few-shot classification problems?

Also, I am curious about the dense classification used in the adaptation phase. Will it achieve similar performance with finetuning using just standard loss?

Btw, according to the formatting instructions, the abstract should be limited in one paragraph.

=========================================================
After Rebuttal:

I thank the author for the response.

I do see there are differences in the way of generating attention masks between the proposed work and (Liu et al.). But the improvements from the attention module is not significant, especially when using all base data.

I keep my original scores.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The paper introduces a problem “few-shot few-shot learning” that aims to firstly transfer prior knowledge from one domain to the domain where the base training tasks reside, and then train a few-shot learning model on training tasks and apply it to novel test tasks. The two “few-shot” in the name refers to base training tasks and novel test tasks. In their algorithm, they use a model pre-trained on another dataset as the prior knowledge and fine-tune it on training tasks. During the test, they use the weighted average of samples’ representations per class as the prototype of each class, where the weight is large for samples with more discriminative prediction over pre-trained domain’s classes. Afterward, classification is reduced to finding the nearest neighbor among the class prototypes. Some experiments show that the pre-trained model can improve few-shot classification accuracy.

My major concerns:

1) They try to propose a new problem, but their description shows that the problem is exactly the same as what most “few-shot learning” works aim to solve: use a pre-trained model, train a meta-learner on few-shot training tasks, and apply it to novel test tasks. 

2) The algorithm does not have any important contributions comparing to existing ones: they define a prototype per class based on the pre-trained model and apply the nearest neighbor classification. The so-called “prototypical classifier” is actually the nearest neighbor classifier since no prototypical network structure is learned in the proposed method.

3) I would not call the weighted average as “attention” because it is not: the weight in attention is computed by a module with learnable parameters, while the weight in this paper is computed by the entropy of a pre-defined model’s output prediction. 

4) The “spatial attention” only makes sense when the pre-trained domain’s classes can describe the main concepts appearing in the images of novel classes. This assumption is too strong since it requires class-level (rather than lower-level) relationships.

5) The base training is not necessary in the algorithm: it is used to only fine-tuning theta and W. As the author said in the beginning of Section 4.1, they can directly solve novel tasks based on the pre-trained model.

6) The experiments show that the pre-trained model is helpful in few-shot learning, which is a known fact.

7) The writing of this paper is very poor: a lot of typos and grammar errors, inconsistency between narratives, abuse of notations, wrong equation reference, even missing punctuations. They make the paper hard to understand.

-------------

Update:

Thanks for the authors' rebuttal! After reading their rebuttal, I still have main concerns about the novelty of the problem and the writing quality. The proposed method tends to be incremental. 
review_title: Official Blind Review #1

Meta-review: This paper tackles the interesting problem of meta-learning in problem spaces where training "tasks" are scarce.  Two criticisms that seems to shared across reviewers are that (i) it is debatable how "novel" the space of meta learning with "few" tasks is, especially since there aren't established standard for how many training tasks should be available, and (ii) the paper could use more comparisons with baseline methods and ablations to understand the contributions.  As an AC, I down-weight criticism (i) because I don't feel the paper has to be creating a new problem definition; it's acceptable to make advances within an existing space.  However, criticism (ii) seems to remain.  After conferring with reviewers it seems that the rebuttal was not strong enough to significantly alter the reviewer's opinions on this issue, and so the paper does not have enough support to justify acceptance.  The paper certainly addresses interesting issues, and I look forward to seeing a revised/improved version at another venue. 
Decision: Reject