Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors have investigated the unsupervised few shot learning problem. They have proposed a method to learn an unsupervised few-shot learner via self-supervised training (UFLST), which consists of two alternate processes, progressive clustering and episodic training. Experimental results on several benchmark data sets validate the effectiveness. Overall, I think it is an interesting scenario and have the following comments.
(1) The first one the about the method. The authors have tried to describe their methods in a concrete way. Nevertheless, it is better to give some justifications from the theoretical aspect. It is vital for readers to convince the superiority.
(2) The second one is about the description. I suggest the authors to express as accurately as possible. For example, in the abstract, the authors state that ‘However, current few-shot learners are mostly supervised and rely heavily on a large amount of labeled examples.’ I do not agree with this statement since in few shot learning, there are only limited exemplars. There is a conflict between them and it will confuse the readers.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper aims to conduct few-shot learning on unlabeled data (instead of on training tasks with few-shot labeled data per task). The proposed algorithm is a trivial combination of existing clustering method and a few-shot learning method, i.e., the clustering provides pseudo labels, from which a series of few-shot training tasks are generated, and then traditional few-shot learning method can be applied afterward. Many existing techniques are integrated, e.g., k-reciprocal Jaccard distance, DBSCAN clustering, prototypical network, triplet loss, etc. The paper reported the experimental results on Omniglot and Market1501.

The paper suffers from several drawbacks: 1) lack of novelty and originality; 2) problem setting is not convincing enough; 3) only a comparison on a very easy and small dataset (Omniglot) is shown, while the comparison on Market1501 is missing; 4) lack of comparison and discussion of other related works.

Detailed comments:

1. A simple combination of two existing techniques (clustering followed by few-shot learning) without any in-depth analysis cannot be justified as a reasonable contribution for an ICLR paper.

2. The unsupervised few-shot learning setting does not make much sense in practice: we can usually collect many labeled data to generate few-shot learning tasks since these labeled data is not required to be drawn from the same distribution as the target few-shot task (e.g., the test tasks). For example, for few-shot image classification, we can always generate training tasks from ImageNet data. In practice, we only suffer from few-shot labeled data on test tasks. Assuming that none of labeled data is available can lead to unnecessary degrade on few-shot learning performance. The authors need to present more evidence or practical application scenarios to support the practical value of this problem setting.

3. Omniglot is too easy and too small for most of the recent few-shot learning methods. More challenging datasets such as ImageNet or at least its easier subsets (e.g., miniImageNet, tiredImageNet) should be considered.

4. No comparison to any baseline is shown on the other dataset Market1501.

5. More related works need to be compared and discussed: 

-Constructing learning tasks via clustering has been studies in “Hsu, K., Levine, S. and Finn, C., 2018. Unsupervised learning via meta-learning. arXiv preprint arXiv:1810.02334.” This paper shares the same idea when constructing the unsupervised episode training tasks. 

-Self-supervised techniques to boost the performance of few-shot learning has also been studies in “Su, J.C., Maji, S. and Hariharan, B., 2019. Boosting Supervision with Self-Supervision for Few-shot Learning. arXiv preprint arXiv:1906.07079.” and “Gidaris, S., Bursuc, A., Komodakis, N., Pérez, P. and Cord, M., 2019. Boosting Few-Shot Visual Learning with Self-Supervision. arXiv preprint arXiv:1906.05186.” This paper uses a self-supervised method called “pseudo labeling”, but other self-supervised methods should also be compared.

----------------------

Update after rebuttal:

Thanks for the new experiments on miniImageNet! However, I still have concerns about the novelty of the proposed idea (clustering+few-shot learning) and its similarity to previous works. In addition, the new results on miniImageNet is not very encouraging (it is necessary to provide a fair comparison). Hence, I will keep my rating unchanged. 

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper considers the problem of learning an image representation for few-shot learning without using image labels during training. This is a well-motivated problem since (as the paper points out) learning such a representation using "episodes" of low-shot learning problems as examples may require a large amount of annotated data. The paper proposes an iterative algorithm which alternates between clustering the images using the current model and updating the model using the clusters as "pseudo-labels". This approach is not particularly elegant as there is no clear objective being optimized, but it may nevertheless be effective. One main claim of the paper is that the iterative nature of this process is key to the success of the algorithm. The choice of model is a multi-layer conv-net which is trained using SGD (Adam). The paper investigates both triplet (with and without hard negatives) and "prototype" losses for learning the model parameters. To find clusters, the paper adopts the DBSCAN algorithm using the Jaccard similarity of the k-reciprocal neighbour sets.

I am not aware of other papers that use self-supervised learning to obtain a representation which is specifically suitable for few-shot learning via nearest-neighbour classification. As is noted in the paper, the absence of supervision during training more closely resembles the scenario of few-shot learning in biological systems.

The design decisions are well motivated throughout the paper.

The appendices are high quality and make the paper much more complete. In particular: the method for choosing epsilon, the empirical study of the effect of epsilon and the discussion of the behaviour of the cluster sizes as training proceeds.

Principal concerns:

(1.1) The proposed approach is particularly similar to DeepCluster (Caron et al.). Besides the use of a different clustering algorithm, it seems that the main high-level difference is the use of "episodic training", in which the algorithm is trained to compare examples to a query example, rather than to classify single examples. I would have preferred to see a comparison to non-episodic training. (It might be necessary to train a linear classifier on top of the final feature representation rather than simply build a nearest-neighbour classifier, but still this is convex, cheap and even closed-form in the case of least-squares regression.)

(1.2) While the algorithm has been demonstrated on real images in the Market1501 dataset, it would have been much more convincing to see it demonstrated on a more widely-used dataset for few-shot learning such as Mini-ImageNet.

(1.3) There are several recent papers on unsupervised feature learning using self supervision, especially as an alternative to ImageNet-classification pre-training. There is no discussion of these approaches, yet they might perform better than the proposed algorithm, especially for tasks with real images. Some examples of such papers are:
- "Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks"
- "Representation Learning with Contrastive Predictive Coding"
- "Unsupervised representation learning by predicting image rotations"
The literature on using auto-encoders to learn feature representations is also relevant.

Issues with details in the paper:

(2.1) The DeepCluster paper observed that non-negligible accuracy could be achieved using a randomly-initialized conv-net (12% when chance is 0.1%). This enabled the use of clusters as pseudo-labels. Is a similar effect observed with your datasets and random initialization?

(2.2) For large datasets, the clustering algorithm might be prohibitively expensive? It would be useful to discuss the complexity of this algorithm.

(2.3) It would be interesting to see the effect of varying the frequency of the clustering during training (i.e. how many gradient steps are taken before updating the clustering).

(2.4) It is concerning that the accuracy drops sharply as \rho increases [6, 7, 8]*10^-5 in Table 4.

(2.5) It is stated that batch-normalization at the network output helps prevent over-fitting. Why? My intuition is that it would be more helpful for avoiding regions of the loss function which have a small gradient magnitude. What happens if you remove it?

(2.6) What is the test time procedure? Do you use the mean feature when there are k > 1 shots (even for the network trained with the hinge loss)? Do you L2-normalize the representation vectors?

(2.7) It seems potentially brittle to use hard negatives in the triplet loss with pseudo-labels? If the labels were wrong, then the hard negatives might not really be negatives. Nevertheless, this does not seem to be an issue, at least with these datasets.

(2.8) There are no error-bars anywhere in the paper.

Minor comments:

(3.1) Tables 3 and 4 would be easier to interpret if plotted on an axis.

(3.2) The word "concurrently" suggests that the clustering and the training are performed simultaneously. I would prefer "alternating".

(3.3) A grammatical review of the paper is required. For example in the abstract: "to a large extent (extend) and approaches the performance of (to the performances of) supervised methods".

(3.4) Remember to use \log and \max in latex to improve the appearance.
review_title: Official Blind Review #2

Meta-review: This paper proposes an approach for unsupervised meta-learning for few-shot learning that iteratively combines clustering and episodic learning. The approach is interesting, and the topic is of interest to the ICLR community. Further, it is nice to see experiments on a more real world setting with the Market1501 dataset.
However, the paper lacks any meaningful comparison to prior works on unsupervised meta-learning. While it is accurate that the architecture used and/or assumptions used in this paper are somewhat different from those in prior works, it's important to find a way to compare to at least one of these prior methods in a meaningful way (e.g. by setting up a controlled comparison by running these prior methods in the experimental set-up considered in this work). Without such as comparison, it's impossible to judge the significance of this work in the context of prior papers.
The paper isn't ready for publication at ICLR.
Decision: Reject