Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Some questions:
- It looks like the model visualized in Fig. 2 has collapsed. All the samples from the same category are nearly identical which would suggest training has failed and this is not a reasonable model to compare. Could you comment on this?
- You haven't compared this method against any other unsupervised techniques, including other unsupervised techniques involving GANs. Could you comment on why these results have been omitted? 
- The novelty of this work is not particularly clear to me. It seems that you are demonstrating that GAN discriminator features are suitable for classification. This however has already been established in prior work (for example the DCGAN paper of Radford et al., among others). I don't see this work as extending or pushing the existing approaches further and so am confused: what is the main contribution of this work?
review_title: pre-review questions

final_score: rating: 3: Clear rejection
review_content: The paper proposes an approach to unsupervised learning based on generative adversarial networks (GANs) and clustering. The general topic of unsupervised learning is important, and the proposed approach makes some sense, but experimental evaluation is very weak and does not allow to judge if the proposed method is competitive with existing alternatives. Therefore the paper cannot be published in its current form. 

More detailed remarks (many of these are copies of my pre-review questions the authors have not responded to):

1) Realted work overview looks incomplete. There has been work on combining clustering with deep learning, for example [1] or [2] look very related. A long list of potentially related papers can be found here: https://amundtveit.com/2016/12/02/deep-learning-for-clustering/ . From the GAN side, for example [3] looks related. I would like the authors to comment on relation of their approach to existing work, if possible compare with existing approaches, and if not possible - explain why.

[1] Xie et al., "Unsupervised Deep Embedding for Clustering Analysis", ICML 2016 http://jmlr.org/proceedings/papers/v48/xieb16.pdf
[2] Yang et al., "Joint Unsupervised Learning of Deep Representations and Image Clusters", CVPR 2016 http://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Yang_Joint_Unsupervised_Learning_CVPR_2016_paper.pdf
[3] J.T. Springenberg, "Unsupervised and semi-supervised learning with categorical generative adversarial networks", ICLR 2016, https://arxiv.org/pdf/1511.06390v2.pdf

2) The authors do not report classification accuracies, which makes it very difficult to compare their results with existing work. Classification accuracies should be reported. They may not be a perfect measure of feature quality, but reporting them in addition to ARI and NMI would not hurt.

3) The authors have not compared their approach to existing unsupervised feature learning approaches, for example feature learning with k-means (Coates and Ng 2011), sparse coding methods such as Hierarchical Matching Pursuit (Bo et al., 2012 and 2013), Exemplar-CNN (Dosovitskiy et al. 2014)

4) Looks like in Figure 2 every "class" consists essentially of a single image and its slight variations? Doesn't this mean GAN training failed? Do all your GANs produce samples of this quality? 

5) Why do you not show results with visual features on STL-10?

6) Supervisedly learned filters in Figure 3 looks unusual to me, they are normally not that smooth. Have you optimized the hyperparameters? What is the resulting accuracy?

review_title: Review

final_score: rating: null
review_content: Hello,

A couple of questions:

1. Looks like in Figure 2 every "class" consists essentially of a single image and its slight variations? Doesn't this mean GAN training failed? Do all your GANs produce samples of this quality? 

2. Could you please report classification accuracies? Otherwise it is impossible to compare your method to most other works.

3. Why have you not compared to existing well-performing unsupervised learning methods? For example, feature learning with k-means (Coates and Ng 2011), sparse coding methods such as Hierarchical Matching Pursuit (Bo et al., 2012 and 2013), Exemplar-CNN (Dosovitskiy et al. 2014) .

4. Why do you not show results with visual features on STL-10?

5. Supervisedly learned filters in Figure 3 looks unusual to me, they are not normally that smooth. Have you optimized the hyperparameters? What is the resulting accuracy?

6. Could you relate your method to Categorical GAN https://arxiv.org/pdf/1511.06390.pdf ?

Thanks!
review_title: pre-review questions

final_score: rating: null
review_content: What is the key novelty of this paper given the InfoGAN paper? In section 3.2, the paper seems to propose a novel “hybrid” approach. However, there seems not much technically novel based on the descriptions in the paper.
 
The number of semantic categories does not necessarily equal to “k”. In this sense, there is no guarantee that the performance is optimal when we set the number of semantic categories equal to “k”. Any comments on this?

In Figure 2, the samples in each row look quite similar. It’s questionable if the model is well-trained.

review_title: questions

final_score: rating: 3: Clear rejection
review_content: The papers investigates the task of unsupervised learning with deep features via k-means clustering. The entire pipeline can be decomposed into two steps: (1) unsupervised feature learning based on GAN framework and (2) k-means clustering using learned deep network features. Following the GAN framework and its extension InfoGAN, the first step is to train a pair of discriminator network and generator network from scratch using min-max objective. Then, it applies k-means clustering on the top layer features from discriminator network. For evaluation, the proposed unsupervised feature learning approach is compared against traditional hand-crafted features such as HOG and supervised method on three benchmark datasets. Normalized Mutual Information (NMI) and Adjusted RAND Index (ARI) have been used as the evaluation metrics for experimental comparison. Although the proposed method may be potentially useful in practice (if refined further), I find the method lacks novelty, and the experimental results are not significant enough.

review_title: review

final_score: rating: 3: Clear rejection
review_content: This paper proposed an unsupervised learning method based on running kmeans on the features learned by a discriminator network in a generative adversarial network setup. Unsupervised learning methods with GANs is certainly a relevant topic but this paper does not propose anything particularly novel as far as I can tell. More importantly, the evaluation methods in this paper are extremely lacking. The authors omit classification results on CIFAR and STL-10 and instead the only quantitative evaluation plot the performance of the clustering algorithm on the features. Not only are classification results not shown, no comparisons are made to the wealth of related work. I list just a few highly related techniques below. Finally, it appear the authors have not train their GANs correctly as the samples in Fig.2 appear to be from a model that has collapsed during training. In summary, the ideas in this paper are potentially interesting but this paper should not be accepted in its current form due to lack of experimental results and comparisons. 

(non-exhaustive) list of related work on unsupervised learning (with and without GANs):
[1] Springenberg. Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks, ICLR 2016 (https://arxiv.org/abs/1511.06390)
[2] Salimans et al. Improved Techniques for Training GANs. NIPS 2016 (https://arxiv.org/abs/1606.03498)
[3] Dosovitskiy et al. Discriminative unsupervised feature learning with convolutional neural networks, NIPS 2014 (https://arxiv.org/abs/1406.6909)

review_title: review

Meta-review: This paper was reviewed by three experts. While they find interesting ideas in the manuscript, all three point to deficiencies (lack of significant novelty, potential problems with GAN training) and unanimously recommend rejection. I do not see a reason to overturn their recommendation.
Decision: Reject