Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a method for few-shot learning using a new image representation called visual concept embedding. Visual concepts were introduced in Wang et al. 2015, which are clustering centers of feature vectors in a lattice of a CNN. For a given image, its visual concept embedding is computed by thresholding the distances between feature vectors in the lattice of the image to the visual concepts. Using the visual concept embedding, two simple methods are used for few-shot learning: a nearest neighbor method and a probabilistic model with Bernoulli distributions. Experiments are conducted on the Mini-ImageNet dataset and the PASCAL3D+ dataset for few-shot learning.

Positives:
- The three properties of visual concepts described in the paper are interesting.

Negatives:
- The novelty of the paper is limited. The idea of visual concept has been proposed in Wang et al. 2015. Using a embedding representation based on visual concepts is straightforward. The two baseline methods for few-shot learning provide limited insights in solving the few-shot learning problem.

- The paper uses a hard thresholding  in the visual concept embedding. It would be interesting to see the performance of other strategies in computing the embedding, such as directly using the distances without thresholding.
review_title: A paper with limited novelty

final_score: rating: 5: Marginally below acceptance threshold
review_content: My main concern for this paper is that the description of the Visual Concepts is completely unclear for me. At some point I thought I did understand it, but then the next equation didnt make sense anymore... If I understand correctly, f_p is a representation of *all images* of a specific layer *k* at/around pixel "p", (According to last line of page 3). That would make sense, given that then the dimensions of the vector f_p is a scalar (activation value) per image for that image, in layer k, around pixel p. Then f_v is one of the centroids (named VCs). However, this doesnt seem to be the case, given that it is impossible to construct VC activations for specific images from this definition. So, it should be something else, but it does not become clear, what this f_p is. This is crucial in order to follow / judge the rest of the paper. Still I give it a try.

Section 4.1 is the second most important section of the paper, where properties of VCs are discussed. It has a few shortcomings. First, iIt is unclear why coverage should be >=0.8 and firerate ~ 1, according to the motivation firerate should equal to coverage: that is each pixel f_p is assigned to a single VC centroid. Second, "VCs tent to occur for a specific class", that seems rather a bold statement from a 6 class, 3 VCs experiment, where the class sensitivity is in the order 40-77%. Also the second experiment, which shows the spatial clustering for the "car wheel" VC, is unclear, how is the name "car wheel" assigned to the VC? That has have to be named after the EM process, given that EM is unsupervised. Finally the cost effectiveness training (3c), how come that the same "car wheel" (as in 3b) is discovered by the EM clustering? Is that coincidence? Or is there some form of supervision involved? 


Minor remarks
- Table 1: the reported results of the Matching Network are different from the results in the paper of Vinyals (2016).
- It is unclear what the influence of the smoothing is, and how the smoothing parameter is estimated / set.
- The VCs are introduced for few-shot classification, unclear how this is different from "previous few-shot methods" (sect 5). 
- 36x36 patches have a plausible size within a 84x84 image, this is rather large, do semantic parts really cover 20% of the image?
- How are the networks trained, with what objective, how validated, which training images? What is the influence of the layer on the performance? 
- Influence of the clustering method on VCs, eg k-means, gaussian, von-mises (the last one is proposed)?

On a personal note, I've difficulties with part of the writing. For example, the introduction is written rather "arrogant" (not completely the right word, sorry for that), with a sentence, like "we have only limited insights into why CNNs are effective" seems overkill for the main research body. The used Visual Concepts (VCs) were already introduced by other works (Wangt'15), and is not a novelty. Also the authors refer to another paper (about using VCs for detection) which is also under submission (somewhere). Finally, the introduction paragraph of Section 5 is rather bold, "resembles the learning process of human beings"? Not so sure that is true, and it is not supported by a reference (or an experiment). 

In conclusion:
This paper presents a method for creating features from a (pre-trained) ConvNet. 
It clusters features from a specific pooling layer, and then creates a binary assignment between per image extracted feature vectors and the cluster centroids. These are used in a 1-NN classifier and a (smoothed) Naive Bayes classifier. The results show promising results, yet lack exploration of the model, at least to draw conclusions like "we address the challenge of understanding the internal visual cues of CNNs". I believe this paper needs to focus on the working of the VCs for few-shot experiments, showing the influences of some of the choices (layer, network layout, smoothing, clustering, etc). Moreover, the introduction should be rewritten, and the the background section of VCs (Sect 3) should be clarified. Therefore, I rate the current manuscript as a reject. 

After rebuttal:
The writing of the paper greatly improved, still missing insights (see comments below). Therefore I've upgraded my rating, and due to better understanding now, als my confidence. 
review_title: Not enough novelty and important details unclear

final_score: rating: 7: Good paper, accept
review_content: The paper adds few operations after the pipeline for obtaining visual concepts from CNN as proposed by Wang et al. (2015). This latter paper showed how to extract from a CNN some clustered representations of the features of the internal layers of the network, working on a large training dataset. The clustered representations are the visual concepts. This paper shows that these representations can be used as exemplars by test images, in the same vein as bag of words used word exemplars to create the bag of words of unseen images.

 A simple nearest neighborhood and a likelihood model is built to assign a picture to an object class.

The results a are convincing, even if they are not state of the art in all the trials. 
The paper is very easy to follows, and the results are explained in a very simple way.


Few comments:
The authors in the abstract should revise their claims, too strong with respect to a literature field which has done many advancements on the cnn interpretation (see all the literature of Andrea Vedaldi) and the literature on zero shot learning, transfer learning, domain adaptation and fine tuning in general.
review_title: Incremental paper, but good results

Meta-review: The paper builds on earlier work by Wang et al (2015) on Visual Concepts (VCs) and explores the use of VCs for few-shot learning setting for novel classes.

The work, as pointed out by two reviewers is somewhat incremental in nature, with main novelty being the demonstration of utilities of VCs for few shot learning. This would not have been a big limitation if the paper had a carefully conducted empirical evaluation providing insights on the effect of various configuration settings/hyperparameters on the performance in few shot learning, which two of the reviewers (Anon3, Anon2) state are missing. The paper falls short of the acceptance threshold in its current form.

PS: The authors posted a github link to the code on Jan 12 which may potentially compromise the anonymity of the submission (though it was after all the reviews were already in) https://openreview.net/forum?id=BJ_QxP1AZ&noteId=BJaIDpBEM

Decision: Reject