Title: Unleashing the Potential of CNNs for Interpretable Few-Shot Learning
Authors: Boyang Deng
Qing Liu
Siyuan Qiao
Alan Yuille
Venue: ICLR 2018
Keywords: Few-Shot Learning
Neural Network Understanding
Visual Concepts

Abstract: Convolutional neural networks (CNNs) have been generally acknowledged as one of the driving forces for the advancement of computer vision. Despite their promising performances on many tasks, CNNs still face major obstacles on the road to achieving ideal machine intelligence. One is that CNNs are complex and hard to interpret. Another is that standard CNNs require large amounts of annotated data, which is sometimes very hard to obtain, and it is desirable to be able to learn them from few examples. In this work, we address these limitations of CNNs by developing novel, simple, and interpretable models for few-shot learn- ing. Our models are based on the idea of encoding objects in terms of visual concepts, which are interpretable visual cues represented by the feature vectors within CNNs. We first adapt the learning of visual concepts to the few-shot setting, and then uncover two key properties of feature encoding using visual concepts, which we call category sensitivity and spatial pattern. Motivated by these properties, we present two intuitive models for the problem of few-shot learning. Experiments show that our models achieve competitive performances, while being much more flexible and interpretable than alternative state-of-the-art few-shot learning methods. We conclude that using visual concepts helps expose the natural capability of CNNs for few-shot learning.

TL;DR: We enable ordinary CNNs for few-shot learning by exploiting visual concepts which are interpretable visual cues learnt within CNNs.

Review discussion:
My main concern for this paper is that the description of the Visual Concepts is completely unclear for me. At some point I thought I did understand it, but then the next equation didnt make sense anymore... If I understand correctly, f_p is a representation of *all images* of a specific layer *k* at/around pixel "p", (According to last line of page 3). That would make sense, given that then the dimensions of the vector f_p is a scalar (activation value) per image for that image, in layer k, around pixel p.
Then f_v is one of the centroids (named VCs).
However, this doesnt seem to be the case, given that it is impossible to construct VC activations for specific images from this definition.
So, it should be something else, but it does not become clear, what this f_p is. This is crucial in order to follow / judge the rest of the paper.
Still I give it a try.
Section 4.1 is the second most important section of the paper, where properties of VCs are discussed. It has a few shortcomings.
First, iIt is unclear why coverage should be >=0.8 and firerate ~ 1, according to the motivation firerate should equal to coverage: that is each pixel f_p is assigned to a single VC centroid.
Second, "VCs tent to occur for a specific class", that seems rather a bold statement from a 6 class, 3 VCs experiment, where the class sensitivity is in the order 40-77%.
Also the second experiment, which shows the spatial clustering for the "car wheel" VC, is unclear, how is the name "car wheel" assigned to the VC? That has have to be named after the EM process, given that EM is unsupervised.
Finally the cost effectiveness training (3c), how come that the same "car wheel" (as in 3b) is discovered by the EM clustering? Is that coincidence? Or is there some form of supervision involved?
Minor remarks
- Table 1: the reported results of the Matching Network are different from the results in the paper of Vinyals (2016).
- It is unclear what the influence of the smoothing is, and how the smoothing parameter is estimated / set.
- The VCs are introduced for few-shot classification, unclear how this is different from "previous few-shot methods" (sect 5).
- 36x36 patches have a plausible size within a 84x84 image, this is rather large, do semantic parts really cover 20% of the image?
- How are the networks trained, with what objective, how validated, which training images? What is the influence of the layer on the performance?
- Influence of the clustering method on VCs, eg k-means, gaussian, von-mises (the last one is proposed)?
On a personal note, I've difficulties with part of the writing.
For example, the introduction is written rather "arrogant" (not completely the right word, sorry for that), with a sentence, like "we have only limited insights into why CNNs are effective" seems overkill for the main research body. The used Visual Concepts (VCs) were already introduced by other works (Wangt'15), and is not a novelty.
Also the authors refer to another paper (about using VCs for detection) which is also under submission (somewhere).
Finally, the introduction paragraph of Section 5 is rather bold, "resembles the learning process of human beings"? Not so sure that is true, and it is not supported by a reference (or an experiment).
In conclusion:
This paper presents a method for creating features from a (pre-trained) ConvNet.
It clusters features from a specific pooling layer, and then creates a binary assignment between per image extracted feature vectors and the cluster centroids. These are used in a 1-NN classifier and a (smoothed) Naive Bayes classifier. The results show promising results, yet lack exploration of the model, at least to draw conclusions like "we address the challenge of understanding the internal visual cues of CNNs". I believe this paper needs to focus on the working of the VCs for few-shot experiments, showing the influences of some of the choices (layer, network layout, smoothing, clustering, etc).
Moreover, the introduction should be rewritten, and the the background section of VCs (Sect 3) should be clarified.
Therefore, I rate the current manuscript as a reject.
After rebuttal:
The writing of the paper greatly improved, still missing insights (see comments below).
Therefore I've upgraded my rating, and due to better understanding now, als my confidence.
Thanks for your detailed comments!
> Unclear description of VCs.
R: We are sorry for the difficulties you encountered in reading this paper. We have thoroughly revised the whole paper and, in particular, improved the clarity of Section 3 and Section 4. For the specific issue you mentioned, L_k is the set of positions in the k-th layer of the CNN for an input image. That means an element p in this set is “a specific position” of the feature maps from “a specific image”. If we assume that our network has C_k feature channels at the k-th layer, then f_p will be a C_k-dimensional vector.
> Problems on the properties of VCs
R:
1. The validity of property 1: On the first property, we list the statistics of 6 categories in Figure. 3(a) for a concise illustration of this property (We have added more examples and stats on more VCs and object categories in the appendix of the updated paper). Our conclusions still hold for more categories. The occurrence percentage of the sensitive categories, though “only” in the order of 40-77%, substantially outnumbers the percentages of other categories and hence can provide useful information for classification. We also replaced the word “dominate” by“fire intensively” to make the meaning clearer.
2. Misunderstanding on the interpretation of VCs such as “car wheel”: On the second and the third property (removed in the updated version), there is a misunderstanding of the "car wheel" VC. The VCs are extracted in an unsupervised manner (e.g. no spatial and image identity information) and are indexed by an integer. We used the term "car wheel" to describe the VC after we visualized it and found the image patches correspond to car wheels. This term was only used informally to give an intuition for the semantic information the VCs represent. It is not a supervision used by the model. In the updated paper, we replaced the informal names with VC index in Figure 1 and Figure 3 to avoid further confusions.
> Minor remarks
R:
1. Please note that results in the paper of Vinyals (2016) used a private split of  Mini-ImageNet.
So it's impossible to re-implement their settings. In this paper, we use a public split of Mini-ImageNet proposed by Ravi & Larochelle, 2017. The results for Matching Network are the same as Ravi & Larochelle, 2017 (We have added remarks for this in the caption of Table. 1).
2. In the original paper, we stated the parameter of our smoothing filter in the second paragraph of Section 5.1 ("For the Gaussian filter used to smooth the factorizable likelihood model, we use a \sigma of 1.2").
Also, we stated in the last sentence of Section 4.4 (original Section 4.3), the use of smoothing is to overcome the spareness of per pixel firing rate in our VC-likelihood model and help to improve the generalizability during testing. With this smoothing operation, our model can better handle small shiftings and deformations in the images. In our experiments, without the smoothing, our VC-likelihood model scores 61.84% in the 5-category 5-shot setting (see Table 1 for comparison). It is just slightly behind the model with smoothing. We use the smoothed results throughout the paper since the smoothing is also part of our model.
3. Previous methods are either learning a metric among categories or learning to learn a classifier within few examples. Our method is learning a composition of semantic concepts, which are extracted from few examples. We argue that our approach differs from previous one in terms of methodology.
Also, our models possess the flexible (Shown in Table 2) and interpretable (shown in Figure 5) characters unlike alternative models.
4. We agree that semantic parts have various scales.
However, for the simplicity of our method, we choose the layer with the most reasonable scale of VCs. In the appendix, we carried out experiments using features from different layers. We see from Appendix Table 3 that using features from the proposed Pool-3 layer achieves the best result.
Meanwhile, we notice that compared with the original features, our VC-based methods are more robust to various scales.
5. We are sorry that we only described the details of network training for PASCAL3D+ in Section 5.2 of the original version. We have added more details in the description for Mini-ImageNet in Section 5.1. The network was trained on the training split of Mini-ImageNet with the objective of cross entropy.
Specifically, we use the training split of the published Mini-ImageNet split from Ravi & Larochelle 2017. We preserve some images per category in the training split to validate our network.
6. VCs were proposed in Wang et al. (2015). We adopted the basic ideas of VCs from Wang et al. (2015) but adapted them to few-shot learning problems. In the appendix, we compared results using K-Means clustering and von Mises-Fisher clustering.
While von Mises-Fisher clustering is mathematically more reasonable based on our assumptions, we find that there are minor differences between different clustering methods empirically. This shows that our VC-based models are robust to different clustering methods.
> Problems on the writing
R: We have revised the writing of our paper thoroughly, removed many of the “bold” claims, and toned down the “arrogance”.
Instead, we concentrated on giving a clearer description of our work, including clarifying the novelty and effectiveness of our models. Please refer to the update list to see the modifications.
The paper adds few operations after the pipeline for obtaining visual concepts from CNN as proposed by Wang et al. (2015). This latter paper showed how to extract from a CNN some clustered representations of the features of the internal layers of the network, working on a large training dataset. The clustered representations are the visual concepts. This paper shows that these representations can be used as exemplars by test images, in the same vein as bag of words used word exemplars to create the bag of words of unseen images.
A simple nearest neighborhood and a likelihood model is built to assign a picture to an object class.
The results a are convincing, even if they are not state of the art in all the trials.
The paper is very easy to follows, and the results are explained in a very simple way.
Few comments:
The authors in the abstract should revise their claims, too strong with respect to a literature field which has done many advancements on the cnn interpretation (see all the literature of Andrea Vedaldi) and the literature on zero shot learning, transfer learning, domain adaptation and fine tuning in general.
Thanks for your comments! We cited previous works on CNN internal representations in Section 2 in the original version, and we modified the paper to cite these works in the introduction as well. The revision will be reflected in our updated version.
The paper proposes a method for few-shot learning using a new image representation called visual concept embedding. Visual concepts were introduced in Wang et al. 2015, which are clustering centers of feature vectors in a lattice of a CNN. For a given image, its visual concept embedding is computed by thresholding the distances between feature vectors in the lattice of the image to the visual concepts. Using the visual concept embedding, two simple methods are used for few-shot learning: a nearest neighbor method and a probabilistic model with Bernoulli distributions. Experiments are conducted on the Mini-ImageNet dataset and the PASCAL3D+ dataset for few-shot learning.
Positives:
- The three properties of visual concepts described in the paper are interesting.
Negatives:
- The novelty of the paper is limited. The idea of visual concept has been proposed in Wang et al. 2015. Using a embedding representation based on visual concepts is straightforward. The two baseline methods for few-shot learning provide limited insights in solving the few-shot learning problem.
- The paper uses a hard thresholding  in the visual concept embedding. It would be interesting to see the performance of other strategies in computing the embedding, such as directly using the distances without thresholding.
Thanks for your comments!
> Lack of novelty in this work.
R: The novelties of this work lie both in new results for VCs and new methods for few-shot learning. Sorry that we did not make this clear enough in the original submission.
1. New results for VCs. It is not obvious that the original VCs described in Wang et al. 2015 can be applied to few-shot learning. We adapt VCs for few-shot learning and our key findings, which are critical for few-shot learning, were not addressed in Wang et al. 2015.
a. Extracting VCs from CNNs trained on different object categories: We learn VCs for objects from a CNN trained on a different set of objects categories (e.g., we learn VCs for vehicles with a CNN trained on a non-vehicle dataset).
By contrast,  in Wang et al. 2015, the VCs were extracted from a subset of the object categories on which the CNN was trained.
b. Extracting VCs from few examples: In this work, we extract VCs from very few examples per category, but Wang et al. 2015 used orders of magnitude more (around 1000 versus 25). Surprisingly, we find that these “few-shot” VCs, when used for VC-encoding, possess similar desirable properties as the traditional VCs and hence are suitable for few-shot object classification task.
c. Extracting VCs without knowing the object category: In this work, we extract VCs without knowing the object category (e.g., by pooling feature vectors from different categories together and clustering over them).
But in Wang et al. 2015, VCs were extracted separately for each object category to obtain category specific VCs. This modification provides sufficient samples to learn high-quality VCs in few-shot setting and encourages VC sharing among different categories. This is useful for improving data efficiency and also makes it easier to apply our VC models on multiple novel categories directly.
d. We found novel properties of VCs, i.e., category sensitivity and spatial pattern, which support the extended application of VCs. We describe these properties in detail in the second and the third paragraph of Section 4.2.
2. A new approach to few-shot learning.
Unlike previous few-shot learning methods, we formulate few-shot learning in terms of compositions of semantic visual cues (i.e., parts). This differs from standard approaches like metric-based methods (e.g., matching network) and meta-learning based methods (e.g., MAML or Meta-Learner).
Moreover, as stated in Table. 2, our proposed models are simple and very flexible (i.e., the same model can be applied with minimal changes to different problem settings, such as 5-category classification, 6-category classification, etc.). We argue that flexibility is an important characteristic of few-shot learning for real-world applications.
> The hard threshold in the VC-Embedding.
R: First, this threshold is for binary encoding of VCs which makes it possible to learn simple Bernoulli distributions  (i.e, we only need to learn a distribution over binary variables which needs very little data). As shown in our experiments, this model is slightly better than nearest neighbor.
Second, we compared Nearest Neighbor using distances to VCs vs VC-Encoding. As shown in Appendix Table 1, the accuracy of using distances directly is slightly worse than using VC-Encoding, but is still better than using original features from CNN.
We thank the assigned and the volunteer reviewers for their comments. We have made a major revision of the paper to tone down the writing and to make it easier to understand. We have also clarified how our work relates to earlier studies of visual concepts to make our contribution clearer. We can release the code by January 19 to address the reproducing issue mentioned by the volunteer reviewer.
Besides the modifications of writing, we also updated our paper with an Appendix which is consisted of:
(1) Ablations studies that include the comparison between VC distance based model and VC-Encoding based model (which was mentioned by AnonReviewer3), comparison between K-Means clustering and vMFM based method (which was mentioned by AnonReviewer2), and the study on VCs of different scales (which is the question of VCs from different layers raised by AnonReviewer2).
(2) More visualization and statistics on VCs to better illustrate their properties.
We have released a primary version of our source code. Please find it on https://github.com/Awcrr/FewshotVC. We will keep completing its documents.

Meta-review:
role: PC
data: comment: The paper builds on earlier work by Wang et al (2015) on Visual Concepts (VCs) and explores the use of VCs for few-shot learning setting for novel classes.

The work, as pointed out by two reviewers is somewhat incremental in nature, with main novelty being the demonstration of utilities of VCs for few shot learning. This would not have been a big limitation if the paper had a carefully conducted empirical evaluation providing insights on the effect of various configuration settings/hyperparameters on the performance in few shot learning, which two of the reviewers (Anon3, Anon2) state are missing. The paper falls short of the acceptance threshold in its current form.

PS: The authors posted a github link to the code on Jan 12 which may potentially compromise the anonymity of the submission (though it was after all the reviews were already in) https://openreview.net/forum?id=BJ_QxP1AZ&noteId=BJaIDpBEM


Decision: Reject