Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors introduce a probabilistic k-shot learning model based on previous training of a CNN on a large dataset. The weights of the softmax layer of the CNN are then used as the MAP solution in a concept learning scenario to put a prior over the soft-max weights of the classifier (logistic regression) for the dataset with k-shot examples. The model is compared against other models for doing k-shot learning in miniImageNet, and against different versions of the same model for CIFAR-100.

The paper introduces a very simple idea that allows transfer knowledge from a network trained with a large-dataset to a network-trained with a smaller amount of data, the data with k-shot examples per class. This is not a general way to do k-shot learning, because it heavily depends on the availability of the large dataset where the weights of the soft-max function can be extracted. But it seems to work for natural image data. 

How many data observations are necessary to estimate \widetilde{W}_{MAP} such that it is still possible to obtain superior performance in the k-shot learning problem? Did you try the methods of Table 1 for CIFAR-100? The experiments on this dataset use models that are variations of the same proposed model. 

review_title: A simple probabilistic method to transfer knowledge from large datasets to smaller datasets (k-shot per class).

final_score: rating: 5: Marginally below acceptance threshold
review_content: Authors present a k-shot learning method that is based on generating representations with a pre-trained network and learning a regularized logistic regression using the available data.  The regularised regression is formulated as a MAP estimation problem with the prior estimated from the weights of the original network connected final hidden layer to the logits — before the soft-max layer.  

The motivation of the article regarding “concepts” is interesting.  It seems especially justified when the training set that is used to train the original network has similar objects as the smaller set that is used for k-shot learning.  Maps shown in Figures 6 and 7 provide good motivation for this approach. 

Despite the strong motivation, the article raises some concerns regarding the method. 
1. The assumption about independence of w vectors across classes is a very strong one and as far as I can see, it does not have a sound justification.  The original networks are trained to distinguish between classes.  The weight vectors are estimated with this goal. Therefore, it is very likely that vectors of different classes are highly correlated. Going beyond this assumption also seems difficult.  The proposed model estimates $\theta^{MAP}$ using only one W matrix, the one that is estimated by training the original network in the most usual way.  In this case, the prior over $\theta$ would have a large influence on the MAP estimate and setting it properly becomes important.  As far as I can see, there is no good recipe presented in the article for setting this prior. 
2. How is the prior model defined?  It is the most important component of the method while precise details are not provided.  How are the hyperparameters set?  Furthermore, this detail needs to be in the main text. 
3. With the isotropic assumption on the covariance matrix, the main difference between logistic regression, which is regularized by L2 norm and coefficient set proportional to the empirical variance, and the proposed method seems to be the mean vector $\mu^{MAP}$.  From the details provided in the appendix — which should be in the main text in my opinion — I believe this vector is a combination of the prior and mean of w_c across classes.  If the prior is set to 0, how different is this vector from 0?  Authors should focus on this in my opinion to explain why methods work differently in 1-shot learning.  In the other problems, the results suggest they are pretty much the same. 
4. Authors’ motivation about concepts is interesting however, if the model bases its prediction on mean of w_c vectors over classes, then I am not sure if authors really achieve what they motivate for.  
5. Results are not very convincing.  If the method was substantially different than baseline, I believe this would have been no problem.  Given the proximity of the proposed method to the baseline with regularised logistic regression, lack of empirical advantage is an issue.  If the proposed model works better in the 1-shot scenario, then authors should delve into it to explain the advantage. 

Minor comments: 
Evaluation in an online setting section is unclear.  It needs to be rewritten in my opinion. 
review_title: interesting interpretation of regularized Logistic regression but missing details and a very strong assumption

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a procedure to efficiently do K-shot learning in a classification setting by creating informative priors from information learned from a large, fully labeled dataset.  Image features are learned using a standard convolutional neural network---the last layer form image features, while the last set of weights are taken to be image "concepts".  The method treats these weights as data, and uses these data to construct an informative prior over weights for new features.

- Sentence two: would be nice to include a citation from developmental psychology.

- Probabilistic modeling section: treating the trained weights like "data" is a good way to convey intuition about your method.  It might be good to clarify some specifics earlier on in the "Probabilistic Modeling" paragraph, e.g. how many "observations" are associated with this matrix. 
 
- In the second phase, concept transfer, is the only information from the supervised weights the mean and estimated covariance?  For instance, if there are 80 classes and 256 features from the supervised phase, the weight "data" model is 80 conditionally IID vectors of length 256 ~ Normal(\mu, \Sigma).  The posterior MAP for \mu and \Sigma are then used as a prior for weights in the K-shot task.  How many parameters are estimated for 

  * gauss iso: mu = 256-length vector, \sigma = scalar variance value of weights
  * log reg: mu = 256-length zero vector, \sigma = scalar variance value of weights
  * log reg cross val: mu = 256-length zero vector, \sigma = cross validated value

If the above is correct, the information boosts K-shot accuracy is completely contained in the 256-length posterior mean vector and the scalar weight variance value?

- Is any uncertainty about \mu_MAP or \Sigma_MAP propagated through to uncertainty in the K-shot weights?  If not, would this influence the choice of covariance structure for \Sigma_MAP? How sensitive are inferences to the choice of Normal inverse-Wishart hyper parameters? 

- What do you believe is the source of the mis-calibration in the "predictied probability vs. proportion of times correct" plot in Figure 2?  

Technical: The method appears to be technically correct.

Clarity: The paper is pretty clearly written, however some specific details of the method are difficult to understand.

Novel: I am not familiar with K-shot learning tasks to assess the novelty of this approach. 

Impact: While the reported results seem impressive and encouraging, I believe this a relatively incremental approach. 
review_title: interesting but perhaps incremental 

Meta-review: This submission presents intriguingly good results on k-shot learning and I agree with the authors that the results are better than the presented previous work, and that the method is simple, so I took a deeper look into the paper despite the overall negative reviews. However, I think in its current form, the paper is not suitable for publication:

- The previous work, that the authors compare to, were not really using comparable architectures: in fact, likely much worse base models with fewer parameters etc. I think any future version of this work would need to control for architecture capacity, otherwise how can we be sure where the gains come from? To me, this is a major unknown in terms of the credit assignment for the great results.
- The authors should be comparing with MAML (and follow-up work) by Finn et al. (2017)
- I don't really understand why the authors claim to have no need for validation sets. That's a very strong claim: are ALL the hyper-parameters (model architectures etc) just chosen in another, principled way? This issue would definitely need to be addressed in a follow-up work.
Decision: Reject