Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary: This paper demonstrates the importance of labels at various levels (no label, basic level label, and superordinate level) as well as in combination to determine the importance of semantic information in classification problems. They train an identical CNN architecture either as an autoencoder (no labels), with the basic label, with the subordinate label, with the basic and subordinate labels, and with basic labels which are fine-tuned with one-hot encodings of superordinate labels, as well as with word vectors. Classification accuracy, t-SNE, cosine similarity matrices and predictions on a human behavior task are used to evaluate the differences across labels types. The authors find that superordinate labels are helpful and important for classification problems. 

Major comments:
- Authors need to include more related work and describe the main related paper they mention (Peterson et al 2018) as well as describe how their work fits in with previous work
- While the idea here is novel and impactful, the experiments used to explain the importance of superordinate labels do have not much compelling information and are not well described
- 4.2 plots for visualization are mentioned to be in the appendix, but are not there

Minor comments:
-	Fig2 large subordinate group text would help
-	Lots of typos throughout and grammar mistakes 
o	Typo ‘use VGG16’ and then ‘Vgg16’ in same paragraph bottom of page 4 
o	Typo top of page 2 “Convolutional neural network(CNN)”
o	Appendix list – ‘banna’ typo under Fruit
o	Page 1 intro ‘for both behavioral and computer vision’ doesn’t really make sense 
o	Page 3 top section ‘new one’ should be ‘new ones’ 
o	Bottom of page 3 ‘room from improvement’ 
o	Last line of conclusion – ‘classificacation’

Consensus: This is a very interesting and potentially impactful idea, but the experiments used to defend and explain the importance of superordinate labels are relatively weak. Significant work on writing and experimental side should be complete, but because this is novel and important work for classification, with some serious revisions, I would suggest accepting this paper. 

review_title: Official Blind Review #5

final_score: rating: 6: Weak Accept
review_content: This paper assesses the effects of training an image classifier with different label types: 1-hot coarse-grained labels (10 classes), 1-hot fine grained labels (30 labels which are all subcategories of the 10 coarse-grained categories), word vector representations of the 30 fine-grained labels. They also compare the representations learned from an unsupervised auto-encoder. They assess the different representations through cosine similarity within/between categories and through comparison with human judgments in an odd-one-out task. They find that (i) the auto-encoder representation does not capture the semantic information learned by the supervised representations and (ii) representations learned by the model depend on the label taxonomy,  how the targets are represented (1-hot vs. wordvec), and how the model is trained (e.g. fine-grained then coarse grained stages), (iiii) the different representations predict human judgements to differing degrees. the first finding is obvious and I'm not even sure why it needs to be stated -- of course semantics of images are not inherently encoded in the pixels of an image! The second point again, is not surprising . This paper starts to get at some interesting questions but does not follow through.  It is also quite confusing to read despite thee simple subject matter. This paper is also missing a related work section! There has been so much word on adding structure to the label space of image classifiers (e.g. models that learn image/text embedding space jointly, models that predict word vectors, graphical model approaches to building in semantic information, etc.) and none of this is discussed. There has also been work on comparing convnet representations to human percepts e.g. https://cocosci.princeton.edu/papers/Peterson_et_al-2018-Cognitive_Science.pdf)and none of this work is discussed! This work needs to be better situated within the context of previous work in this field. Please write a related work section.

Detailed comments/questions:
- It would be good to add a super-basic model to table 1 for comparison (i.e. first train of coarse level categories and then fine-tine on the more fine-grained taxonomy). 
- It would be good to compare the use of word vector representations at both the basic and superordinate levels; the 1-hot vs word vector targets and the basic vs superordinate taxonomy seem like orthogonal axess to explore and I'm not sure why the authors didn't test all combinations. 
- the authors found the imagenet categorical representations were most predictive of human judgements in the odd-one-out task. This seems highly unsurprising since (i) the humans saw images from the Imagenet dataset (not THINGS) and (ii) humans leverage semantic information when making similarity judgements. 
- What categories had the least inter-rater agreement.. was there any relationship between these categories and the similarity of representations learned by the convnet?
- It seems the odd-one-out comparison always involves averaging image representations at the basic category level. In the case where the items come from three different superordinate classes it would bee interesting to see the results when averaging over superordinate classes as well.
- In table 3, what does the FastText column just list "true"/"false" rather than accuracies? I would expect this column to show the accuracy when the FastText embeddings for the three words are used to compute similarity. I don't understand what the "true"/"false" is meant to indicate. Also it's not clear to be what the two rows in table three are meant to correspond to?
- The authors claim "Surprisingly, the kind of supervised input that proved most effective in matching human performance on the triplet odd-one-out task was training with superordinate labels". This should be qualified to say that, when there are two or more superordinate classes represented in the triplet, the superordinate labels are highly effective when the three items come from three different superordinate classes. I'm also not clear why this would be surprising? Could the authors elaborate?
- I'm surprised more space isn't given to discussing the wordvec representations since these should capture some of the semantic information that the 1-hot encodings might miss. In fact, the word vectors targets seem to perform as good as or close to the other representations on the odd-one-out

In short, I really like the overall idea of comparing convnet representations with human perceptions of images. However, this work barely scratches the surface of what could be done here and mostly reveals incredibly obvious results. There are so many interesting questions to ask regarding the relationship between how humans perceive similarity and what is encoded in a convnet representation. For example, it would have been very interesting to test the effects of asking the human rates to cue in on different aspects of the image. Focusing on semantic similarity, visual similarity, etc. would all likely give different ratings. 



----------------------------------------------------------
Update (in light of rebuttal)

I appreciate the authors lengthly and considered response. In particular, the updated related work and expansion of the empirical experiments. While I am more comfortable with this paper being accepted that previously (and have updated by score to "weak accept" to reflect this), I still think the paper has a lot of room for improvement. In particular, I suggest a more expansive analysis of human perceptions and a discussion f the implications of the findings. 
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The authors conduct a comparative study of several variants of CNNs trained on imagenent things category with different types of labeling schemes (direct, superordinate, word2vec embedding targets, etc.) They also use a human judgement dataset based on odd-one-out classification for triplets of inputs as comparison to evaluate whether the CNNs are able to capture the linguistic structure in the label categories as determined by the relation of the superordinate labels to the basic labels.   

The authors used the t-SNE embeddings to visualize the representations learned and evaluate whether these cluster related classes close enough. Not suprisingly, training with the word2vec targets produced the best representations for similarity between/within category. Interestingly, the autoencoder failed to learn representations that are easily interpretable by the analysis tools they were using. 

This is an interesting study. The core claim being made as follows:

"The representations learned by the models are shaped enormously by the kinds of supervision the models get suggesting that much of the categorical structure is not present in the visual input, but requires top-down guidance in the form
of category labels. "

The fact that the representations being learned are shaped strongly by the supervision is probably not surprising or in contention. However, it is not clear that the representations being learned can be exhaustively interpreted by convenient visualization tools. In my opinion, absence of evidence here is not clearly an evidence of absence. However, I still think these are interesting analyses so I am giving weak accept.


review_title: Official Blind Review #1

Meta-review: This paper explores training CNNs with labels of differing granularity, and finds that the types of information learned by the method depends intimately on the structure of the labels provided.

Thought the reviewers found value in the paper, they felt there were some issues with clarity, and didn't think the analyses were as thorough as they could be. I thank the authors for making changes to their paper in light of the reviews, and hope that they feel their paper is stronger because of the review process.
Decision: Reject