Effects of Linguistic Labels on Learned Visual Representations in Convolutional Neural Networks: Labels matter!

1 INTRODUCTION . A critical distinction between human category learning and machine category learning is that only humans have a language . A language means that human learning is not limited to a one-to-one correspondence between a visual input and a category label . Indeed , the users of a language are known to actively seek out categorical relationships between objects and use these relationships in making perceptual similarity judgments and in controlling behavior ( Hays , 2000 ; Lupyan & Lewis , 2017 ) . A premise of our work is that a language provides a semantic structure to labels , and that this structure contributes to the superior efficiency and flexibility of human vision compared to any artificial systems ( Pinto et al. , 2010 ) . Of course , the computer vision literature on zero-shot and fewshot learning has also made good progress in leveraging semantic information ( e.g. , image captions , attribute labels , relational information ) to increase the generalizability of a model ’ s performance ( Lampert et al. , 2013 ; Sung et al. , 2018 ; Lei Ba et al. , 2015 ) . Still , this performance pales in comparison to the human ability for classification , where zero-shot and few-shot learning is the norm , and efficiently-acquired category knowledge is easily generalized to new exemplars ( Ashby & Maddox , 2005 ; Ashby & Ell , 2001 ) . One reason why machine learning lags behind human performance may be because of a failure to fully consider the semantic structure of the ground-truth labels used for training , which can be heavily biased by basic or subordinatelevel categories . This might result in models learning visual feature representations that may not be best for generalization to new , higher-level categories . For example , ImageNet ( Deng et al. , 2009 ) contains 120 different dog categories , making the models that are trained using these labels dog experts , creating an interesting but highly atypical semantic structure . Here we study how the linguistic structure of labels influences what is learned by models trained on the same visual inputs . Specifically , we manipulated the labels used to supervise the training of CNN models , each having the same architecture and given identical visual inputs . For example , some of these models were trained with basic-level labels only , some with only superordinate-level labels , and some with both . We then compare visual representations learned by these models , and predict human similarity judgement that we collected using an Odd-one-out task where people had to select which of three object images was the most different . With this dataset , and using categorical representations extracted from our trained models , we could predict human similarity decisions with up to 74 % accuracy , which gives us some understanding of the labels needed to produce human-like representations . Our study also broadly benefits both computer vision and behavioral science ( e.g. , psychology , neuroscience ) by suggesting that the semantic structure of labels and datasets should be carefully constructed if the goal is to build vision models that learn visual features representations having the potential for human-like generalization . For behavioral science , this research provides a useful computational framework for understanding the effect of training labels on the human learning of category relationships in the context of thousands of naturalistic images of objects . 2 RELATED WORK NEW 2.1 SEMANTIC LABEL EMBEDDING . Although many computer vision models perform well in image classification , generalization tasks such as zero-shot and few-show learning remain challenging . Several studies have attempted to address this problem by embedding semantic information into a model ’ s representations using text description ( Lei Ba et al. , 2015 ) , attribute properties ( Lampert et al. , 2013 ; Akata et al. , 2015 ; Chen et al. , 2018 ) , and relationships between objects ( Sung et al. , 2018 ; Annadani & Biswas , 2018 ) . More related to our work , some studies even directly leveraged the linguistic structure of labels . For example , Lei et al . ( 2017 ) and Wang & Cottrell ( 2015 ) found that training CNNs with coarse-grained labels ( e.g. , basic-level categories ) improve classification accuracy for finer-grained labels ( e.g. , subordinate-level labels ) . Also , Frome et al . ( 2013 ) re-trained a CNN to predict the word vectors learned by a word embedding model , instead of using one-hot labels , and found improved zeroshot predictions ; the model was able to predict thousands of novel categories that were never seen with 18 % accuracy . These results suggest that different semantic structures of labels , such as word hierarchy , an order of learning , or semantic similarity between words , affect learned visual representations in CNNs to differing degrees . The current study provides a more systematic investigation of this question . 2.2 UNDERSTANDING HUMAN VISUAL REPRESENTATION . The human visual system is unparalleled in its ability to learn feature representations for objects that are robust to large changes in appearance . This tolerance to variability , not only enables accurate object recognition , but also facilitates generalization to new exemplars and categories ( DiCarlo et al. , 2012 ) . Understanding how humans learn these visual representations is , therefore , an enormously important question , but one that is difficult to study because human learning in the real world is affected and confounded by many factors that are difficult to control experimentally . Recently , work has addressed this issue by computationally modeling and simulating human representation . For example , Hebart et al . ( 2019 ) studied human visual representations by fitting probabilistic models to human similarity judgement , and found that human visual representations are composed of semantically interpretable units , with each conveying categorical membership , functionality , and perceptual attributes . Peterson et al . ( 2018 ) , the study most similar to ours , trained CNNs with labels that differed in hierarchy ( e.g. , subordinate-level vs. basic-level ) . They found that training on coarsergrained labels ( either as standalone or as coming after finer-grained ) induces a more semantically structured representation , and produces more human-like generalization performance . The current study builds on this earlier work by 1 ) including CNNs trained with no labels ( autoencoder ) or very fine-grained labels ( word vector ) , 2 ) testing on a large-scale dataset of human similarity judgement , and 3 ) comparing superordinate vs. basic levels . 3 MODEL TRAINING . Our goal is to study how linguistic labels change the visual representations learned by CNNs . To do this , we trained equivalently designed CNNs for classification , but each with different linguistic labels as ground-truth . In addition , we trained a Convolutional autoencoder , which encodes the images using the same convolutional structure as the other models but , instead of being supervised to predict the class of the image , the objective of this model is to generate an output image that is the same as the input . This Conv . Autoencoder , therefore , represents a model that was not trained with any linguistic label , in contrast to the other models that were each trained with some type of linguistic labels . The description of each model and the labels used for training are provided below . • Conv . Autoencoder : Autoencoder with Convolutional encoder and decoder trained to output the same image as input • Basic labels : CNN model trained with one-hot encoding of basic-level categories , n=30 • Superordinate labels : CNN model trained with one-hot encoding of superordinate-level categories , n=10 • Basic + Superordinate : CNN model trained with two-hot encoding of both basic and superordinate-level categories , n=40 ( 10+30 ) • Basic then Superordinate : CNN model trained with one-hot encoding of basic-level cat- egories first ( n=30 ) , and then finetuned with one-hot encoding of superordinate categories ( n=10 ) • Superordinate then Basic : CNN model trained with one-hot encoding of superordinatelevel categories first ( n=10 ) , and then finetuned with one-hot encoding of basic categories ( n=30 ) NEW • Basic FastText vectors : CNN model trained with basic-level word vectors extracted from FastText word embedding model ( Bojanowski et al. , 2017 ) , dimension=300 • Superordinate FastText vectors : CNN model trained with superordinate-level word vectors extracted from FastText word embedding model ( Bojanowski et al. , 2017 ) , dimension=300 NEW The identical CNN architecture was used for each model in our labeling manipulation , except for the output layer and its activation function . This general pipeline is described in Figure 1 . Our CNN models consist of five blocks of two Convolutional layers , each followed by Max pooling and Batch normalization layers . For all Convolutional and Max pooling operations , zero padding was used to produce output feature maps having the same size as the input . Rectified linear units ( ReLU ) were used to obtain an activation function after each convolution . The flattened output of the final Convolutional layer , the ” bottleneck ” feature that we later extract and use as a model ’ s visual representation ( dim=1568 ) , was then fed into one fully connected dense layer . For Conv . Autoencoder , the same Convolutional architecture was used for encoding and decoding , with the hidden layer in the model ( dim=1568 ) serving as the bottleneck feature for analysis . The final predicted output , ” label vector ” is either one-hot or word embedding according to the model ’ s target labels . Output activation functions differed depending on what label vector was used : a sigmoid function for Basic + Superordinate CNN , a linear function for the Conv . Autoencoder and FastText vectors CNNs , and a softmax for the rest of CNNs . All models were trained and validated on the images of 30 categories from the IMAGENET 2012 dataset ( Deng et al. , 2009 ) , and tested on images of the same 30 categories from the THINGS dataset ( Hebart et al. , 2019 ) . These 30 basic-level categories were grouped into 10 higher-level , superordinate categories , which included : ’ mammal ’ , ’ bird ’ , ’ insect ’ , ’ fruit ’ , ’ vegetable ’ , ’ vehicle ’ , ’ container ’ , ’ kitchen appliance ’ , ’ musical instrument ’ , and ’ tool ’ . A list of all 30 categories , with their superordinates , are provided in the Supplementary 7.1 . All input images were converted from RGB to BGR and each channel was zero-centered with respect to the ImageNet images . Different loss functions were used for training different models : Binary Crossentropy loss for Basic + Superordinate CNN , and Mean Squared Error loss for both Conv . Autoencoder and FastText vectors CNNs , and Categorical Crossentropy loss was used for the rest of the CNNs . All models were trained using Adam optimization ( Kingma & Ba , 2014 ) , with a mini-batch size of 64 . During training , early stopping was implemented and the model with the lowest validation loss was used for the following analysis . 4 BEHAVIORAL DATA . To compare the visual representations learned by our trained models with those of humans , we collected human similarity judgments in an Odd-one-out task , as in Zheng et al . ( 2019 ) . Participants were shown three images of objects per trial , a triplet , and were asked to choose which object was most different from the other two . Each triplet consisted of three exemplar objects from the 30 categories used for our model training . All exemplar objects came from Zheng et al . ( 2019 ) , except for ’ crate ’ , ’ hammer ’ , ’ harmonica ’ , and ’ screwdriver ’ , which were replaced with new exemplars to increase image quality and category representativeness . There are 4060 possible triplets that can be generated from all 30 categories , but we collected behavioral data on only a subset of these to reduce the time and cost of data collection . This subset includes 1 ) the ten triplets having objects coming from the same superordinate category , e.g. , ’ orangutan ’ , ’ lion ’ , ’ gazelle ’ 2 ) all 435 triplets where two objects came from the same superordinate category , e.g. , ’ orangutan ’ , ’ lion ’ , ’ minivan ’ , and 3 ) 1375 triplets where all objects came from different categories , e.g. , ’ orangutan ’ , ’ minivan ’ , ’ lemon ’ , yielding 1820 unique triplets in total . 51 Amazon Mechanical Turk ( AMT ) workers participated in this task , each making responses on ∼200 triplets . After removing responses with reaction times below 500ms , we collected 9697 similarity judgments where each triplet was viewed by 5.6 workers , on average ( min=4 , max=51 ) .

Summary: This paper assesses the effects of training an image classifier with different label types: 1-hot coarse-grained labels (10 classes), 1-hot fine grained labels (30 labels which are all subcategories of the 10 coarse-grained categories), word vector representations of the 30 fine-grained labels. They also compare the representations learned from an unsupervised auto-encoder. They assess the different representations through cosine similarity within/between categories and through comparison with human judgments in an odd-one-out task. They find that (i) the auto-encoder representation does not capture the semantic information learned by the supervised representations and (ii) representations learned by the model depend on the label taxonomy,  how the targets are represented (1-hot vs. wordvec), and how the model is trained (e.g. fine-grained then coarse grained stages), (iiii) the different representations predict human judgements to differing degrees. the first finding is obvious and I'm not even sure why it needs to be stated -- of course semantics of images are not inherently encoded in the pixels of an image! The second point again, is not surprising . This paper starts to get at some interesting questions but does not follow through.  It is also quite confusing to read despite thee simple subject matter. This paper is also missing a related work section! There has been so much word on adding structure to the label space of image classifiers (e.g. models that learn image/text embedding space jointly, models that predict word vectors, graphical model approaches to building in semantic information, etc.) and none of this is discussed. There has also been work on comparing convnet representations to human percepts e.g. https://cocosci.princeton.edu/papers/Peterson_et_al-2018-Cognitive_Science.pdf)and none of this work is discussed! This work needs to be better situated within the context of previous work in this field. Please write a related work section.