Categories vs Semantic Features: What shape the similarities people discern in photographs of objects?

Authors: siddharth  suresh
wei-chun  huang
kushin  mukherjee
timothy t rogers
Venue: ICLR_cc_2024_Workshop_Re-Align
Published: 2024-02-09 06:11:00

Abstract: In visual cognitive neuroscience, there are two main theories about the function of
the ventral visual. One suggests that it serves to classify objects (classification
hypothesis); the other suggests that it generates intermediate representations
from which people can generate verbal descriptions, actions, and other kinds of
information (distributed semantic hypothesis). To adjudicate these, we trained
two deep convolutional AlexNet models on 330,000 images belonging to 86
categories, representing the intersection of Ecoset images and the semantic
norms collected by the Leuven group. One model was trained to produce
category labels (classification hypothesis), the other to generate all of an item’s
semantic features (distributed semantic hypothesis). The two models learned very
different representational geometries throughout the network. We also estimated
the human semantic structure of the 86 classes by using a triadic comparison
task. The representations acquired by the feature-generating model aligned better
with human-perceived similarities amongst images, and better predicted human
judgments in a triadic comparison task. The results thus support (distributed
semantic hypothesis).

Main paper: 
 introduction: 
 Theories about human visual object recognition have generally oriented around two somewhat different views. The first proposes that the recognition system serves to match a perceived item to one of several stored object categories (Riesenhuber & Poggio, 2000; Serre et al., 2007) . such matching can then index into the human semantic system to allow for retrieval of other information about the perceived item (Jolicoeur et al., 1984; Coltheart, 2004) . On this classification hypothesis, visual object recognition is contained solely within the visual system and provides access to a separate, potentially amodal semantic store. The second perspective proposes that visual object recognition is not separate from, but is deeply influenced by, semantic processing. On this distributed semantic hypothesis, object recognition does not function to match a visually-perceived object to a discrete category representation. Instead, visual object processing generates a distributed, cross-modal representation that expresses the perceived item's semantic or conceptual similarities to other known items (Rogers et al., 2004) . As a consequence, the representational structure expressed within recognition systems can be influenced, not only by visual structure, but also by the rich multi-modal semantic knowledge the perceiver possesses about the object.The current paper assesses which view provides a better account of the similarities that people discern amongst photographs of objects, by training models with identical encoding architectures on identical distributions of images, but with different objectives corresponding to the two different views. We first consider whether the objective matters, showing that the two model variants learn different representational geometries that are most profound in deeper layers but persist all the way to the shallowest layers. We then consider which variant provides a better account of human-perceived similarities, by considering how well model-based representations explain human similarity judgements, and how well model representational structure aligns with structure derived from human judgements.Relationship to prior work. A broad range of recent work has considered how the internal representations acquired by contemporary deep vision models can help to understand aspects of human visual perception and/or the neural encoding of visual object information. Yamins et al. (2014) has closely considered how the behavior of deep convolutional image classifiers relate to human behavior, and how such models may explain fine-grained organization of featuredetectors within primate visual cortex when constrained with additional losses (Margalit et al., 2023) . Muttenthaler et al. (2023) have considered how representations in deep image classifiers can be directly regularized to show better accordance with human similarity judgements. Fel et al. (2022) have showed that deep image classifiers can be regularised with the features that humans pay attention to while perceiving images. Mehrer et al. (2021) showed that DCNNs better capture patterns of human behavior when trained on more representative corpora of images than the usual ImageNet dataset (Deng et al., 2009) .The current work contributes to these efforts by considering an apples-to-apples comparison of models matched in all aspects except for the two losses considered, each corresponding to a different cognitive hypothesis about the functioning of the visual object recognition system. Even though layers that are deeper and densely connected exhibit a stark contrast in their geometries, differences are observable across all layers, indicating that the distinct objectives of the model variants lead to systematically divergent representational geometries throughout the network.The dataset used to train our models was constrained by two goals. First, because we aim to understand human-perceived similarities amongst photographs, we wanted it to include a representative and ecologically realistic distribution of images. We therefore began with the ECOSET (Mehrer et al., 2021) , a compilation of images belonging to 565 basic-level object categories that arguably represent the distribution of concepts with which people are typically familiar. Second, we wished to compare models that classify photographs by assigning them to the appropriate basic-level category with models that instead learn to generate a distributed semantic representation of the items. For this, we employed the Leuven norms, a dataset of semantic norms collected by the Leuven group (De Deyne et al., 2008) , which indicates which of 2057 propositional features (e.g. "has wings", "is big," "can move," etc) are commonly judged to be true of 300 different living and nonliving categories. For such feature vectors, the feature overlap between two concepts provides an indication of their semantic relatedness (McRae et al., 2005) , so that binary feature vectors encoding the norm data can serve as proxies for distributed output representations that encode semantic similarity structure. To create the dataset for this study, we therefore took the intersection of concepts appearing in both the ECOSET and the Leuven norms-a set that includes 330,000 images, each depicting an item belonging to one of 86 possible basic-level categories, and with each such category associated with a binary semantic feature vector containing 2057 elements(1). We will call this the ECOSET-86. 
assessing whether the objective matters.: 
 Study 1 assessed whether a DCNN will acquire similar or different internal representations when trained with losses corresponding to the two hypotheses. Specificaly, categorization models learned to generate a one-hot basic-level category label for each image, while distributed semantic models learned to generate all of the semantic features associated with an image. Each model type adopted the same encoding architecture, viewed the same distribution of images, and was trained to the same performance criterion. We then assessed, at each model layer, how acquired representations compared within and between model types.Models were variants of the AlexNet architecture (Krizhevsky et al., 2017) trained to categorize using standard cross-entropy loss (Cat-Alexnet (Table 2 )) or to generate semantic features using an independent binary cross-entropy loss on each output unit (Sem-Alexnet (Table 3 )). Model accuracy was measured as the proportion of times (a) the most active label was the correct label (for Cat-Alexnet) or (b) the output vector was more similar to the target vector than any other vector(for Sem-Alexnet). Five instances of each variant, differing only in initial random weight configuration, were trained to the same accuracy criterion of 70%. We then evaluated similarity of the representational geometries acquired at each model layer within and between model variants, following the procedure described by (Mehrer et al., 2020) : at each layer we computed pairwise cosine distances in the elicited representations for a large test set of photographs. For each pair of layers across all models, we then computed the pearson correlation in the resulting distance matrices-producing a layer-to-layer distance matrix whose entries indicate dissimilarity in the representational geometries acquired by each model layer. Finally, we reduced this matrix to three components using classical multidimensional scaling.Results are shown in Figure 1 . Layers within a model instance are connected by a line; the two model variants (categorization vs distributed semantic) are indicated with different colors; and distances within the 3D embedding indicate dissimilarity of representational geometries between layers. These tend to be quite similar within variant, dissimilar across variants at each layer. The difference is most obvious in deeper densely-connected layers (well-separated green vs red points toward the right) but is discernable through all layers, even to the most shallow. Thus the different objectives produce systematically different representational geometries throughout the network.Study 2 assessed whether the difference in learned representations would lead to differences in the ability of the models to predict human behavior on a triplet judgement task.Triadic comparison task Our primary objective is to test the representational alignment of different model-based visual-semantic representations with human perceived visual-semantic representations. In pursuit of this, we require a means of assessing human semantic representations for our objects of interest. Here, we focus on naturalistic images of a set of inanimate and animate concepts. We adopt a method used in several recent studies investigating human conceptual structure, namely triadic similarity judgements Hebart et al. ( 2023 2A ). This entails presenting participants with sets of image triplets and instructing them to discern which among the two option images bears greater resemblance to a third reference image. By collecting enough image triplet judgements, we estimate a latent embedding (Figure 2B ) for each of the unique images that participants are shown during the experiment Jamieson et al. (2015) ; Hornsby & Love (2020); Sievert et al. (2023) . These embedding can then be leveraged to estimate a conceptual space that best predicts the judgements that humans made during the task. Below, we describe the procedure for our human experiment.Stimuli. Of the 86 categories, 46 were animate and 40 were were inanimate concepts. Within the animate domain, images belonged to parent categories like Mammals, Insects, and Reptiles. Within the inanimate domain, images belonged to parent categories like Household Objects, Instruments, Vehicles, and Tools. Each category had a total of 3 images resulting in a total of 258 images.Participants. We recruited 91 participants on Prolific to complete our task (Palan & Schitter, 2018) . Each participant provided informed consent in compliance with our Institutional IRB.Procedure. On each trial, participants viewed a target image displayed above two option images, and were instructed to choose via mouse click or arrow keys which of the two option images was most similar to the target image. Each participant completed 570 trials. On 520 of the trials, the triplet on each trial was sampled randomly with uniform probability from the space of all possible triplets. The remaining 50 trials (randomly dispersed during the experiment) constituted 'validation triplets'. Each participant saw 50 'validation trials' from a curated pool of 258 triplets. The study yielded a total of 49970 judgments, an order of magnitude larger than the minimal needed to estimate an accurate 5-D embedding from random sampling according to estimates of sample complexity in this task (Jamieson et al., 2015; Sievert et al., 2023) . 80 % of the non-validation judgments were used to estimate a 5-D embedding that minimized the crowd-kernel triplet loss on the training set of triplet judgement trials (Tamuz et al., 2011) . The resulting embedding was then tested by assessing its accuracy in predicting human judgments on the non-validation held-out data. The final embeddings predicted human decisions on held-out triplets with 75% accuracy. For a direct comparison between human judgments and our model's predictions, we expose our trained models to the same set of images that were used in the triplet judgement task conducted on human participants. We used activations from intermediate layers that were present in Sem-Alexnet, Cat-Alexnet, and Alexnet pre-trained on ImageNet and ECOSET and investigated differences in learned representational geometries. To assess the structural coherence of model representations with human embeddings, we computed the the average parent category-wise Procrustes correlation of the embeddings of the 258 images derived from humans and models . This metric, analogous to Pearson r, indicates the extent to which variations in pairwise distances from one representational space are reflected in the other. To emulate human decision-making during each trial using the model, we computed the cosine similarity between the hidden layer representations of the images shown in the trial, thereby determining the model's choice and using accuracy as a metric of human-model alignment.Procrustes alignment. We find that the highest Procrustes correlation between model and human representations is achieved by the Sem-AlexNet in the deepest layer (classifier.5) that is common to all the models ( Figure 2C ) . This shows that models trained on semantic feature elicitation are better aligned with human representations. It is widely believed that the model performance improves as the training dataset size increases and becomes more diverse Lei et al. (2018) . To see if using an AlexNet trained on datasets that were diverse and contained more image classes improved human-model alignment, we also compared the alignment of our trained models to an AlexNet pre-trained on the entirety of Ecoset and on ImageNet. We still find that the highest Procrustes correlation is achieved by classifier.5 layer of the Sem-AlexNet which is trained on only ∼ 20-25% of the images (330,000) compared to ECOSET (1.5 million), and Imagenet-1k (1.2 million) images. This suggests that a training objective that is centered around human-generated semantic features allows vision models to acquire more human-like semantic representations relative to models trained on categorization objectives. These results thus lends support to the distributed semantic hypothesis.Triplet judgement prediction. In addition to investigating the alignment of model-human representational geometries, we were also interested in how accurate judgments that were based in model representations were. We specifically investigated how well models could predict human judgements on the 258 validation triplets. The 258 triplets belonged to one of three types based on how the the three images in the triplet were sampled. A third of the validation triplets were sampled such that all the three images belonged to the same parent category (eg. (cat, dog, zebra) are all mammals), a third were sampled such that two (including the anchor) belonged to the same parent category and one belonged to a different parent category (eg. (cat, dog, ant) because cat and dog are mammals and an ant is an insect), and a third were sampled such that all the three images belonged to different parent categories (eg. (cat, ant, helicopter) ). We simulated the triplet judgement task using cosine distance between images from the intermediate layer activations of the model. The image that was closer in representational space to the reference image was chosen as the 'answer' on each trial. We also measured the human accuracy using the human estimated 5-D embeddings. We found that the highest triplet prediction was achieved by classifier.5 layer of the Sem-AlexNet on all three types of triplets, being the only model to reach human levels of accuracy, computed using human embedding, in two of the three triplet types and even exceeding the human accuracy when all the three images in a triplet belong to the same parent category. Predicting accuracy using a mixedeffect logistic regression model1 also revealed that model type (Sem-Alexnet vs. Cat-Alexnet) was a significant predictor with Sem-Alexnet having the higher accuracy (β = -0.05, SE = 0.02, p <.001). These results (Figure 2D , Figure 3 ) suggest that Sem-Alexnet was more aligned with humans on the triplet decision task relative to Cat-Alexnet providing additional support to the distributed semantic hypothesis.People might gauge similarity between a triplet of images not simply using category information but visual features of the images. Is Sem-Alexnet more sensitive to this information relative to Cat-Alexnet? To test this, we compare each models' triplet judgement performance on true triplet sets shown to participants during the human experiment and on simulated 'swapped' triplet sets, where the categories in the triplet set were kept the same but the images were swapped to one of the other images belonging to each category. In other words, if the original triplet was (cat1, dog1, zebra1), we would use (cat2, dog2, zebra2) to simulate a swapped triplet. We swap the images 100 times for each triplet. We then compute the means of each of the 100 runs to the accuracy obtained when using real images. We find that Sem-Alexnet is more sensitive to image-specific properties because it is more accurate while simulating triplets than Cat-Alexnet (Figure 2D ). This was supported by a significant model type × trial type (original vs. simulated) interaction effect in our mixed-effect logistic regression model (β = .03, SE=.02, p <.05). This effect is driven by triplets of the type where all the triplets belong to the same parent category. This suggests that training models to elicit semantic features leads to learning within-category structure that is more aligned with that of humans. 
discussion: 
 While numerous models addressing visual semantics and visual perception have been proposed, our study distinguishes itself in its unique comparative methodology. To our knowledge, this is the first investigation that contrasts representations across varied models under equitable conditions-specifically,ensuring that the models are trained on identical datasets and achieve comparable performance metrics (e.g., classification accuracy).Moreover no prior work has compared image classifiers to alternative models aligning with the distributed semantic hypothesis-that is, models in which representations in ventral stream are partially shaped by linguistic and/or semantic knowledge. Furthermore, our study would be the first to collect a large dataset of human similarity judgments to evaluate model-human representational alignment complementing earlier such efforts Hebart et al. (2022) . This dataset stands to benefit not only cognitive scientists but also computer scientists with interests in human-AI alignment. From a pragmatic standpoint, our models hold potential in the domain of medical research, particularly in the study of neurodegenerative disorders such as semantic dementia (Rogers et al., 2004) . By being the targets of controlled perturbation studies, these models stand to support controlled simulations and could elucidate why certain therapeutic interventions exhibit heightened efficacy at specific junctures in a disorder's progression -a revelation of paramount significance to the medical community. In this study, the mixed-effects logistic regression models were fit using the lme4 package in R, specifically version 1.1.30. The bobyqa optimizer was utilized for the fitting of these mixed-effects models.Sem-Alexnet is more accurate at predicting human judgements and it is also more accurate when the judgements are predicted using original images vs swaped images We fit a model predicting the accuracy of a triplet judgement from the model (Sem-Alexnet or Cat-Alexnet), the type of image (Real vs swapped), and their interaction while controlling for the layer (features.1 ,features.4, features.7, features.9, features.11). We included random intercepts for the type of triplet (all within parent category, two within parent category, and all different parent category) and the Model (Sem-Alexnet or Cat-Alexnet). refer to Appendix for statistical model details 
 
 Appendix:  



Reviews: review: title: title: Interesting assessment of theories of visual perception through the lens of model-human alignment.
main_review: main_review: **Summary:** This paper aims to assess two theories regarding the function of the ventral visual: the *classification hypothesis* (that human recognition simply groups items into categories) and the *semantic hypothesis* (that the recognition system generates detailed semantically-informed representations). The papers examines these theories by training AlexNet models using training objectives that are aligned with these two hypotheses, and then testing which models are better aligned with human similarity judgements; they find better alignment with the feature-generating model, thus supporting the semantic hypothesis.

**Strengths:**
- The research question is well-formulated, and of interest to both cognitive science and machine learning researchers.
- The results around model representational geometry (Fig 1) are particularly interesting, and the trend regarding greater differences in deeper layers makes intuitive sense given that the training objective is the main difference between the two models.
- The main finding -- that the feature generating model aligns better with human judgements -- is well-supported by the experiments.

**Weaknesses**:
- The claims regarding the novelty of the collected dataset are not supported. The authors collect a dataset of image triplets with human similarity judgements, and state that they are  "the first to collect a large dataset of human similarity judgements to evaluate model-human representational alignment". However, to my knowledge, there are at least three existing, large datasets of human similarity judgements on image triplets: BAPPS [1], THINGS [2], and NIGHTS [3], all of which have been used in machine learning literature to evaluate model-human alignment. The authors cite [2] however do not explain how their dataset differs. To support statements of the dataset's novelty, they should cite existing datasets in the related works section and explain how theirs is unique.
- In plots assessing human-model alignment, the baseline is denoted by the accuracy of embeddings that predict human decisions. However this leads to results, such as in Fig. 3, where some models appear to achieve higher alignment than the human embeddings. It seems a bit strange for any model to outperform the human metric on a human alignment task -- it may be more informative for the authors to instead (or additionally) plot the inter-rater agreement as a noise ceiling.
- It may not be immediately clear to all readers why the triplet judgement task in particular is a good test of model-human representational alignment. The authors explain this well in the Section 3, but it may be helpful to include some of this reasoning in the introduction.

**Questions:**
- How much does the choice of model architecture impact these findings? Is it possible that different trends may emerge for larger models?
- It would be interesting to see if these model-human alignment results are consistent across different similarity judgement datasets.

**References**
[1] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, O. Wang.  The Unreasonable Effectiveness of   
Deep Features as a Perceptual Metric. In CVPR, 2018.
[2] Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kidder, Anna Corriveau, Maryam Vaziri-Pashkam, Chris I Baker (2023) THINGS-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior eLife 12:e82580.
[3] Fu, S., Tamir, N., Sundaram, S., Chai, L., Zhang, R., Dekel, T., & Isola, P. (2023). DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data. NeurIPS (2023).
strength_weakness: strength_weakness: This paper examines a meaningful, significant question about human visual object recognition, with interesting parallels to existing work on machine learning on model-human representational alignment. It aligns well with the goals of the workshop and should spark interesting discussions between cognitive scientists and machine learning researchers.
title: title: Solid experiments supporting a hypothesis for ventral visual stream function; clarity could be improved
main_review: main_review: Clarity & correctness: The point of the paper and writing style are clear. Experiments are mostly described thoroughly, though some clarifications could be made in the text to improve paper quality:
1. Training details: What is the 70% accuracy threshold evaluated over (and why not train for n epochs or until val accuracy saturates)? What is the train/val/test split?
2. What is the "classical multidimensional scaling" used to visualize the distance matrix?
3. Why learn five dimensions for the human representation? If this is justified in Jamieson et al. or Sievert et al., it would still be useful to briefly justify the low dimensionality. Experimentally, learning a >5 dimensional embedding and showing that it does not encode much additional variance would be quite convincing.
4. My understanding of Procrustes analysis is that it can compare two shapes (like distributions) by performing a set of transformations. I may just not be familiar with the correlation part of the analysis, but it is not clear to me what a correlation coefficient has to do with the shape-transformation angle that Procrustes analysis usually takes. It would be helpful to provide a citation or more rigorous formulation for how this correlation is calculated.
5. While I appreciate the citations pointing towards one hypothesis or the other, it is not clear to me that these two hypotheses are the main camps that are at odds with each other. Are there no other major theories sharing this problem space?
6. There are some typos and grammatical errors, but they are not too distracting - I would still suggest at least doing a pass to smooth over spellings and sentence readability. 

The experimental design seems sound to me. Some design decisions need to be explained (see above) for thoroughness.

Novelty & interest to community: This paper provides a new way to computationally support a hypothesis for the function of the ventral visual pathway. Perhaps the learned embeddings themselves can be useful to the ML/cog-sci communities as well as the conclusions drawn from the work.
strength_weakness: strength_weakness: As detailed in the review, this paper meets the criteria for the workshop.
title: title: Official Review of Submission73 by Reviewer 17rU
main_review: main_review: **Summary:** \
In the study, the authors have created a novel dataset (ECOSET-86; 86 classes and 333,000 images), being the intersection of the widely used ECOSET and the lesser-known Leuven Norms. A less-known dataset consisted of semantic feature vectors (2057 elements) for each image. This dataset is used to train two AlexNet Models (Cat-AlexNet; Sem-AlexNet) with which the authors aim to test whether humans utilize objects/categories or semantic features during visual perception. First, the authors demonstrate that each network learns unique representational geometries, making them suitable models to test the hypothesis. This is followed by testing the alignment of these models with human object recognition behavior in the form of a triadic comparison task. Their results show that Sem-AlexNet seems to be more aligned with human behavioral judgments indicating that visual perception

---

**Clarity:** good \
**Correctness:** good \
**Novelty:** good \
**Interest to the community:** good 

---

**Strong points:** 
- **Well-written manuscript** (Some typos and missing words)
- **Good clarity:** The novel dataset is well explained and seems to be a great addition to the field to test the hypothesis. The models are also well described (although there seems to be a typo in the supplementary figure stating the SEM-AlexNet model). The used methods seem appropriate, well described, and could be replicated. The results are stated clearly, offer new insights, and are interpreted and discussed in a balanced way. 
- **Good Visualizations**: The Figures in the paper clearly state the results, seem to be correct, and support the findings as well.  
The research done in this paper is relevant and at the core of the workshop. The dataset seems to be an interesting venue to test human-AI alignment. 

**Weak points:**
- The supplementary figure 3 for the Sem-AlexNet Architecture seems to have a typo as it states out_features=86 (the same as the object classes in ECOSET-86) but it should be 2057 as this is the dimension of the semantic feature vector. 
- The authors test for the effect of dataset diversity in the training set by also including AlexNets trained on the full ECOSET and ImageNet. They do not control the number of classes that they use. While indeed ECOSET-86 has way fewer images and SEM-AlexNet an even slightly higher correlation with human behavior this might be because SEM-AlexNet has the highest number of output classes (2057); Cat-AlexNet (86); ECOSET (353) and ImageNet(1k classes). I think this would be a great addition to the paper to test the effect of the number of output classes for the SEM-AlexNet to ensure that the effect comes from semantic information and not just the amount of output classes.

---

**Questions:**

Question 1: Can you ensure that the different model representational geometries go back to the semantic information and are not just a result of a more diverse (larger number) of output classes? \
Question 2: Is the number of output classes wrongly stated in the supplementary Figure of SEM-AlexNet?
strength_weakness: strength_weakness: Reason 1: Well-written manuscript, using well-established methods to answer and relevant questions to the community. \
Reason 2: The novel dataset seems to be an interesting new venue to test human-AI alignment for visual perception beyond object labels which can benefit the community.