Connecting Concept Convexity and Human-Machine Alignment in Deep Neural Networks

Authors: teresa  dorszewski
lenka  tětková
lorenz  linhardt
lars kai hansen
Venue: NLDL_org_2025_Conference
Published: 2024-09-05 11:52:14

Abstract: Understanding how neural networks align with human cognitive processes is a crucial step toward developing more interpretable and reliable AI systems. Motivated by theories of human cognition, this study examines the relationship between convexity in neural network representations and human-machine alignment based on behavioral data. We identify a correlation between these two dimensions in pretrained and fine-tuned vision transformer models. Our findings suggest the convex regions formed in latent spaces of neural networks to some extent align with human-defined categories and reflect the similarity relations humans use in cognitive tasks. While optimizing for alignment generally enhances convexity, increasing convexity through fine-tuning yields inconsistent effects on alignment, which suggests a complex relationship between the two. This study presents a first step toward understanding the relationship between the convexity of latent representations and human-machine alignment.

Main paper: 
 introduction: 
 As machine learning models are increasingly integrated into various aspects of daily life, understanding and improving human-machine alignment holds the promise of a safe, reliable and fair deployment of these technologies, ensuring that they operate in ways that are ethical, transparent, and beneficial to society [1] . Subsequently, we see a rising academic interest in the alignment of humans and machine learning models [2] [3] [4] [5] [6] [7] , as well as the agreement between different alignment measures [8, 9] . This paper explores the connection between two quantities characterizing the structure of neural network representations inspired by cognitive science: a theory-based measure of convex regions in neural networks [10] , inspired by Gärdenfors' theory of conceptual spaces [11] , and an empirical measure of human-machine alignment based on a triplet odd-one-out task [7] .Gärdenfors' theory posits that human cognition can be understood in terms of geometric struc-Figure 1 . Toy example with four potential cases of alignment vs. convexity in representation space. Apples and peaches (or cars and airplanes) should have high cosine similarity to achieve high alignment with humans. All objects should be neighboring objects (gray connection) of the same type for high convexity. In principle, high convexity does not necessarily imply high alignment and vice versa. We investigate which of the scenarios best reflects the representation structure of trained neural networks.tures, where concepts are represented as convex regions. Convexity is argued to assist generalization and communication of representations between humans [11] . Tětková et al. [10] introduced a framework to measure the convexity of conceptual regions in neural networks and discovered pervasive convexity across many data domains and models. They find that greater convexity leads to better generalization, which aligns with human cognition theories. In this work, we ask the fundamental question of whether these convex regions align with concepts used in human decision-making.Human-machine alignment can be measured in several ways [8] , one of which is by comparison with humans' relative similarity judgments (e.g. [4, 7] ). This may involve an odd-one-out image triplet task ("Which of three images is the most different?") using model representations and comparing the outcomes with the response of humans on the same task [7] . We chose this alignment metric as it can be easily evaluated for any neural network and additionally, methods for increasing this form of alignment exist [12] .We experimentally investigate the relationship between convex conceptual regions in neural network representations and alignment with human similarity judgments. We address two primary research questions:(1) Correlation between Convexity and Human-Machine Alignment: To what extent are convexity and human-machine alignment correlated in real-world models? Figure 1 illustrates potential scenarios of high and low levels of convexity and alignment. In particular, high convexity does not necessarily imply high alignment or vice versa. Yet, in non-pathological real-world scenarios, there may be a relationship between alignment and convexity. We perform extensive experiments suggesting that these two quantities are indeed correlated.(2) Impact of Improving Human-Machine Alignment on Convexity: Does increasing human-machine alignment lead to increased convexity? To understand whether the connection between alignment and convexity is causal, we increase human-machine alignment by applying the latent space transformations developed by Muttenthaler et al. [7, 12] and examine the effect on the convexity of conceptual regions. Additionally, we look at the effect of fine-tuning on convexity and alignment. Our findings indicate a complex relationship, showing an increase in convexity when optimizing pretrained models for alignment, but inconsistent effects on alignment during fine-tuning.The motivation of this work is two-fold. First, both convexity and the investigated measure of human-machine alignment are proposed as indicators of human-likeness in neural network representations. If these measures were to exhibit a strong positive correlation, it could suggest a shared underlying phenomenon that needs to be better understood. This, in turn, could provide valuable insights into how neural networks represent inputs, which has practical implications for fields such as mechanistic interpretability, with the goal of enhancing neural network robustness and safety. Conversely, if convexity and the alignment measure show no or a negative correlation, it might indicate that one of these measures is a flawed indicator of humanlikeness.Second, both convexity and the alignment measure have been shown to improve task performance in certain contexts [10, 12] . Examining the interaction between these two measures could reveal whether their performance improvements can be expected to be cumulative -suggesting that networks should be optimized for both -or whether they stem from the same underlying mechanism. 
background: 
 In this work, we investigate the connection of two already developed measures, namely the graph convexity score [10] and the odd-one-out accuracy [7] .The graph convexity score measures the convexity of concepts in latent spaces of neural networks [10] . It extends the common definition of convexity for Euclidean spaces to curved manifolds and utilizes sampled data. Graph convexity is defined as follows [10, 13] :Definition 1 (Graph Convexity). Let (V, E) be a graph and A ⊆ V . A is convex if for all pairs x, y ∈ A, there exists a shortest pathP = (x = v 0 , v 1 , v 2 , . . . , v n-1 , y = v n ) and ∀i ∈ {0, . . . , n} : v i ∈ A.The graph convexity score measures the ratio of points within the same class that constitute the shortest path connecting any two points of the same class. It is constructed in the following way: first, we extract the latent representations of all data points in a given layer (V ). Then, we construct a nearest neighbor graph (with N=10) with the Euclidean distance measure. Next, for each pair of data points that belong to the same class (x, y ∈ A), we find the shortest path (P ) in the neighbor graph and calculate the proportion of points along these paths that belong to the same class. The average of these proportions across all pairs is called the graph convexity score. See Appendix A.2 for a pseudocode procedure for calculating the graph convexity score.As a measure of human-machine alignment, we use the triplet odd-one-out accuracy (OOOA) [7] . The underlying triplet task is based on the THINGS dataset [14] , which contains images of natural objects. From this dataset, image triplets have been sampled, and human judgments of which of each triplet's images is the odd one out have been collected [15] . The alignment between humans and a neural network on the THINGS triplet task is measured by how well the human-designated odd one out can be directly identified using cosine-similarity of the images in latent space. For this, we first construct a similarity matrix S ∈ R 3×3 where S i,j := x ⊤ i x j / ∥x i ∥ 2 ∥x j ∥ 2 , the cosine similarity between a pair of representations, for the representations x 1 , x 2 and x 3 of images of a given triplet. We center all representations by subtracting the mean representation over the THINGS dataset and divide them by the standard deviation. We identify the closest pair of images in the triplet as arg max i,i<j S i,j .The remaining image is the odd one out. The oddone-out accuracy is defined as the proportion of matching responses between humans and the model. Important to note is that the observed agreement between humans is 67.22% and this value thus upper bounds the expected OOOA values [15] .To improve human-machine alignment (i.e. OOOA), we use the naive transform defined by Muttenthaler et al. [12] . It affinely transforms the latent space to maximize the alignment between human similarity judgments and the network's representations. This transformation consists of a square matrix W and bias b obtained as the solution to arg minEQUATIONwhereZ ij = (W x i + b) ⊤ (W x j + b)where {i, j} index any pair of representations in the dataset. The log-likelihood is defined as [16] :EQUATIONHere, n is the number of triplets, and the probability of a pair, {a, b} of the triplet {i, j, k}, being most similar, is given by the softmax over the representation similarities in Z.3 Methods 
data: 
 For all experiments, we used the THINGS dataset [14] and its attached human triplet responses [15] . The THINGS dataset consists of over 26.000 images of 1854 classes of natural objects, which can be grouped into 27 high-level categories (determined by human crowd-sourcing), which we will refer to as superclasses. The dataset features over 4.7 million human triplet odd-one-out responses, based on one image per class. We used the same images chosen for the triplet experiment to determine the OOOA and 100 randomly chosen images per superclass from the THINGS dataset to determine the graph convexity score.Although convexity and human-machine alignment are domain-independent concepts, we restricted ourselves to vision models, as the THINGS dataset used to measure alignment in prior work [7, 12, 17] consists of images. We compared three different transformer-based vision models: Vision Transformers (ViT) [18] , Bidirectional Encoder representation from Image Transformers (BEiT) [19] , and data2vec [20] . For each model, we compared a base and a large architecture and both in their pretrained and fine-tuned (for ImageNet-1k [21] classification) version. 1 The models consist of a feature extractor followed by 12 or 24 transformer layers [22] with an embedding size of 768 or 1024 respectively. While ViTs were pretrained for classification, BEiT and data2vec were pretrained using a self-supervised objective, where the goal was to predict masked-out input (data2vec) or representation tokens (BEiT). All models were pretrained on ImageNet-21k [21] .Correlation between Convexity and Human-Machine Alignment. To answer whether convexity and human-machine alignment are related, we perform a correlation analysis between the graph convexity and the odd-one-out-accuracy (OOOA). For this, we extracted the latent representations of the images used for the triplet after each transformer layer and measured the OOOA of the centered representations (as described in subsection 2.2). For the convexity analysis, we extracted the latent representations of 100 images for each superclass of the THINGS dataset after each transformer layer. We averaged the convexity scores of all classes to get one score per layer. We performed a correlation analysis between the two scores using Pearson's R on a layer-wise basis across all models.Impact of Improving Human-Machine Alignment on Convexity. After addressing the correlation between the human-machine alignment and the convexity score, we ask how the optimization of one score will influence the other. To answer this question, we trained the naive transform described in subsection 2.2 to increase the OOOA of the latent representations after the first, middle, and last transformer layers. We then applied the transform to the latent representations of the remaining images and evaluated the graph convexity of these transformed latent representations. Furthermore, we examined the impact of fine-tuning on both the convexity and OOOA. 
correlation between convexity and human-machine alignment: 
 First, we examine the progression of the convexity score and the OOOA across layers of different models and investigate the correlation between the two. The graph convexity increases consistently for the first half of all evaluated models and then continues to increase for fine-tuned models but plateaus or Correlation between convexity and human-machine alignment for ViT [18] , BEiT [19] and data2vec [20] (all in base and large architecture, pretrained and fine-tuned (ft)) across all layers: a) Graph convexity of the THINGS superclasses across all models. Convexity increases steadily for fine-tuned models and peaks in the middle layers for pretrained models. Dotted line indicates the lower bound (random labeling). b) Human-machine alignment measured by OOOA across all models. OOOA peaks in the middle layers. Dotted lines indicate the lower bound (chance level) and upper bound (inter-human consistency level). c) Correlation between convexity and OOOA for pretrained and fine-tuned models respectively. The correlation is calculated across all models per layer (for large models we take every second layer, corresponding to the same relative position in the network as in the small networks). There is a strong positive correlation in the first half of the model. Fine-tuned models show a strong negative correlation in late layers.decreases for pretrained models (Figure 2a ). The convexity is generally higher in fine-tuned models, consistent with prior work [10] . The bell curve pattern observed in pretrained unsupervised models (BEiT and data2vec) may be attributed to their training objective, which employs a reconstruction loss. As the models learn to rebuild the masked input data, the later layers increasingly build latent representations akin to those in the earlier layers. Thus, intermediate layers may be the ones with the highest level of conceptual abstraction. A similar phenomenon has been observed previously in speech representation models [23] [24] [25] .The OOOA also follows a bell-shaped curve, peaking around the middle layers of the model and decreasing again toward the end (Figure 2b ). Current human-machine alignment analyses focus mainly on the last or penultimate layer, as this bell curve was not observed in the supervised and contrastive models examined in previous work [7] . Our results suggest that the intermediate layers of the studied vision transformer models are better aligned with the human similarity space than the last layers. This is in line with observations on denoising diffusion models [17] and should motivate a careful approach for layer selection in future research on the alignment of neural networks. In contrast to our results on convexity, the bell curve can be observed for alignment in pretrained and fine-tuned models alike, albeit the latter generally achieve a higher maximum OOOA. This indicates that fine-tuning affects OOOA and convexity differently.The correlation between the OOOA and convexity follows two main trends (see Figure 2c ). In the first half of the models, both the convexity and the OOOA increase and are highly correlated, whereas the correlation decreases in the second half of the models. There is a considerable difference between pretrained and fine-tuned models in the later layers: for pretrained models, there is a mostly positive correlation that increases again in the last layers, while there is a strong negative correlation in late layers for fine-tuned models.In general, we observe a high correlation between human-machine alignment (OOOA) and the convexity of the 27 high-level concepts in many scenarios. This suggests that the networks form convex regions in their latent spaces that align with human-defined categories, which is also consistent with the similarity relations humans use to solve the triplet task. Especially in the first half of the models, which we interpret as the concept built-up phase, the correlation is strong. For the later layers, which can be viewed as processing the built-up concepts, e.g. for the purpose of classification, the correlation decreases and even becomes negative. This potentially indicates that the models initially learn to distinguish concepts, where human-like categorization and similarity judgments are beneficial, especially in unsupervised settings. In the subsequent phase and during fine-tuning, the similarity between classes seems to become less critical. For reconstructionbased models, the reason may be that high-level information captured during the first phase may be decomposed into low-level features in later layers to solve the task (i.e. pretraining objective). For classification models, the separation of classes for classification might not necessitate retaining the similarity structure across concepts.We investigate how increasing human-machine alignment influences convexity. Using the naive transform [12] , the OOOA of all models improves significantly -on average by 13.7% for pretrained models and by 13.0% for fine-tuned models. We find that improving the OOOA also leads to higher convexity of representations of the THINGS superclasses in pretrained models; the convexity of the transformed latent representations increases for all but one model by on average 3.1% (see Table 1 ). This relation is strongest in the last layers, but also holds in other layers of the models (see Table A .2 and A.3 in the appendix).For fine-tuned models, this relation does not hold consistently (see Table A .4-A.5 in the appendix). For most models, convexity decreases marginally, which is in line with the negative correlation scores in the late layers. We hypothesize that the training strategy (supervised vs. self-supervised) also has an impact on the effect of alignment optimization on convexity (preliminary results in Appendix A.4).Increasing convexity does not necessarily lead to higher alignment, as observed in the fine-tuned models. Although classification performance and convexity increase during fine-tuning, this does not have a consistent effect on human-machine alignment. While fine-tuning increases OOOA for ViT, it de- +26.9 ± 17.3 +0.58 ± 6.9creases OOOA for data2vec and has only a marginal impact on OOOA for BEiT (see Table 2 ). This indicates a complex interplay between alignment and convexity, which potentially depends on factors such as training objectives as we also observed above.Another promising direction for future research is to develop methods to increase convexity (and performance) that also consider human-machine alignment. Positive relations between human-machine alignment and performance/robustness have been previously found [12, 26] , although alignment is not a sufficient condition for high performance and optimizing solely for alignment can worsen performance by unlearning properties of representations that are not necessary to achieve high OOOA. 
conclusion: 
 We presented first evidence of a relationship between convexity in neural network representations and empirical human-machine alignment. Our findings indicate a significant correlation between these two measures in some scenarios and that the intervention to promote alignment can also increase convexity. The correlation suggests that neural networks form convex regions in their latent spaces that to some extent align with human-defined categories and reflect the similarity relations humans use in tasks such as the odd-one-out triplet task. In the studied models, we find the highest human-machine alignment in the middle layers, which could help inform future research.We observed that optimizing for human-machine alignment through latent space transformations not only increases the odd-one-out accuracy (OOOA) but can also increase the convexity of the representations. This intervention effect underscores the potential for optimizing models to be both more aligned with human cognition and more effective in their performance.Our results highlight that in the higher layers, convexity and alignment can exhibit a strong positive or a strong negative correlation. This indicates a complex interplay between alignment and convexity that may hinge on factors such as training objective, architecture, or training data. Further research is warranted to explore under which conditions convexity and human-machine alignment align.Overall, our study offers first insights into the relationship between the convexity of latent representations and their alignment with human similarity judgments, further connecting cognitive science and deep learning research. Future work should investigate the causal relationships between alignment and convexity and explore new techniques to enhance both simultaneously, potentially leading to more aligned and generalizing models. gLocal and https://vicco-group.github.io/ thingsvision/Alignment.html.All investigated models were obtained from huggingface.co, the exact models are in Table A .1.We extract the representations after each transformer block, which have the dimensions of 768×197 for base models and 1024×197 for large models. For ViT we use the first embedding patch (the cls token, corresponding to the first row of length 768/1024), which represents the class token of the embedding, which is later used by the classifier head. For the other models, we average across the rest of the patch embeddings (resulting in a vector of length 768), as this is used by the classifier head. We observe a high correlation for the first half of all models (R=0.91), while the correlation is lower for the second half of pretrained models (R=0.4) and even negative for fine-tuned models (R=-0.54). 
 
 Appendix: Table A .2 and Table A .3 show the change in OOOA and convexity after optimizing the latent space for alignment (OOOA) using the naive transform [12] . The improvement in convexity after the first and middle transformer block is not as high as after the last transformer block (see Table 1 ), but still significant, indicating a causal relation between alignment and convexity.Table A .4-A.6 show the change in convexity after optimizing for OOOA. The positive relation we observed for pretrained models does not hold in this case. For most models, convexity slightly decreases in the middle and last layer while convexity increases in the first layer, which aligns with the negative correlation scores observed in Figure A.1 for late layers in fine-tuned models and positive correlation in early layers.Since the two scores explored in this paper use different distance metrics to assess the similarity between representations (Euclidean distance in graph convexity and cosine distance in OOOA), we explore whether this discrepancy could influence the results. The correlation coefficient between graph convexity using Euclidean distance and cosine distance is 0.9931 (across all models and layers used in this paper) and 0.9989 for OOOA. Therefore, the influence is negligible.When comparing the correlations between convexity and alignment on a model basis, there is a notable difference between self-supervised models (BEiT, data2vec) and supervised models (ViT, all fine-tuned models). Self-supervised models show a very high correlation, which decreases drastically with finetuning (see Table A .7), while the correlation is lower but stable for the supervised model (ViT). While this hints at a complex relationship, a large-scale study is needed to provide clear conclusions on the role of the objective function. Generally, we hypothesize that there are multiple factors that could potentially mediate the relationship of human-machine alignment convexity. While we can currently only offer speculations, we hope to provide directions for future work. Pretraining objective: It has been suggested in prior work that training objective plays a significant role in determining the human-machine alignment of neural network [7] . A larger-scale study would be necessary to systematically gauge how the training objective influences the relationship between convexity and alignment. We hypothesize that at least the layer where each metric peaks could be affected. Models trained with multi-modal objectives would be another interesting model type to explore, since textual supervision may encourage the formation of conceptual regions. Fine-tuning: Fine-tuning is of high practical relevance, given the proliferation of publicly available foundation models. The effect of fine-tuning specifically on human alignment and convexity metrics is as of yet underexplored. Potentially, fine-tuning a model pretrained on a self-supervised task using a supervised objective could exhibit properties on the interpolation between the two. Weight initialization: While prior work on alignment and convexity analyzes trends across larger model sets (e.g. [7, 10] ), the extent to which parameter initialization affects the learned representations' alignment or convexity has not been determined and could be an interesting direction of future research. Training dataset size: All models evaluated in this work have been trained on ImageNet-1k or ImageNet-21k. It may be that the size or diversity of these datasets is not large enough to enable the accurate learning of human-like concepts. Future work could provide an analysis of the effect of training dataset size on the relationship of convexity and alignment to elucidate whether ImageNet variants are sufficient to properly learn the evaluated concepts. Overall, we believe that our work provides a promising starting point for future lines of investigation that may lead to a more integrated view on the human-likeness of neural network representations. Proceedings of the 6th Northern Lights Deep Learning Conference (NLDL), PMLR 265, 2025. 2025 Teresa Dorszewski, Lenka Tětková, Lorenz Linhardt, & Lars Kai Hansen. This is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/).All models were obtained from huggingface.co, for exact models see TableA.1). 


Decision: The paper examines the relationship between convexity in the representations produced by neural networks and human-machine alignment. They establish a correlation in real-world models between these quantities. Additionally, they find that convexity tend to increase when optimizing for alignment. 

The reviewers agree that the topic of the paper is interesting and provides insights worth publication. Given the nature of the reviews and that the authors have improved the paper by addressing the concerns raised by the reviewers during the rebuttal phase, I recommend accepting the paper.

Reviews: review: title: title: Interesting unusual metrics with potential benefits
paper_summary: paper_summary: The paper describes a novel approach of measuring convexity in neural networks and relating it to the alignment with humans, by measuring the performance on the OOOA (odd one out task). The study demonstrates that convexity in neural networks seem to be related with human alignment in the way information is organized in the latent space. They show that optimizing for OOOA task have a positive effect in convexity in Neural networks.
main_review: main_review: Given the positive reviews that other reviewers have raised. I upgraded my score.
strength_weakness: strength_weakness: I might have missed it, but I feel that while the concept of convexity is nicely introduced,  a more clear understanding on why is it desirable is missing? What do we get out models that are more convex? Are they more interpretable?  The authors mention briefly in the intro that convex systems are more robust, but there is no evidence presented in the paper about this. Could a robust study be included to show how the increased convexity helped the models after the alignment was provided? 

Also not sure why there are no mention on any convolutional models that have been shown more aligned with humans in terms of architecture.
title: title: Connecting Concept Convexity and Human-Machine Alignment in Deep Neural Networks
paper_summary: paper_summary: The paper provides insights into how convexity in latent representations and human-machine alignment are connected. It looks at convexity scores and alignment for 3 different transformer-based networks at different stages/layers using both simply pretrained and fine-tuned versions, as well as how directly improving human-machine alignment via a dedicated transform affects the convexity. 
While the results are not entirely consistent across settings, they identify some positive correlation between alignment and convexity and  a potential causal relationship between them.
main_review: main_review: The authors have further improved the paper according to the reviews. All the comments point to the fact that while the experiments (or argumentation for some of the choices made) are not the strongest, the presented idea and insights are interesting for the wider community.
strength_weakness: strength_weakness: A few points I would like to be adressed for clarity:
- what is meant by 'centered representations' in 3.3?
- You mention that the convexity and OOOA follow bell-shaped curves. However there are sudden increases again at the last layers for some of the methods. Any comments on that?
- Figure 2C has inadequate caption/description: For which network is that plotted? Since they don't all have the same amount of layers... Is it averaged?
- seems that convexity changes most (for the best) at transformations at first layer, while last layer is more important for OOOA. Any thought on why that could be?
- I find the part on confounding factors (eg. in A.3) lacking/not at all worked out. I would expect you to at least speculate about the topic and further research directions here, instead of simply mentioning the results. After all, this paper is not a method contribution paper, but trying to look under the hood and explain or at least indicate the underlying relationships.

- I'm not entirely convinced the findings on pretrained vs finetuned are generalizable; have you looked at the specific tasks (both for finetuning and pretraining) and how well they resemble the alignment task? The potential problem I see here is that this task for  measuring alignment is biased towards models trained for similar tasks (eg classification of natural images). I would thus be careful when using its results in arguments comparing different networks that were trained for sufficiently different tasks. 



Some smaller comments that would further improve the paper:
- you only provide a definition of graph convexity, then describe the convexity score inside text. I would recommend you provide even a formal definition of the score in addition to this. 
- legend missing in Figure A1
- more networks and specifically different tasks would be needed to be able to evaluate properly and in a more generalizable way, although I suppose the scope of this paper was really just to present the idea and not a thorough evaluation.
- mention the existence of additional results (specifically the results from other than last layers)  that are in the supplementary clearly in main text
title: title: Review
paper_summary: paper_summary: The topic is representation learning. In particular, the paper studies graph convexity and human-machine alignment of the representations learned by neural networks trained either self-supervised or supervised on ImageNet-data. Graph convexity is measured for each class in the THINGS dataset. For human-machine alignment, triplet odd-one-out-accuracy (OOOA) on the THINGS dataset is used. The paper finds that human-machine alignment is typically highest in middle layers of networks, while graph convexity typically increases monotonically with deeper layers. In the authors' words, correlation scores of the two measures over several networks "indicate a significant correlation between these two measures in some scenarios". The paper also studies what happens to the graph convexity when optimizing an affine transformation to increase the OOOA. The graph convexity increases for some networks and decreases for others. The scale of the experiments (12 networks) is ultimately too small to say anything conclusively about the relationship between OOOA and graph convexity. As the authors note, "Further research is warranted to explore under which conditions convexity and human-machine alignment align.".
main_review: main_review: The authors have addressed the concerns of all the reviewers. The paper contains insights worth discussing at the conference, although the experiments are perhaps not large-scale enough to draw far-reaching conclusions.
strength_weakness: strength_weakness: - It is not clear why the correlation between graph convexity and OOOA is studied. Is the aim to test Gärdenfors' theory by arguing that if human concepts are convex in neural network latent spaces, then human concepts are convex? If so, what are the conclusions from this study?
- The experiments are too small-scale to give any conclusive results. 
- The paper divides the studied networks into "pretrained" and "finetuned", but the networks were pretrained in different ways. As noted in Appendix A.3, perhaps the more important division is self-supervised vs supervised. Again, the number of studied networks is too small to determine this.
- For graph convexity, Euclidean distance is used to build the graph. However, cosine similarity is used to measure similarity in the OOOA measure. This means that if a class has embeddings close to zero, it can be graph convex while having embeddings in all directions from zero, leading automatically to low OOOA. Do the conclusions from the experiments change if the same similarity is used for both measures? One could switch to either use cosine similarity to build the graph for measuring graph convexity or use Euclidean distance to measure similarity in the OOOA.
title: title: Review of "Connecting Concept Convexity and Human-Machine Alignment in Deep Neural Networks"
paper_summary: paper_summary: The authors empirically study, for various transformer-based vision models, the relationship between the two concepts _human-machine alignment_ and _concept convexity_. For both of these concepts they evaluate an existing proxy/metric, namely _graph convexity_ and the _odd one out accuracy_, and evaluate these on the latent representations of pictures from based classes after various transformer layers in the model. The authors mention that this study provides the first evidence of a relationship between the two concepts. Although it is difficult to draw conclusions from correlation coefficients, the authors also performed an intervention study where they optimize for odd-one-out-accuracy to see the influence on convexity, showing that especially for models that are not finetuned, also the convexity increases.
main_review: main_review: The high-level story is interesting to read and it addresses a concrete version of the interesting question how human-machine alignment and latent representations in machine-learning models relate. I also think it is nice that the experiments have been done and that anybody can look at the numbers and ponder about what they mean. That also immediately gives the counterside: it is a a bit difficult to interpret back the results in terms of the higher level question. 

I also think the authors responded well to the reviews, by incorporating concerns in the text.
strength_weakness: strength_weakness: In contrast to the larger story, the details were more difficult to figure out for me, and I am still uncomfortable pointing out exactly how various quantities are computed. The authors mention for instance "We performed a correlation analysis of the two scores using Pearson’s R on a layer-wise basis across all models." It was originally unclear to me whether there the number of datapoints in the computation of such a correlation would be just be the twelve averaged scores corresponding to the twelve trained models, or whether there would be a data point for every triplet of classes, for instance. A formula might help, or an even more precise wording.

Similarly, the appendix mentions "The latent representations were extracted according to the way they were used by the classifier in the original implementation. Hence, we took the vector corresponding to the classifier token for ViT and averaged over all the other tokens for data2vec and BEiT." I have difficulty understanding what this means. In general, I am unsure how and where the latent representation of a class is actually computed. This is relevant for the interpretation of the convexity concept.

Although not necessarily a weakness of the paper, it is in a way difficult to connect the proxy to the intended concept, i.e. graph convexity to a measure of convexity, and out-of-order-alignment with human-machine alignment. For instance, whereas the authors in the conclusion talk about the formation of convex regions, it is not necessary for achieving a high correlation score that latent representations of a certain class form a mathematically convex set. On a related, note, the mathematical concept of convexity would not be affected by an affine transformation, while the graph convexity does, because of the construction of the nearest-neighbor graph. The article might benefit from a discussion on these issues.

More detailed remark:
I am wondering why the vectors appearing in the inner product in the definition of Z are not normalized.