Decoupling Semantic Similarity from Spatial Alignment for Neural Networks.

Authors: tassilo  wald
constantin  ulrich
gregor  köhler
david  zimmerer
stefan  denner
michael  baumgartner
fabian  isensee
priyank  jaini
klaus h maier-hein
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 18:56:04

Abstract: What representation do deep neural networks learn? How similar are images to each other for neural networks? Despite the overwhelming success of deep learning methods key questions about their internal workings still remain largely unanswered, due to their internal high dimensionality and complexity. To address this, one approach is to measure the similarity of activation responses to various inputs.
Representational Similarity Matrices (RSMs) distill this similarity into scalar values for each input pair.
These matrices encapsulate the entire similarity structure of a system, indicating which input lead to similar responses.
While the similarity between images is ambiguous, we argue that the spatial location of semantic objects does neither influence human perception nor deep learning classifiers. Thus this should be reflected in the definition of similarity between image responses for computer vision systems. Revisiting the established similarity calculations for RSMs we expose their sensitivity to spatial alignment. In this paper we propose to solve this through _semantic RSMs_, which are invariant to spatial permutation. We measure semantic similarity between input responses by formulating it as a set-matching problem. Further, we quantify the superiority of  _semantic_ RSMs over _spatio-semantic_ RSMs through image retrieval and by comparing the similarity between representations to the similarity between predicted class probabilities.

Main paper: 
 introduction: 
 Deep neural networks are trained to extract powerful feature representations for a wide range of downstream tasks. Despite this, their inner workings are highly-complex, making understanding how networks solve tasks and what they learn challenging. To obtain a better understanding of these fundamental questions, researchers in the fields of neuroscience, cognitive science, and machine learning independently developed various methods to interpret and relate representations [29] . In the realm of machine learning, a lot of prior-methods have been proposed to meaningfully measure the similarity between intermediate representations of ANNs [15, 31, 25, 19, 12] . Klabunde et al. [10] provides a comprehensive summary and categorizes them into measures based on a) Canonical Correlation Analysis (CCA) [25, 19] b) Alignment measures c) Representational Similarity Matrices (RSMs) d) Nearest Neighbors e) Topologies and f) Descriptive Statistics. Of all these methods, Representational Similarity Matrix (RSM) [13] based measures have enjoyed the most attention over the last years. RSMs consist of sample-to-sample comparison, measuring the similarity between the (intermediate) responses of the same network to two different samples. Based on many such comparisons, the RSM represents the similarity structure of what a model considers similar. This representation of a system's behavior reduces the highly-dimensional, complex internal structure, that the model of interest may possess, to a N × N Matrix for N samples. It enables the comparison of the similarity structure of any system, as long as one can input the same samples and measure the similarity between the responses. In the machine learning domain, this concept was introduced by Kornblith et al. [12] in conjunction with Centered Kernel Alignment (CKA) to compare the similarity between RSMs of different layers within a model or across models. CKA superseded previously popular Canonical Correlation Analysis (CCA) metrics [25, 19] , since they need a vast amount of samples to measure similarity. Consequently, CKA was used in various applications, to measure the similarity between Transformers and CNNs [26] or wide and deep networks [21] and to understand catastrophic forgetting [27] or transfer learning [20] to name a few.In this paper, we revisit the key component of the most used similarity measure in the field: Representational Similarity Matrices and how they are constructed in the vision domain.The key contributions of our work are summarized as follows:• We highlight that current RSMs are constructed in a way that couples localization and semantic information, which constraints one only to measure similarity if, spatial and semantic information aligns between two samples. • To address this issue we propose semantic RSMs, which are invariant to spatial permutation and exclusively measure semantic similarity, by formulating it as a set-matching problem. • We show that the inter-sample similarity of semantic RSMs leads to improved retrieval performance and better reflects the similarity between representations of classifiers and their predictive behavior. • Moreover, due to the computational complexity of the proposed algorithm, we introduce approximations that significantly reduce computation time.The Code is available here. 
representational similarity: 
 Formalization To establish the concept of representational similarity in the context of computer vision, we provide a brief formalization of the problem. Let x ∈ {x 0 , . . . , x N } denote the input samples for which we collect input responses Z 1 ∈ {z 1,0 , . . . , z 1,N } and Z 2 , which we call representations. The representations can take different shapes, with Z CNN ∈ R N ×C×W ×H denoting the responses of a CNN with C channels and a spatial extent of W, H or Z ViT ∈ R N ×D×T denoting the responses of a ViT with depth D and tokens T . For the purpose of simplification and without loss of generality, we unify the spatial dimensions W, H for CNNs and T for ViTs into a joint spatial dimension S, resulting in Z ∈ R N ×C×S . Given representations Z, we can construct RSMs K, L ∈ R N ×N , with values K ij = k(z 1,i , z 1,j ) and L ij = l(z 2,i , z 2,j ) measuring how similar the representation z i is to z j given the kernels k or l. The kernels define the measure of similarity between the Our proposal focuses solely on measuring semantic similarity. We achieve this by determining the optimal permutation between two representations and introducing sample-wise permutation invariance.representation vectors and hence play an important role. We introduce an exemplary kernel in Section 3 and all kernels used in this paper with some properties in Appendix A.With the two RSMs K and L at hand, it is possible to compare the similarity structure between the two models. As introduced in Kornblith et al. [12] one can use the Hilbert-Schmidt independence criterion (HSIC) [5, 28] to calculate the level of independence through Centered Kernel Alignment, providing a measure of similarity of the two representations Z 1 and Z 2 with H denoting the centering matrix.EQUATIONCKA(K, L) = HSIC(K, L)EQUATIONAlternatively, a variety of different measures based on RSMs are possible for which we refer to Section 3.3 of Klabunde et al. [10] . As highlighted above, the similarity calculation based on RSMs is a two-step process with the first being the calculation of the RSMs and the latter being the comparison of the RSMs. In this paper, we focus on the first step, by quantifying the importance of disentangling semantic similarity from spatial alignment.While not the focus of this paper, we provide qualitative examples of the downstream effect on CKA measures in Appendix G.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: In Section 5, a comprehensive discussion of the results and their limitations, such as the runtime, is provided. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper. • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speechto-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
the semantic representational similarity matrix: 
 Representational Similarity Matrices (RSMs) are designed to reflect the system behavior of interest. The RSM K, originally introduced by Kriegeskorte et al. [13] , represents the similarity structure of a system given a set of inputs x ∈ {x 0 , . . . x N }. Each value K ij in K quantifies how similar the responses of two inputs z i and z j are to each other. The definition of what symmetries between representations similarity measures should be invariant to is a central point of debate. Previous work proposed permutation invariance [15] , invariance to orthogonal transformations [12] , or invariances to invertible linear transformations [25, 19] . While arguments for any of these invariances are valid, we believe that an important aspect has been neglected in the calculation of RSMs: The spatial alignment between the representations!The dependency on spatial alignment Revisiting the structure of representations of a CNN, channels C correspond to semantic concepts while the spatial position corresponds to where the semantic concept is localized in the input image [34] . Consequently, one can reformulate the representation z i of a sample x i to be fully defined by a set of semantic concept vectors v, one for each spatial location S: z i = {v 0 , . . . , v S } with v ∈ R C . In the case of linear CKA [12] , the RSMs are then calculated, between semantic concept vectors at the same spatial location. For instance, when employing the linear kernel K ij can be expressed as:EQUATIONThis formulation emphasizes the coupling of semantic similarity and localization during similarity calculation, which we term as spatio-semantic RSMs. This coupling can lead to issues, e.g. when comparing an image to a translated version of itself. Due to the quasi translation-equivariant nature of CNNs3 semantic vectors are translated similarly, changing the alignment of pairs of v, leading to a low perceived similarity despite highly similar semantic vectors. This issue is visualized in a small toy example in Fig. 1 3As shown above, current RSMs compare different input samples without accounting for the lack of spatial alignment. Previous work of Williams et al. [32] recognized this and introduced translation invariance to RSMs by finding the optimal translation a, b of the representations z ′ j = {v 0+a,0+b , . . . , v w+a,h+b } to maximize similarityK ij max a,b = ⟨z i , z ′ j ⟩ through circular shifts.While this is an improvement to no spatial alignment and emulates a CNN's inherent translation equivariance, we argue that the measure of representational similarity should not be constrained to what the underlying model is invariant to, but the similarity measure should be invariant to the possible spatial configurations of semantic features in the input image.To motivate this, we propose a thought experiment: Imagine we have trained a classifier with an augmentation pipeline including rotations. Given an image and a rotated version of the image, we extract representations z at layer i once for the normal z i and once for the rotated image z i,rot . Due to the initial rotation, these representations may differ in earlier layers i, due to the network extracting different edges and corners. However, if the network successfully learned to become invariant to the augmentation, it may have learned to map it to the same semantic vector v at a later layer but at a different spatial location. For such cases, we argue that the similarity between the two representations should be high. Should the model be sensitive to the rotation, no semantically similar representations may be expressed at a later layer, which should lead to a low similarity. This reasoning can be extended to all kinds of shifts, be they artificial augmentations like shearing or mirroring or natural variations of the input manifold. Subsequently, we argue that the similarity measure should be invariant to as many spatial shifts as possible. This alone allows one to measure the similarity of representations a model is invariant to, be these learned or designed invariances. Such variable shifts cannot be captured with simple translation operations.To impose as minimal constraints on spatial structure as possible, we propose to make K ij invariant to all spatial permutations of the semantic concept vectors v. Formalizing this we demand that the similarityK ij = k(z i , z j ) = k(z i , P ij z j ) with P ij ∈ R S×Sbeing a unique permutation matrix for the pair of z i and z j . To accomplish this, we propose to find the optimal permutation matrix P ij that maximizes the similarity K ij .EQUATIONTo find the optimal permutation matrix P ij , we decide to use the linear kernel ⟨•, •⟩ to maximize both the magnitude of activation and the direction of vectors, as both magnitude of activation and direction of the vectors matter [12] . This allows us to calculate an affinity matrix A ij ∈ R S×S measuring the similarity between all concept vectors:EQUATIONWith this affinity matrix, bipartite set-matching algorithms, such as Hungarian matching, can be employed to find the optimal permutation matrix P ij that maximizes the inner product between z i and z ′ j . Finding all P ij for all pairs i, j and applying the chosen kernel k yields the semantic RSM. This semantic RSM is invariant to any arbitrary, unique spatial 3The patch embedding can similarly translate semantics to a different position in the sequence. The results indicate that spatio-semantic RSMs struggle to detect largely identical but translated images, while semantic RSMs exhibit an enhanced off-diagonal in the RSMs and a significant gap between distributions. This demonstrates the capability of our method to detect the same semantics even when translated.permutation P ij for each pair of representations, and, depending on the choice of kernel k, invariant to orthogonal transformations U ∈ R C×C along the channel dimension. These semantic RSMs can be used as a drop-in replacement for any other RSM, e.g. for applications such as calculating CKA(K, L) to measure the similarity between systems.Computational Complexity Finding the optimal permutation matrix P ij is NP-hard and needs to be repeated for each pair of representations z i , z j . With N samples, this results in N •(N +1) 2 unique permutations that need to be computed for a semantic RSM. The overall complexity of bipartite matching algorithms grows with the spatial dimensions cubed, resulting in O(N 2 ) × O(S 3 ). The outer O(N 2 ) complexity can be parallelized, or reduced by decreasing the batch size. However, the inner permutation can become timeconsuming, especially with large spatial dimensionality. To address this, we provide various approximations to reduce the complexity, which are detailed in Section 4.4. For all later experiments, except the translational toy example, we use the Batch-Optimal approximation with windows size b 512, with the batch referring to batches of semantic concept vectors v and not samples. The pseudo-code for calculating semantic RSMs is visualized in the Appendix under Algorithm 1. 
experiments: semantic vs spatio-semantic rsms: 
 Given the novel permutation-invariant similarity definition, we evaluate the utility of our semantic RSMs relative to spatio-semantic RSMs for various similarity kernels, architectures, and tasks. Across all experiments we compare the linear kernel, the radial basis function (RBF) kernel, and the cosine similarity kernel, see Appendix A for details. Figure 3 : Relaxing the constraint of spatial alignment leads to better retrieval. We leverage general feature extractors to embed images of the EgoObjects dataset. We then compare these embeddings either with or without permutation invariance. PI: Permutation InvariantTo illustrate the problems of coupling semantic content and localization a toy dataset is created using 84 × 84 pixels large, downsampled images of ImageNet [2] . For each image two 64 × 64 crops are extracted, one from the upper-left and one from the lower-left corner, resulting in two images that share 44 × 64 identical pixels (Fig. 2 left ). Ten upper-left and ten lower-left crops are then used to extract representations of a ResNet18 [7] , which are subsequently used to calculate spatio-semantic and semantic RSMs at different layers of the architecture (Fig. 2 middle ). As kernel, we use the radial basis function, as it provides bounded similarity values allowing a better visualization.As expected, the spatio-semantic RSM measures low similarity between pairs of overlapping crops, due to the semantic concept vectors not aligning. Only in the last layer, after many pooling operations, the off-diagonal is slightly expressed. Conversely, our semantic RSM is capable of detecting the high semantic similarity of the partially overlapping crops throughout the entire depth of the architecture, as evident by the highly similar off-diagonal.Aggregating the similarity values between partially-overlapping and between different images across multiple batches, allows us to measure the distribution of similarity values between overlapping crops, and non-related image comparisons. Throughout the entire depth of the architecture, the similarity distributions show that our measure better separates overlapping images from different images. Notably, the similarity distribution in spatiosemantic RSMs shows a significant overlap of the distributions of partially overlapping images and non-related images, making differentiation between them difficult (Fig. 2 A similar toy experiment for a ViT-B/16 [4] , is provided in Appendix B.To test the impact of the semantic RSMs in real-world applications, we now investigate the common task of image retrieval. Each entry in an RSM quantifies a sample-to-sample similarity value, which can be directly used for retrieval. While not specifically designed for it, we argue that better retrieval performance reflects a better inter-sample similarity. This allows us to quantify improvements in the RSM structure. To measure retrieval performance the EgoObjects dataset [35] is used. It contains frames of video that capture the same scene from different viewing perspectives and lighting conditions. This results in object centers being distributed across the extent of the image. By randomly sampling 2000 query images and 5000 database images from the test set and using general feature extractors to extract embeddings from them we construct RSMs that allow us to do retrieval. As feature extractors we use CLIP (ViT/B32) [24] , CLIPSeg (Rd64) [16] , DinoV2-Giant [22] , SAM (ViT/B32) [9] and BIT-50 [11] and as kernels for similarity calculation we use the cosine similarity, RBF and the inner product. For all RSMs, we retrieve the most similar image that is not part of the same video -the same scene but different conditions are allowed. As multiple objects can be present in each scene, we quantify retrieval performance by the F1-Score, measuring the overlap of annotated objects between images. Due to the rather complex dataset, we elaborate this in more detail in Appendix D.1.Across all architectures and metrics, the inclusion of permutation invariance (PI) for the similarity calculation improves retrieval performance relative to the non-invariant similarity, in some cases with a dramatic difference in performance, see Fig. 3 . For models designed for dense downstream tasks like SAM or CLIPSeg, the retrieval performance changes particularly much, while models with more global reasoning, like BiT improve less, relatively.Qualitative Similarity Aside from a quantitative comparison, we visualize the most similar retrieved images for two exemplary queries of SAM in Fig. 4 as case examples.Left Query: The image displays various utensils scattered on a desk. When retrieving with the permutation-invariant similarity metric two images of the same scene but a very different perspective are successfully retrieved as most similar. Retrieving with the non-invariant similarity metric fails to retrieve similar images, due to lack of spatial alignment of the semantic concepts. Instead, it retrieves images of a whiteboard, possibly due to its spatial alignment with the paper on the desk. Right Query: The image features a blender on a counter. The retrieval based on nonpermutation-invariant similarity fails to retrieve any of the semantically similar scenes and returns images with a light switch, likely due to the spatial alignment of the light-switchlooking object to the right of the blender. Contrary, the retrieval based on permutationinvariant similarity correctly returns the blender in all cases from different perspectives.Additional qualitative examples are provided in Appendix D.3.These experiments display clearly, that demanding spatial alignment can be a significant shortcoming when semantically similar concepts are misaligned. In Fig. 4 , the network learned to represent the objects very similarly, despite a shift in perspective, but due to the same objects not aligning anymore, spatio-semantic similarity fails to recognize this. This effect should generalize to other datasets where objects are not heavily centered. For datasets with heavy object-centric behavior, like ImageNet, this should be less pronounced. While the retrieval experiments relate to a rather human notion of similarity, one can raise the question if semantic RSMs are also better at measuring the similarity for classifiers.For each pair of samples, we can compare how similar the predicted class probabilities of a model are and compare this to the representational similarity. A commonly used metric for this is the Jensen-Shannon Divergence (JSD), which quantifies how dissimilar the two probability distributions are from one another. More details are provided in Appendix E.Consequently, we use various classifiers trained to predict ImageNet1k from Huggingface and compare the Pearson correlation ρ between their JSD and the representational similarity of their last hidden layer. We chose to use the Pearson correlation, as it allows observing a direct linear behavior between representational similarity and predictive similarity. Again we measure semantic similarity and spatio-semantic similarity with different kernels. Due to JSD measuring dissimilarity, we want the correlation to be as negative as possible. As models we use multiple ResNets [7] , ViTs [4] , a fine-tuned DinoV2 [22] classifier from and a convnextv2 [33] classifier.The results, displayed in Table 1 , show that for almost all architectures and kernels tested, the permutation invariant similarities are better at capturing the notion of what a classifier deems similar. While better than the spatio-semantic similarity, overall correlations are generally low, indicating that either, the similarity metric is confounded by irrelevant representations, or that the kernels should be improved. Moreover, the RBF kernel sometimes provides a positive correlation indicating it is unsuitable to predict the similarity of output probabilities, whereas the Cosine Similarity and the Inner Product both are consistently negative for all architectures tested.Since we find the best possible permutation matrix through linear sum assignment algorithms that maximize the inner product of two samples, we can guarantee that the K ij,semantic ≥ K ij ∀i, j. This provides us with an upper bound of similarity that can be leveraged to measure how much of the maximally achievable semantic similarity was measured by the spatio-semantic similarity. Additionally, it can be used as a baseline to estimate the quality of permutation matrices P ij provided by faster, approximative assignment algorithms.Determining the optimal permutation between samples poses a substantial computational challenge with a complexity of O(S 3 ) for each of the O(N 2 ) pairs in the same mini-batch, particularly for early layers with large spatial resolution S. Although, in theory, the calculation of the K matrix needs to be conducted only once for the desired representations, applying the method to representations with larger spatial extents becomes impractical with the demands of optimal matching. To mitigate runtime, two options are available: reducing the batch size N to lessen the number of permutation calculations or decreasing the time spent on finding the permutation. Given that scenarios like image retrieval often desire larger batches, our focus is on minimizing the time required to obtain suitable assignments. When aligning semantic concepts we observe that similarity changes heterogeneously, indicating that some pairs of samples have more spatially misaligned semantic concepts than others.Solving the optimal bipartite matching between semantic concept vectors is equivalent to the well-known assignment problem [18, 1] . We attempted to find existing approximate algorithms for this purpose. Unfortunately, most established algorithms primarily focus on optimal solutions, and existing approximate algorithm implementations, such as those based on the auction algorithm [6] , are not runtime-optimized, often taking longer than optimal algorithms in our experiments. To enhance computational efficiency nonetheless, we explored three tailored approximation algorithms:A) A Greedy breadth-first matching (Greedy) B) An optimal matching of the TopK values based on their Norm, followed by the Greedy algorithm for the remaining samples (TopK-Greedy) C) Optimal matching of smaller batches, with samples batched by their Norm (Batch-Optimal)For explicit details on the approximation algorithms, we refer to Appendix F. We conducted a comprehensive comparison between the approximate algorithms and the optimal algorithm. We compare their runtime per sample and the quality of matches, quantified by the average relative similarity k k optimal . The evaluation utilized representations from a ResNet18 on TinyImageNet, as illustrated in Fig. 5 . It can be seen that the measured spatio-semantic similarity for TinyImageNet samples are, on average, 30% lower with layers of higher spatial resolution exceeding 40%. This suggests a notable misalignment of semantic concept vectors. Notably, the Batch-Optimal approximation stands out as a reliable approximation for optimal matching. The fastest of the Batch-Optimal approximation methods shows < 8% error while improving run-time ×36 relative to the fastest optimal algorithm for spatial extent 4096, while no spatial alignment shows 42% deviation from the optimal matching. Moreover, we highlight the time vs accuracy trade-off of the different optimal and approximate algorithms in Appendix F.1. Furthermore, it can be seen that the changes between spatio-semantic and semantic RSMs are anisotropic, as highlighted in Fig. 6 , indicating scale invariant downstream applications may be influenced. 
discussion, limitations, and conclusion: 
 The concept of Representational Similarity Matrices (RSMs) is a powerful tool to represent the similarity structure of complex systems. In this paper we revisit the construction of such RSMs for neural networks of the vision domain, question the current state, and propose semantic RSMs, warranting discussion. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The paper introduces a new concept that has no direct related methods to compare with. Therefore, no statistical analysis or ranking methods are needed.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: The presented work does not rely on specific compute requirements. However, in Appendix F.1 we compare the runtime of multiple approximation algorithms to solve the assignment proble.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
 
 Appendix: Being aware that current, spatio-semantic RSMs demand semantic concepts to be aligned is highly relevant to understand what RSMs are sensitive to. Previous work [32] identified this shortcoming and proposed translation invariance, partially addressing this issue. We argue translation invariance is insufficient, since models may learn invariances during training, which the translation invariant metric would not be sensitive to. Subsequently, we propose a new -spatially permutation invariant -similarity measure between samples that allows the detection of similarity whenever a model expresses similar semantic vectors in its representations, irrespective of spatial geometry. To highlight the benefits of our similarity, we propose that better similarity measures should allow more accurate retrieval when comparing last-layer representations and should allow better predictions about the similarity of class probabilities of a classifier. However, we acknowledge certain limitations in our current evaluation. Specifically, we have not yet compared our method to more established retrieval techniques. Traditional retrieval methods are often not applied to representations directly but utilize a lower-dimensional non-spatial, global vector representing the entire sample. In contrast, we chose to limit ourselves to methods that are directly applied to the representations.Computational Complexity Aside from quantitative or qualitative benefits, the construction of semantic RSMs is time-consuming, limiting its applicability. This complexity mostly affects layers of large spatial extent, which mostly corresponds to early CNN layers while later layers and ViTs are unproblematic. Our proposed Batch-optimal approximation alleviates this partially, yet application to large-scale representations at higher resolution, like at the output of a segmentation architecture with spatial extents of s=65.536 would be too costly. We leave optimizing the compute efficiency or finding better approximations for future work.In conclusion, our investigation into semantic RSMs has shed light on the limitations of spatio-semantic RSMs and introduced a novel approach to disentangle spatial alignment from semantic similarity. The proposed method provides a more accurate measure of how representations capture underlying semantic content, showcasing its potential in various applications, particularly in scenarios where spatial alignment cannot be assumed. While challenges such as computational complexity and scalability need to be addressed, the findings open avenues for further research and improvement in the analysis of neural network representations.Defining the notion of similarity between two vectors is a matter of preference and wanted properties. In this work, we include the Radial Basis Function kernel Eq. ( 6) and the linear kernel Eq. ( 7), as well as the cosine similarity kernel Eq. ( 8). We include the former two as they were proposed for RSM construction by Kornblith et al. [12] and the cosine similarity due to its popularity in the retrieval domain, despite generally being applied to class probabilities.We denote that in our manuscript we refer to the linear kernel as the inner product and dot product interchangeably as they correspond to the same operation.EQUATIONEQUATIONEQUATIONDifferent properties The three selected kernels all have different properties: The RBF kernel and the Cosine similarity are bounded between k RBF (x, y), k cosine (x, y) ∈ [0, 1] ∀x, y ∈ R, while the k linear (x, y) is not bounded. Moreover, the RBF kernel is parametrized by σ, which influences at which rate the distance between the representations results in a decrease in similarity. For all our experiments we choose σ as the square root of the median Euclidean distance of all distances within a mini-batch.Similarly to the translated Tiny-ImageNet experiment, we prepared a very similar experiment for a ViT-B/16 vision transformer that was pre-trained on ImageNet1k. We utilize the implementation and weights provided by torchvision [17] .Given the larger ImageNet images, we resample them to 324 × 324 pixels and crop two partially overlapping images of size 224 × 224 from it. Given these image pairs, we calculate semantic and spatio-semantic RSMs again, see Fig. 7 .It is important to note that our approach intentionally avoids achieving a perfect translation that would lead to the same patchified tokens, as we shift the image by a factor that is not divisible by 16, the patching window size. We believe this realistic imperfection is preferable to a perfect overlap, where identical tokens would be formed from the exact same set of pixels.Similarly to the previous partially overlapping crop experiment, we can observe that spatio-semantic RSMs are incapable of identifying the largely identical content of the two partially overlapping crops due to their different localization. The spatio-semantic RSMs can capture this notion of similarity as evidenced by the off-diagonal and the larger gap in the distribution of similarity between partially overlapping samples and independent samples, Fig. 7 (bottom). Contrary to the CNN example in the main, overall similarity between samples is much lower overall and separation between translated image pairs and random image pairs follows a vastly different trajectory. While there is a profound difference, the origin of this difference cannot be clearly made out. We hypothesize that this may be due to the different ways that Transformers process information and learn different representations, as highlighted in Raghu et al. [26] .Moreover, we emphasize, that calculating the optimal permutation is significantly faster for ViTs than the CNNs, as the early tokenization reduces the spatial dimension S substantially at an early stage, whereas the iterative downsampling of CNNs makes comparing representations of early layers very costly. In addition to our provided explanation of the algorithm in the main manuscript, we provide the pseudo-code used to compute semantic RSMs in Algorithm 1. The only difference between semantic and spatio-semantic RSMs algorithmically is where the optimal permutation is calculated that maximally aligns the two representations. The current definition of spatio-semantic RSMs assumes that spatial locations are corresponding, while semantic RSMs calculate correspondence through similarity matching.In addition to the provided retrieval examples in the main manuscript, we provide more details on the retrieval experiments in Appendix D.1, an additional table holding the quantitative data of Fig. 3 with varying database sizes in Appendix D.2 and lastly, additional qualitative retrieval examples for each model including direct comparisons of all models for the same query image in Appendix D.3.For the retrieval experiment, we utilize the EgoObjects dataset [35] . It contains multiple frames from multiple videos, with multiple videos capturing the same scene under different shifts like lighting conditions, distances, viewing angles, and different motion trajectories.For each frame, multiple objects of different categories can be present and are annotated Algorithm 1: Semantic RSM calculation. We calculate the optimal permutation matrix, resulting in maximal similarity between the representations of two samples.Data: Z ∈ R N ×C×S ; Kernel k Result: Semantic RSM K K = 0 ∈ R N ×N for i from 0 to N do for j from 0 to N do if i < j then cont. if i ̸ = j then P ij = argmax Pij ⟨z i , P ij z j ⟩; z ′ j = P ij z j ; else z ′ j = z j ; K ij = k(z i , z ′ j ); K ji = K ij ;/* is symmetric */ through a bounding box. Moreover, frames can vary in spatial resolution, yet a large fraction was captured in 16:9 format of 1920 × 1028 pixels.For our experiments, we utilize the EgoObjects test set, which is comprised of 29.5K images. Of these 29.5k we remove all images not in 16:9 format and resize the remaining to the 1920 × 1028 format. This discards roughly 10k images.Of all remaining images, we then draw 2k query images and 5k database images used for extracting embeddings for similarity calculation and later retrieval. Naturally, we sample in a way to keep the 2k query and 5k database image sets non-overlapping. When passing the images to the models for feature extraction each image is preprocessed according to the corresponding Huggingface ImagePreprocessor. This mostly represents resizing the image by the shortest edge to the expected image input dimensions and normalizing the image. The only exception is SAM, of which we use the official implementation, which handles feature extraction and embedding of the image itself.As mentioned in the main manuscript we use 1. CLIP (ViT/B32) [24] 2. ClipSeg (Rd64) [16] 3. DinoV2-Giant [22] 4. SAM (ViT/B32) [9] and 5. BIT-50 [11] as general feature extractors as they were trained on a vast amount of data.4.Of all models, we use the last hidden layer as image embeddings should they not per-default provide image embeddings as output. After extracting representations we calculate the mean from the database embeddings to zero-center all representations by, query and database representations alike.RSM construction Given all 2000 query embeddings and 5000 database embeddings, we calculate the RSMs. To parallelize this process we mini-batch the representations into 100 × 100 pairs and populate the 2000 × 5000 matrix in this fashion. We denote that this proved to be necessary for models with large spatial embedding dimensions like SAM, starring 64 × 64 spatial extent. Moreover, we denote that, due to the RBF choosing its 4Last accessed on 22nd of May 2024 parameter based on the median of the measured values within one batch this patch-wise calculation is not optimal for this kernel. The inner product and the cosine similarity kernels are not affected by this.Given the RSMs containing a sample-to-sample similarity measure, we can retrieve the most similar sample of the database for each query from it.As each image can contain multiple objects measuring retrieval performance is not trivial.For both images we quantify how many objects of each class are present in the image, resulting in a count of class instances for each image. With the query image representing the ground truth (GT) and the database representing the prediction, we match class instance counts. Each correctly matched GT instance represents a TP, each missed an FN and all unmatched database instances represent an FP.To formalize: Let Q c be the number of instances of class c in the query image and D c be the number of instances of class c in the database image. With this, the used F1 metric can be expressed asEQUATIONEQUATIONEQUATIONEQUATIONIn addition to the results highlighted in Fig. 3 we provide retrieval results for varying database sizes to retrieve from EgoObjects. Specifically, results for database sizes of 2.5k, 5k, and 10k samples are given in Table 2 .Table 2 : Quantitative results of the EgoObject retrieval experiment for multiple models and multiple database sizes. Database Size 5.000 represents Fig. 3 of the main manuscript. It can be observed that models with a greater spatial extent (CLIPSeg and SAM) show greater improvement in retrieval performance over models with a lower spatial extent (DinoV2-Giant, BiT-50, CLIP). Moreover, it can be observed that retrieval improvements decrease with growing database size. This is likely due to the increasing odds of finding one image where spatial position and semantics align w.r. In addition to the two qualitative examples provided for SAM in the main, additional examples are provided. Qualitative examples are picked from the first 50 query images when retrieving from a database size of 5.000 images. Additional qualitative retrieval examples are provided for DinoV2 in Fig. 8 , CLIP in Fig. 9 , BiT-50 in Fig. 10 , CLIPSeg in Fig. 11 and SAM in Fig. 12 . Moreover, we highlight a direct comparison between all models for the same images in Fig. 13 and Fig. 14 . Across all models, it can be observed that models retrieve images where semantic content and spatial positions are aligned when using standard cosine similarity. This leads to larger-scale objects like desks, tiled floors, or countertops dominating retrieval when imaged from a similar perspective. When decoupling spatial-alignment from semantic content, images of the same scene but a different perspective get retrieved more often, leading to the same scene appearing more regularly in the 5 nearest neighbors. This effect is currently not quantified in the F1 metric, due to only comparing object presence between the query and the 1st neighbor and not the average F1 between the query and the top 5 neighbors. We opted against using this, as retrieval metrics most commonly consider the maximum match in the top 5 neighbors.Moreover, it can be observed that models with lower spatial extent can retrieve quite different neighbors as opposed to models with higher spatial extent, see Fig. 13 . While DinoV2 and CLIP retrieve very similar 4th and 5th neighbors of the same book object without the smaller mouse and headphone object, CLIPSeg and SAM retrieve scenes with these two objects still present instead.Since previous results were limited to the EgoObjects dataset we provide an additional quantitative experiment on CityScapes. Analog to before, we use N=500 validation images as the query dataset and the remaining N=2975 training images as the database for retrieval. We utilize the IoU metric to compare the presence of semantic classes between the images. Due to the lack of instance labels in the CityScapes dataset, we can't take the quantity of objects into account as before, but can only consider if a semantic class is present or absent. As apparent from Table 3 permutation allows for consistently improved retrieval when utilizing cosine similarity or RBF kernels. To measure the correlation between the inter-sample representational similarity and the prediction probability inter-sample similarity we utilize pre-trained classifiers and the ImageNet1k [2] dataset. Unlike during the retrieval results this constraints the possible model selections to models trained for classification.Image preprocessing Test set images of ImageNet1k are randomly sampled without applying any filtering to them. In total, we utilize a subset of 2k ImageNet test set samples in this experiment. This may appear small, yet provides a sufficient basis as the combinatoric growth increases the absolute number of measurements substantially.As mentioned in the main manuscript we use 1. ResNet18 [7] 2. ResNet50 [7] 3. ResNet101 [7] 4. a DinoV2-Giant based classifier [22] 5. ConvNeXt V2 [33] 6. ViT-B/16 [4] and 7. ViT-L/32 [4] and as pre-trained classifiers for predicting the ImageNet1k classes.5. For each sample, we extract the last hidden layer's representations and center them analog to before. For the same sample, we extract the logits and obtain the probability distribution through the softmax, saving the pair for later comparisons.For each pair of representations and probabilities, we calculate the similarities between their representations for all three kernel functions, once permutation invariant and once not. Additionally, we calculate the Jensen-Shannon Divergence (JSD)5Last accessed on 22nd of May 2024 between the predicted class probability distributions for P and Q. A formal definition of the JSD is provided in Eq. ( 13).EQUATIONwhere M the pointwise mean of P and Q:EQUATION)and D KL is the Kullback-Leibler divergence defined as:EQUATIONGiven the paired JSD and Similarity K between all samples i, j we utilize the Pearson correlation ρ to calculate the correlation between the two. Due to the JSD being 0 for identical probabilities and increasing for more dissimilar values and the Similarity being 1 for perfectly similar representations and 0 for dissimilar representations, the desired correlation between the two should be negative.In addition to the Pearson correlation between the Jensen-Shannon-Divergence (JSD) and inter-image similarity, we also present the results of their relationship measured by the Spearman correlation, as shown in Table 4 .While the Pearson correlation demonstrated consistently stronger correlations with cosine similarity, the inner product, and the RBF kernel, the Spearman rank correlations are less stable across these methods.For ResNets, we observe a significant decline in correlation consistency and strength with the exception of ResNet18. Opposed to this, ViTs display notably higher negative correlation values when using cosine similarity and radial basis function kernels, in contrast to the Pearson correlation results. The computational complexity of determining optimal matchings using the Jonker-Volgenant algorithm [8] scales significantly with O(s 3 ), resulting in substantial computation time for input-patch sizes with spatial dimensions s = 64 2 . To address this challenge, we propose alternative approximate algorithms with reduced computational complexity. In all our approximations, we take advantage of additional information, specifically the L2-norm ∥v i ∥ 2 of each semantic concept vector. We assume that achieving a high degree of matching involves pairing vectors with the highest norms and high cosine similarity. This assumption guides our design of more efficient matching algorithms.1. Greedy: The simplest approach we employ is breadth-first matching. We determine the order in which to match v i by considering the L2-norm ∥v i ∥ 2 in descending order. We then match the current v i with the best, non-assigned v j based on A ij . The sorting complexity is O(s log(s)), making this the fastest approximate algorithm among those tested.2. TopK-Greedy: Recognizing that the TopK norm concept vectors ∥v i ∥ 2 might have a significantly higher impact on the final similarity, we attempt to find the optimal matching for only the highest TopK norm concept vectors v i and v j . The remaining lower norm concept vectors are assigned using the Greedy algorithm as described above. The process involves an initial sorting based on the semantic concept vectors' L2-norm, followed by optimal matching with O(k 3 ) complexity for the k TopK values and the greedy matching for the remaining values.3. Batch-Optimal: If the TopK norm concept vectors do not sufficiently approximate an optimal matching, we apply optimal matching for the remaining concept vectors in batches. To achieve this, we create s//b smaller batches, with semantic concept vectors assigned to batches according to their L2-norm. All values within a batch are then optimally matched, leading to a matching complexity of ( s b ) • O(b 3 ).6 Evaluating the various approximations, we observe that the Greedy matching yields suboptimal approximation quality and offers marginal to no improvement over the current same-position assignment. Although we do not present the details of the greedy matching, it is important to highlight that it is guaranteed to be worse or equal to the TopK-Greedy matching with a k value of 128, as shown in Fig. 5 . We include the Greedy algorithm for completeness as a simple baseline. Furthermore, it is noteworthy that the TopK-Greedy matching demonstrates that exclusively matching the largest norm concept vectors is insufficient for a good approximation of the optimal matching. This insight suggests that a substantial portion of the overall similarity is contributed by semantic concept vectors not included in the set of highest norms. Lastly, we observe that the Batch-Optimal approximation, using a small batch size of 128 samples, provides an approximation with less than 10% error compared to the optimal matching. This result underscores the effectiveness of our batching approach based on the L2-norm of the concept vectors. It offers a reliable estimate for overall similarity, simplifying the matching process significantly.In order to assess algorithm performance across different spatial resolutions, we conducted a benchmarking study. For each resolution, we randomly selected 10,000 pairs of samples from a ResNet101 trained on Tiny-ImageNet. Affinity matrices (Aij) were pre-computed to facilitate permutation (Pij) calculations. The average time taken per matching was then reported for the same single CPU core, as outlined in Table 5 . We observe that the OR-Tools implementation outperforms other alternatives, being four times faster than the lapjv implementation 7. However, even this optimal approach requires 1.52 seconds per pair on a 64 × 64 resolution. Despite the potential for parallelizing sample-wise matching, optimal algorithms face scalability challenges with larger spatial dimensions S. In contrast, the Batch-Optimal approximation offers a compelling balance between computation time and approximation quality. Importantly, its complexity scales linearly with S due to the fixed batch size.Building upon the success of Linear/RBF (spatio-semantic) CKA to compare systems, we provide some preliminary qualitative comparisons gauging how CKA comparisons are affected by our differently proposed RSM. Unfortunately, hardly any quantitative benchmark 6There is an error in the current version in the main regarding the square root of s, which will be corrected in a revision. We apologize for any confusion 7Implementation on Github https://github.com/src-d/lapjvTable 5 : We compare different implementations of the optimal Jonker-Volgenant algorithm [8] against our linearly scaling Batch-Optimal approximation and no matching. The table presents average run-times per pair (T ) and average similarity relative to the optimal value ( k kopt ) for 1,000 randomly chosen image pairs. Utilizing representations of varying sizes from a ResNet101 trained on Tiny-ImageNet, the optimal solutions are reported relative to the maximum similarity achieved by the optimal algorithms. The Batch-Optimal approximation demonstrates a substantial fraction of optimal matching performance with significantly improved scaling. In all following experiments, all representations are extracted globally and zero-centered along the sample dimension. Given the zero-centered representations spatio-semantic and semantic RSMs are computed in mini-batches of 250 samples. To calculate the semantic RSMs on CIFAR an optimal bipartite matching algorithm is used, while for Tiny-ImageNet and ImageNet we utilize the Batch-Optimal approximation with window size b 512.As initial inspection, we evaluate how different the similarity structures of a model measured through spatio-semantic RSMs are to a model measured through semantic RSMs. We do this by constructing both RSMs from the same representations of a model. Subsequently, we compare the two alternative RSMs of the same representations to each other through CKA Eq. ( 2). The diagonal of this matrix represents a direct comparison of identical representations, just with another definition of what is considered "similar". This is evaluated for three ResNet18s and three ResNet34s on Tiny-ImageNet with the linear and RBF kernels respectively.We display the average CKA matrices across architecture seeds for both architectures, as well as the diagonal values of the CKA matrix. Results are shown in Fig. 15 .Examining these CKA matrices multiple observations can be made: A) Despite the diagonal representing a comparison between identical representations, the CKA values are not 1. This indicates that the different way of constructing RSMs changes the perceived similarity structure of the system, as measured by CKA. B) Inspecting the diagonal shows, that earlier layers with greater spatial extent express higher differences in similarity, whereas layers at a later layer and lower spatial extent are less dissimilar. This is consistent with the expectation that, with shrinking spatial extent, alignment of semantic concepts gets more likely.Given these large changes in CKA similarity, we conclude that the definition of what a model perceives as similar can highly influence inter-system similarity. This is especially relevant when comparing systems across domains, where RSM construction may be domain-specific, disallowing to be consistent with RSM construction. Exemplary when comparing ML vision systems to human vision models, in particular when comparing representations of high spatial extent. In the previous paragraph, semantic RSMs were directly compared to spatio-semantic RSMs. While this can influence measurements when models are compared across domains (e.g. CNN Vision to Biological Vision), it does not need to imply that the CKA similarity of vision models changes substantially.Given that within the same domain, the RSM construction will likely be chosen consistently, one can opt to either: Calculate only spatio-semantic RSMs or only semantic RSMs, due to personal opinions or preferences. Subsequently, the question is not "Are spatio-semantic RSMs similar to semantic RSMs", but "Does the CKA similarity between spatio-semantic RSMs change when calculating semantic RSMs".To address this question we: A) Compare the CKA matrix difference of the CKA matrix based on semantic RSMs to the CKA matrix of spatio-semantic RSMs when comparing a model with itself (Intra Model) and B) when comparing between different models (Cross-Model).Intra-Model CKA Similarly to before, we extract representations and form semantic and spatio-semantic RSMs. We extract representations on CIFAR100 [14] (32×32), Tiny-ImageNet (64×64) and ImageNet-1k [2] (160×160) datasets, from 3 differently seeded and trained from scratch ResNet18, ResNet34 and ResNet101 architectures. The semantic RSMs are calculated utilizing the Batch -Optimal matching with b 512 for matching on Tiny-ImageNet and ImageNet. We calculate semantic and spatio-semantic RSMs with a mini-batch size of 250, subsequently using them for Canonical Correlation Analysis (CKA) calculations. The corresponding cka matrices and their differences are displayed in Fig. 16 . If not further specified the experiment uses the linear kernel.Introspecting the results it can be seen that across all Architectures ResNet18, ResNet34, and ResNet101 largely the same change in similarity structure can be observed. For CIFAR100, the very first layers are perceived as less similar to semantic RSMs than with spatio-semantic RSMs, while the CKA between the middle to later layers is more similar. This structure, though does not remain consistent across datasets: When moving from CIFAR100 to Tiny-ImageNet earlier layers appear to become more similar while intermediate layers become less so. On ImageNet1k CKA on semantic RSMs seem to indicate models are more similar. This trend indicates that the influence of semantic RSMs on spatio-semantic RSMs seems to be largely dataset-dependent. Moreover, the overall maximum change in CKA similarity in these matrices is between -0.2 and +0.2 for Tiny-ImageNet, indicating a modest change in overall CKA.Cross-Model CKA Aside from evaluating only CKA similarity of RSMs of the same model we extend to comparing RSMs between models, as commonly done when comparing models through CKA. ResNet18/101 models trained on CIFAR100 are used with RSMs constructed identically to previously specified. Results are displayed in Fig. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: The authors have strictly adhered to the ethical guidelines.Guidelines: 


Decision: This paper deals with extending the concept of Representational Similarity Matrices (RSM) to semantic RSM, by measuring semantic similarity between input responses via solving a set-matching problem, in order to achieve permutation invariance.  The idea is validated on image retrieval and probabilistic similarity comparisons.

It has received 2x weak accepts, 1x borderline accept, and 1x borderline reject.  Reviewers have major concerns on lacking quantitative comparative data, lacking thorough discussions and analysis of experimental results, computational costs with the utilization of Hungarian algorithms, conceptual inadequacy of semantics disregarding spatial configuration, applicability under noisy semantic content detection and in complex scenarios with multiple objects of the same class, lacking more visual examples and questionable metrics etc.  The authors have provided extensive rebuttal with additional experimental results, which reviewers find mostly helpful and render an overall acceptance consensus.

Representation similarity is an ambiguous concept.   The reality lies between global feature similarity and set-matched local feature similarity for individual images.  Nevertheless, the AC recommends acceptance based on its different perspective, even at one extreme of permutation invariant matching.   Please revise the paper per reviewers' comments and additional materials in the rebuttal, strengthening the position of the paper with more convincing results and discussions.   Thank you.

~~~~~~~~~~~~~~~~~~~~~~~~
Here are extra comments from SAC.
(1) Please remove the period at the end of title. In addition, check the usage of punctuation in Line 31-33.
(2) Please use the official template. The current one suffers from the issue of paragraph spacing. 
(3) It would be nice to adjust orphan lines.
(4) Please coordinate the font sizes in figures, where some fonts are too large in Figure 1 and some fonts are too small in Figure 2.

Reviews: review: paper_summary: paper_summary: This paper makes a contribution to the construction of RSMs in the field of vision neural networks and puts forward the concept of semantic RSMs, which is innovative and theoretical.
strength_weakness: strength_weakness: This paper lacks the experimental verification of specific downstream tasks, such as detection and segmentation, on semantic RSMs. I need to know which scenario is more suitable for RSMs and semantic RSMs.

Lack of quantitative comparative data. It is suggested to add tables or charts to show specific performance comparison data between semantic RSMs and existing methods in different tasks (such as image retrieval, class probability similarity comparison, etc.), including accuracy, time complexity and other indicators.

The discussion of the experimental results was not thorough enough. It is recommended to add a detailed analysis of the experimental results to explain why semantic RSMs perform better on certain tasks, as well as possible reasons and limitations.

 "aligns" to "align" in line 57.
questions: questions: It is suggested to further elaborate the potential and specific scenarios of the research in practical applications to enhance readers' understanding of its practical value.
limitations: limitations: see weekness
paper_summary: paper_summary: The authors introduce semantic RSMs, which are designed to be invariant to the spatial arrangement of elements within images. These semantic RSMs assess similarity by treating the problem as one of set-matching, where the focus is on matching semantic content rather than spatial details. This approach not only aligns more closely with human perception but also improves the relevance and accuracy of image retrieval tasks. The paper claims that semantic RSMs offer a more robust measure of similarity by comparing them to traditional spatio-semantic methods.
strength_weakness: strength_weakness: - Employing algorithms like Hungarian matching to find the optimal permutation matrix can be computationally expensive.

- The effectiveness of this approach relies heavily on accurate identification and parsing of semantic concepts within images, which can be challenging in complex scenes or under conditions of visual ambiguity.

- While focusing on semantic content is generally advantageous, completely ignoring spatial information can sometimes omit useful contextual cues that contribute to overall image understanding. For example, 

[contextual cues] a picture of a dining table with plates, utensils, and food arranged in a specific way might convey a meal setting, which could be lost if the spatial relationships are ignored. 

[object interactions] Images where interactions between objects are important, such as a cat sitting on a mat, might lose their interpretative meaning if spatial information is disregarded. The semantic content (cat, mat) remains the same, but the relationship changes based on their arrangement.

[abstract content] In abstract art or images with non-literal interpretations, spatial composition itself can carry meaning and affect how the content is perceived and classified.
questions: questions: - How well does the method scale to very large datasets or to more complex neural networks that handle highly varied or abstract visual content?

- How does the method perform under noisy conditions or when semantic parsing is imperfect due to occlusions or poor image quality?
limitations: limitations: - To better evaluate the impact of semantic RSMs, a set of diverse metrics should be established: Evaluate semantic RSMs against traditional spatio-semantic RSMs and other state-of-the-art similarity measures to highlight the improvements or shortcomings.

- The authors should explicitly state the primary objectives of employing semantic RSMs. Show that semantic RSMs can make neural network decisions more interpretable by aligning more closely with how humans perceive images.

- Apply semantic RSMs in specific use cases like medical imaging, satellite image analysis, and autonomous driving where ignoring spatial arrangements can be particularly detrimental or beneficial, providing a nuanced view of their applicability.
paper_summary: paper_summary: This paper proposes Semantic RSMs to understand the internal representations in deep neural networks. The authors argue that the current RSMs are limited by their coupling of semantic and spatial information, which restricts the assessment of similarity. The proposed semantic RSMs are spatial permutation invariant and focus solely on semantic similarity. The proposed method is shown to enhance retrieval performance and provide a more accurate reflection of the predictive behavior of classifiers.
strength_weakness: strength_weakness: 1. While the paper does highlight the high computational complexity as a limitation, it would benefit from a more detailed discussion on the scalability of the proposed method to larger models and datasets and the approximation error.
questions: questions: I'm not an expert in this field, so I tend to start by looking at what other reviewers think of the paper.
limitations: limitations: Yes, the authors have addressed the limitations.
paper_summary: paper_summary: In this paper, the authors propose a new method to measure similarity between responses of deep neural networks in vision. They reformulate the commonly used strategy to compute Representational Similarity Matrices (RSMs) by acknowledging the superiority of the semantic information over the spatio-semantic information in the creation of RSMs. The authors perform different experiments to show the improvement over the baseline method caused by their reformulation.
strength_weakness: strength_weakness: - The authors should better discuss the differences between the results obtained for ViTs and CNNs in their study, which are quite well visible. E.g. while in the case of an examined CNN, the spatio-semantic RSM does not reflect well the similarities between translated images, in the case of the examined ViT (appendix), these similarities can be observed. The other thing is that the experiment is slightly different, because in the experiment with CNNs much smaller images are used than in the ViT experiment. Also, the differences are also visible in Table 1 (ResNets obtain much higher absolute correlation values for the baseline and the proposed methods than ViTs). 
- The authors provided few visual examples of the results of their method. It would be good to provide more of them (e.g. for different similarity metrics used, for more images and for more netorks) to enable more comprehensive qualitative evaluation (they could be placed in the appendix). 
- The use of some methods at work is not well justified (e.g. Pearson correlation).
questions: questions: - Why do the authors use Pearson correlation to examine the relationship between the Jensen-Shannon Divergence and the representational similarity? E.g. Kendall/Spearman correlation can be more robust.
- The authors should focus on the differences between the results obtained for CNNs and ViTs (see the comment in the weaknesses section). 
- The statement in the introduction “we argue that the spatial location of semantic objects does neither influence human perception nor deep learning classifiers.” is a little bit too bold - the paper does not examine human perception, therefore it would be better to leave only the deep learning here. 
- Also, a minor thing is that some typos, grammatical and formatting errors can be found in the paper (e.g. the sentence starting in l79, l205: a RSM -> an RSM, l25: SAMor CLIPSeg ,the retrieval performance)
- The authors could provide more examples of their method for different networks to enable their better qualitative assessment which is now limited (e.g. in the appendix).
limitations: limitations: The authors adequately discussed the biggest limitation of their work (the computational cost of their method). They could also better highlight the limited data used in the experiments caused by the mentioned computational constraints.

Comments: title: Discussion response
Post-rebuttal response
comment: Thank you for engaging with our rebuttal. We are pleased that we have addressed most of your initial concerns. Regarding the remaining issues:

>What are the derived insights from the proposal? e.g., Do ViT and CNN exhibit different behaviors? What can we learn from the results? A deeper analysis is needed beyond just showing that the proposal can predict output probabilities (e.g., Table 1).

We believe our paper provides a plethora of insights and is not limited to "just show[ing] that the proposal can predict output probabilities". In our paper, we:
1. Highlight that current RSM construction is flawed and has limitations of requiring spatial alignment, which has not been adequately discussed previously.
2. Introduce an algorithm that addresses this by utilizing set-matching, offering a novel approach.
3. Demonstrate the utility of our method through two applications, revealing significantly improved retrieval performance with five general-purpose feature extractors across the EgoObjects and Cityscapes datasets, and better correlation between the Jensen-Shannon Divergence of output probabilities and inter-sample similarity.

We believe our work underscores a critical gap in current RSM-based methods and sets the foundation for further exploration into RSM construction, effective inter-sample similarity measures, and their applications for retrieval. 

>There are counterexamples where neglecting location information leads to issues (e.g., visual anagrams), where semantically different images might not be distinguishable by the proposal since it matches features at a set level.

We understand your concerns regarding location information. However, __i)__ our experiments on the EgoObjects dataset, which includes varied contextual scenarios such as the meal table example, indicate that our method outperforms existing RSMs. This suggests that issues related to spatial location might either be rare or of lesser importance. __ii)__ Our method is based on learned representations. _Should a visual anagram exist where similar representations are expressed by the network, we believe this is to be a failure of the model and not our approach_. We don't believe a retrieval method should try to compensate for this. __iii)__ There are other examples where spatial position is neglected entirely: E.g. the ViT paper [1] shows (in Appendix D.4) that ViT's are able to learn meaningful representations for classification without any positional embedding, making the input a bag-of-words. 

> Also, how can this proposal be applied in complex scenarios, such as when the data contains multiple objects of the same class?

Our method has already been successfully applied in scenarios involving multiple objects of the same class. Both retrieval datasets, EgoObjects (Figure 3) and Cityscapes (Rebuttal PDF) contain multiple objects of the same class. Additionally, one currently provided qualitative example in the paper and the additional qualitative examples in our rebuttal PDF illustrate our propose methods effectiveness in exactly such complex scenarios, highlighting retrieval on images with multiple instances of glasses or bowls.

>Last but not least, the permutation matrix required for set-matching introduces high computational overhead, which could limit the applicability of the proposal to high-resolution data.

We acknowledge the computational demands introduced by the permutation matrix for set-matching. Nevertheless, we have applied our method to high-resolution images (1920x1080) from the EgoObjects and Cityscapes datasets, which demonstrates its feasibility on large-scale data. If your concerns are aimed towards retrieval, we want to emphasize that retrieval is generally a two stage process (see [2] Section 4), with the first being a ranking by global cosine similarity and the second being a re-ranking of the top 100-400 cases which are most similar through e.g. our method. Hence such a two-stage application can enable our method to scale easily to any number of retrieval dataset size. 
Aside from this, we transparently address this limitation in our paper and have expanded on this topic in the appendix during the rebuttal phase. Moreover we'd kindly refer you to our general response which goes into more detail on the scalability of our method and would be happy to answer any more specific concerns regarding the scalability of our method should there be any remaining.    

We hope this addresses your concerns and look forward to any further questions you might have.

[1] Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020).

[2] Cao, Bingyi, Andre Araujo, and Jack Sim. "Unifying deep local and global features for image search." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16. Springer International Publishing, 2020.
Thanks for the clarification.

The paper focuses on computing representational similarity by comparing semantics with location similarity. Instead of considering both simultaneously, the proposal focuses solely on semantics. This is achieved by extracting semantic vectors from the feature tensor and comparing them as a set, rather than matching the vectors individually in spatial terms.

The concept is straightforward, but the application is unclear, and the derived insights may not be sufficient for publication. Here are the remaining concerns:

- What are the derived insights from the proposal? e.g., Do ViT and CNN exhibit different behaviors? What can we learn from the results? A deeper analysis is needed beyond just showing that the proposal can predict output probabilities (e.g., Table 1).

- There are counterexamples where neglecting location information leads to issues (e.g., visual anagrams), where semantically different images might not be distinguishable by the proposal since it matches features at a set level.

- Also, how can this proposal be applied in complex scenarios, such as when the data contains multiple objects of the same class?

- Last but not least, the permutation matrix required for set-matching introduces high computational overhead, which could limit the applicability of the proposal to high-resolution data.
I would like to thank the authors for their response along with some additional results and wish them good luck!