Discriminating image representations with principal distortions

Authors: Jenelle Feather, David Lipshutz, Sarah E Harvey, Alex H Williams, Eero P Simoncelli
Primary area: applications to neuroscience & cognitive science
Keywords: representational similarity metric; Fisher information; information geometry; perception

Abstract:
Image representations (artificial or biological) are often compared in terms of their global geometric structure; however, representations with similar global structure can have strikingly different local geometries. Here, we propose a framework for comparing a set of image representations in terms of their local geometries. We quantify the local geometry of a representation using the Fisher information matrix, a standard statistical tool for characterizing the sensitivity to local stimulus distortions, and use this as a substrate for a metric on the local geometry in the vicinity of a base image. This metric may then be used to optimally differentiate a set of models, by finding a pair of "principal distortions" that maximize the variance of the models under this metric. As an example, we use this framework to compare a set of simple models of the early visual system, identifying a novel set of image distortions that allow immediate comparison of the models by visual inspection. In a second example, we apply our method to a set of deep neural network models and reveal differences in the local geometry that arise due to architecture and training types. These examples demonstrate how our framework can be used to probe for informative differences in local sensitivities between complex models, and suggest how it could be used to compare model representations with human perception.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730620317536

thread_type:
official_review

<thread_object>
creator: reviewer_2ndt

<content>
summary:
The paper proposes a method to compare image representations by analyzing local geometries using the Fisher information matrix, enabling differentiation of models based on local sensitivity to distortions. For comparison, it has been applied to both early visual system models and deep neural networks, the method enables comparison of multiple models (>2) and can be used to compare model representations with human perception.

strengths:
I found the paper's topic interesting and overall the language of the paper was clear. The paper targets an important problem in cognitive science and is a step toward understanding visual models better. The figures and qualitative results were sufficient to understand the concept and the differences. The paper is also based on a sound experimental framework to investigate the source of variability in results from both training data and architectural design.

weaknesses:
I have a few questions and would like to know the authors' response to them:

1. (minor) It has been claimed in Figure 4 that the differences between AlexNet and ResNet come from the architecture than the training procedure. There is always randomness in the training where initialization of the model's weights and also the random split of data can effect one single model's performance. In fact, the model output may change if we train it on different seeds. I am wondering wouldn't that be a better approach to replicate the results on models trained on at least three different seeds? (basically 3 different versions of a model)

2. (minor) I expected to see a candidate from Transformers in the experiments, however, there was no candidate from those (currently) popular models. Can authors justify why they have not included any transformers such as ViT or Swin?

3. With Transformers, it has been a trend to show the attention maps (on tokens of an image) and show the behavior of the model. I am wondering if the authors can comment on this. Can the distortion maps be correlated to attention maps? This can ideally (if correlated) help with the explainability of attention-based models.

4. (major) Currently the manuscript is heavily populated with qualitative results. I strongly suggest the authors add a per-class sensitivity distribution for the dataset they are using for different models. This helps to understand on a population level and over different classes which models are more sensitive to high freq. localities and which ones are not.

5. I could not find a Dataset section in the manuscript :) Adding an independent dataset section would make it easier to understand the scale of experiments. currently, it is hard to find within the text

questions:
mentioned everything in weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730639244433

thread_type:
official_review

<thread_object>
creator: reviewer_zmdx

<content>
summary:
The paper introduces a new method for comparing vision models that map input images to stochastic representations, leveraging the Fisher Information Matrix (FIM) to analyze the local geometry of image representations. This approach allows for the comparison of multiple models, including computational models of the visual system and deep neural networks (DNNs). By focusing on relative sensitivities to principal distortion directions in the stimulus (image) space, the method reveals differences between models that are not captured by global geometric approaches such as representational similarity analysis (RSA). The authors demonstrate the utility of their method through computational studies, showing consistent findings with previous research on early visual system models and uncovering differences in local image representations between models like AlexNet and ResNet50.

strengths:
- The paper is clearly written and easy to follow, with a strong motivation that outlines how it builds on previous work.
- Presents a new method to compare the local geometry of image representations between models, generalizing prior approaches to accommodate multiple models.
- The method uncovers differences invisible to global geometric approaches, providing deeper insights into model behaviors.
- Includes several computational experiments that support the method's effectiveness and relevance.
- Discusses potential applications in studying biological vision, DNNs, and the interplay between them.
- Offers a well-rounded discussion that contextualizes the findings within the field.

weaknesses:
- The technical advancement may be seen as an incremental improvement over existing metrics.
- The method primarily applies to stochastic representations, which are uncommon in DNNs; applying it to deterministic models may seem arbitrary.
- It is not entirely clear how the proposed local metric can be aggregated to provide global insights across different stimuli or datasets.

questions:
1. How do you plan to extend this work to achieve more quantitative results? For instance, do you have ideas on quantifying the distinction between "noisy" versus "smooth" image distortions?
2. The authors essentially present a metric between models, which is based on the model’s representations for a single stimulus. It seems to me that the results and conclusions drawn from them could potentially disagree for different stimuli. How do you either choose a representative stimulus or how do you choose multiple and how could the results be aggregated?
3. How does your approach compare to, or how could it be integrated with, explainability methods based on saliency or relevance maps?
4. Could you clarify how you scaled the relative sensitivities between models of the early visual system (as mentioned on pages 5 and 6)? A revision of this explanation might enhance understanding.
5. Can you speculate on how the observed differences in sensitivity to noisy distortions between AlexNet and ResNet50 relate to their architectural differences or inductive biases?

Additional feedback:
- For deterministic representations in DNNs, the application of the FIM might seem less direct. It would be beneficial to elaborate on this aspect, perhaps by relating it to concepts like the pullback of the Euclidean metric onto the image space.
- Strengthening the discussion of experimental results and their key takeaways would enhance the paper. Specifically, clarifying how principal distortions inform us about a model's alignment with human visual representation, responses to adversarial examples, and contributions to interpretability would be valuable.
- Introducing more quantitative analyses or metrics could solidify the results and provide stronger evidence of the method's effectiveness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710064242

thread_type:
official_review

<thread_object>
creator: reviewer_s6hs

<content>
summary:
This paper proposes a framework for comparing image representations regarding their local geometries through the Fisher information matrix by finding a pair of “principal distortions” that maximize the variance of the models under this metric. The experiments include (1) comparing a set of simple models of the early visual systems and 2) comparing a set of deep neural network models to reveal differences in the local geometry that arise due to architecture and training types.

strengths:
1. The idea is simple but effective -- as the first approach to compare more than two models.
2. The problem statement and method description are well-written and clear to understand.
3. The experiment result is interesting, with meaningful discussions about texture bias and adversarial vulnerability. Visualizations are very helpful. 
4. The supplementary is very informative.

.

weaknesses:
1. It is not clear how this paper's principal distortions relate to human sensitivity. There seem to be no experiments to prove this statement, such as using human observers for evaluation of the principle distortions [1].
2. For example, in 4.1 Early Vision Models, it is not stated how to determine the effectiveness of this approach. There is no quantitative evaluation, and it is unclear how to understand the visualizations of the principle distortion images. 
3. Are there some practical applications of this approach? Or how to use visualizations of principle distortions?
4. What kind of networks can be compared with this approach? Only networks for classifications? How about the models for detections, segmentations, and other applications?
5. This paper is over nine pages.

[1] Alexander Berardino, Johannes Balle, Valero Laparra, and Eero Simoncelli. Eigen-distortions of ´
hierarchical representations. Advances in Neural Information Processing Systems, 30:3531–3540,
2017.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730765445412

thread_type:
official_review

<thread_object>
creator: reviewer_7znr

<content>
summary:
The paper proposes a novel framework to compare local geometry within image representations, arguing that image representation encompasses both global and local geometry information. The authors introduce a method to quantify the local geometry of image representations using the Fisher information matrix and statistical techniques sensitive to local stimulus distortion. This approach aims to identify "principal distortions pairs," which maximize model variances and serve as optimal discriminative tools for evaluating model performance.

strengths:
The framework's focus on comparing local geometry is innovative, and the use of the Fisher information matrix and sensitivity to local distortions represents a unique approach to quantifying image representation. This novel method to derive principal distortion pairs that maximize model variance offers a potentially valuable tool for model discrimination.

weaknesses:
Model Selection: The authors demonstrate the metric’s functionality using older architectures, specifically AlexNet and ResNet. Given that AlexNet is largely obsolete in current practical applications, the paper would benefit from extending the evaluation to more contemporary and widely-used networks (e.g., EfficientNet, Vision Transformers). Demonstrating the framework's effectiveness across a variety of modern architectures would strengthen the claim that the metric is universally applicable and capable of distinguishing models.

Practical Utility and Generalizability: The manuscript does not clearly establish the practical relevance and utility of this framework in real-world applications. A more explicit discussion on the potential benefits of this metric in actual deployment scenarios, or in improving model interpretability and selection, would add significant value. Additionally, empirical results supporting the framework’s effectiveness across diverse, practical model architectures are essential to substantiate the generalizability and robustness of the proposed approach.

questions:
Model Selection: The authors demonstrate the metric’s functionality using older architectures, specifically AlexNet and ResNet. Given that AlexNet is largely obsolete in current practical applications, the paper would benefit from extending the evaluation to more contemporary and widely-used networks (e.g., EfficientNet, Vision Transformers). Demonstrating the framework's effectiveness across a variety of modern architectures would strengthen the claim that the metric is universally applicable and capable of distinguishing models.

Practical Utility and Generalizability: The manuscript does not clearly establish the practical relevance and utility of this framework in real-world applications. A more explicit discussion on the potential benefits of this metric in actual deployment scenarios, or in improving model interpretability and selection, would add significant value. Additionally, empirical results supporting the framework’s effectiveness across diverse, practical model architectures are essential to substantiate the generalizability and robustness of the proposed approach.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731104316673

thread_type:
official_review

<thread_object>
creator: reviewer_wqar

<content>
summary:
This paper introduces a novel metric on image representations to measure differences in local geometry. The authors then leverage this metric to generate "principal distortions" that maximize variance across a set of models. Experimental results of the proposed method reveal qualitative differences in the local geometry of ResNet50 and AlexNet, which is quite interesting.

strengths:
- This is well-written and organized paper; hence it is easy for readers to follow. 
 
- The concept of exploring the local geometry of deep networks and examining the interplay between local geometry and the global structure of images is compelling.

- Developing a novel metric to compare two image representations is highly innovative.

weaknesses:
- While the proposed method of using the Fisher Information Matrix to measure the sensitivity of a representation to a stimulus distortion seems reasonable, its effectiveness as a metric is unclear, or difficult to justify.

- Identifying the types of principal distortions to which the network is most sensitive is an interesting idea. However, the proposed method lacks a good validation plan to confirm the accuracy or reliability of these findings.

questions:
- How is the stimulus-dependent function $f(s)$, defined after Eq. (1), computed in practice?  
- If $I(s)$ is positive semi-definite, this seems to cause issues with the metric defined in Eq. (3) when it approaches zero.   
- How are two image representations obtained from a single image $S$? Are $I_A(S)$ and $I_B(S)$ learned by different neural networks? Please clarify. 
- How are the coefficients for $\epsilon_1$ and $\epsilon_2$ determined in the proposed approach?  
- The interpretation and justification of the estimated principal distortions are difficult to justify. For instance, in Fig. 3, while the finding that AlexNet and ResNet are more sensitive to complementary parts of the images is interesting, its validity is unclear. Providing additional justification would strengthen the claims. One possible approach is to introduce adversarial noise to different parts of the images and evaluate its impact on downstream classification tasks, e.g., comparing the performance drop across different parts of the images.  
- This reviewer finds it difficult to connect the identified distortions with the concept of local geometry. The method appears to focus more on differences in image intensities and textures. However, it is possible that this reviewer has misunderstood some aspects of the proposed approach. Please clarify.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732476069210

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Thank you for your thoughtful questions, comments, and suggestions. We are pleased that reviewers found our principal distortion method “innovative”, and drew connections to many different research areas. As a result of the reviewers' comments, we have performed several new analyses studying the principal distortions and included 5 new supplementary figures. Here, we address general questions raised by multiple reviewers. Specific responses to each reviewer are below.
# DNN result quantification
In our DNN experiments, we found that AlexNet is consistently more sensitive to distortions that are concentrated on the “stuff” in the image (portions of the image with higher variability) whereas ResNet50 is more sensitive to distortions that appear concentrated on relatively constant parts of the image. We now quantify this observation with a controlled experiment (**Supp Fig 15**) in which we generate principal distortions for the DNNs trained on ImageNet or Stylized Image Net after blurring the foreground or background of a base image. The principal distortion that ResNet50 is more sensitive to (regardless of training) is consistently concentrated on the blurred portions of the image (areas lacking high spatial frequencies, regardless of whether this is in the foreground or background), while AlexNet is more sensitive to the principal distortion that is concentrated on non-blurred parts of the image.

We also measured whether the principal distortions changed the network's classification decision (**Supp Fig 13**). We find that the distortions differentially change the classification decisions. The model that is more sensitive to distortion $\boldsymbol{\epsilon}_1$ changed its classification decision when $\boldsymbol{\epsilon}_1$ was added to the base image, while the other model did not change its classification decision. This points to future work that could investigate distortions explicitly generated to differentially change models’ classification decisions (we focused on intermediate representations). 
# Early visual model clarification
There was some confusion about how we scaled the distortions before adding them to the base image in Fig 2C. We have modified the wording on **Lines 316-319** to clarify. We have also added **Supp Fig 11**, detailing how human perceptual experiments could be carried out to identify which model is best aligned with the human visual system. 
# On the choice of AlexNet & ResNet50
Several reviewers asked why we did not generate principal distortions for modern DNN architectures. We opted to present controlled experiments on well-studied architectures with varied training procedures that were publicly available. Despite the fact that these networks have been well studied, the principal distortions revealed what appear to be novel and interpretable differences between the local sensitivities of AlexNet and ResNet50. We have made this choice more explicit on **Lines 357-360**. 

That said, our method can be applied to any differentiable model. To directly demonstrate this with modern architectures, we generate principal distortions for layers of EfficientNet and a ViT (**Supp Fig 12**). We find that the distortions reliably differentiate the two networks and capture the hierarchical structure, are consistent across many base images, and are qualitatively distinct from the distortions found when comparing AlexNet vs ResNet50 (for instance, there are notable blocks corresponding to the patch size in the distortion that ViT is most sensitive to). We are also working on a Jupyter notebook that easily generates principal distortions from models in PyTorch Image Models (Timm), which we hope will encourage others to use principal distortions for model analysis. 
# Practical applications and model interpretability
Multiple reviewers asked about practical applications of principal distortions and how they contribute to model interpretability. Principal distortions can be used as an exploratory analysis method for understanding how networks differ in terms of local sensitivity, which can then lead to further investigation and experimentation. This touches on many aspects of model interpretability. For example, principal distortions can be used to identify what a model is sensitive to relative to other models, while most other interpretability tools focus on single models. We added lines to the introduction (**Lines 53 and 72-74**) and discussion (**Lines 534-536**) to make this more explicit. 

We also hope that the clarification about the early visual models demonstrates the practical applications in the domain of neuroscience/cognitive science. Overall, we think the examples in our paper highlight the broad applicability of our principal distortions method, and only scratch the surface of potential analyses and experiments to run with our approach. We are excited by the ways our method can be used for both neuroscience/cognitive science and in ML fields such as model interpretability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733243174630

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank you

comment:
We'd like to express our gratitude to the reviewers for your thoughtful and detailed feedback on our submission. Your insights encouraged us to refine several aspects of our work, including clarifying the experiments on early visual models, quantifying and expanding the DNN results, and elaborating on the practical applications of our method, particularly highlighting its connection to model interpretability. Your constructive feedback significantly enhanced our submission and underscored the broad applicability of our approach. We are excited about the potential uses now highlighted in the discussion!

Following the discussion period, we were pleased that all of the reviewers who responded indicated that we successfully addressed all of their questions and concerns. Thank you again for your feedback and enthusiasm about our principal distortion method!
</content>
</thread_object>

</thread>


Metareview:
metareview:
The current paper applies Fisher information w.r.t. multiple neural networks to find local distortions in the input space of image representations. The reviews highlighted the novelty and clear presentation. Most reviewers acknowledged that the authors' rebuttal has sufficiently addressed their concerns, and that the work is meaningful to extends such local metrics to multiple neural networks and useful for model interpretability.

In preparing the final version, please consider carefully the comments raised by all reviewers. Additionally, the authors mentioned that eq.(3) is a proper metric between positive semi-definite matrices, this is not true considering degenerate cases such as  $\epsilon'=\lambda \epsilon$, or $I_A=\lambda I_B$, or when $\epsilon$ or $\epsilon'$ lies in the kernel of $I_A$ or $I_B$ as the authors have mentioned in the rebuttal. Note a metric distance is strictly positive for any two different points. Please reword related sentences accordingly with proper citations and/or proofs.