Discriminating image representations with principal distortions

## Abstract

Abstract Image representations (artificial or biological) are often compared in terms of their global geometry; however, representations with similar global structure can have strikingly different local geometries. Here, we propose a framework for comparing a set of image representations in terms of their local geometries. We quantify the local geometry of a representation using the Fisher information matrix, a standard statistical tool for characterizing the sensitivity to local stimulus distortions, and use this as a substrate for a metric on the local geometry in the vicinity of a base image. This metric may then be used to optimally differentiate a set of models, by finding a pair of “principal distortions” that maximize the variance of the models under this metric. We use this framework to compare a set of simple models of the early visual system, identifying a novel set of image distortions that allow immediate comparison of the models by visual inspection. In a second example, we apply our method to a set of deep neural network models and reveal differences in the local geometry that arise due to architecture and training types. These examples highlight how our framework can be used to probe for informative differences in local sensitivities between complex computational models, and suggest how it could be used to compare model representations with human perception.

## 1 Introduction

Biological and artificial neural networks transform sensory stimuli into high-dimensional internal representations that support downstream tasks, and these representations are often described in terms of their neural population geometry (Chung & Abbott, ).
This idea has led to a multitude of proposed measures of representational similarity (Kriegeskorte et al., ; Yamins & DiCarlo, ; Kornblith et al., ; Williams et al., ; Klabunde et al., ), and these measures are often used to compare representations within a computational model to the representations within a brain.
However, despite differing in architectures and training procedures, many computational models of perceptual or neural responses are equally performant on these representational similarity measures (Schrimpf et al., ; Tuckute et al., ; Conwell et al., ). Are these models functionally interchangeable, or are the datasets and methods that are used to test them simply insufficient to reveal their differences?

Often the similarity between two representations is quantified by measuring alignment of the representations over a set of natural stimuli that are relatively far apart in stimulus space.
In this way, these measures capture notions of global geometric similarity between representations.
However, systems with similar global structure can have strikingly different local geometries.
For example, Szegedy et al. () found image distortions that were imperceptible to humans but led artificial neural networks to misclassify images, which motivated methods for training artificial neural networks so as to minimize their susceptibility to these adversarial examples (Goodfellow et al., ; Madry, ).
These observations suggest a need for metrics that compare the local geometries of image representations and, in particular, highlight the differences between systems even when global structure seems similar.

How can we quantify and compare the local geometry of different image representations?
A brute-force comparison clearly is prohibitive:
the space of images is extremely high-dimensional, and the set of potential distortions equally high-dimensional.
Estimating the local geometry of representations over a moderately dense sampling of this full set of possible distortions is impractical, and estimating human sensitivity to such a set is essentially impossible.
As such, it is worthwhile to develop a method for judicious selection of stimulus distortions that can be used when comparing a set of models.

We take inspiration from Zhou et al. ().
For a pair of models and a base image, they synthesize distortions along which the two models’ sensitivities maximally disagree.
This bears conceptual similarity to other methods that construct stimuli to optimally distinguish a pair of models (Wang & Simoncelli, ; Golan et al., ) or cluster neurons by cell type (Burg et al., ), and builds on earlier work that examined “eigen-distortions” along which individual models are maximally/minimally sensitive (Berardino et al., ).
Specifically, Zhou et al. measure the local sensitivity of a model in terms of its Fisher Information matrix (FIM, Fisher, ), a classical tool from statistical estimation theory, and choose the pair of “generalized eigen-distortions” that maximize/minimize the ratio of the two models’ sensitivities.
Once these image distortions have been computed, they may be added in varying amounts to a base image to determine the level at which they become visible to a human.
These measured human sensitivities can then be compared to those of the models, with the goal of identifying which model is better aligned with the local geometry of the human visual system.
However, there is no principled method for selecting image distortions for use in comparing more than two models.

Here we define a novel metric for comparing model representations in terms of their relative sensitivities to image distortions.
We then use this metric to generate a pair of distortions that maximize the variance across two or more models under this metric.
In analogy with principal component analysis, our method can be viewed as a dimensionality reduction technique that preserves as much of the variability in the local representational geometry as possible. As such, we refer to these as the “principal distortions” of the set of models.

We apply our method to a nested set of hand-crafted models of the early visual system to identify distortions that differentiate these models and can potentially be used to evaluate how well these models predict human visual sensitivities.
We then apply our method to a set of visual deep neural networks (DNNs) with varying architectures and training procedures.
We find distortions that allow for visualization of differences in the sensitivities between layers of the networks and neural network architectures. We further explore differences between standard ImageNet trained networks and their shape-bias enhanced counterparts, and between standard networks and their adversarially-trained counterparts.
In all cases, we illustrate how the method generates novel distortions that highlight differences between models.

Portions of this work were initially presented in (Lipshutz et al., ).

Figure: Figure 1: Comparing the local geometry of image representations. A) Each model maps stimuli to stochastic responses in a model’s representation space—biological neurons are inherently noisy, while a putatively deterministic model can be made stochastic by assuming additive Gaussian response noise. For example, Model A maps the stimulus ${\mathbf{s}}$ (solid black circle $\bullet$) to a conditional density $p({\mathbf{r}}_{A}|{\mathbf{s}})$ on Model A’s representation space (the solid blue circle $\bullet$ and surrounding transparent blue ellipse represent the mean and covariance of ${\mathbf{r}}_{A}$ conditioned on ${\mathbf{s}}$). B) A model’s sensitivity at the base image ${\mathbf{s}}$ to local distortions can be mapped back to the stimulus domain via the model’s positive semidefinite FIM. In the top panel “Eigen-distortions (Berardino et al., )”, the blue ellipse represents the unit level set $\{{\bm{v}}:d_{A}({\bm{v}})=1\}$ of the norm induced by Model A’s FIM ${\bm{I}}_{A}$, which is the set of distortions of the base stimulus ${\mathbf{s}}$ that appear equally distorted according to Model A’s representation. The eigenvectors of the FIM (${\bm{\epsilon}}_{1}$, ${\bm{\epsilon}}_{2}$) can equivalently be interpreted as the distortions that maximize the magnitude of the log ratio of the model’s sensitivities, which is represented as a solid blue circle $\bullet$ on the number line. In the middle panel “Generalized Eigen-distortions (Zhou et al., )”, the blue ellipse is copied from the top panel and the orange ellipse is the level set of Model B’s FIM ${\bm{I}}_{B}$. The generalized eigenvectors of ${\bm{I}}_{A}$ and ${\bm{I}}_{B}$ (${\bm{\epsilon}}_{1}$, ${\bm{\epsilon}}_{2}$) can equivalently be interpreted as the vectors that maximize the difference between the log ratios of the models’ sensitivities. In the bottom panel “Principal distortions (this paper)”, the blue and orange ellipses are as in the above panels and the green ellipse represents the level set of Model C’s FIM ${\bm{I}}_{C}$. We propose a general method for comparing an arbitrary number of models, by selecting two stimulus distortions (${\bm{\epsilon}}_{1}$, ${\bm{\epsilon}}_{2}$) that maximize the variance of the log ratios of model sensitivities.
Refer to caption: x1.png

## 2 Problem statement and existing methods

Given a collection of image representations, our goal is to develop a method for comparing the local geometries of these representations in the vicinity of some base input image.
In this section, we define the local geometry of an image representation in terms of the FIM and review existing methods for selecting image distortions based on model FIMs.

### 2.1 Local information geometry of stochastic image representations

We assume that each image representation has an associated conditional density $p({\mathbf{r}}|{\mathbf{s}})$, where ${\mathbf{s}}$ is a $K$-dimensional vector of image pixels and ${\mathbf{r}}$ is a vector of stochastic responses (e.g., biological neuronal firing rates or deterministic neural network activations with additive response noise).
Figure 1A depicts a two-dimensional stimulus space and three models mapping stimuli ${\mathbf{s}}$, ${\mathbf{s}}+{\bm{\epsilon}}_{1}$, and ${\mathbf{s}}+{\bm{\epsilon}}_{2}$ to conditional densities.
Note that the dimension of the responses may vary across representations.

The sensitivity of the representation to a small distortion ${\bm{\epsilon}}$ depends on the overlap between the conditional distributions $p({\mathbf{r}}|{\mathbf{s}})$ and $p({\mathbf{r}}|{\mathbf{s}}+{\bm{\epsilon}})$, with less overlap corresponding to higher sensitivity (Green & Swets, ).
This sensitivity can be precisely quantified in terms of the Fisher-Rao metric (Rao, ; Amari, ), a Riemannian metric on the stimulus space (Fig. 1B) that is defined in terms of the positive semi-definite FIM (Fisher, ):

$$ $\displaystyle{\bm{I}}({\mathbf{s}}):=\mathbb{E}_{{\mathbf{r}}\sim p({\mathbf{r }}|{\mathbf{s}})}\left[\nabla_{{\mathbf{s}}}\log p({\mathbf{r}}|{\mathbf{s}}) \nabla_{{\mathbf{s}}}\log p({\mathbf{r}}|{\mathbf{s}})^{\top}\right],$ $$

where $\nabla_{\mathbf{s}}\log p({\mathbf{r}}|{\mathbf{s}})$ denotes the gradient of $\log p({\mathbf{r}}|{\mathbf{s}})$ with respect to ${\mathbf{s}}$.
The FIM is a standard tool in statistical estimation theory that locally approximates the expected log likelihood ratio (or KL divergence) between the conditional distributions $p({\mathbf{r}}|{\mathbf{s}})$ and $p({\mathbf{r}}|{\mathbf{s}}+{\bm{\epsilon}})$, and provides a lower bound on the variance of any unbiased estimator of ${\mathbf{s}}$ (Cramér, ; Rao, ).
The FIM has also been used to link neural representations to perceptual discrimination
(Paradiso, ; Seung & Sompolinsky, ; Brunel & Nadal, ; Averbeck & Lee, ; Seriès et al., ; Ganguli & Simoncelli, ; Wei & Stocker, ).
Given the FIM, we can define the sensitivity of the representation of stimulus ${\mathbf{s}}$ to a distortion ${\bm{\epsilon}}$ as:

$$ $\displaystyle d({\mathbf{s}};{\bm{\epsilon}}):=\sqrt{{\bm{\epsilon}}^{\top}{ \bm{I}}({\mathbf{s}}){\bm{\epsilon}}},$ (1) $$

which quantifies how well an ideal observer could detect small perturbations of the base stimulus in the direction ${\bm{\epsilon}}$.

As a tractable example, suppose that the conditional response ${\mathbf{r}}$ is Gaussian with stimulus-dependent mean ${\bm{f}}({\mathbf{s}})$ and constant covariance ${\bm{\Sigma}}$.
Then the FIM at ${\mathbf{s}}$ is

$$ $\displaystyle{\bm{I}}({\mathbf{s}})={\bm{J}}_{f}({\mathbf{s}})^{\top}{\bm{ \Sigma}}^{-1}{\bm{J}}_{f}({\mathbf{s}}),$ $$

where ${\bm{J}}_{f}({\mathbf{s}})$ is the Jacobian of ${\bm{f}}(\cdot)$ at ${\mathbf{s}}$.
From this expression, we see that the sensitivities of a representation to a distortion ${\bm{\epsilon}}$ depend on how the mean response is changing in the direction ${\bm{\epsilon}}$ relative to the noise covariance.

### 2.2 Eigen-distortions of an image representation

Given a model image representation, Berardino et al. () proposed the use of extremal eigenvectors of the model FIM (termed “eigen-distortions”, Fig. 1B, top panel) as model predictions of the most- and least-noticeable image distortions.
For a set of early vision models and deep neural networks whose parameters were optimized to match a database of human image quality ratings (Ponomarenko et al., ), they computed the eigen-distortions of each model.
Despite the fact that these models all fit the image quality data equally well, their eigen-distortions were quite different, and human perceptual judgements of the severity of these eigen-distortions varied substantially.
The eigen-distortions of each image representation correspond to its most distinct distortion predictions, which can then be tested with human perceptual experiments.
However, if the eigen-distortions of two models are similar, they will not be useful in distinguishing the models, since this method is insensitive to differences in the non-extremal eigenvectors.

### 2.3 Generalized eigen-distortions for comparing two image representations

Zhou et al. () proposed comparing two image representations $A$ and $B$ along distortions in which their local sensitivities maximally differ, which is conceptually similar to previous methods that construct stimuli that maximize disagreement between models (Wang & Simoncelli, ; Golan et al., ).
Specifically, they chose distortions to extremize the generalized Rayleigh quotient:

$$ $\displaystyle{\bm{\epsilon}}_{1}({\mathbf{s}})$ $\displaystyle=\operatorname*{arg\,max}_{\bm{\epsilon}}\frac{{\bm{\epsilon}}^{ \top}{\bm{I}}_{A}({\mathbf{s}}){\bm{\epsilon}}}{{\bm{\epsilon}}^{\top}{\bm{I}} _{B}({\mathbf{s}}){\bm{\epsilon}}},$ $\displaystyle{\bm{\epsilon}}_{2}({\mathbf{s}})=\operatorname*{arg\,min}_{\bm{ \epsilon}}\frac{{\bm{\epsilon}}^{\top}{\bm{I}}_{A}({\mathbf{s}}){\bm{\epsilon} }}{{\bm{\epsilon}}^{\top}{\bm{I}}_{B}({\mathbf{s}}){\bm{\epsilon}}}.$ (2) $$

Since these distortions correspond to the extremal eigenvectors of the generalized eigenvalue problem ${\bm{I}}_{A}({\mathbf{s}}){\bm{\epsilon}}=\lambda{\bm{I}}_{B}({\mathbf{s}}){
\bm{\epsilon}}$, we refer to them as “generalized eigen-distortions” (Fig. 1B, middle panel).
However, this method is limited to comparisons of pairs of models, or of a single model against the average of other models.

## 3 Principal distortions of image representations

We propose a natural extension of generalized eigen-distortions that allows for comparisons among more than two image representations.
We show that the generalized eigenvalue problem suggests a metric on image representations, which can be used to optimally choose image distortions that distinguish more than two models.

### 3.1 A metric on the local geometry of image representations

We can re-express the generalized eigen-distortions defined in Equation 2 as the solutions of a single optimization problem:

$$ $\displaystyle\{{\bm{\epsilon}}_{1},{\bm{\epsilon}}_{2}\}=\operatorname*{arg\, max}_{{\bm{\epsilon}},{\bm{\epsilon}}^{\prime}}\left\{\log\frac{{\bm{\epsilon} }^{\top}{\bm{I}}_{A}{\bm{\epsilon}}}{{\bm{\epsilon}}^{\top}{\bm{I}}_{B}{\bm{ \epsilon}}}-\log\frac{{\bm{\epsilon}}^{\prime\top}{\bm{I}}_{A}{\bm{\epsilon}}^ {\prime}}{{\bm{\epsilon}}^{\prime\top}{\bm{I}}_{B}{\bm{\epsilon}}^{\prime}} \right\},$ $$

where we’ve simplified notation by omitting the dependence of ${\bm{I}}_{*}$ and ${\bm{\epsilon}}_{*}$
on ${\mathbf{s}}$.
We can regroup the numerators and denominators of the log quotients by model rather than by distortion, to re-express the optimization problem as follows:

$$ $\displaystyle\{{\bm{\epsilon}}_{1},{\bm{\epsilon}}_{2}\}=\operatorname*{arg\, max}_{{\bm{\epsilon}},{\bm{\epsilon}}^{\prime}}m_{{\bm{\epsilon}},{\bm{ \epsilon}}^{\prime}}({\bm{I}}_{A},{\bm{I}}_{B}),$ $$

where

$$ $\displaystyle m_{{\bm{\epsilon}},{\bm{\epsilon}}^{\prime}}({\bm{I}}_{A},{\bm{I }}_{B}):=\left|\log\frac{d_{A}({\bm{\epsilon}})}{d_{A}({\bm{\epsilon}}^{\prime })}-\log\frac{d_{B}({\bm{\epsilon}})}{d_{B}({\bm{\epsilon}}^{\prime})}\right|$ (3) $$

and we have used the definition of the sensitivity in Equation 1 and dropped a factor of 2.
For any pair of distortions ${\bm{\epsilon}},{\bm{\epsilon}}^{\prime}$, the function $m_{{\bm{\epsilon}},{\bm{\epsilon}}^{\prime}}(\cdot,\cdot)$ is a proper metric on positive semi-definite matrices. Specifically, it is non-negative, symmetric, obeys the triangle inequality, and is zero when ${\bm{I}}_{A}={\bm{I}}_{B}$.
This metric has several appealing properties:

- •
Invariance to scaling of the FIMs by positive constants $c_{A},c_{B}>0$:
$\displaystyle m_{{\bm{\epsilon}},{\bm{\epsilon}}^{\prime}}({\bm{I}}_{A},{\bm{I
}}_{B})=m_{{\bm{\epsilon}},{\bm{\epsilon}}^{\prime}}(c_{A}{\bm{I}}_{A},c_{B}{
\bm{I}}_{B}).$
This is a desirable property since we are interested in identifying relevant image distortions that depend on the shape of the FIMs, independent of scaling factors.
- •
Invariance to permutation of the distortions ${\bm{\epsilon}}$ and ${\bm{\epsilon}}^{\prime}$:
$\displaystyle m_{{\bm{\epsilon}},{\bm{\epsilon}}^{\prime}}({\bm{I}}_{A},{\bm{I
}}_{B})=m_{{\bm{\epsilon}}^{\prime},{\bm{\epsilon}}}({\bm{I}}_{A},{\bm{I}}_{B}).$
- •
When ${\bm{\epsilon}}_{1}$ and ${\bm{\epsilon}}_{2}$ are the generalized eigen-distortions of ${\bm{I}}_{A}$ and ${\bm{I}}_{B}$, $m_{{\bm{\epsilon}}_{1},{\bm{\epsilon}}_{2}}({\bm{I}}_{A},{\bm{I}}_{B})$ is an approximation of the Fisher-Rao distance between mean-zero Gaussian distributions with covariances ${\bm{I}}_{A}$ and ${\bm{I}}_{B}$ (up to scaling factors, see Appx. A).
This interpretation suggests a principled extension of the metric to more than two distortions.
- •
The metric compares stochastic representations back in stimulus space via their FIMs.
This avoids the problem of aligning the stochastic representations, which is a computationally intensive step that is required when evaluating existing metrics on stochastic representations (Duong et al., ).

### 3.2 Principal distortions for comparing multiple image representations

To optimize a pair of image distortions for distinguishing $N>2$ representations, $A_{1},\dots,A_{N}$, we choose ${\bm{\epsilon}}_{1},{\bm{\epsilon}}_{2}$ to maximize the sum of the squares of all pairwise distances between the FIMs under the metric defined in Equation 3:

$$ $\displaystyle\{{\bm{\epsilon}}_{1},{\bm{\epsilon}}_{2}\}=\operatorname*{arg\, max}_{{\bm{\epsilon}},{\bm{\epsilon}}^{\prime}}\sum_{n=1}^{N}\sum_{m=1}^{N}m_{ {\bm{\epsilon}},{\bm{\epsilon}}^{\prime}}^{2}({\bm{I}}_{A_{n}},{\bm{I}}_{A_{m} }).$ (4) $$

This is equivalent to maximizing the variance of the image representations’ log sensitivity ratios:

$$ $\displaystyle\{{\bm{\epsilon}}_{1},{\bm{\epsilon}}_{2}\}=\operatorname*{arg\, max}_{{\bm{\epsilon}},{\bm{\epsilon}}^{\prime}}\sum_{n=1}^{N}\left|\log\frac{d _{A_{n}}({\bm{\epsilon}})}{d_{A_{n}}({\bm{\epsilon}}^{\prime})}-\frac{1}{N} \sum_{m=1}^{N}\log\frac{d_{A_{m}}({\bm{\epsilon}})}{d_{A_{m}}({\bm{\epsilon}}^ {\prime})}\right|^{2}$ $$

We refer to $\{{\bm{\epsilon}}_{1},{\bm{\epsilon}}_{2}\}$ as the “principal distortions” of the models, analogous to principal component analysis (Fig. 1B).
For a gradient-based optimization algorithm, see Appx. B.

Note that there are several other natural extensions of generalized-eigendistortions when considering $N>2$ models.
For example, for any $p\geq 1$, one can choose distortions ${\bm{\epsilon}}_{1},{\bm{\epsilon}}_{2}$ that maximize the sum of the $p$^th power of all pairwise distances, which amounts to replacing $m_{{\bm{\epsilon}},{\bm{\epsilon}}^{\prime}}^{2}({\bm{I}}_{A_{n}},{\bm{I}}_{A_
{m}})$ in Equation 4 with $m_{{\bm{\epsilon}},{\bm{\epsilon}}^{\prime}}^{p}({\bm{I}}_{A_{n}},{\bm{I}}_{A_
{m}})$.
Here we focus on the case of maximizing the variance ($p=2$) and leave an exploration of higher-order moments to future work

## 4 Experimental results

As a demonstration of our method, we generated principal distortions for computational models previously proposed to capture aspects of the human visual system. All models were implemented in PyTorch (Ansel et al., ) and simulations were performed on NVIDIA GPUs (RTX A6000 and A100 models). As the models are deterministic, we calculate the FIM by assuming the network output is corrupted by additive Gaussian noise, as in (Berardino et al., ).
In this case,
${\bm{I}}({\mathbf{s}})={\bm{J}}_{f}({\mathbf{s}})^{\top}{\bm{J}}_{f}({\mathbf{
s}})$,
where ${\bm{J}}_{f}({\mathbf{s}})$ is the Jacobian of the model ${\bm{f}}(\cdot)$ at input ${\mathbf{s}}$.

Figure: Figure 2: Principal distortions of a set of early visual models. A) Log sensitivity ratios of the two principal distortions and two random distortions for each of the four models (Berardino et al., ). Models in nested order (LN is the most basic, LGN is the full model). Principal distortions (filled circles) separate the log ratios, while random distortions (hollow circles) do not. B) Natural image $\mathbf{s}$ and corresponding optimized principal distortions $\{{\bm{\epsilon}}_{1},{\bm{\epsilon}}_{2}\}$. C) Natural image corrupted by principal distortions, with each pair scaled so as to be equally detectable by the corresponding model (abbreviations above). Models are ordered by the log ratio of their sensitivities (panel A). If a model’s thresholds are comparable to human thresholds, the two scaled distortions should be equally visible in the top and bottom images. Note: Distorted images are best viewed at high resolution.
Refer to caption: x2.png

### 4.1 Early visual models

We generated principal distortions for a nested family of models designed to capture the response properties of early stages (specifically, the lateral geneiculate nucleus, LGN) of the primate visual system (Fig. 2). The full model (LGN) contains two parallel cascades representing ON and OFF center-surround filter channels, rectification, and both luminance and contrast gain control nonlinearities. The other models are reduced versions of this model. LGG removes the OFF channel, LG additionally removes the contrast gain control, and LN removes both gain control operations. The filter sizes, amplitudes, and normalization values of each model were previously fit separately to predict a dataset of human distortion ratings (Berardino et al., , see details in Appx. C).

As these models were explicitly trained to predict human distortion thresholds, we provide a qualitative comparison of each model’s sensitivities to human distortion sensitivity (Fig. 2C). We adjusted the relative scaling of the principal distortion so as to be equally detectable by that model while constraining the sum of the Euclidean norms of the two distortions to be a fixed value of 100; that is, we chose positive scalars $c_{1},c_{2}$ such that $d(c_{1}{\bm{\epsilon}}_{1})=d(c_{2}{\bm{\epsilon}}_{2})$ and $c_{1}+c_{2}=100$. If a model’s thresholds are comparable to human thresholds, then these rescaled distortions should be equally detectable when added to the image ${\mathbf{s}}$. Visual inspection of these images reveals that both distortions are visible when rescaled for the LGN model and the LN model, suggesting that these models are closest to human distortion thresholds. For LG, the scaled ${\bm{\epsilon}}_{2}$ distortion is not visible, while the scaled ${\bm{\epsilon}}_{1}$ distortion is immediately apparent, suggesting a strong mismatch with human observer thresholds. The same is true of the LGG model, with the roles of the two distortions swapped.
These qualitative observations are consistent with the results of (Berardino et al., ), in which experiments on eigen-distortions suggested that the LGN model was the best of these models in terms of consistency with human distortion sensitivity.
Analogous perceptual experiments could be used to quantify the visibility of the principal distortions arising from our analysis.

An advantage of our framework is that it can dramatically reduce the number of distortions that are needed to differentiate a set of models.
For instance, to judge how well these models of the early visual system capture human perceptual discrimination thresholds, Berardino et al. () computed the extremal eigen-distortions for each of the four models, and then measured human perceptual discrimination thresholds to all eight distortions.
In general, their method requires assessing visibility of $2N$ distortions, for $N$ models.
Zhou et al. () considered a pair of generalized eigen-distortions for each pair of models, for a total of $N(N+1)$ distortions.
In contrast, our method always selects the two distortions that maximize the variance across the models, independent of $N$.
Human sensitivities to these distortions can then be estimated in perceptual discrimination experiments to judge which model(s) are closest in terms of the metric we defined Equation 3.
The models whose sensitivities are far from human sensitivities can be discarded and this procedure can be repeated to best differentiate the remaining models, and so on. If one could reduce the number of models by, say, a factor of two on each iteration of this process, the total number of stimuli to be assessed scales as $2\log_{2}(N)$.
This dramatic improvement in efficiency could enable the comparison of significantly larger sets of models than feasible with previous methods.

### 4.2 Deep neural networks

Figure: Figure 3: AlexNet versus ResNet50. A) Example base images. B) Optimized principal distortions (scaled by a factor of 20 for visibility, and using the convention $\|{\bm{\epsilon}}\|=1$ here and in other figures). C) Base image plus principal distortions. D) Log sensitivity ratios of principal distortions when comparing image representations at multiple layers of in AlexNet and ResNet50. Assignment of ${\bm{\epsilon}}_{1}$ and ${\bm{\epsilon}}_{2}$ was chosen so that the final tested layer of AlexNet has a positive log ratio. E) Log sensitivity ratios averaged across 100 base images (error bars are standard deviation). The principal distortions organize the networks by architecture—the log sensitivity ratios of AlexNet and ResNet50 are separated and early layers have smaller log ratios than late layers. AlexNet is more sensitive to distortion ${\bm{\epsilon}}_{1}$, which is concentrated on higher contrast or textured parts of the image (often the foreground object). ResNet50 is more sensitive to distortion ${\bm{\epsilon}}_{2}$ which concentrates power on relatively smooth parts of the image, such as regions of constant intensity/color.
Refer to caption: x3.png

Figure: Figure 4: Comparison of AlexNet and ResNet50 variants trained to increase “Shape Bias” of networks. A) Log sensitivity ratios of principal distortions for networks trained on standard ImageNet and Stylized ImageNet (SIN). Average log sensitivity is computed across 100 base images (error bars are standard deviation) and the choice of ${\bm{\epsilon}}_{1}$ is set such that $d({\bm{\epsilon}}_{1})\geq 0$ for the last tested layer of the standard AlexNet. B) Example base image and the optimized principal distortions. Similar to the analysis of the ImageNet-trained ResNet50 and AlexNet (Fig. 3), ${\bm{\epsilon}}_{1}$ is concentrated on parts of the image with higher spatial frequencies while ${\bm{\epsilon}}_{2}$ is concentrated on parts of the image with relatively solid patches. Both AlexNet architectures are more sensitive to ${\bm{\epsilon}}_{1}$ while both ResNet50 architectures are more sensitive to ${\bm{\epsilon}}_{2}$, suggesting that the differences in local sensitivities of these networks depend more on differences in architecture than training procedure.
Refer to caption: x4.png

Deep Neural Networks (DNNs), originally developed for object recognition, have also been examined as models of the primate visual system (Yamins & DiCarlo, ; Schrimpf et al., ; Lindsay, ). A plethora of models, varying in architecture and training techniques, have been proposed, but many of these models perform quite similarly on behavioral tasks or neural benchmarks (Schrimpf et al., ; Tuckute et al., ; Conwell et al., ).
This situation offers a well-aligned opportunity for use of our principal distortion method.

We first measured the FIM of a set of layers from two different architectures trained on the ImageNet object classification task—AlexNet (Krizhevsky et al., ) and ResNet50 (He et al., )—and generated the principal distortions that maximally separate these models (Fig. 3, see Sec. D of the supplement for layer choices and model details).
Although these architectures are not currently state-of-the-art at image recognition or neural prediction, they have been widely used and trained with various optimization strategies. Notably, the hierarchical structure of the models is reflected in the log ratios of the sensitivities, where early layers of the models are closer together in the metric space (Fig. 3D), and late layers of AlexNet are always more sensitive to ${\bm{\epsilon}}_{1}$ when late layers of ResNet50 are more sensitive to ${\bm{\epsilon}}_{2}$.
There is additional structure revealed by these principal distortions—AlexNet is more sensitive to the principal distortion that generally appears concentrated on regions of the image that have more variability, i.e., the “stuff” in the image (distortion ${\bm{\epsilon}}_{1}$ in Fig. 3B), while ResNet50 is more sensitive to distortions that occur in the relatively constant regions of the image (distortion ${\bm{\epsilon}}_{2}$ in Fig. 3B).
The separability of the models and the sensitivity of the networks to the two distortion types was remarkably consistent across a set of 100 base images chosen from the ImageNet dataset (Fig. 3E, see Fig. 6 of the supplement for additional examples). This distinction was also found in images designed explicitly to test for possible differences between models due to edge artifacts or contrast sensitivity (Fig. 7 of the supplement). As far as we know, this qualitative difference in sensitivities of the architectures to distortions located at different places of the image has not been documented, demonstrating that our method can reveal interpretable differences in the local sensitivities of complex computational models.

#### Networks trained to reduce texture bias

The architectural difference observed between ImageNet-trained AlexNet and ResNet50 suggested that the texture of the image may be driving some of the differences in local geometry. Previous work demonstrated that standard DNNs exhibit strong “texture bias” (Geirhos et al., ) and proposed models that explicitly reduce the texture bias by training on Stylized ImageNet (SIN), a set of images that retains the content of each ImageNet image but overlays details matched to particular texture. Training on these images can reduce the model’s reliance on texture for classification. If this training set strongly affected the local geometry of the networks, we might expect that principal distortions generated for a mixture of architectures and training sets would be driven by this training set distinction. We find evidence that this is not the case. We generated principal distortions for 100 base images using a set of layers from two architectures (ResNet50 and AlexNet) and two different training datasets (ImageNet and Stylized ImageNet) (Fig. 4A).
The principal distortions appeared qualitatively similar to those generated when we investigated only the standard networks (Fig. 4B, more examples in Fig. 8 of the supplement). Both AlexNet architectures, regardless of the training type, were more sensitive to the perturbations of higher variability parts of the image, while both ResNet50 models were more sensitive to the distortions that were mainly targeting constant areas of the image.

#### Networks trained to reduce adversarial vulnerability

Another well-known example of the use of training set modifications to achieve robustness in object recognition is that of adversarial training (AT).
Adversarial examples are generated at each step of model training and these stimuli are used to update the model weights, with the “true” category label used for the update. Previous work has found that adversarially-trained networks are more aligned with those of biological systems (Madry, ; Feather et al., ; Gaziv et al., ). As adversarial examples are constrained to be very small perturbations, it seems plausible that the local geometry for AT models would differ from their standard counterparts. Indeed, we see that the principal distortions generated from the set of models that included standard and AT trained AlexNet and ResNet50 models reliably separate the model classes by training type, rather than architecture (Fig. 5A). Most layers of the adversarially trained models are more sensitive to relatively smooth changes of patches of color in the image, or to shading around edges, while most layers of the standard models are more sensitive to what appears as unstructured noise (Fig. 5B, additional examples in Fig. 9 of the supplement). This is in stark contrast to the style-trained networks of the previous section, for which principal distortions reliably separated the models by architecture. These examples provide a compelling demonstration that our method can be used to separate collections of similar models, and points to its utility in probing complex high-level representations.

Figure: Figure 5: Comparison of AlexNet and ResNet50 variants trained to reduce adversarial vulnerability. A) Log sensitivity ratios of principal distortions for different layers of standard-trained and adversarially-trained (AT) models. Average log sensitivity is computed across 100 base images (error bars are standard deviation) and the choice of ${\bm{\epsilon}}_{1}$ is set such that $d({\bm{\epsilon}}_{1})\geq 0$ for the last tested layer of the standard AlexNet. B) Example base image and the generated principal distortions. Distortion ${\bm{\epsilon}}_{1}$ appears as less structured noise, and both AlexNet and ResNet50 standard networks are more sensitive to this perturbation, while the AT DNNs are more sensitive to ${\bm{\epsilon}}_{2}$ which focuses color changes around the content of the image, suggesting that the differences in local sensitivities of these networks depend more on differences in training procedure than architecture.
Refer to caption: x5.png

## 5 Discussion

We have introduced a metric on image representations that quantifies differences in local geometry, and used it to synthesize “principal distortions” that maximize the variance of the metric over a set of models.
When applied to hand-engineered models of the early visual system and to deep neural networks, our approach produced novel distortions for distinguishing the corresponding models. In particular, with the DNN analysis, we revealed that there are qualitative differences in local geometries of ResNet50 and AlexNet architectures, that adversarial training dramatically changes the local geometry, but stylized ImageNet training does not. Although our qualitative examples in this targeted set of neural networks do not fully elucidate the recent observations that many different models are equally good at capturing brain representations, our method provides a direct approach to begin to tease apart the interplay between local geometry and global structure.

There are several natural methodological extensions.
The metric is closely related to the Fisher-Rao metric between mean zero Gaussians and this relation suggests a natural extension to synthesizing more than two distortions (Appx. A), analogous to using additional principal components to capture more variance within a set of high-dimensional vectors.
Additionally, while we focus on computing principal distortions that differentiate a finite set of models, there is a natural extension to computing the principal distortions that differentiate continuous families of models with a prior distribution over models.

There are several limitations in our formulation of principal distortions.
First, our framework is based on local differential analyses of a model at a base image, so the sensitivity estimates we obtain via these analyses can only be guaranteed to hold in an infinitesimally small neighborhood of the base image.
If a model is highly nonlinear in the vicinity of a base image, then the local linear approximation may not accurately reflect model sensitivities.
Second, to compute the FIM of a deterministic model, we assume additive Gaussian response noise, which is not generally representative of neural responses in the brain.
Poisson variability could be used to better capture neural responses.
Another approach is to fit the model noise structure to measurements of a neural system; see (Ding et al., ) for work along these lines.

Principal distortions provide an efficient method for comparing computational models with human observers, for whom experimental time for acquiring responses to stimuli is generally severely limited. Although we only presented qualitative comparisons and examples related to human perception in this paper, the optimized distortions are a parsimonious choice of stimuli that can be readily incorporated into psychophysics experiments. For example, the distortions generated from the early visual models could be used for a perceptual discrimination experiment with human observers similar to those performed by Berardino et al. (), to compare the human log-ratio sensitivity for the optimized distortions to that predicted by the models.

The results with DNNs reveal some intriguing properties: some models have stronger sensitivity biases in the local geometry for perturbing higher variability regions of the space, while others have more sensitivity to perturbations in relatively blank regions (see Fig. 7 of the supplement for direct evidence of this).
This also provides and interesting question for future distortion detection: in what contexts are humans more sensitive to perturbations of the “stuff” of the image compared to perturbations in empty parts of the image? And how do these observed differences relate to previous work investigating how human and neural networks rely on on spatial frequencies for classification (Subramanian et al., )? Finally, beyond direct comparisons with human observers using psychophysics experiments, our method may be useful in the domain of neural network interpretability, where it may be useful to have direct comparisons of the local distortions that will maximally differentiate sets of models.

## Acknowledgements

The Flatiron Institute is a division of the Simons Foundation. The computations reported in this paper were performed using resources made available by the Flatiron Institute. We thank David Brainard, Thomas Yerxa, and the Laboratory for Computational Vision at NYU and the Flatiron Instiute for their feedback.

## 6 Reproducibility Statement

The code used to generate principal distortions and details of loading the models is uploaded as a .zip file for the submission, and will be made available via a public GitHub repository upon publication. We aim for the distortion generation to be easy for others to use to probe new models.
The mathematical derivation of the algorithm is provided in Appx. B.
We have included details in Appx. C with the parameters of the early visual models, and Appx. D details where checkpoints were obtained for the tested deep neural networks.

## Appendix A Relation to the Fisher-Rao metric

The Fisher-Rao distance between two mean zero $K$-dimensional Gaussian distributions with positive definite covariance matrices ${\bm{A}}$ and ${\bm{B}}$ is defined as:

$$ $\displaystyle\delta^{2}({\bm{A}},{\bm{B}}):=\|\log({\bm{B}}^{-1/2}{\bm{A}}{\bm {B}}^{-1/2})\|_{F}^{2}=\sum_{i=1}^{K}(\log\lambda_{i})^{2},$ $$

where $\{\lambda_{i}\}$ denote the eigenvalues of the generalized eigenvalue problem ${\bm{A}}{\bm{v}}=\lambda{\bm{B}}{\bm{v}}$ (Pinele et al., ).
We’d like a metric that’s invariant to arbitrary scalings ${\bm{A}}\mapsto c_{A}{\bm{A}}$ or ${\bm{B}}\mapsto c_{B}{\bm{B}}$ for $c_{A},c_{B}>0$, which suggests the following definition:

$$ $\displaystyle\gamma^{2}({\bm{A}},{\bm{B}})=\min_{c_{A},c_{B}>0}\delta^{2}(c_{A }{\bm{A}},c_{B}{\bm{B}})=\min_{c\in\mathbb{R}}\sum_{i=1}^{K}(c+\log\lambda_{i} )^{2}=K\text{Var}(\{\log\lambda_{i}\}),$ (5) $$

where the final equality uses the fact that the optimal $c$ is the mean of $\{-\log\lambda_{i}\}$.

Using the facts that

$$ $\displaystyle K\text{Var}(\{\log\lambda_{i}\})$ $\displaystyle=\frac{1}{2K}\sum_{i=1}^{K}\sum_{j=1}^{K}(\log\lambda_{i}-\log \lambda_{j})^{2},$ $$

and $(\log\lambda_{i}-\log\lambda_{j})^{2}\leq(\log\lambda_{1}-\log\lambda_{K})^{2}$ for all $i,j$, we have

$$ $\displaystyle\frac{1}{K}(\log\lambda_{1}-\log\lambda_{K})^{2}\leq\gamma^{2}({ \bm{A}},{\bm{B}})\leq\frac{K-1}{2}(\log\lambda_{1}-\log\lambda_{K})^{2}.$ $$

When ${\bm{A}}={\bm{I}}_{A}$ and ${\bm{B}}={\bm{I}}_{B}$, then $d_{A}({\bm{\epsilon}})=\sqrt{{\bm{\epsilon}}^{\top}{\bm{A}}{\bm{\epsilon}}}$ and $d_{B}({\bm{\epsilon}})=\sqrt{{\bm{\epsilon}}^{\top}{\bm{B}}{\bm{\epsilon}}}$.
If ${\bm{\epsilon}}_{1}$ and ${\bm{\epsilon}}_{K}$ denote the extremal generalized eigenvectors associated with $\lambda_{1}$ and $\lambda_{K}$, respectively, then

$$ $\displaystyle\log\lambda_{1}=2\log\frac{d_{A}({\bm{\epsilon}}_{1})}{d_{B}({\bm {\epsilon}}_{1})},$ $\displaystyle\log\lambda_{K}=2\log\frac{d_{A}({\bm{\epsilon}}_{K})}{d_{B}({\bm {\epsilon}}_{K})}.$ $$

Therefore,

$$ $\displaystyle\frac{2}{\sqrt{K}}\left|\log\frac{d_{A}({\bm{\epsilon}}_{1})}{d_{ B}({\bm{\epsilon}}_{1})}-\log\frac{d_{A}({\bm{\epsilon}}_{K})}{d_{B}({\bm{ \epsilon}}_{K})}\right|\leq\gamma({\bm{A}},{\bm{B}})\leq\sqrt{2(K-1)}\left| \log\frac{d_{A}({\bm{\epsilon}}_{1})}{d_{B}({\bm{\epsilon}}_{1})}-\log\frac{d_ {A}({\bm{\epsilon}}_{K})}{d_{B}({\bm{\epsilon}}_{K})}\right|,$ $$

and so

$$ $\displaystyle\frac{2}{\sqrt{K}}m_{{\bm{\epsilon}}_{1},{\bm{\epsilon}}_{K}}({ \bm{I}}_{A},{\bm{I}}_{B})\leq\gamma({\bm{A}},{\bm{B}})\leq\sqrt{2(K-1)}m_{{\bm {\epsilon}}_{1},{\bm{\epsilon}}_{K}}({\bm{I}}_{A},{\bm{I}}_{B}).$ $$

#### Extension to more than two distortions

Equation 5 suggests a natural extension for defining a metric between positive definite matrices using $M>2$ distortions.
Specifically, it suggests choosing the distortions to maximize the variance across the log ratios of the sensitivities.
To this end, we can define the squared distance between the local geometries to be the variance (across distortions) of the log ratio of the the sensitivities:

$$ $\displaystyle m_{{\bm{\epsilon}}_{1},\dots,{\bm{\epsilon}}_{M}}^{2}({\bm{A}},{ \bm{B}})=M\text{Var}\left(\left\{\log\frac{d_{A}({\bm{\epsilon}}_{m})}{d_{B}({ \bm{\epsilon}}_{m})}\right\}\right).$ $$

A set of $M$ principal distortions can then be chosen to maximize the variance across models under this metric.

## Appendix B Computing the top two optimal distortions

Suppose we have $N$ models with sensitivities $\{d_{n}({\bm{\epsilon}})\}$.
The optimal distortions $\{{\bm{\epsilon}}_{1},{\bm{\epsilon}}_{2}\}$ are solutions to the optimization problem

$$ $\displaystyle\operatorname*{arg\,max}_{{\bm{\epsilon}}_{1},{\bm{\epsilon}}_{2} }L({\bm{\epsilon}}_{1},{\bm{\epsilon}}_{2}),$ $\displaystyle L({\bm{\epsilon}}_{1},{\bm{\epsilon}}_{2}):=\sum_{n=1}^{N}\left \{\log\frac{d_{n}({\bm{\epsilon}}_{1})}{d_{n}({\bm{\epsilon}}_{2})}-\frac{1}{N }\sum_{m=1}^{N}\log\frac{d_{m}({\bm{\epsilon}}_{1})}{d_{m}({\bm{\epsilon}}_{2} )}\right\}^{2}.$ $$

Differentiating $L$ with respect to ${\bm{\epsilon}}_{1}$ yields

$$ $\displaystyle\nabla_{{\bm{\epsilon}}_{1}}L({\bm{\epsilon}}_{1},{\bm{\epsilon}} _{2})$ $\displaystyle=2\sum_{n=1}^{N}\left\{\log\frac{d_{n}({\bm{\epsilon}}_{1})}{d_{n }({\bm{\epsilon}}_{2})}-\frac{1}{N}\sum_{m=1}^{N}\log\frac{d_{m}({\bm{\epsilon }}_{1})}{d_{m}({\bm{\epsilon}}_{2})}\right\}\left\{\frac{{\bm{I}}_{n}({\mathbf {s}}){\bm{\epsilon}}_{1}}{d_{n}^{2}({\bm{\epsilon}}_{1})}-\frac{1}{N}\sum_{m=1 }^{N}\frac{{\bm{I}}_{m}({\mathbf{s}}){\bm{\epsilon}}_{1}}{d_{m}^{2}({\bm{ \epsilon}}_{1})}\right\}$ $\displaystyle=\sum_{n=1}^{N}\left\{\log\frac{d_{n}^{2}({\bm{\epsilon}}_{1})}{d _{n}^{2}({\bm{\epsilon}}_{2})}-\frac{1}{N}\sum_{m=1}^{N}\log\frac{d_{m}^{2}({ \bm{\epsilon}}_{1})}{d_{m}^{2}({\bm{\epsilon}}_{2})}\right\}\left\{\frac{{\bm{ I}}_{n}({\mathbf{s}}){\bm{\epsilon}}_{1}}{d_{n}^{2}({\bm{\epsilon}}_{1})}- \frac{1}{N}\sum_{m=1}^{N}\frac{{\bm{I}}_{m}({\mathbf{s}}){\bm{\epsilon}}_{1}}{ d_{m}^{2}({\bm{\epsilon}}_{1})}\right\},$ $$

where we have used the fact that

$$ $\displaystyle\nabla_{{\bm{\epsilon}}}\log d({\bm{\epsilon}})$ $\displaystyle=\frac{1}{2}\nabla_{{\bm{\epsilon}}}\log({\bm{\epsilon}}^{\top}{ \bm{I}}{\bm{\epsilon}})=\frac{{\bm{I}}{\bm{\epsilon}}}{{\bm{\epsilon}}^{\top}{ \bm{I}}{\bm{\epsilon}}}=\frac{{\bm{I}}{\bm{\epsilon}}}{d^{2}({\bm{\epsilon}})}.$ $$

Similarly, differentiating $L$ with respect to ${\bm{\epsilon}}_{2}$ yields:

$$ $\displaystyle\nabla_{{\bm{\epsilon}}_{2}}L({\bm{\epsilon}}_{1},{\bm{\epsilon}} _{2})$ $\displaystyle=-\sum_{n=1}^{N}\left\{\log\frac{d_{n}^{2}({\bm{\epsilon}}_{1})}{ d_{n}^{2}({\bm{\epsilon}}_{2})}-\frac{1}{N}\sum_{m=1}^{N}\log\frac{d_{m}^{2}({ \bm{\epsilon}}_{1})}{d_{m}^{2}({\bm{\epsilon}}_{2})}\right\}\left\{\frac{{\bm{ I}}_{n}({\mathbf{s}}){\bm{\epsilon}}_{2}}{d_{n}^{2}({\bm{\epsilon}}_{2})}- \frac{1}{N}\sum_{m=1}^{N}\frac{{\bm{I}}_{m}({\mathbf{s}}){\bm{\epsilon}}_{2}}{ d_{m}^{2}({\bm{\epsilon}}_{2})}\right\}.$ $$

Combining, we have the following gradient-based optimization algorithm.

Figure: Algorithm 1 Computing the principal distortions via projected gradient descent

## Appendix C Methods: Early Visual Model Experiments

PyTorch implementations of the early visual models were obtained from (https://github.com/plenoptic-org/plenoptic, Duong et al., ). The early visual models were adopted from Berardino et al. (), with parameters chosen to maximize the correlation between predicted perceptual distance and human ratings of perceived distortion, for a wide range of images and distortions provided in the TID-2008 database (Ponomarenko et al., ). Parameters are reported in Table 1. Note that although the models are nested in their construction and parameterization, the model parameters differ across tested models since they are optimized for each model independently.

**Table 1: Parameters for early visual models, obtained from (Berardino et al., ).**
| LN Model |  |
| --- | --- |
| center-surround, center std | 0.5339 |
| center-surround, surround std | 6.148 |
| center-surround, amplitude ratio | 1.25 |
| LG Model |  |
| luminance, scalar | 14.95 |
| luminance, std | 4.235 |
| center-surround, center std | 1.962 |
| center-surround, surround std | 4.235 |
| center-surround, amplitude ratio | 1.25 |
| LGG Model |  |
| luminance, scalar | 2.94 |
| contrast, scalar | 34.03 |
| center-surround, center std | 0.7363 |
| center-surround, surround std | 48.37 |
| center-surround, amplitude ratio | 1.25 |
| luminance, std | 170.99 |
| contrast, std | 2.658 |
| LGN Model | (Two channels) |
| luminance, scalar | [3.2637, 14.3961] |
| contrast, scalar | [7.3405, 16.7423] |
| center-surround, center std | [1.237, 0.3233] |
| center-surround, surround std | [30.12, 2.184] |
| center-surround, amplitude ratio | 1.25 |
| luminance, std | [76.4, 2.184] |
| contrast, std | [7.49, 2.43] |

## Appendix D Methods: Deep Neural Network Experiments

The analyzed DNNs were obtained from the model loading code and checkpoints available at https://github.com/jenellefeather/model_metamers_pytorch which were used in Feather et al. (), and allowed for easy loading and selection of the intermediate layer stages for many models that had previously been proposed as models of human visual perception. The checkpoints for the standard ResNet50 and AlexNet models were obtained from the public pytorch checkpoints ((Ansel et al., )). The Stylized Image Net AlexNet (alexnet_trained_on_SIN) and ResNet50 (resnet50_trained_on_SIN) were obtained from https://github.com/rgeirhos/texture-vs-shape associated with Geirhos et al. (). The checkpoint for the ResNet50 $\ell_{2}(\epsilon_{\text{train}}=3.0)$ adversarially trained model was obtained from Engstrom et al. (), and the checkpoint for the Alexnet $\ell_{2}(\epsilon_{\text{train}}=3.0)$ adversarially trained model was obtained from Feather et al. (). For all experiments, we only included intermediate layers before the final classification stage in the principal distortion analysis, and the subset of layers followed those chosen in Feather et al. (). Specifically, for the AlexNet models we included layers relu0, relu1, relu2, relu3, relu4, fc0_relu, and fc1_relu in each set of anlayses. For the ResNet50 models we included conv1_relu1, layer1, layer2, layer3, layer4, and avgpool in each set of analyses.

### D.1 Principal distortion optimization

For the deep neural network experiments, we use a subset of 100 ImageNet images where each image was chosen from a unique class (randomly chosen from the set of images at https://github.com/EliSchwartz/imagenet-sample-images). For each image and each comparison, we ran the gradient descent procedure for principal distortion optimization for 2500 iterations, using an exponentially decaying learning rate that started at 10.0 and decayed to 0.001 by the final step.

We used a target distortion size of $\alpha=0.1$ and at each step of the optimization, we also scaled ${\bm{\epsilon}}$ so that the image ${\mathbf{s}}+1000{\bm{\epsilon}}$ would not be clipped when the RGB value was represented between 0–1; that is, we scaled ${\bm{\epsilon}}$ so that $0\leq s[i]+1000\epsilon[i]\leq 1$ for each value $i$ in the image. This constraint could potentially bias the perturbations to be more spread out across the image (because all of the amplitude for the perturbation cannot be focused at a small set of pixels). However, removing this constraint did not lead to quantitatively different results (Supplementary Fig. 10), but the inclusion allows for more viable comparisons with human perception (as the perturbation can be scaled while maintaining valid values in the image gamut).

### D.2 Supplemental Figures

Figure: Figure 6: Principal distortions for ResNet50 and AlexNet for six example base images (${\mathbf{s}}$, middle row). Each row shows base images with differently scaled additive perturbations of ${\bm{\epsilon}}_{1}$ and ${\bm{\epsilon}}_{2}$. Distortions $\{{\bm{\epsilon}}_{1},{\bm{\epsilon}}_{2}\}$ are shown in isolation in the top/bottom rows, respectively. The log ratio plot for each image is at the top of the column.
Refer to caption: x6.png

Figure: Figure 7: Principal distortions for ResNet50 and AlexNet for base images that have been constructed to have either (A) texture only in the periphery and a blank center (B) texture only in the center and a blank surround and (C) a full image of texture. The log ratio plot for each image is at the top of the column. Panels A and B highlight that the AlexNet architecture is more sensitive to the perturbation with power around the “stuff” of the image (i.e., the non-blank areas), while ResNet50 is more sensitive to a distortion focused on blank areas of the image, and these perturbations are not sensitive to the choice of low contrast (“black”) or high luminance (“white”) for the blank part of the space. Panel C highlights that in the case where there is not obvious blank area of the image, the perturbations become harder to interpret, and this reflects the locally adaptive property of the FI to the input image.
Refer to caption: x7.png

Figure: Figure 8: Principal distortions for standard ImageNet trained and Shape Image Net Trained (SIN) AlexNet and ResNet50 architectures for example base images.
Refer to caption: x8.png

Figure: Figure 9: Principal distortions for standard ImageNet trained and adversarially trained AlexNet and ResNet50 architectures for example base images.
Refer to caption: x9.png

Figure: Figure 10: Example principal distortions for ResNet50 vs. AlexNet comparison where the pixelwise min/max value for the perturbation has not been constrained to avoid clipping once the perturbation is scaled. Similar to the results in the main text, we see that the principal distortion that AlexNet is more sensitive to is focused on the part of the image with high-contrast texture features, while ResNet50 is more sensitive to the perturbation targeting relatively blank parts of the image.
Refer to caption: x10.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 4
questions: - How is the stimulus-dependent function $f(s)$, defined after Eq. (1), computed in practice?  
- If $I(s)$ is positive semi-definite, this seems to cause issues with the metric defined in Eq. (3) when it approaches zero.   
- How are two image representations obtained from a single image $S$? Are $I_A(S)$ and $I_B(S)$ learned by different neural networks? Please clarify. 
- How are the coefficients for $\epsilon_1$ and $\epsilon_2$ determined in the proposed approach?  
- The interpretation and justification of the estimated principal distortions are difficult to justify. For instance, in Fig. 3, while the finding that AlexNet and ResNet are more sensitive to complementary parts of the images is interesting, its validity is unclear. Providing additional justification would strengthen the claims. One possible approach is to introduce adversarial noise to different parts of the images and evaluate its impact on downstream classification tasks, e.g., comparing the performance drop across different parts of the images.  
- This reviewer finds it difficult to connect the identified distortions with the concept of local geometry. The method appears to focus more on differences in image intensities and textures. However, it is possible that this reviewer has misunderstood some aspects of the proposed approach. Please clarify.
soundness: 3
strengths: - This is well-written and organized paper; hence it is easy for readers to follow. 
 
- The concept of exploring the local geometry of deep networks and examining the interplay between local geometry and the global structure of images is compelling.

- Developing a novel metric to compare two image representations is highly innovative.
summary: This paper introduces a novel metric on image representations to measure differences in local geometry. The authors then leverage this metric to generate "principal distortions" that maximize variance across a set of models. Experimental results of the proposed method reveal qualitative differences in the local geometry of ResNet50 and AlexNet, which is quite interesting.
weaknesses: - While the proposed method of using the Fisher Information Matrix to measure the sensitivity of a representation to a stimulus distortion seems reasonable, its effectiveness as a metric is unclear, or difficult to justify.

- Identifying the types of principal distortions to which the network is most sensitive is an interesting idea. However, the proposed method lacks a good validation plan to confirm the accuracy or reliability of these findings.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Model Selection: The authors demonstrate the metric’s functionality using older architectures, specifically AlexNet and ResNet. Given that AlexNet is largely obsolete in current practical applications, the paper would benefit from extending the evaluation to more contemporary and widely-used networks (e.g., EfficientNet, Vision Transformers). Demonstrating the framework's effectiveness across a variety of modern architectures would strengthen the claim that the metric is universally applicable and capable of distinguishing models.

Practical Utility and Generalizability: The manuscript does not clearly establish the practical relevance and utility of this framework in real-world applications. A more explicit discussion on the potential benefits of this metric in actual deployment scenarios, or in improving model interpretability and selection, would add significant value. Additionally, empirical results supporting the framework’s effectiveness across diverse, practical model architectures are essential to substantiate the generalizability and robustness of the proposed approach.
soundness: 3
strengths: The framework's focus on comparing local geometry is innovative, and the use of the Fisher information matrix and sensitivity to local distortions represents a unique approach to quantifying image representation. This novel method to derive principal distortion pairs that maximize model variance offers a potentially valuable tool for model discrimination.
summary: The paper proposes a novel framework to compare local geometry within image representations, arguing that image representation encompasses both global and local geometry information. The authors introduce a method to quantify the local geometry of image representations using the Fisher information matrix and statistical techniques sensitive to local stimulus distortion. This approach aims to identify "principal distortions pairs," which maximize model variances and serve as optimal discriminative tools for evaluating model performance.
weaknesses: Model Selection: The authors demonstrate the metric’s functionality using older architectures, specifically AlexNet and ResNet. Given that AlexNet is largely obsolete in current practical applications, the paper would benefit from extending the evaluation to more contemporary and widely-used networks (e.g., EfficientNet, Vision Transformers). Demonstrating the framework's effectiveness across a variety of modern architectures would strengthen the claim that the metric is universally applicable and capable of distinguishing models.

Practical Utility and Generalizability: The manuscript does not clearly establish the practical relevance and utility of this framework in real-world applications. A more explicit discussion on the potential benefits of this metric in actual deployment scenarios, or in improving model interpretability and selection, would add significant value. Additionally, empirical results supporting the framework’s effectiveness across diverse, practical model architectures are essential to substantiate the generalizability and robustness of the proposed approach.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: 1. The idea is simple but effective -- as the first approach to compare more than two models.
2. The problem statement and method description are well-written and clear to understand.
3. The experiment result is interesting, with meaningful discussions about texture bias and adversarial vulnerability. Visualizations are very helpful. 
4. The supplementary is very informative.

.
summary: This paper proposes a framework for comparing image representations regarding their local geometries through the Fisher information matrix by finding a pair of “principal distortions” that maximize the variance of the models under this metric. The experiments include (1) comparing a set of simple models of the early visual systems and 2) comparing a set of deep neural network models to reveal differences in the local geometry that arise due to architecture and training types.
weaknesses: 1. It is not clear how this paper's principal distortions relate to human sensitivity. There seem to be no experiments to prove this statement, such as using human observers for evaluation of the principle distortions [1].
2. For example, in 4.1 Early Vision Models, it is not stated how to determine the effectiveness of this approach. There is no quantitative evaluation, and it is unclear how to understand the visualizations of the principle distortion images. 
3. Are there some practical applications of this approach? Or how to use visualizations of principle distortions?
4. What kind of networks can be compared with this approach? Only networks for classifications? How about the models for detections, segmentations, and other applications?
5. This paper is over nine pages.

[1] Alexander Berardino, Johannes Balle, Valero Laparra, and Eero Simoncelli. Eigen-distortions of ´
hierarchical representations. Advances in Neural Information Processing Systems, 30:3531–3540,
2017.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. How do you plan to extend this work to achieve more quantitative results? For instance, do you have ideas on quantifying the distinction between "noisy" versus "smooth" image distortions?
2. The authors essentially present a metric between models, which is based on the model’s representations for a single stimulus. It seems to me that the results and conclusions drawn from them could potentially disagree for different stimuli. How do you either choose a representative stimulus or how do you choose multiple and how could the results be aggregated?
3. How does your approach compare to, or how could it be integrated with, explainability methods based on saliency or relevance maps?
4. Could you clarify how you scaled the relative sensitivities between models of the early visual system (as mentioned on pages 5 and 6)? A revision of this explanation might enhance understanding.
5. Can you speculate on how the observed differences in sensitivity to noisy distortions between AlexNet and ResNet50 relate to their architectural differences or inductive biases?

Additional feedback:
- For deterministic representations in DNNs, the application of the FIM might seem less direct. It would be beneficial to elaborate on this aspect, perhaps by relating it to concepts like the pullback of the Euclidean metric onto the image space.
- Strengthening the discussion of experimental results and their key takeaways would enhance the paper. Specifically, clarifying how principal distortions inform us about a model's alignment with human visual representation, responses to adversarial examples, and contributions to interpretability would be valuable.
- Introducing more quantitative analyses or metrics could solidify the results and provide stronger evidence of the method's effectiveness.
soundness: 4
strengths: - The paper is clearly written and easy to follow, with a strong motivation that outlines how it builds on previous work.
- Presents a new method to compare the local geometry of image representations between models, generalizing prior approaches to accommodate multiple models.
- The method uncovers differences invisible to global geometric approaches, providing deeper insights into model behaviors.
- Includes several computational experiments that support the method's effectiveness and relevance.
- Discusses potential applications in studying biological vision, DNNs, and the interplay between them.
- Offers a well-rounded discussion that contextualizes the findings within the field.
summary: The paper introduces a new method for comparing vision models that map input images to stochastic representations, leveraging the Fisher Information Matrix (FIM) to analyze the local geometry of image representations. This approach allows for the comparison of multiple models, including computational models of the visual system and deep neural networks (DNNs). By focusing on relative sensitivities to principal distortion directions in the stimulus (image) space, the method reveals differences between models that are not captured by global geometric approaches such as representational similarity analysis (RSA). The authors demonstrate the utility of their method through computational studies, showing consistent findings with previous research on early visual system models and uncovering differences in local image representations between models like AlexNet and ResNet50.
weaknesses: - The technical advancement may be seen as an incremental improvement over existing metrics.
- The method primarily applies to stochastic representations, which are uncommon in DNNs; applying it to deterministic models may seem arbitrary.
- It is not entirely clear how the proposed local metric can be aggregated to provide global insights across different stimuli or datasets.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: mentioned everything in weaknesses
soundness: 4
strengths: I found the paper's topic interesting and overall the language of the paper was clear. The paper targets an important problem in cognitive science and is a step toward understanding visual models better. The figures and qualitative results were sufficient to understand the concept and the differences. The paper is also based on a sound experimental framework to investigate the source of variability in results from both training data and architectural design.
summary: The paper proposes a method to compare image representations by analyzing local geometries using the Fisher information matrix, enabling differentiation of models based on local sensitivity to distortions. For comparison, it has been applied to both early visual system models and deep neural networks, the method enables comparison of multiple models (>2) and can be used to compare model representations with human perception.
weaknesses: I have a few questions and would like to know the authors' response to them:

1. (minor) It has been claimed in Figure 4 that the differences between AlexNet and ResNet come from the architecture than the training procedure. There is always randomness in the training where initialization of the model's weights and also the random split of data can effect one single model's performance. In fact, the model output may change if we train it on different seeds. I am wondering wouldn't that be a better approach to replicate the results on models trained on at least three different seeds? (basically 3 different versions of a model)

2. (minor) I expected to see a candidate from Transformers in the experiments, however, there was no candidate from those (currently) popular models. Can authors justify why they have not included any transformers such as ViT or Swin?

3. With Transformers, it has been a trend to show the attention maps (on tokens of an image) and show the behavior of the model. I am wondering if the authors can comment on this. Can the distortion maps be correlated to attention maps? This can ideally (if correlated) help with the explainability of attention-based models.

4. (major) Currently the manuscript is heavily populated with qualitative results. I strongly suggest the authors add a per-class sensitivity distribution for the dataset they are using for different models. This helps to understand on a population level and over different classes which models are more sensitive to high freq. localities and which ones are not.

5. I could not find a Dataset section in the manuscript :) Adding an independent dataset section would make it easier to understand the scale of experiments. currently, it is hard to find within the text

Meta-review: The current paper applies Fisher information w.r.t. multiple neural networks to find local distortions in the input space of image representations. The reviews highlighted the novelty and clear presentation. Most reviewers acknowledged that the authors' rebuttal has sufficiently addressed their concerns, and that the work is meaningful to extends such local metrics to multiple neural networks and useful for model interpretability.

In preparing the final version, please consider carefully the comments raised by all reviewers. Additionally, the authors mentioned that eq.(3) is a proper metric between positive semi-definite matrices, this is not true considering degenerate cases such as  $\epsilon'=\lambda \epsilon$, or $I_A=\lambda I_B$, or when $\epsilon$ or $\epsilon'$ lies in the kernel of $I_A$ or $I_B$ as the authors have mentioned in the rebuttal. Note a metric distance is strictly positive for any two different points. Please reword related sentences accordingly with proper citations and/or proofs.

Decision: Accept (Poster)