Local vs distributed representations: What is the right basis for interpretability?

## Abstract

Abstract Much of the research on the interpretability of deep neural networks has focused on studying the visual features that maximally activate individual neurons.
However, recent work has cast doubts on the usefulness of such local representations for understanding the behavior of deep neural networks because individual neurons tend to respond to multiple unrelated visual patterns, a phenomenon referred to as “superposition”.
A promising alternative to disentangle these complex patterns is learning sparsely distributed vector representations from entire network layers, as the resulting basis vectors seemingly encode single identifiable visual patterns consistently. Thus, one would expect the resulting code to align better with human-perceivable visual patterns, but supporting evidence remains, at best, anecdotal.
To fill this gap, we conducted three large-scale psychophysics experiments collected from a pool of 560 participants.
Our findings provide ( i ) strong evidence that features obtained from sparse distributed representations are easier to interpret by human observers and ( ii ) that this effect is more pronounced in the deepest layers of a neural network.
Complementary analyses also reveal that ( iii ) features derived from sparse distributed representations contribute more to the model´s decision. Overall, our results highlight that distributed representations constitute a superior basis for interpretability, underscoring a need for the field to move beyond the interpretation of local neural codes in favor of sparsely distributed ones.

## 1 Introduction

One of the goals of explainable AI (XAI) in computer vision is to identify the visual features and characterize the representations used by deep neural networks (DNNs) to categorize images (Ribeiro et al., ; Sundararajan et al., ; Smilkov et al., ; Petsiuk et al., ; Selvaraju et al., ; Linsley et al., ; Fel et al., ; ; ; Novello et al., ; Zhou et al., ; Bau et al., ; Cammarata et al., ; Kim et al., ; Ghorbani et al., ). In general, identifying these features requires uncovering the visual patterns that drive the activation of units within a network.

Figure: Figure 1: (a) $\bullet$ Local (neuron) versus $\bullet$ Distributed (sparsely distributed vector) visual representations. The activation of individual neurons may be driven by multiple unrelated visual elements (depicted in the images at the bottom) whereas distributed representations, obtained via dictionary learning methods, break down complex patterns into simpler ones corresponding to single visual features. (b) In practice, dictionary learning methods “disentangle” local activations to yield a new vector basis whose activation is driven by single features. The hope for interpretability is that those features align better with the set of features that humans can interpret $S=\{f_{1},f_{2},...,f_{n}\}$.
Refer to caption: extracted/5982294/Figures/fig1_final.png

This goal is shared with the study of biological vision, where there is extensive research over the last several decades focused on identifying the “preferred stimulus” of individual neurons in the visual cortex (Hubel & Wiesel, ; Lettvin et al., ; Tsunoda et al., ; Wang et al., ; Pasupathy & Connor, ; Quiroga, ). This approach to visual neuroscience reflected the dominant theory at the time, known as the “grandmother (cell)” theory
, which postulates that information in the visual system is stored locally – at the level of single neurons – and that the visual system contains specific neurons that respond to particular objects or people (including one’s own grandmother). Early XAI advances inspired by neuroscience research similarly focused on understanding local representations (Zhou et al., ; Bau et al., ). This led to the development of more sophisticated optimization methods to synthesize maximally activating images for individual neurons (Erhan et al., ; Zeiler & Fergus, ; Yosinski et al., ; Olah et al., ; ; Nguyen et al., ; ; Cammarata et al., ).

This parallelism between XAI and neuroscience extends beyond vision as recent work has found neurons in multi-modal systems that respond to very high-level concepts beyond simple image appearance, including hand-drawing and text (Goh et al., ). Interestingly, the authors identified a “Halle Berry” neuron in CLIP, reminiscent of the neuroscience finding reported two decades ago in the human brain (Quiroga, ).

At the same time, a paradigm shift is taking place in neuroscience, where the study of neural populations is quickly superseding the study of single neurons (for a review see Ebitz & Hayden ()) because the neural code is believed to be sparse and distributed rather than local (Haxby et al., ; Quiroga et al., ; ).
Interestingly, a similar shift is emerging in XAI, because local representations are known to suffer from the “superposition” problem (Arora et al., ; Cheung et al., ; Olah et al., ; Elhage et al., ; Fel et al., ): the number of features captured by DNNs might be larger than the number of neurons. Therefore, the neurons’ activations might be driven by multiple unrelated features.
To address this challenge, the XAI community has started using dictionary learning methods (Fel et al., ; ; Bricken et al., ; Templeton et al., ) to project the activations of DNNs onto a new basis of vectors, each activated by a single distinct feature.
From an interpretability perspective, representations driven by single features are more desirable because they are expected to be easier to understand by a human, *i.e.*, a unit responding to a single visual pattern –compared to multiple patterns– is inherently easier to interpret. An implicit hypothesis is that applying dictionary learning methods to network activations helps break down complex visual patterns into simpler ones corresponding to single features (Fig. 1.a), which, in turn, might be easier for humans to interpret (Fig. 1.b).
Scaling up standard interpretability evaluations to study representations poses significant challenges (Colin et al., ). A practical alternative is to assess the ambiguity—or perplexity—of the visual features derived from these representations (Borowski et al., ) (see 3.2.1 for a more thorough elaboration on this point).

Despite the growing consensus that distributed representations constitute a stronger basis for interpretability compared to local ones (single neurons), empirical evidence remains scarce. This paper aims to fill this gap. Specifically, we contend that a representation can be considered superior if the features derived from it are more intelligible, *i.e.*, easier for humans to make sense of while being demonstrably used by the model in its decision-making process. By means of computational and psychophysics experiments, we set out to find which of the local vs. distributed representations better meets these two conditions.

In sum, the main contributions of this paper are as follows:

- •
We conduct three large-scale psychophysics experiments for a total of 15,720 responses from a pool of 560 participants, to evaluate the visual ambiguity of the features derived from local vs distributed representations (see Fig 1). In the process, we identify a potential semantic bias in the experimental protocols commonly used in the field (Borowski et al., ; Zimmermann et al., ), and provide an approach to at least partially mitigate it.
- •
Our findings provide strong evidence that (i) features derived from distributed representations are significantly easier for humans to interpret than features derived from local representations, and (ii) this effect is even more pronounced in the deepest layers of a neural network.
- •
Additionally, we observe that (iii) models rely significantly more on features derived from distributed representations compared to those derived from local representations. Overall, our results suggest that distributed representations provide a substantially better foundation for the interpretability of models than local representations.

## 2 Related Work

##### From local to distributed representations in XAI.

Early research on explainable AI (XAI) in computer vision developed attribution methods (Zeiler et al., ; Zeiler & Fergus, ; ; Sundararajan et al., ; Smilkov et al., ; Fong & Vedaldi, ; Ancona et al., ; Shrikumar et al., ; Chattopadhay et al., ; Fel et al., ; Novello et al., ) to understand specific model predictions. These methods predominantly aimed to identify *”where”* are the most important pixels of an image, given a specific model prediction. Unfortunately, these methods fell short in explaining the *”what”* (Kim et al., ; ; Taesiri et al., ; Colin et al., ), *i.e.*, the visual features that the models rely on to make their predictions. As a result, new XAI methods, including Feature Visualization approaches (Nguyen et al., ; Olah et al., ), were developed to provide insights into the features that neurons or model layers respond to. These efforts revealed that the neurons’ activations can be driven by visually distinct features (Nguyen et al., ; ; Cammarata et al., ; Bricken et al., ).
A similar behavior was observed in other fields, and notably in Natural Language Processing (NLP) (Elhage et al., ).

The neurons’ tendency to respond to multiple unrelated visual elements indicates that single neurons might not align well with the model’s internal representations. This phenomenon, known as superposition (Arora et al., ; Cheung et al., ; Olah et al., ; Elhage et al., ; Fel et al., ) or feature collapse (Fel et al., ), suggests that there could be significantly more features than neurons in a model.
Consequently, interpreting neurons might be no more meaningful than interpreting arbitrary directions in the feature space. To achieve effective model interpretation, it is, therefore, essential to identify an interpretable basis that facilitates the extraction of meaningful features. This observation has led to increased interest, over the past five years, towards examining deep learning models by considering the distributed nature of latent space representations. More specifically, it spurred the development of concept extraction methods (Ghorbani et al., ; Zhang et al., ; Fel et al., ; ; Graziani et al., ; Vielhaben et al., ) that leverage dictionary learning, especially overcomplete dictionaries. In NLP, sparse autoencoders (SAEs) (Elhage et al., ; Bricken et al., ; Cunningham et al., ; Tamkin et al., ) are extensively studied and regarded as a promising direction for discovering interpretable bases. In this paper, we are interested in comparing the suitability of local vs distributed representations to serve as a basis for the interpretability of deep learning models in computer vision.

##### Human-evaluation of interpretability.

Since the ultimate goal of XAI is to make complex models understandable to humans, it is essential to incorporate human evaluation in measuring interpretability through psychophysics experiments. Human evaluation serves as a critical benchmark for assessing whether the explanations provided by XAI systems are comprehensible, useful, and actionable for end-users. This human-centric approach ensures that interpretability methods are not just theoretically sound but also practically effective in real-world applications.

To the best of our knowledge, Borowski et al. () were the first to quantify the interpretability of features through psychophysics experiments. Their approach involved visualizing unit responses by contrasting maximally and minimally activating stimuli. Specifically, Borowski et al. () focused on studying features from local representations (i.e., single unit activations) of an Inception V1 (Szegedy et al., ) trained on ImageNet (Deng et al., ; Russakovsky et al., ). In their experiments, participants were shown maximally and minimally activating images from the ImageNet ILSVRC 2012 validation set (Russakovsky et al., ) to illustrate a specific feature. They were then asked to select which of two query images also activated the feature (see Fig. 2 for an example trial). As a proxy for interpretability, they measured the visual coherence of maximally activating stimuli of a given feature. It is maximal when the visual feature is unambiguous.
The main conclusion from these experiments is that, from a human-centric perspective, maximally activating natural images are more effective for studying features than synthetically generated feature visualizations (Olah et al., ).  Zimmermann et al. () proposed a variation of this task to investigate if humans gained causal insights from those visualizations. They tested the participants’ ability to predict the effect of an intervention, such as occluding a patch of the image, on the activation of the unit. Both by means of a large-scale crowdsourced psychophysics experiment and measurements with experts, they found that synthetic feature visualizations (Olah et al., ) helped humans perform the task successfully. However, these visualizations did not provide a significant advantage over other visualizations, such as exemplary images. Finally, Zimmermann et al. () extended the work of Borowski et al. () to a broader set of deep learning architectures used for computer vision tasks, including a ResNet50 (He et al., ). Their main conclusion is that increasing the scale of the model does enhance the interpretability of the features.
While both our work and that of Zimmermann et al. () build upon the experimental protocol of Borowski et al. (), they focus on scaling Borowski et al. ()’s insights, whereas we adapt the protocol to generate insights into a different and novel research question: comparing the suitability of features derived from local vs distributed representations for interpretability.

Interestingly, in a similar vein but within the context of NLP, (Bricken et al., ) compare the interpretability of 162 features extracted from both local and distributed representations. One of the authors scored each feature using samples drawn uniformly across the spectrum of activation. Their findings indicate that features obtained from distributed representations are substantially more interpretable than those from single neurons. The research presented in this paper differs from this work in the application domain, the experimental design, single participant in their study vs 15,720 responses from a pool of 560 participants in our study.

## 3 Methodology

In this section, we first provide an overview of the technical methods used to create the conditions for the three psychophysics experiments described below.

### 3.1 Technical Methods

##### Model

All the experiments described in this paper were performed on a ResNet50 (He et al., ) from the Torchvision (Marcel & Rodriguez, ) library, pre-trained on ImageNet-1k (Deng et al., ).

##### Sparse Distributed Representations

To compute sparse distributed representations for the distributed condition in our psychophysics experiments (see Section 3.2), we employed CRAFT (Fel et al., ), an off-the-shelf dictionary learning method. Specifically, given a model $\bm{f}:\mathcal{X}\to\mathcal{A}$ that maps from an input space $\mathcal{X}\subseteq\mathbb{R}^{d}$ to an activation space $\mathcal{A}\subseteq\mathbb{R}^{p}$ (*e.g.*, any layer of the network), we compute the activations $\mathbf{A}=\bm{f}(\mathbf{X})\in\mathbb{R}^{n\times p}$, where $\mathbf{X}=[\bm{x}_{1},\ldots,\bm{x}_{n}]^{\top}\in\mathbb{R}^{n\times d}$ represents a set of $n$ input data points. Each row $\bm{a}_{i}\geq\mathbf{0}$ of $\mathbf{A}$ contains the non-negative activations for a given data point $\bm{x}_{i}$, due to the use of ReLU activations. To obtain the sparse distributed representations, Non-negative Matrix Factorization (NMF) is applied to approximate $\mathbf{A}$ as:

$$ $(\mathbf{Z}^{\star},\mathbf{D}^{\star})=\operatorname*{arg\,min}_{\mathbf{Z} \geq\mathbf{0},\ \mathbf{D}\geq\mathbf{0}}\left\|\mathbf{A}-\mathbf{Z}\mathbf{ D}^{\top}\right\|_{F},$ $$

where $\left\|\cdot\right\|_{F}$ denotes the Frobenius norm. Here, $\mathbf{Z}\in\mathbb{R}^{n\times k}$ are the codes (or concept coefficients), and $\mathbf{D}\in\mathbb{R}^{p\times k}$ forms the dictionary (or concept bank). Both $\mathbf{Z}$ and $\mathbf{D}$ are constrained to have non-negative entries and tend to be sparse due to the properties of NMF. The dictionary matrix $\mathbf{D}$ provides a new set of basis vectors (concepts) aligned with the activation patterns of the neural network, while $\mathbf{Z}$ contains the coefficients representing the original activations $\mathbf{A}$ in terms of these concepts. CRAFT is particularly well suited for our purposes as it has been shown to extract interpretable features from deep neural networks (Fel et al., ).

##### Feature importance

For an image $\bm{x}_{i}$, let $\mathbf{v}_{c}$ be the vector it activates most strongly from an intermediate representation—it can be a sparse distributed vector in $\mathbf{D}$ or a neuron in $\mathcal{A}$—, and let $\bm{y}_{i}$ be the logit score for $\bm{x}_{i}$. To assess $\mathbf{v}_{c}$’s importance, we perform an ablation by setting the activation along this dimension to zero for $\bm{x}_{i}$, resulting in modified activations $\mathbf{a}_{i}^{\prime}$. These modified activations are then propagated to the output layer to compute a new logit score, $\bm{y}_{i}^{\prime}$. One can assess the importance of $\mathbf{v}_{c}$ by measuring as the difference in logit scores:

$$ $\Delta\bm{y}_{i}=\bm{y}_{i}-\bm{y}_{i}^{\prime}$ (1) $$

The more important the feature driving the activation of $\mathbf{v}_{c}$ is for the model, the higher the drop in logit score.

### 3.2 Psychophysics Studies

Figure: Figure 2: Illustration of a trial. Example of a trial in our study corresponding to Experiment I, distributed representation condition of a unit located in $layer2.0.bn2$. Two panels of 9 reference images are located on the left and right-hand side of the display, separated by 2 query images in the center. Participants were asked to select the query image they believed shared the same visual elements as the reference images displayed on the right panel, corresponding to maximally activating stimuli. The less ambiguous this shared visual element is—the more visually coherent the set of images—the more likely participants are to select the correct query. In this case, the correct query is the bottom image depicting a yellow tram.
Refer to caption: extracted/5982294/Figures/trial_illustration_new.png

#### 3.2.1 Experimental protocol

Interpretability is a human-centric attribute. Hence, to shed light on how well local vs distributed representations serve as a basis for the interpretability of deep neural networks, we performed three large-scale online psychophysics experiments. The three experiments focus on evaluating the interpretability of features extracted from both types of representations. In practice, given the challenges of scaling up standard interpretability evaluations to study representations (Colin et al., ), we adapt the experimental protocol of Borowski et al. () to measure the ambiguity, or perplexity,
of the visual feature as a proxy for its interpretability. More precisely, given a feature and a set of images that illustrate it—*e.g.*, maximally activating images—, this protocol measures how visually coherent humans find this set of images. The more visually coherent it is, the more likely they are to correctly identify another member of this set, namely the correct query image. Hence, in our results, we report the visual coherence of a feature as the proportion of participants that correctly identified the query image. This visual coherence should be maximal when the feature represents a single visual pattern that people can interpret.

Each participant was assigned to one of two conditions: local or distributed representation. After successfully finishing a practice session composed of 9 trials, participants were asked to sequentially perform 40 different trials of the same task, where each trial corresponded to a different network unit.
For each selected network unit and trial, participants were shown on the left and right-hand side of the screen two panels of 9 reference images each, separated by 2 images in the center, which were denoted as queries, as illustrated in Figure 2. The task across all experiments consisted of selecting the query image that participants believed shared the same visual elements as the 9 reference images displayed on the right panel. The selection of the reference images shown in each panel differed depending on the experiment, as explained below.

#### 3.2.2 Neuron and Stimuli Selection

##### Neuron selection

In line with the literature (Zimmermann et al., ), we aimed to obtain a representative sample of units within the different layers of the ResNet50. As a starting point, we selected the 80 units reported in (Zimmermann et al., ) for a ResNet50 (He et al., ) model. Given that CRAFT requires positive activations, we replaced the neurons from the convolutional layers (which could have non-positive activations) to their equivalent in the following batchnorm layer ($layer1.1.conv1$ neuron $53$ -> $layer1.1.bn1$ neuron $53$). As a result, we selected 80 neurons distributed across 43 layers that are grouped in four blocks. Thus, the names of the layers used in this paper ($layer1$ through $layer4$) are directly obtained from the modules of the ResNet50 Pytorch implementation.

##### Stimuli selection for the local representation condition

For each of the selected neurons, we identified $2,900$ images from the validation set of ImageNet ILSVRC 2012 (Russakovsky et al., ) obtained as follows: the $2,500$ most strongly activating images and the $400$ least strongly activating images. Following the procedure described by Borowski et al. (), we illustrated a visual feature through both maximally activating (images that possess the feature, see Fig. 2 right panel) and minimally activating (images that do not possess the feature, see Fig. 2 left panel) stimuli. Again, like in previous work (Borowski et al., ),
for the former, we selected a random sample of 9 images from the top 150 images; for the latter, we uniformly sampled 9 images from the bottom 20 images. We created 10 different trials per feature following this procedure to ensure image independence in the results. When participants were assigned to a feature, we randomly sampled one of those 10 trials to illustrate the feature.

##### Stimuli selection for the distributed representation condition

Our hypothesis is that applying dictionary learning methods to local representation allows us to disentangle the superposition of features that might drive the activation of a neuron. Hence, to test this hypothesis, all the distributed representations studied in this work are obtained from the local representation.
In practice, for each neuron in the local condition, we selected the top 300 maximally activating images, *i.e.*, those that most strongly led to the activation of the neuron. We used CRAFT to identify a new basis of distributed vectors through which their activations can be expressed. Following the recommended procedure in (Fel et al., ), we constrained the dimensionality of this new basis to $d=10$. From these $10$ directions, we selected the one that was the most frequently the most activated across the $300$ images. Once a direction was chosen, we ranked the $2,900$ images according to their activation in that direction to obtain the images that illustrate a visual feature in the distributed representation.

#### 3.2.3 Experiments

In this section, we present the details of each of the three psychophysics experiments conducted in this work.

##### Experiment I

This experiment is an adaption of the methodology proposed by (Borowski et al., ) with two conditions: local vs distributed representations. An illustration of a trial belonging to Experiment I can be found in Figure 2. In this experiment, we performed a total of 1,600 trials (80 neurons $\times$ 2 conditions $\times$ 10 trials per unit).

##### Experiment II

The main objective of Experiment II was to control for potential semantic confounding variables present in Experiment I. If the reference images that possess/do not possess the feature of interest belong to very different semantic groups (classes), then it could be possible to solve the task through simple semantic grouping. Figure
3 illustrates this phenomenon: in the example depicted in the figure, it is easier to solve the trial by inferring that the feature of interest is not about a monkey than it is to infer the actual visual feature present in the reference images. The goal of Experiment II is to control for this potential semantic confound as follows: given a set of reference images that possess the feature of interest, we extract their semantic labels from ImageNet and aim to find within the 400 minimally activating images a set of 10 images that share the same distribution of semantic labels. We define four levels of semantic similarity ($level$ 0 0 to $level$ $3$). In $level$ 0 0, the labels from ImageNet are used to determine the semantic similarity, whereas in $levels$ $1$ through $3$ we obtain the labels by moving up one branch in the WordNet (Fellbaum, ) hierarchy.
In practice, we performed an iterative search for each trial starting from $level0$. If there are not 10 images in the 400 minimally activating set that share labels with the reference images at a given level of semantics, the process is restarted, this time using labels from the next and broader level of semantics. We continued this process until 10 images from the minimally activating set shared the distribution of semantic labels with the reference images. In cases where it was impossible to identify 10 images, the feature was excluded from the experiment. Only one feature was fully excluded for both the local and distributed conditions due to this factor.

##### Experiment III

Finally, recognizing that the adopted experimental protocol was originally designed to assess whether features are better understood using natural or synthetic visualizations, we devised Experiment III to explore the impact of combining natural images with feature visualizations (See more details in Section D).

Figure: Figure 3: Illustration of the role of semantics. Example of a trial from Experiment I in the local representation condition. In this case, the task can be trivially solved by simply relying on semantics. By observation of the minimally activating stimuli (left panel), it is easy to conclude that the neuron of interest is not a monkey detector, yet, it is hard to articulate what is the visual feature captured by the neuron (images in the right panel).
Refer to caption: extracted/5982294/Figures/semantic_bias.png

#### 3.2.4 Participants

A total of 560 participants were recruited to take part in Experiments I, II, and III through the online platform Prolific (^1^11www.prolific.com). All participants were native English speakers who reported not being visually impaired and completed the study on a laptop or desktop computer (not a mobile phone). They
provided informed consent electronically and were compensated $\$2.75$ for their time ($\sim 10-13$ min). The protocol was approved by the University IRB. Based on the power analysis of (Zimmermann et al., ), a minimum of 60 participants per condition (120 participants per Experiment) was needed to obtain statistically robust results. Furthermore, participants were required to (1) succeed in at least 5 of the 9 practice or instruction trials and (2) correctly answer at least 4 of the 5 attentiveness tests (catch trials) that were randomly inserted in the experiment. As a result, we analyzed the data corresponding to 138, 133, and 122 participants from Experiments I through III, respectively.

## 4 Results

In this section, we summarize the results obtained from analyzing the responses from the 138, 133, and 122 participants who successfully completed Experiments I, II, and III, as well as the results from our feature importance analysis.

Unless stated otherwise, all of our behavioral data analyses consist of generalized logistic mixed-effects regressions (GLMER), with trial accuracy (1 vs. 0) as the dependent variable. The random-effects structure included both a by-participant and a by-unit (^2^22Here, a unit refers to either a neuron in the local representation condition or to a specific direction of the dictionary in the distributed representation condition.) random intercept, as well as a by-unit random slope for the condition variable. We used the lme4 (Bates et al., ) package in R (R Core Team, ) to fit the models and lmerTest to obtain p-values for the fixed effects, with an $\alpha$-level of 0.05 for statistical significance.

Figure: (a)
Refer to caption: extracted/5982294/Figures/exp1.png

##### Reproduction of previous results.

Given that the experimental protocol used in Experiment I, with the local representation condition, is the same as the one described in (Zimmermann et al., ), we first evaluate to which degree our results corroborate previous research. For the ResNet50, Zimmermann et al. (Zimmermann et al., ) report an average task performance of $83.0\%\pm 2.0$(^3^33Values inferred from Figure 3 in their paper.). In our experiment, we obtain an average performance of $78.8\%\pm 1.5$. Given the results’ similarity and that the selected units are not exactly the same (as described in Section 3), we conclude that Experiment I reproduces previously reported findings regarding local representations. This result also serves as an external validation of our experimental protocol.

##### Human performance is superior in the distributed representation condition.

Based on our main hypothesis that a distributed representation constitutes a better basis for interpretability than a local representation, we predicted that participants would be better at selecting the maximally activating query image in the distributed condition. In Experiment I, the average performance across participants in the distributed condition was $83.5\%\pm 1.4$ (^4^44The values reported correspond to a 95% confidence interval.), when compared to $78.8\%\pm 1.5$ in the local condition. A GLMER with condition as a predictor revealed a statistically significant disadvantage for the local condition: $\beta_{condition}=-0.47,SE=0.23,z=-2.04,p=0.04$. This result was corroborated both in Experiments II (see examples of trials in Fig 8, 9, 10) and III, with a mean performance of participants in the distributed condition of $85.1\%\pm 1.4$ vs. $76.2\%\pm 1.6$ in the local condition, $\beta_{condition}=-0.93,SE=0.24,z=-3.93,p=<.001$ and a mean performance of $80.0\%\pm 1.6$ (distributed) vs. $74.1\%\pm 1.7$ (local), $\beta_{condition}=-0.42,SE=0.17,z=-2.47,p=0.01$, respectively.

##### Semantic control matters, but not significantly.

Figures 4(a) and 4(b) depict the results obtained without (Experiment I) and with (Experiment II) a semantic control applied to the stimuli, respectively. At first glance, the results seem consistent in both scenarios, except for the results corresponding to neurons in layer 4 where the performance of participants in the local condition suffers a drastic decrease in Experiment II. This result is coherent with the intuitive observation that the model aggregates class-specific information the closer the layer is to the output layer. Hence, neurons located in deeper layers are more likely to respond highly to features that correspond to certain classes and not at all to features belonging to other classes, *i.e.*, the semantic confounds are expected to be higher for those neurons. To investigate further the role of controlling the semantics, we pooled the data from Experiments I and II and included both experiment and an interaction term condition:experiment as predictors in a GLMER. We found a significant main effect for condition, $\beta_{neuron}=-0.47,SE=0.23,z=-2.04,p=0.04$, but not for experiment.
Interestingly, we did not obtain evidence for a significant interaction between experiment and condition, $\beta_{neuron:Exp2}=-0.46,SE=0.30,z=-1.53,p=0.13$. As it was not possible to semantically control all trials in Experiment II to the same extent, we also tested for an interaction between condition and semantic control instead of the variable experiment. Semantic control was coded as a categorical variable with 4 levels: no control, 1, 2, and 3, corresponding to the previously described levels in Section 3.2.3. We compared two GLMERs: one including only the main effects of condition and semantic control, and another also considering the interaction between these two variables. Based on the Akaike Information Criterion, the model without the interaction was preferred (9116 vs. 9120). In sum, while the nominal disadvantage of the neuron condition is larger in Experiment II (semantic control) than in Experiment I (no semantic control), none of the performed statistical tests yielded a significant difference between these two experiments. This result can be partially explained by the unequal distribution of trials across semantic levels (see fig 11) and by the quality of the semantic control achieved once getting to a certain level in the Word-Net hierarchy.
We leave to future work a finer-grained study of the role of semantics in the interpretability of features.

Figure: Figure 5: Feature importance. We measure the importance of a feature as the average drop in logit score $\Delta y$ for the 300 most activating images when the feature is occluded. Except for $layer1$, we find that the model relies significantly more on features derived from the distributed representation than on features from local representations, $z=-5.86$, $p<.001$ (Mann-Whitney U test).
Refer to caption: extracted/5982294/Figures/importance.png

##### The deeper the layer, the more prominent the benefits of the distributed representation.

Figures 4(a), 4(b) and 7 illustrate the per-layer results obtained in Experiments I, II and III, respectively. We find evidence to suggest that the benefits of the distributed representation increase with the depth of the layer from which we select the unit that the participants were interpreting. Indeed, we identified a significant main effect for unit and also for an interaction term unit:condition, *i.e.*, the advantage of the distributed condition increases as the units belong to deeper layers. This result is consistent across the 3 experiments: $\beta_{depth}=0.08,SE=0.02,z=3.92,p<.001$ and $\beta_{depth:Exp1}=-0.06,SE=0.03,z=-2.22,p=0.03$ for Experiment I; $\beta_{depth}=0.07,SE=0.02,z=3.27,p=0.001$ and $\beta_{depth:Exp2}=-0.09,SE=0.03,z=-3.6,p<.001$ for Experiment II; and $\beta_{depth}=0.05,SE=0.02,z=2.6,p=0.01$ and $\beta_{depth:Exp3}=-0.06,SE=0.03,z=-2.59,p=0.01$ for Experiment III.

##### The model relies more on features derived from the distributed representation.

We evaluate the importance of every feature used in our psychophysics experiment by measuring the average drop in logit score for the 300 most activating images when the feature is occluded (see Eq. 1). Overall, we find that the model relies significantly more on features derived from distributed representations than from local representations: $z=-5.86,p<.001$ (Mann-Whitney U Test), the only exception being $layer1$ (Fig 5). Interestingly, and in line with the rest of our results, the deeper the layer, the larger the difference between the local vs distributed representations.

## 5 Discussion

##### Conclusion

In this work, we have investigated and compared the suitability of local and distributed representations to serve as a basis for the interpretability of deep neural networks. We contend that the best-suited basis possesses a dictionary of features that is (a) more aligned with the set of features that human observers can interpret while (b) being demonstrably more important for decision-making by the model.
Through psychophysics experiments, we consistently find that features derived from distributed representations are easier for humans to interpret, particularly when features are derived from deeper layers in the network. Equally important, we find that the model relies significantly more on those features to make its decisions. To the best of our knowledge, our results provide the first empirical evidence of the superiority of distributed representations over local ones for the interpretability of deep neural networks.

##### Limitations

Our study is not exempt from limitations. First, the methodology proposed by Borowski et al. () and followed by Zimmermann et al. () utilizes the entire ImageNet validation set with the goal of studying a broad range of stimuli and thereby increasing the likelihood of identifying stimuli that are representative of neurons’ selectivity. While such motivation is sound, it poses a significant challenge when performing psychophysics experiments: neurons that are selective to class-specific features (*e.g.*, fish scale) will be maximally activated when presented with stimuli that belong to the corresponding classes (*e.g.*, fish) and minimally activated when provided with stimuli that belong to other classes (*e.g.*, dogs). In those cases, the task can be solved trivially using semantics. The design of Experiment II reflects an initial attempt to mitigate this confounding factor. However, both quantitative results and a manual exploration of the trials by the authors hint at only a partial success of the adopted methodology to address this challenge. We leave to future research a further refinement of the experimental protocol.

Second, while the original work (Borowski et al., ) this manuscript builds on aimed at measuring the *interpretability* of features, we have been more conservative with the terminology employed in this paper as we believe that the current experimental protocol only measures a loose proxy of interpretability. Interpretability, or understanding, of a complex visual pattern usually requires procedural learning with feedback (Ashby & Maddox, ). Given that, in practice, this protocol measures a score based on a set of images presented in a single trial, it seems unreasonable to expect this score to capture interpretability.
In contrast, it seems more reasonable to expect this score to characterize something more visual, like the ambiguity of the features, i.e., how complex it might be to interpret the features if one had to. Such ambiguity can be inferred, at least in part, by the visual coherence of the set of images.   have indeed shown that responses from participants can be accurately predicted based on visual coherency—or perceptual similarity—of a trial.

While we believe that these metrics are useful proxies and that features achieving a high score at them are desirable for interpretability, we want to emphasize that we do not claim that it means that they *are* interpretable. Perceptual similarity differs from interpretability; being able to group images correctly does not necessarily imply a deep understanding of the underlying factors driving those groupings or the ability to explain why the images belong together based on intrinsic features or causal relationships.

Nevertheless, we believe that the adopted experimental protocol serves as a solid foundation for developing a scalable and robust paradigm to measure the interpretability of features in deep learning models. We leave to future work the development of such a paradigm.

## Acknowledgement

This work was funded by the ONR grant (N00014-24-1-2026), NSF grant ( IIS-2402875 and EAR-1925481) and the ANR-3IA Artificial and Natural Intelligence Toulouse Institute (ANR-19-PI3A-0004). The computing hardware was supported in part by NIH Office of the Director grant #S10OD025181 via the Center for Computation and Visualization (CCV) at Brown University. J.C. and N.O. have been partially supported by Intel Corporation and funding from the Valencian Government (Conselleria d’Innovació, Industria, Comercio y Turismo, Direccion General para el Avance de la Sociedad Digital) by virtue of 2022-2023 grant agreements (convenios singulares 2022, 2023). J.C. has also been partially supported by a grant by Banco Sabadell Foundation.

## Appendix

## Appendix A Practice Trials

All participants were shown 9 practice trials to help them understand the core idea of the task. The images for these trials did not overlap with the images used in any of the experiments, preventing them from influencing our results. They were selected to represent specific features on the maximum-activating grid: checkerboard, veiny, green, round, blue, rough fur, yellow, straight lines, and magenta. Images on the left side of the grid in practice trials had no coherent pattern, and were randomly sampled without replacement from a set of images with features including whiskers, spikes, droplets, and liquid flow.

Practice trials were customized for each experiment, to keep all variables controlled. For the feature visualization psychophysics, practice trials were generated using the same methods as the experimental images. They were chosen from neurons in convolutional layers to show the same features that were included in the practice trials for other experiments.

## Appendix B Attentiveness Tests

Attentiveness tests were included at random points in the study. Following the methodology adopted in previous work (Borowski et al., ), they were based on a simple premise: if the query image is also present in the grid of maximum-activating images, participants paying attention should get near-perfect accuracy. We generated attentiveness tests for each experiment using the same procedure as was used for the experimental trials. To ensure that the attentiveness tests did not overlap with experimental trials, we chose maximum- and minimum-activating images from neurons from convolutional layers. Rather than sampling 10 maximum-activating images, we sampled 9 and put the query in a random position in the maximum-activating grid. The attentiveness tests were generated for all experiments using the same neurons from the model to guarantee consistency.

## Appendix C Image section for the distributed representation condition.

We originally tested 2 ways to select the direction from the distributed representation obtained by CRAFT to be kept for the main experiment. The first one is the one explained in the main section of the paper, namely: from the 10 directions, we selected the direction that was the most often the most activating one across the 300 images. However, we considered a second alternative where we selected the direction that was the most often the most activating one across the 2900 images. We ran the psychophysics experiments on both alternatives. The average performance obtained on this alternative was of $81\%\pm 1.4$ and we find no effect of the condition variable between this distributed condition vs the local condition describe in the main paper: $\beta_{condition}=-0.34,SE=0.25,z=-1.39,p=0.164$. Further quantitative and qualitative analysis done on both alternatives have led us to keep the first alternative for the reminder of the experiments.

## Appendix D Supplementary information about experiment III

##### Experimental protocol

Given that the adopted experimental protocol was originally designed to evaluate the informativeness of feature visualizations, we devised Experiment III to shed light on the value that a state-of-the-art feature visualization method (MACO) would add to the task. We subscribe to the idea that feature visualizations are more suitable to complement natural images than to replace them. Thus, Experiment III implements a protocol similar to the $mixed$ condition reported by Zimmermann et al. (Zimmermann et al., ). In practice, Experiment III was based on Experiment II, but in each trial, 4 of the natural images displayed both on the left and right panels were replaced by 4 feature visualizations, as depicted in Figure 6.

##### Method

To synthesize feature visualizations for Experiment III (see Section 3.2), we used MACO (Fel et al., ), inspired by the method by Olah et al. (Olah et al., ). MACO generates feature visualizations with a natural Fourier amplitude spectrum by fixing the amplitude spectrum to the empirical mean derived from natural images. Specifically, MACO optimizes directly in the Fourier domain $\mathcal{F}(\bm{x})$ by modifying the phase $\bm{\varphi}$ of the target image while keeping the magnitude $\bm{r}$ of the spectrum fixed. This constraint on the magnitude helps prevent high-frequency artifacts and ensures that the resulting images remain visually coherent.
Formally, let $\bm{r}$ denote the average amplitude spectrum computed across the ImageNet dataset, and let $\bm{d}$ be a target concept direction to be maximized. MACO solves the following optimization problem:

$$ $\bm{\varphi}^{\star}=\operatorname*{arg\,max}_{\bm{\varphi}}\left(\bm{f}\left( \mathcal{F}^{-1}\left(\bm{r}\circ e^{i\bm{\varphi}}\right)\right)\cdot\bm{d} \right),~{}~{}~{}\text{and}~{}~{}~{}\bm{x}^{\star}=\mathcal{F}^{-1}\left(\bm{r }\circ e^{i\bm{\varphi}^{\star}}\right).$ $$

where $\bm{x}^{\star}$ is the feature visualization obtained after optimization, $\mathcal{F}$ denotes 2-D Discrete Fourier Transform (DFT) on $\mathcal{X}$, $\mathcal{F}^{-1}$ its inverse and $\circ$ represents element-wise multiplication. Additionally, MACO introduces an attribution-based transparency mask to highlight spatially important regions in the visualizations, enhancing interpretability.

##### The value of feature visualization remains unclear.

Finally, Figure 7 depicts the results obtained from Experiment III, which combined natural images with synthetic images corresponding to feature visualizations using MACO. Similarly to (Zimmermann et al., ), we do not find that mixing stimuli from feature visualization with natural images helped participants perform better at the task. In fact, participants performed overall significantly worse in Experiment III than in Experiment II: $\beta_{depth:Exp3}=-0.63,SE=0.19,z=-3.3,p<0.001$.
Interestingly, the worst performance was observed in the local condition for units located in the deepest layers of the network, with a drop in performance from $80.2\%$ in Experiment I to $68.3\%$ in Experiment III. Furthermore, the difference in the per-layer task performance obtained in layer 4 in each condition is $8.2\%$, $17.1\%$, and $18.9\%$, respectively.
We leave to future work the investigation into the specific factors that contributed to this decline in performance. Potential avenues for future research include exploring the cognitive load imposed by mixed stimuli and identifying optimal conditions under which feature visualization might enhance rather than hinder task performance.

Figure: Figure 6: Illustration of a trial from Experiment III. Example of a trial from $layer4.2.bn3$ in the distributed representation condition from Experiment III. Note how the reference images on the left and right panels contain a mix of feature visualizations using MACO and natural images.
Refer to caption: extracted/5982294/Figures/fv_figure.png

Figure: Figure 7: Per-layer results for Experiment III. Given a feature and a set of images to illustrate it, we assess how visually coherent participants find this set of images—or how unambiguous the feature is. More precisely, we measure the proportion of participants that are able to identify the query image which is also part of this set of images. Similarly to the results from experiment I and II, a clear trend emerges where features appear significantly less ambiguous in the distributed representation than in the local representation condition, particularly in the deeper layers of the network.
Refer to caption: extracted/5982294/Figures/exp3.png

## Appendix E Further illustration of local vs distributed trials

Figure: (a) Local representations
Refer to caption: extracted/5982294/Figures/l2_n.png

Figure: (a) Local representations
Refer to caption: extracted/5982294/Figures/l3_n.png

Figure: (a) Local representations
Refer to caption: extracted/5982294/Figures/l4_n.png

Figure: Figure 11: Number of trials per semantic control level, for both local and distributed representation.
Refer to caption: extracted/5982294/Figures/nb_trials_SC.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: What are the assumptions behind the logistic regression analysis in Section 4? It would be good to make these more explicitly so a naive reader would have a better sense of how the results should be interpreted. 


Line 301-303, it was not exactly clear in terms of how the particular direction was selected. What does “the most frequently the most activated” mean? Please clarify. 

Line 130: Re: “neurons’ tendency to respond to multiple unrelated visual elements”
What does “unrelated visual elements” mean? This seems to be fairly arbitrary. 

Are the comparisons in Fig. 4 statistically significant? What do the error bars represent?

Line 277-281, what were special about the 80 neurons selected? Are these a representative subset of all units in the network? If not, how do we know if the results would generalize to other neurons in the network?
soundness: 3
strengths: Originality: While this is not the first study to use psychophysics to understand the interpretability of deep network features, human interpretability of deep network features remains an under-explored direction. Overall, the combined experimental and modeling approach is considered to be novel. 


Quality: The study is generally well-executed. Behavioral responses were collected from a large number of human subjects.  Three experiments were conducted. The first one was the main experiment. The remaining two were control experiments to help further interpreting the results. 


Clarity: The method and approach are generally well described, except for a few sections (see comments below about Section 4).


Significance: Understanding and interpolating the features of deep nets is an interesting and timely question. The approach used here, while not entirely novel, remains useful.
summary: This paper studies the question of human interpretability of the features derived from deep neural network. They considered two types of features: (i) features that triggered maximal responses (maximally activating images) in certain network units; (ii) features that were obtained by applying sparse dictionary learning to the maximally activating images. They defined these two classes as local representation v.s. distributed representation, respectively. Using psychophysics experiments, the authors studied which type of features were more interpretable. They reported some evidence suggesting that the sparse features was more interpretable to human observers. Overall, this seems to be an interesting piece of work, despite that the effect size obtained in the experiments is fairly small.
weaknesses: - The formulation of local v.s. distributed representation (two concepts that are essential to this paper) is messy and problematic in the current version. In the Introduction, the author referred to the grandmother cell as the local representation, and cited the paper by Quiroga et al (2005). I think the results in Quiroga et al (2005) was misinterpreted. The fact that, from a small of recorded neurons in the hippocampus in Quiroga et al (2005) one could already find a neuron that selective to a specific person, suggests that there is a large population of neurons that are selective to that person (yet being sparse). Thus, the results from Quiroga et al (2005) as well as other related studies instead favor a sparse and distributed representation.  I wonder if a better distinction would be “dense” v.s. “sparse”, rather than “local” v.s. “distributed”.

Thus, I think the overall pitch of the paper, i.e., the formulation of “local” v.s. “distributed”, is problematic. 


- One potential explanation for the main experimental results might be that the complexity of the features from the two classes (original maximally activating images v.s. sparse features obtained from them) did not match. Perhaps the sparse features used in the experiments are simpler and have less entropy compared to the maximally activating images. Would it be able to rule this out?

- The work only tested one version of ResNet50. It remains unclear whether the results would generalize to other deep nets. I understand that it would be too much to re-run the experiments using features from a different network for the current study. However, if the authors could compare the features in different networks (i.e., running these psychophysics in silico), that would likely strengthen the paper.

- The main effect, while being statistically significant, the effect size as reported in Section 4 was fairly small.  The paper needs to take the effect size into account when interpreting the results.

- While the number of subjects was large, the number of responses collected in each experiment (~5k) did not exceed the number in typical psychophysical experiments. Some of the statements regarding the scale of experiments should be toned down accordingly. 


- The AIC comparison in line 450-451 implies a small difference (i.e, 4) between the two models. I think the interpretation should be more nuanced. 

- How the Result section was presented can be improved. It would be useful if the authors could further unpack the meaning of the analyses performed there.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Can the results be used to improve interpretability somehow?

Can the results be reused for other networks in some way?
soundness: 3
strengths: In general, interpretability of neural networks is a pressing matter and the sparse bases for interpretability are a recent development that is worth testing. To do so, the authors present a coherent set of experiments with state of the art methods. Both in terms of the psychophysical task and in terms of the statistical analysis, I believe the results are solid.
summary: In this manuscript, the authors present the results of 3 psychophysics experiments on the interpretability of internal representations of a ResNet-50 trained on ImageNet. Following an established paradigm the authors showed images that highly activate a dimension and images that lowly activate a dimension in the deep neural network and asked subjects to infer whether a new test image activates the dimension or not. The main comparison is between the original neural dimensions and distributed directions determined by CRAFT, a dictionary learning method. The authors find better performance for the distributed representations. Nulling the most activated distributed direction also impaired network performance more than nulling the most active neuron. All effects get stronger for deeper layers in the network.
weaknesses: While the paper seems sound enough to me, the I doubt the importance of these results. It appears to be a solid confirmation of ideas we had before with established methods. This is good to do, but of somewhat limited impact.

- The results presented here are limited to a single network for a single network for a single task. Thus it remains unclear whether the effects described are general.
- All the technical steps were established before, i.e. there is no technical innovation in this paper
- As sparse dictionaries were developed as an XAI method, it is already a general believe in the literature that they are indeed more interpretable than the original neural activities, especially for deeper representations 
- The effect on interpretability appears to be quite gradual, i.e. while the difference is clearly significant, the features extracted by CRAFT are still far from optimally interpretable especially for earlier layers. Thus, the problem of interpreting DNN activations remains unsolved effectively.
- There could be somewhat deeper analysis of the data, for example which features are more or less interpretable, what proportions of features are interpretable, etc.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Minor comments, suggestions and questions:

Line 214 reads “neuron in \mathcal{A}”. Shouldn’t this be a component of a vector $a \in \mathcal{A}”?

Line 213: why is v_c indexed by c if it is made to correspond to an image x_i indexed by i?

Line 215: notation for modified activations a’_i is using a font different from the one in the definition in the previous paragraph.

Line 242: “by measuring as the difference” should presumably be “by measuring the difference”

The parallel between the problems of XAI and neuroscience, highlighted in the second and third paragraphs of the introduction, has been drawn explicitly and elaborated on by Vilas et al (2024) in the context of interpretability.

Please clarify mathematical notation. For instance, in equation 1, is y_i a vector? How does equation 1 relate to importance?

Stimuli selection: is it possible that the selection of maximally and minimally activating images contribute to mischaracterization of the full neural function? Relatedly, why are there more strongly activating images than weakly activating images?

Could the authors plot/overlay the data points in the graphs (i.e., not just bars)? It could foster a more truthful appreciation of the effects.

Line 460 references figure 7, which appears in the appendix, as if it were in the main text. There are possibly other instances of this throughout.

Given the authors' conclusions favoring distributed representations, the results could be relevant to the following literature exploring feasible interpretability queries, most of which only explore axis-aligned queries: Adolfi et al., 2024; Barcelo et al., 2020.

It might be useful to relate local versus distributed representations to discussions of axis-aligned versus non-aligned features.

Barceló, P., Monet, M., Pérez, J., & Subercaseaux, B. (2020). Model Interpretability through the lens of Computational Complexity. Advances in Neural Information Processing Systems, 33, 15487–15498. https://proceedings.nips.cc/paper_files/paper/2020/hash/b1adda14824f50ef24ff1c05bb66faf3-Abstract.html

Adolfi, F., Vilas, M. G., & Wareham, T. (2024). The Computational Complexity of Circuit Discovery for Inner Interpretability (No. arXiv:2410.08025). arXiv. http://arxiv.org/abs/2410.08025

Vilas, M. G., Adolfi, F., Poeppel, D., & Roig, G. (2024). Position: An Inner Interpretability Framework for AI Inspired by Lessons from Cognitive Neuroscience. Proceedings of the 41st International Conference on Machine Learning, 49506–49522. https://proceedings.mlr.press/v235/vilas24a.html
soundness: 2
strengths: * Well written
* Well referenced
* Addresses an important issue for the interpretability community
* Provides evidence of interpretability from human perception
* Provides evidence of feature importance for model decisions

I find it hard to comment on originality (e.g., the work is important/useful regardless of whether the methods themselves are novel). The 4th bullet above is to me an important component which is not always assessed rigorously in other work.
summary: The authors perform computational and psychophysics experiments to determine whether local representations are more interpretable and useful for model decisions than distributed representations. They conclude that distributed representations are more interpretable and models rely more on them.
weaknesses: I provide a summary first and then elaborate.

* The paper makes general claims about distributed vs local representations but does not provide empirical evidence for this generality or theoretical arguments for why it should be expected.
* Does not make explicit the model of human cognition/perception that motivates the psychophysics experiments and their analyses.
* Does not provide theoretical insight for why sparse distributed representations obtained via NMF should match human-interpretable features.
* Dichotomizes local vs distributed in the interpretation of the results, even though the results indicate the difference might be one of degree.
* Properties of the sampling of neurons are unclear.

Line 093: “…growing consensus that distributed representations constitute a stronger basis for interpretability compared to local ones...” This is the main point of the introduction and should be elaborated and explained. It merits more space than the generic survey-like paragraphs that come before it. In particular, what is the theoretical rationale for why this should be the case? Or is this ‘consensus’ purely driven by empirical findings? In any case, it would be helpful to clarify.

Line 084: “representations driven by single features are more desirable because they are expected to be easier to understand by a human” Why is this so? Single features can be non-intelligible to humans. Otherwise the definition of feature would have human-intelligibility baked in, in which case the quoted statement is circular.

Line 257: “Measure the ambiguity, or perplexity, of the visual feature as a proxy for its interpretability.” But a neuron can be interpretable even though it is ‘ambiguous’ (e.g., because it responds to more than one feature). For instance, it seems to me that responding to a disjunction of features is an interpretable response.

Perhaps these issues can be clarified if the authors make explicit a (ideally formal) notion of human-intelligible feature.

The model of human perception is left implicit in the manuscript. What is the model of human cognition that motivates the predicition that coherence relates to interpretability?

Line 275 mentions “Representative sample of units”. What are they representative of? And how is the representativeness achieved? Relatedly, can the authors clarify why they select the 80 units reported in Zimmerman? How was this selection made?

Do the results generalize beyond ResNet50 and ImageNet? Do they generalize across modalities (vision, audition, language, etc)? As far as I can see, there is no empirical evidence of generalization in these directions in the paper, nor theoretical arguments why it should be expected. Nevertheless, the paper makes a general claim about local versus distributed representations. It would be helpful to have experiments testing generalization in these directions and/or rigorous theoretical arguments.

Figure 1: As with other figures in the manuscript, it might benefit from unambiguously pointing to figure elements in the text. For instance, “depicted in the images at the bottom” should presumably point only to the visual elements in orange, not to all visual elements at the bottom of panel (a). Furthermore, panel (a) contains multiple other elements in the graphs which are not discussed in the figure caption (e.g., axes, black and colored data points, “n_1” labels, etc.). It would be great if the authors can take the reader through each panel explaining how the schematic depicts the concepts the authors want us to take away. Additional issues include: orange and green colors are referenced in the figure caption but gray is not. Likewise, the explanation in Figure 4 could be clearer. For example, you can clearly identify each experiment and avoid ambiguoous referents such as “this set of images”.

Participants perform quite well in the local representation condition. This means, following the authors’ rationale, that the local representation is quite interpretable as well. However, the authors express a more dichotomic view throughout. Why is this so? It seems to me that a dichotomic interpretation of the results is not warranted. At the very least, consideration of mechanisms that use both local and distributed representations, and what each of these contribute to, is called for.

Meta-review: This paper examines the interpretability of distributed versus local representations in neural networks through psychophysical experiments with human participants. While the experimental methodology is sound and provides empirical evidence regarding interpretability of these two kinds of representations, the work has several critical limitations that preclude acceptance. The theoretical foundations explaining why distributed representations should enhance interpretability are inadequately developed. The generalizability of findings remains limited even with the addition of the VGG16 experiments, as results are confined to vision tasks and two specific architectures. In addition, the moderate effect sizes and above-chance performance with both representation types suggest a less dramatic difference between approaches than the paper's framing implies.

Decision: Reject