Labeling Neural Representations with Inverse Recognition
Authors: Kirill Bykov
Laura Kopf
Shinichi Nakajima
Marius Kloft
Marina MC Höhne
Venue: NeurIPS 2023 poster (2023)
Keywords: Explainable AI
Mechanistic Interpretability
Machine Learning
Deep Neural Networks
TL;DR: We propose a new scalable method called Inverse Recognition (INVERT), designed to label neurons in Deep Neural Networks, based on their capability to identify concepts.

Abstract: Deep Neural Networks (DNNs) demonstrate remarkable capabilities in learning complex hierarchical data representations, but the nature of these representations remains largely unknown. Existing global explainability methods, such as Network Dissection, face limitations such as reliance on segmentation masks, lack of statistical significance testing, and high computational demands. We propose Inverse Recognition (INVERT), a scalable approach for connecting learned representations with human-understandable concepts by leveraging their capacity to discriminate between these concepts. In contrast to prior work, INVERT is capable of handling diverse types of neurons, exhibits less computational complexity, and does not rely on the availability of segmentation masks. Moreover, INVERT provides an interpretable metric assessing the alignment between the representation and its corresponding explanation and delivering a measure of statistical significance. We demonstrate the applicability of INVERT in various scenarios, including the identification of representations affected by spurious correlations, and the interpretation of the hierarchical structure of decision-making within the models.

Labeling Neural Representations with Inverse Recognition
Kirill Bykov"
Laura Kopf UMI Lab ATB Potsdam Potsdam, Germany
UMI Lab ATB Potsdam Potsdam, Germany kbykov@atb-potsdaan de
Stopffatb-potssam.cc
Shinichi Nakajima
Marius Kloft
Marina M.-C. Höhne
Machine Learning Group
Machine Learning Group
UMI Lab
TU Berlin
RPTU Kaiserslautern-Landaa Kaiserslautem, Germany kloft0cs. uni-kl.de
ATB Potsdam
Berlin, Germany nakaj wwwetumerreleenn..ommmmm de
University of Potsdam, Germany
mhoehne@atb wotttttb-podddmm...mmm de
Abstract
Deep Neural Networks (DNNs) demonstrate remarkable capabilities in learning complex hierarchical data representations, but the nature of these representations remains largely unknown. Existing global explainability methods, such as Network Dissection, face limitations such reliance segmentation masks, lack of statis- tical significance testing, and high computational demands. We propose Inverse Recognition (INVERT), scalable approach for connecting learned representations with human-understandabbe concepts by leveraging their capacity to discriminate between these concepts. In contrast to prior work, INVERT is capable of handling diverse types of neurons, exhibits less computational complexity, and does not rely on the availability of segmentation masks. Moreover, INVERT provides an interpretable metric assessing the alignment between the representation and its corresponding explanation and delivering measure of statistical significance. We demonstrate the applicability of INVERT in various scenarios, including the identi- fication of representations affected by spurious correlations, and the interpretation of the hierarchical structure of decision-making within the models.
1 Introduction
Deep Neural Networks (DNNs) have demonstrated exceptional performance across broad spectrum of domains due to their ability to learn complex, high-dimensional representations from vast volumes of data (I]. Nevertheless. despite these impressive accomplishments, our comprehension of the concepts encoded within these representations remains limited. The "black-box" nature of representa- tions, combined with the known susceptibility of networks learn spurious correlations biases [5] and harmful stereotypes [6] poses significant risks for the application of DNN systems, particularly in safety-critical domains
7
To tackle the problem of the inherent opacity of DNNs, the field of Explainable AI (XAI) has emerged 88 9 [0]. The global explanation methods aim explain the concepts and abstractions learned within the DNNs representations. This is often achieved by establishing associations between by visualizing the stimuli responsible
neurons and uumaan-nneestanddbble concepts II
"Correspondigg author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

18]. Such methods demonstrated themselves be capable of detecting the malicious behavior and identifying the specific neurons responsible [T9 20
for provoking high neural activation levels 16
In this work, we introduce the Inverse Recognition (INVERT) method for labeling neural representa- tions within DNNs. Given specific neuron, INVERT provides an explanation of the function of the neuron in the form of a composition of concepts, selected based on the ability of the neuron to detect data points within the compositional class. Unlike previous methods, the proposed approach does not rely on segmentation masks and only necessitates labeled data, is not constrained by the specific type of neurons, and demands fewer computational resources. Furthermore, INVERT offers statistical significance test to confirm that the provided explanation is not merely a random occurrence. We evaluate the performance of the proposed approach across various datasets and models, and illustrate its practical use through multiple examples.
2 Related work
Post-hoc interpretability, a subfield within Explainable AI, focuses on explaining the decision decision-making strategies of Deep Neural Networks (DNNs) without interfering with the original training process
21 12. Within the realm of post-hoc methods, a fundamental categorization arises concerning the
scope of explanations they provide. Local explanation methods aim to explain the pocision-making
process for individual data points, often presented in the form of attribution maps [23 24 25 On the
other hand, global explanation methods aim to explain the prediction strategy learned by the machine
across the population and investigate the purpose of its individual components [26] 27
Inspired by principles from neuroscience the in-depth examination of individual model components and their functional purpose [3T. Often, global explainability is referred to as mechanistic interpretability, particularly in the context of Natural Language Processing (NLP) [32 33 34 33). Global approach to interpretability allows for the exploration of concepts learned by the model [36]
28
29 30
global explainability directs attention towards
38 39 and explanation of circuits computational subgraphs within the model that learn the transformation of various features [1] Various methods were proposed to interpret the learned features, including
37
Maximisation (AM) methods [15]. These methods aim to explain what individual neurons or groups of neurons have learned by visualizing inputs that elicit strong activation responses. Such input signals can
either be found in an existing dataset [16] or generated synthetically [42 17
I8]
AM methods
demonstrated their utility in detecting undesired concepts learned by the model 19 43 20). However,
these methods require substantial user input to identify the concepts embodied in the Activation- Maximization signals. Recent research has demonstrated that such explanations can be manipulated while maintaining the behavior of the original model [44 45 46).
Another group of global explainability methods aim to explain the abstraction learned by the neuron within the model, by associating it with the human-understaan understandable concepts. The Network Dissection (NetDissect) method 47 was developed to provide explanations by linking neurons to concepts, based on the overlap between the activation maps of neurons and concept segmentation masks, quantified using the Intersection over Union (IoU) metric. Addressing the limitation that neurons could only be explained with a single concept, the subsequent Compositional Explanations of Neurons (CompExp) method was introduced, enabling the labeling of neurons with compositional concepts [12. Despite their utility, these methods generally have limitations, as they primarily applicable to convolutional neurons and necessitate a dataset with segmentation masks, which significantly restricts their scalability (a more comprehensive discussion of these methods can be found in Appendix (.2).. Other notable methods include CLIP-Dissed [3), MILAN [48), and FALCON [49]. However, these methods utilize additional model to produce explanations, thereby introducing a new source of potential unexplainability stemming from the explainer model.
INVERT: Interpreting Neural Representations with Inverse Recognition
In the following, we introduce method called Inverse Recognition (INVERT). This method aims to explain the abstractions learned by a neural representation by identifying what compositional concept representation is most effective detecting in binary classification scenario. Unlike the general
"The code can be accessed via the following link: https///gthaab.oom/ppppaa//rrrrt

---

objective of Supervised Learning (SL) [50,, which is to learn representations that can detect given concepts, the central idea behind INVERT is learn compositional concept that explains a given representation the best.
Let M C Rm where m N is the number of dimensions of data, be the input (data) space. We use the term neural representations to refer to a sub-function of a network that represents the computational graph from the input of the model to the scalar output (activation) of a specific neuron, or any combination of neurons, that results in scalar function.
Definition 1 (Neural representation). A neural representation f E F is defined as real-valued function f ID + R. which maps the data domain D to the real numbers IR. Here, F represents the space of real-valued functions on ID)
Frequently, in DNNs, particular neurons, like convolutional neurons, produce multidimensional outputs. Depending on the specific needs of the application, these multidimensional functions can be interpreted either as a set of individual scalar representations or the neuron's 's output can be aggregated to yield single sealar output, e.g. with pooling operations, such average- max-pooling. Unless stated otherwise, we utilize average-poole poollgg- as the standard aggregation measure.
We define concept as mapping that represents the human process of attributing characteristics to data.
Definition 2 (Concepts), A concept E C is defined as a function: c: D - {0, 1}, which maps the data domain ID to the set of binary numbers. value of 1 indicates the presence of the concept in the inpul, and 0 indicates its absence. Here, C corresponds to the space of all concepts, that could be defined on D.
In practice, given the dataset C D. concepts are usually defined by labels, which reflect the judgments made by human experts. We define C = {c., Ca} C as set of NN atomic concepts, that are induced by labels of the dataset (also referred to as primilivee concepts or primilives). Within the context of this work, we permit concepts be non-disjoint, signifying that each data point may have multiple concepts attributed to it. Additionally, we define vector C = [ci. cal E C².
A key step for explaining the abstractions learned by neural representations relies the choice of the similarity measure between the concept and the representation. INVERT evaluates the relationship between representation and concepts by employing the non-parametric Area Under the Receiver Operating Characteristic (AUC) metric, measuring the representation' 's ability to distinguish between the presence and absence of a concept.
Definition 3 (AUC similarity). Let f E F be a neural representation, dataset D C ID and concept E C. We define similarity measure d F. x C -> [0, 1]
d(f,c)= E(= E(y LEI [[f(a)< < f(v) 0 Z
(I)
{v y E D,cy))
where / [f(x) < f()) is an indicator function that yields if f() < fv) and 0 otherwise.
AUC provides interpretable measure to assess the ability of the representation systematically output higher activations for the datapoints, where the concept is present. An AUC denotes a perfect classifier, while AUC of 0.5 suggests that the classifier's performance is no better than random chance.
Given that various concepts have different numbers of data points associated with them, for concept c E C we can compute concept fraction, corresponding to the ratio of data points that are positively labeled by the concept:
T(c) = { r E D, c(x) =
1r x DED)
3.1 Finding Optimal Compositional Explanations
Given representation f E F, the INVERT's objective is to identify the concept, that maximizes the AUC similarity with the representation, or, in other words finding the concept that representation is detecting the best. Due to the ability of representations to detect shared features across various concepts explaining a representation with a single atomic concept from C may not provide a com- prehensive explanation. To surmount this challenge, we adopt the existing compositional concepts
3

---

Neuron, fxa
Labeled dataset, D
Distribution far activations across the dataset
images alosivate several Nigheadivatee signals
Ciraleg sloce
P
Cftetial at steck
1.0
u.e
Fimity
Chrmettr
0A-
0.2
ReshAuTS,, AvgPaol layor NNooo
InieglMet 2012 volidation ael
20
INVERT explanation
interpesting a neuron by identtyyng the concept that delecte boet
"clock"
"odoneter"
"clock"
"cookk
"ddometer"
AND
"wal.1 clock"
AUC 0.9061
AUC = 0.9260
AUC = 0.9573
Dendtt
0.6
0.0
pow
0.8
c.8
0.0
0.6
posi
0.4
0.4
0.0
=
L= 3
Figure 1: Demonstration of the INVERT method (B = 1,a = 0.35%) for the neuron fa3 from Re AvgPool layer (Neuron 33), using ImageNet 2012 validation dataset. The resulting explanations can be observed in the bottom part of the figure, where three steps of the iterative process are demonstrated from L to 3. It can be observed that INVERT explanations align with the neurons high-activating images, illustrated in the top right figure.
approach [2]. and we augment the set of atomic concepts C' by introducing new generic concepts, as logical combination of existing ones. These logical forms involve the composition of AND, OR. and NOT operators, and they are based on the atomic concepts from C.
Definition 4 (Compositional concept). Given vector of atomic concepts C, compositional concept is a higher-order interpretable function that maps C to a new, compositional concept:
Cd C.
xp
For example, let C. = {c1, o2] be a set of atomic concepts with corresponding vector C. Let e₁ be a concept for "dog", and C2 concept for "Ilama". Then (oCC) ca OR C2 "dog" OR "Ilama" is compositional concept with the length L = 2. The (C)) is a concept in itself (i.e. (cCC E C) and corresponds to a concept that is positive for all images of dogs or Ilamas in the dataset.
Evaluating the performance of all conceivable logical forms across all of the d concepts from C is generally computationally infeasible. Consequently, the set of potential compositional concepts & is restricted to a form of predetermined length L E N, where L is a parameter of the method. The objective of INVERT, in this context, can be reformulated as:
arg max d (f, (CC). We
To determine the optimal compositional concept that maximizes AUC, we employ approach similar to that used in [1]. utilizing Beam- Search optimization. Parameters of the proposed method include predetermined length L E N. the beam size B E N. Additionally, during the search process explanations could be constrained to the condition (C)( E [a, 3). where 0 S a < 3 < 0.5. In Section 4.1 we further demonstrate that by imposing such constraint on the concept fraction resulting explanations could be made more comprehensive. We refer to the standard approach when 0, B 0.5. In experiments, unless otherwise specified, the parameter , is set to 0.5. Additional details and a description of the algorithm can be found in Appendix [A.3
Figure T Iillustrates the INVERT pipeline for explaining the neuron from ResNet18 88 Average Pooling layer (51). For this, we employed the validation set ImageNet2012 [52) as the dataset D, in the INVERT process. This subset contains 50,000 images from 1,000 distinct, non-overlapping

---

VIT Features layer (CLS token), Neuron 592
"Bad" concept
INVERT, L=
"wtter buttalo"
cup
"bevrrage"
substance
AUC - 0.5385 T-
0.1%
AUC - 0.9114 P
7= 0.1%6
AUG 0.8134 (1)
T 0.4%
AUC 0.8124
7 0.5% T
p=
regetis
rapstim
p 0.005
p 0.005
( 0
Ct 0.2 %
(1- 0.6 %
Figure 2: The figure illustrates the contrast between poor explanation (on the left) and INVERT explanations with L = -1 and varying parameter a, for neuron 592 in the VIT B 16 feature-extractor layer. The INVERT explanations were computed over the ImageNet 2012 validation set. The figure demonstrates that as the parameter increases, the concept fraction ''' also increases, indicating that more data points belong to the positive class. Furthermore, this figure showcases the proposed methods ability to evaluate the statistical significance of the result. The poor explanation fails the statistical significance test (double-sided alternative) with a w p-value of 0.35, while all explanations
provided by INVERT exhibit p < 0.005.
classes, each represented by 50 images. Notably, since ImageNet classes are intrinsically linked to WordNet [53], we extracted an additional 473 hypernyms, or higher-level categories, and assigned beige color to
labels for these overarching classes. In Figure [ and subsequent figures, represent individual ImageNet classes and orange color to represent hypernyms. In the density plot graphs, the orange density illustrates the distribution of data point activations that belong to the explanation concept, while blue represents the distribution of activations of data points corresponding to the negation of the explanation.
3.2 Statistical significance
IoU-based explanations, such those provided by the Network Dissection method (1. often report small positive IoU scores for the resulting explanations. This raises concerns about the potential randomness of the explanation. The AUC value is equivalent to the Wilcoxon-Mamn Whitney statistic [54] and can be interpreted as a measure based on pairwise comparisons between classifications of the classes. Essentially, it estimates the probability that the classifier will rank randomly chosen positive example higher than a negative example 55
Given the concept C, this connection to the Mann Whitney U test allows to test if the distri- bution of the representations activations on the data points where concept e is positive significantly differs from the distribution of activations report the corresponding p-value (against a double- or one-sided alternative), which helps avoid misinterpretations due to randomness, thereby improving the reliability of the explanation process, as shown in Figure 2 In all subsequent figures, the explanations provided by INVERT achieve statistical significance (against double-sided alternative) with standard significance level (0.05).
points where the concept is negative. We can then
4 Analysis
In this section, we provide additional analysis of the proposed method, including the effect of constraining the concept fraction of explanations and comparison of the INVERT to the prior methods.
4.1 Simplictty-Precision tradeoff
The INVERT method is designed to identify the compositional concept that has the highest AUC similarity to given representation. However, the standard approach neglects to account for the class imbalance between datapoints that belong and do not belong to particular concept, often leading to precise but narrowly applicable explanations due to the small concept fraction. To mitigate this
5

---

ResNet18
GoogLeNet
EfficientNet B0
VIT 16
1.0
1.0
0.0
0.9
1.8
1.7-
0.0
0.0-
0.6-
0.5
Random
Kandem
T.mmuu length
formus leogth
2,0%
1.0
2.0%
1.0
2.0%
1.0
2.0
0.9
1.5%
0.9
1.5%
0.9
1.5%
0.9
1.54
2.00
goa
1,0%
1.0%
1.0%
0.7
0.6
0.5%
0.-
3.%
0.5%
0.6
0.5%
0.5
0.0%
0.5
2.0%
0.5
0.0%
0.5
0.0%
Foction
Mit Fraction
Fourtoe
Figure 4: Impact of the parameter a and formula length L on the resulting explanations. The first row of the figure shows the average AUC of optimal explanations for 50 randomly sampled neurons from the feature-extractor part of each one of the four ImageNet pre-trained models, conditioned by different values of parameter a in different colors. These graphs indicate that neurons generally tend to achieve the highest AUC for one individual class with f, = and a 0. The second row presents the distribution of AUC scores alongside the distribution of concept fractions T for the INVERT explanations of length L 5, for each model. Here, we can observe clear trade-off between the precision of the explanation in terms of AUC measure and concept size T.
issue, we can modify the INVERT process work exclusively with compositional concepts where the fraction equals or exceeds a specific threshold, represented as a.
For this experiment, the INVERT method was uti- lized on the feature extractors of four different mod- els trained on ImageNet. These models include ResNet18 [51] GoogleNet [56). EfficientNet BO (57). and ViT B 16 [58] In this experiment, we exam- ined 50 randomly chosen neurons from the feature- extractor layer of each model. We utilized the Im- ageNet 2012 validation dataset D,, which was out- lined in the previous section, generate INVERT explanations with B 3 varying the explanation length L between 1 and and parameter a, re- sponsible for the constraining the concept fraction, a (0, 0.002, 0.005, 0.013.
Neuron 88, ResNet18, AvgPool layer
INVERT, L=
"recreational vchicte
OR
"eratter tousk"
parbace crack"
1.00
AUC 0.9601, 0
negalliee positvve
a=0%
Denaty
0.75
0.50
0.25
0.00
8
10
("sccraational vehicte")
("otckup""
0.055% %.
0.75
AUC 0.5234, T %
negative positive
0.50
0.25
0.00
The experiments results are depicted in Figure 4 For all models, we can see effect that call the simplicinn-precisoon tradeoff: the explanations with the highest AUC typically involve just one individ- ual class with low concept fraction, achieved in an unrestricted mode with parameter a set to O. By con- straining the concept fraction and increasing the explanation length L, we can improve AUC scores while still maintaining the desired concept fraction. Still, this indicates that more generalized, broader explanations come at the cost of loss in precision terms of the AUC measure. Figure 3 demonstrates how the change of parameter affects the resulting explanation.
10
Ppupiac Lrunspacs
Trotso www.cc
NOT
1.00
AUC 0.8233,
T= 1.6% %
negative positve
a=1% %
Denaity
0.75
0.50
0.00
Figure 3: Three different INVERT explana- tions, computed by adjusting the parameter a for the Neuron 88 in ResNet18 8 AvgPool layer. Higher values of this parameter lead to broader explanations, albeit at the cost of precision, thus resulting in a lower AUC. The visualization of the WordNet taxonomy for the hypernyms is provided the Appendix B
4.2 Evaluating the Accuracy of Explanations
While it is generally challenging to obtain ground-truth explanations for the latent representations in Deep Neural Networks (DNNs), in Supervised Learning, the concepts of the output neurons are

---

Table 1: A comparison of explanation accuracy between NetDissect and INVERT. The accuracy is computed by matching identified classes with the ground truth labels.
Model
Dataset
NetDissect
INVERT
MaskRCNN ResNet50 FPN
MS COCO 95.06%
98.77 %
FCN ResNet50
MS COCO
95.24% %
95.24%
8
ImageNet 19.2% 19.7%
73.2%
GoogleNet
ImageNet
82.2
DenseNet161
ImageNet 19.1%
86.9%
defined by the specific task. the subsequent experiment, we compared the performance of INVERT and Network Dissection in accurately explaining neurons when the ground truth is known.
For this experiment, employed 5 different models: 2 segmentation models and classification models. For image segmentation, we employed MaskRCNN ResNet50 FPN model [59] pre-trained MS COCO dataset (60) and evaluated a subset of 24,237 images of MS COCO train2017.
containing 80 distinct classes, and FCN ResNet50 model [61], pre-trained on MS COCO, and evaluated
subset of MS COCO val2017, limited the 20 categories found in the Pascal VOC dataset [62]. For classification models we employed ImageNet pre-trainee ResNet18 [5]]
DenseNet161 [63] and GoogleNet [64], with 1,000 output neurons, each neuron corresponding to the individual class in the ImageNet dataset.
The outputs from the segmentation models were converted into pixel-wise wise confidence scores. These scores were ar- ranged in the format [NB, .e, H, W], where NB repre- sents the number of images in a batch, and N₂ signifies the number of classes. Each value indicates the likelihood of specific pixel belonging to particular class. To aggregate multidimensional activations, the INVERT method used a max-pool operation.
Computational Cost Comparison
10
INVERT
CompExp
Hours
6
All the classification models that used had 1,000 dimensional output neurons. The evaluation process for both explanation methods was carried using subset of 20,000 images from the ImageNet-2012 validation dataset. For the Network Dissection method, which necessitates segmentation masks, these masks were generated from the bounding boxes included in the dataset. Both Network Dissection and INVERT methods were implemented using standard parameters.
2
0
Formula Length
Figure 5: Comparing the computational cost INVERT with Compositional Ex-
planations of Neurons method (Com- pExp) in hours with varying formula lengths.
Table presents the outcomes of the evaluation process. It is noteworthy that INVERT exhibits superior equiv- alent performance to Network Dissection across all tasks. Importantly, INVERT accurately identify concepts in image segmentation networks using only the labels of images, in comparison the Network Dissection method that uses segmentation masks.
Computational cost comparison
Methods such as Network Dissection and Compositional Explanations (CompExp) of neurons have been observed exhibit computational challenges mainly due to the operations on high- dimensional masks. While CompExp and INVERT share a beam-search optimization mechanism, proposed approach allows for less computational resources since logical operations are performed on binary labels, instead of masks. Figure S showcases the running time of applying INVERT and Compositional Explanations for explaining 2048 neurons in layer of the PCOS-ResNet5O-FPP
model (65) pre-trained on the MS COCO dataset [60] on singe Tesla -1000-PCCI-3300 GPU.

---

The time comparison of varying formula lengths demonstrates the advantage of INVERT being more effective computationally, which leads to reduced running time and computational costs.
5
Applications
In this section, we outline some specific uses of INVERT, including auditing models for spurious correlations, explaining circuits within the models, and manually creating circuits with desired characteristics.
5.1 Finding Spurious Correlations by Integrating New Concepts
Due to the widespread use of Deep Neural Networks across various domains,
ResNet18, AvgPool Layer, Neuron 154
crucial to investigate
whether these models display spurious correlations, backdoors, or base their decisions on undesired con-
"african chameleen")
INVERT with
ImageNet
2- (- o8
0.8
AUC = 0.95, T 0.1%
bositive
cepts. Using the known spurious dependency of
0.-
ImageNet-trained models on watermarks written in Chinese [19 66 67] we illustrate that INVERT pro- vides a straightforward method to test existing hy- potheses regarding the models dependency on spe- cific features and allows for identification of the par- ticular neurons accountable for undesirable behavior.
04
10
25
20
INVERT with imagehet
"chinese text
text Text concepts
L- 1 006
1.0
AUC:
nogotiic
Demaly
R g S.
To illustrate this, we augmented the ImageNet dataset Dr, with an additional dataset, DD, comprising 100 images. This new dataset contains images for each of two distinct concepts: Chinese textual water- marks and Latin textual watermarks (see Appendix A.4) We created examples of these classes by ran- domly selecting images from the ImageNet dataset and overlaying them with randomly generated textual watermarks. Figure/o]depice depicts the change in the expla- nation process conducted the original dataset and its expanded version. Since the original dataset didn't 't include the concept of watermarked images, the label "African chameleon" was attributed to the representation. However, after augmenting the dataset with two new classes, the explanation shifted to the "Chinese text" concept, with the AUC measure increasing to 0.99. This demonstrates the capability of INVERT pinpoint sources of spurious behavior within the latent representations of the neural network.
10
Figure 6: Difference of INVERT (L = L,a 0) explanations of Neuron 154 in Average Pooling layer of ImageNNtt-rrained ResNet18 model before (top) and after (bottom) integra- tion of new concepts the dataset.
5.2 Explaining Circuits
INVERT could be employed for explaining circuits - computational subgraphs within the model, demonstrating the information flow within the model [41]. The analysis of circuits enables us to
understand complex global dicision-making strategies by examining how features transform from one layer to another. Furthermore, this approach can be employed for glocal explanations [68 local explanation of a particular data point can be deconstructed into local explanations for individual neurons in the preceding layers, explained by INVERT.
To illustrate this, we computed INVERT explanations (J. = 3, a = 0.002) for all neurons in the average pooling layer of R8sNet18. This was based on the augmented dataset from the preceding section. In ResNet18, the neurons in the Average Pooling layer have a linear connection to the output class logits. Figure 7 (left) illustrates the circuit of the three most significant neurons (based on the weight of linear connection) linked to the "carton" output logit. It could be observed that this class depends on Neuron 296, a "box" detector, and Neuron 154, which identifies the "Chinese text" concept. Furthermore, the right side of Figure depicts the decomposition of local explanations: given an image of a carton box, we can dissect the GradCam (69 local explanation of a "carton" class-logit into the composition of local explanations from individual neurons. It is noticeable how Neuron 296 assigns relevance to the box, while Neuron 154 assigns relevance solely to the watermark present in the image. More illustrations of different circuits can be found in Appendix A.8

---

AND NOT
crunky
0.2844
Neuron
0.3127
AND NOT Cperett
478, carton
Neuron 154
3.3032
1D20/1356
478, carton nG2971356
Output ligat Neuron 478, "caronn 2
ArgRont Neuron 296
Auggool Neuron 154 AND LOT OCLAmeD Blaac
soxt OR "slper
[dlanenbacck
TRL MOD ABD "OENNAL 2A
Figure 7: The figure illustrates the "carton" circuit within the ResNet18 model. The left part of the figure showcases the three most significant neurons (in terms of the weight of linear connection) and their corresponding INVERT explanation linked to the class logit "carton". The right part of the figure demonstrates how the local explanation from the class logit can be decomposed into individual explanations of individual neurons from the preceding layer.
5.3 Handcrafting Circuits
In this section, we demonstrate that its somewhat feasible to use the knowledge of what concepts are detected by neurons to combine them into manually designed circuits that can detect novel concepts. Just compositional concepts formed using logical operators, we employed fuzzy logic operators between neurons to construct meaningful handcrafted circuits with desired properties.
In contrast to conventional logic, fuzzy logic operators allow for the degree of membership to vary from 0 to 1 70] For this experiment we employed the Gödel norm that demonstrated the best performance among other fuzzy logic operators (see Appendix A.5 details). For the two functions f. D [0, 1]. the Gödel AND (T-norm) operator defined as min(f. f. g) and the OR (T-conorm) is defined as max(S,g) 9). Negation is performed by the f operation.
We utilized the ImageNet-trained ViT L. 16 model 58 specifically 1024 representations from the Ceature-extractor layer. The output of each of these representations was mapped to the range [o. 1] by first normalizing the output based on their respective mean and standard deviation across the ImageNet 2012 validation dataset, and then applying the Sigmoid transformation. In this experiment, for each of the 1473 ImageNet atomic concepts (which includes 1000 classes and 473 hypernyms), we identified neuron from the Peature-extractor layer that showed the highest AUC similarity. For instance, for the concept "boat", Neuron 61 exhibited the highest AUC similarity (denoted as fboa), for the concept "house", Neuron 899 showed the highest AUC similarity (denoted as fhouse, and for the concept "lakeside". Neuron 575 showed the highest AUC similarity (denoted as ftakeside).
Further, we manually constructed six different compositional formulas using concepts from ImageNet that were designed resemble different concepts from the Places365 [7]] dataset. For example, for the "boathouse" class from Places365, we assumed that images from this class would likely include "boat", "house", and water, represented by the concept "lakeside". As such, we constructed compositional formula "boat" AND "house" AND "lakeside" using concepts from the ImageNet dataset. Finally, using the neurons, that detect these concepts (e.g. fowa, Jhowse flakeside) we manually constructed the circuits using Gödel fuzzy logic operators. That is, for "boathouse" example, final circuit was formed as g(x) min
min(fosa (x). fhose (I), fusessie (=)) using the Gödel AND operator. The performance of the resulting circuits was evaluated on the Places365 dataset in terms of AUC similarity with the concept. essence, by labeling representations using the ImageNet dataset and manually building a circuit guided by intuition, we evaluated how this newly created function can perform in detecting a class in the binary classification task on a different dataset.
Figure 8 iillustrates the "boathouse" example and three other handcrafted circuits derived from ViT representations (the other two circuits can be found Appendix T6]. We found that after performing this manipulation, the AUC performance in detecting the Places365 class improved compared to the performance of each individual neuron. This example shows that by understanding the abstractions behind previously opaque latent representations, we can potentially construct meaningful circuits and utilize the symbolic properties of latent representations. In Appendix [A.6] we further demonstrate that when labels of the target dataset overlap or are similar to the dataset used for explanation, fine-tuning of the model can achieved by simply employing representations with explanations matching the target labels.

---

Handcrafted circuits from ViT representations
ImageNet single-class
InageNet hyper-class
Places365 single-class
"Lakeside" AND "boat" AND "house"
"boachouse"
*i Littrryyunnoom uni_oun"
AND "cank"
Party basee
Neuron 575
fin(fe:fass)
Neuron 560
minofroo.
AUROC: 0.712
AND
AUROC: 0.908
AND
AND
Wel mititary
Neuron 899
"house" AUROC
ALIOCC
AUROC:
busines iness
place of
AND 'monitor" OR "spotllight"
teeevisioo studio"
AND
'housing
"bedzoon"
Neuron 407
Neuron 648
Neuron
mintlan, mantFan
THOYL COC"
"plass DE
"Dod AUROC 0.740
min(foo..aao)
AUROC: 0.728
AND
AND
froeh
("spart on mamtttr
Neuron 213
AND "housing
AUROC: 0.829
"houssing" AUROC: 0.758
AUROC: 0.760
Figure 8: The figure presents four distinct handerafted circuits, created from the latent representations from the ImageNet-trained ViT L 16 feature-extracto layer to detect classes from the Places365 dataset. For each neuron, combination of neurons, provide the Area Under the Receiver Operating Characteristic (AUROC) score for the Places365 concept in a binary classification task, distinguishing between the presence and absence of this concept.
6 Disscussion and Conclusion
In our work, we introduced the Inverse Recognition (INVERT) method, a novel approach for interpreting latent representations Deep Neural Networks. INVERT efficiently links neurons with compositional concepts using an interpretable similarity metric and offers a statistical significance test to gauge the confidence the resulting explanation. We demonstrated the wide-ranging utility of our method, including its capability for model auditing to identify spurious correlations, explaining circuits within models, and revealing symbolic- like properties connectionist representations.
While INVERT mitigates the need for image segmentation masks, it still relies on a labeled dataset for explanations. In future research, we plan to address this dependency. Additionally, will explore different similarity measures between neurons and explanations, and investigate new ways to compose cumman--undrsttddable concepts.
The widespread use Deep Neural Networks across various fields underscores the importance of developing reliable and transparent intelligent systems. We believe that INVERT will contribute to advancements in Explainable AI, promoting more understandable AI systems.
Acknowledgements
This work was partly funded by the German Ministry for Education and Research (BMBF) through the project Explaining 4.0 (ref. 011S200551). Shinichi Nakajima was supported by the German Ministry for Education and Research (BMBF) as BIFOLD Berlin Institute for the Foundations of Learning and Data under the grant BIFOLD23B. Marius Kloft acknowledges support by the Carl-Zeiss Foundation, the DFG awards KI. 2698/2-1, KI. 2698/5-- KI. 2698/6-1 1, and Kl. 2698/7-1,
and the BMBF awards 03/B0770E and 011521010C.
10

---

References
[1] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1825 2013.
[2] Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 10: 1096, 2019.
[3] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 (11)):65-613 663556673220 2020.
[4] Kirill Bykov, Laura Kopf, and Marina M.-C. Höhne. Finding spurious correlations withafunnttion-seannis contrast analysis. In Luca Longo, editor, Explainuble Artificial Intelli- gence, pages 549-572, Cham, 2023. Springer Nature Switzerland.
[5] Laleh Seyyed- Kalantari, Haoran Zhang, Matthew A. McDermott, Irene Y. Chen, and Marzyeh Ghassemi. Underdiagnosis Bias of Artificial Intelligence Algorithms Applied to Chest Radio- graphs in Under-Served Patient Populations. Nature Medicine, 27(12):2176-211 December 2021.
[6] Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Dehora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. Easily Accessible Text-to- Image Generation Amplifies Demographic Stereotypes at Large Scale, November 2022.
[7] Oliver Willers, Sebastian Sudholt, Shervin Raafatnia, and Stephanie Abrecht. Safety concerns and mitigation approaches regarding the use of deep learning in safety-critical perception tasks. In Computer Safety, Reliability, and Security. SAFECOMP 2020 Workshops: DECSoS 2020, DepDevOps 2020, USDAI 2020, and WAISE 2020, Lisbon, Portugal, September 15, 2020, Proceedings 39, pages $36-350. Springer, 2020.
[8] Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller. Explainable Al: interpreting, explaining and visualizing deep learning, volume 11700. Springer Nature, 2019.
[9] Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pages 80-89. IEEE, 2018
[10] Feiyu Xu, Hans Uszkoreit, Yangzhou Du, Wei Fan, Dongyan Zhao, and Jun Zhu. Explainable ai:
A brief survey on history, research areas, approaches and challenges. In Natural Language Pro- cessing and Chinese Computing: 8th CCF International Conference, NLPCC 2019, Dunhuang, China, October 9-14, 2019, Proceedings, Part 11 8, pages 563-574. Springer, 2019.
[11] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissec- tion: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541-6549, 2017.
[12] Jesse Mu and Jacob Andreas. Compositional explanations of neurons. Advances in Neural Information Processing Systems, 33:17153-17163, 2020.
[13] Tuomas Oikarinen and Tsui-Wei Weng. Clip-dissect: Automatic deseription of neuron repre- sentations in deep vision networks. arXiv preprint arXiv:2204. 20w.:2204..005, 2022.
[14] Adriano Lucieri, Muhammad Naseer Bajwa, Stephan Alexander Braun, Muhammad Imran Malik, Andreas Dengel, and Sheraz Ahmed. Exaid: A multimodal explanation framework for computer-aider aided diagnosis of skin lesions. Computer Methods and Programs in Biomedicine, 206:10 2022.
[15] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.
[16] Judy Borowski, Roland Simon Zimmermann, Judith Schepers, Robert Geirhos, Thomas SA Wallis, Matthias Bethge, and Wieland Brendel. Natural images are more informative for interpreting cnn activations than state-of-the-art synthetic feature visualizations. In NewrIPS 2020 Workshop SVRHM, 2020.
11

---

[17] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11):e7, 2017.
[18]
Thomas FEL. Thibaut Boissin, Victor Boutin, Agustin Martin Picard, Paul Novello, Julien Colin, Drew Linsley, Tom ROUSSEAU, Remi Cadene, Lore Goetschalckx, Laurent Gardes, and Thomas Serre. Unlocking feature visualization for deep network with MAgnitude constrained optimization. In seirth-seventh Conference on Neural Information Processing Systems, 2023.
[19]
Kirill Bykov, Mayukh Deb, Dennis Grinwald, Klaus Robert Muller, and Marina MC Höhne. DORA: Exploring outlier representations in deep neural networks. Transactions Machine Learning Research, 2023.
[20] Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 6(3)=e30, 2021.
[21] Sajid Ali, Tamer Abuhmed, Shaker El-Sappagh, Khan Muhammad, Jose M Alonso-Moral, Roberto Confalonieri, Riccardo Guidotti, Javier Del Ser, Natalia Diaz-Rodríguez, and Francisco Herrera. Explainable artificial intelligence (xai): What we know and what is left to attain trustworthy artificial intelligence. Information Fusion, 99: 99:101805, 2023.
[22] Daniel Vale, Ali El-Sharif, and Muhammed Ali. Explainable artificial intelligence (xai) post- hoc explainability methods: Risks and limitations in non-discrimination law. Al and Ethics, 2(4):815 826, 2022.
[23] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How to explain individual classification decisions. Journal of Machine Learning Research, (JJun): (attun)::800000 803-1831, 2010.
[24] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130144 2015.
[25] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions.
Advances in neural information processing systems, 30, 2017.
[26] Rabia Saleem, Bo Yuan, Fatih Kurugollu, Ashiq Anjum, and Lu Liu. Explaining deep neural networks: A survey on the global interpretation methods. Newrocomputing, 2022.
[27] Vijay Arya, Rachel KE Bellamy, Pin-Yu Chen, Amit Dhurandhar, Michael Hind, Samuel C Hoffman, Stephanie Houde, Q Vera Liao, Ronny Luss, Aleksandra Mojsilovic, al. One explanation does not fit all: A toolkit and taxonomy of ai explainability techniques. arXiv preprint arXiv: 00..9999101... 2019.
[28] David H Hubel and Torsten N Wiesel. Receptive fields of single neurones in the cat's striate cortex. The Journal of physiology, 148(3):574, 1959.
[29] Vernon Mountcastle. Modality and topographic properties of single neurons of cat's somatic sensory cortex. Journal of neurophysiology. 20(4):408-434 1957.
[30] John OeKeefe and Jonathan Dostrovsky. The hippocampus as a spatial map: preliminary evidence from unit activity in the freely-moving rat. Brain research, 1971.
[31] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5()) 001, 2020.
[32] Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring sequence model trained on synthetic task. urXiv preprint arXiv:22 13382, 2022.
[33] Neel Nanda, Lawrence Chan, Tom Liberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.052//1 2023.
[34] N Elhage, N Nanda, C Olsson, T Henighan. N Joseph, Mann, A Askell, Y Bai, A Chen, T Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
[35] Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, llya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. URL httx://openaipublic blob. core. windows, nelheeuro-explainnrriue per/index. html, (Date accessed: 14.05, 2023), 2023.
12

---

[36] Johanna Vielhaben, Stefan Bluecher, and Nils Strodthoff. Multi-dimensional concept discovery (med): A unifying framework with completeness guarantees. Transactions on Machine Learning Research, 2023.
[37] Laura O' O'Mahony, Vincent Andrearezyk, Henning Müller, and Mara Graziani. Disentangling neuron representations with concept vectors. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 3769-3774, 2023.
[38] Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept- based explanations. Advances in neural information processing systems, 32, 2019.
[39] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (teav). In International conference on machine learning, pages 2668-2677. PMLR, 2018.
[40] Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:221 1.00993, 2022.
[41] Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. Thread: Circuits. Distill, 2020. https://dis-
till:pub/2020/cirruuttt.
[42] Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in neural information processing systems, 29, 2016.
[43] Stephen Casper. Yuxiao Li, Jiawei Li, Tong Bu, Kevin Zhang, Kaivalya Hariharan, and Dylan Hadfield-Menell. Red teaming deep neural networks with feature synthesis tools. arXiv preprint arXiv:2302. 10894, 2023.
[44] Dilyara Bareeva, Marina M. C. Höhne, Alexander Warnecke, Lukas Pirch, Klaus-Rober Müller, Konrad Rieck, and Kirill Bykov. Manipulating feature visualizations with gradient slingshots, 2024.
[45] Robert Geirhos, Roland S Zimmermann. Blair Bilodeau, Wieland Brendel, and Been Kim. Don't trust your eyes; on the (un) reliability of feature visualizations. arXiv preprint arXiv:2306.04719, 2023.
[46] Jonathan Marty, Eugene Belilovsky, and Michael Eickenberg. Adversarial attacks on feature visualization methods. In NewrlPS ML. Safety Workshop, 2022.
[47] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B Tenenbaum, William Freeman, and Antonio Torralba. GAN dissection: Visualizing and understanding generative adversarial networks. arXiv preprint 181i 105997 2018.
[48] Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Ja- cob Andreas. Natural language descriptions of deep visual features. In International Conference on Learning Representations, 2022.
[49] Neha Kalibhat, Shweta Bhardwaj. C Bayan Bruss, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi. Identifying interpretable subspaces in image representations. 2023.
[50] Pádraig Cunningham, Matthieu Cord, and Sarah Jane Delany. Supervised learning. In Machine
learning techniques for multimedia, pages 21-49. Springer, 2008
[51] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[52] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[53] George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11):39-41, 1995.
[54] James A Hanley and Barbara J McNeil. The meaning and use of the area under receiver operating characteristic (roc) curve. Radiology, 143(1):29-36, 1982.
[55] Corinaa Cortes and Mehryar Mohri. Confidence intervals for the area under the roc curve. Advances in neural information processing systems, 17, 2004.
13

---

[56] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 1-9, 2015.
[57] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-61 14. PMLR, 2019.
[58] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 20Xv:2010.11129, 2020.
[59] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[60] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitmick. Microsoft COCO: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision ECCV 2014, pages 740-755, Cham, 2014. Springer International Publishing.
[61] Jonathan Long. Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
[62] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303-338, June 2010.
[63] Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2017.
[64] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
[65] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), October 2019.
[66] Kirill Bykov, Klaus-Robert Müller, and Marina M-C Höhne. Mark my words; Dangers of watermarked images in imagenet. arXiv preprint arXiv:2303.00 00049, 2023.
[67] Zhiheng Li, Ivan Evtimov, Albert Gordo, Caner Hazirbas, Tal Hassner, Cristian Canton Ferrer, Chenliang Xu, and Mark Ibrahim. A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20071 2023.
[68] Reduan Achtibat, Maximilian Dreyer, Iloaa Eisenbraun, Sebastian Bosse, Thomas Wiegand, Wo- jeiech Samek, and Sebastian Lapuschkin. From attribution maps to human uuman-underssandabbe expla- nations through concept relevance propagation. Nature Machine Intelligence, 5(9):1006-1019, 2023.
[69] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via brase localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[70] Emile van Krieken, Erman Acar, and Frank van Harmelen. Analyzing differentiable fuzzy logic operators. Artificial Intelligence, 302: 2022.
[71] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A
10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
[72] Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008
14

---

[73] Ziwei Liu, Ping Luo, Xiaogang Wang. and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018):11, 2018.
[74] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[75] Fei-Fei Li, Marco Andreeto, Marc' Aurelio Ranzato, and Pietro Perona. Caltech 101, apr 2022.
[76]
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), July 2017.
15

---

A Appendix
A.1 Broader Impact
Our proposed INVERT method contributes to enhancing the transparency and safety of Deep Neu- ral Networks. By providing human understandable and interpretable explanations for neurons in black-box models, our approach offers valuable insights into their internal operations, improving understanding. Moreover, our method is able identify potentially spurious representations. An important advantage of our method is its notable reduction in computational cost compared to previous approaches. This reduction not only improves efficiency but also minimizes the harmful environmental impact associated with excessive GPU usage.
It important note that cannot make definitive claims regarding specific groups of people benefiting from or being disadvantaged by our method. The general applicability and potential implications of our approach should be explored further and with caution.
A.2 Prior work
Let's consider a function, g: D + B that represents a convolutional neuron within a model that produces activation maps of dimensions is x k along with concept c C. Both Network Dissection [1] and Compositional Explanations of Neurons [12] methods make use of the Intersection over Union (IoU) similarity metric to measure the degree of correlation between function and concept. A prerequisite for these methodologies are segmentation masks of concepts, meaning for every concept e C, there exists a corresponding function M D {0, 1) 1) which generates a binary mask for the specific concept, of the same size as the original input.
To evaluate the similarity between function g and concept c, the multi-dimennsonal outputs from g subjected to thresholding based on neuron-specd peuron-specific percentiles (i.e., values above chosen percentiles are converted to and the remaining to 0), and upscaled to match the dimensions of the original image. We can define the resulting function that produces binary masks of the same size as the input as G : D + {0, 1 The final similarity (IoU) score between g and can be computed as the Intersection over Union score between concept masks M and function G
diou(g EZED (Mo(z) G(x))
(5)
U(x))
In section 4.2, the method of Compositional Explanations of neurons was applied using a 7x7 input map for each feature. Conversely, the INVERT approach uses strategy that computes a scalar value by calculating the average of the input map.
A.3 INVERT algorithm
Given neural representation D - I,, dataset D C DD, set of atomic concepts C C, and vector C E Cd the INVERT approach seeks to identify a compositional concept ø², which is formed as logical operation the concepts, optimize AUC similarity (C),,"((C)) For this purpose, utilized an optimization process similar to that of the CompExpl methodology 12]. employing Beam search to find the optimal compositional concept.
This method requires the configuration of certain parameters, namely the predetermined formula length I E N, the beam size B E N, and additionally, the parameters a, 3. Beam search intends to iteratively combine concepts, starting with the atomic concepts (primitives) from C. At every iteration of the process, the top B best-performing compositional concepts selected, and all feasible formulas are computed with primitives (i.e. atomic concepts). Subsequently, only the top B best-performing concepts are selected, and the process continues until the formula reaches the predetermined length.
In detail, firstly, we define a set of primitives T- - set of compositional concepts that correspond to the set of concepts C and their negation. The set comprises 24 compositional concepts, with each concept corresponding to either the base concept or its negation. Next, all 2k concepts are evaluated in terms of AUC similarity with a given function, and the top best performing compositional concepts, that satisfy < (()()) < 8 are selected, leading to the formation of the set Φ" where
16

---

- B. referred to as Beam. These are the top B best-performing compositional concepts with length of 1, satisfying the requisite condition on their positive fraction in the dataset. Subsequently, the following operations are iteratively performed until the predetermined formula length T. is met:
1. Each of the B compositional concepts in the beam 4* is combined with all primitives (concepts from )) using either the AND or OR operation, thereby augmenting the formula length by 1, resulting in a total of 4Bk new formulas.
2. All newly generated formulas are evaluated based on their similarity to the representation, and the beam 4h is updated to include the top B performing formulas, which satisfy the condition a < ((p(C)) < 3.
Upon reaching the predetermined formula length L, the Beam-Search procedure concludes by identifying the compositional concept yp with the highest observed AUC.
A.4 Integrating Datasets from Different Sources
Let Dj and D2 be two separate datasets. Each of these datasets is linked to its unique set of concepts, represented as C1 and C2 respectively. By merging these datasets, we can form a consolidated dataset, symbolized as D = D2 U D22 This unified dataset will encompass a combined set of concepts, denoted as C C1 UC2
The key requirement for this integration is the mutual definition: the concepts in C1 should be defined within the dataset D2, and conversely, the concepts C2 should be defined within the dataset Dj. While this does necessitate supplementary labeling. it becomes straightforward when it is evident that the concepts from both datasets do not overlap semantically. For instance, flower concepts from the Oxford Flowers 102 [72) and faces from the CelebA [73] can be effortlessly combined. This is accomplished by designating the output of concepts within the non-native dataset as negative.
A.5 Comparing Fuzzy Logic operators
Fuzzy logic operators 770 serve as essential instruments within the domain of fuzzy logic, mathe- vague. This contrasts with conventional logic where an element strictly either belongs to a set or not; fuzzy logic allows for the degree of membership to vary from 0 to thereby allowing for partial membership.
matical construct designed for modeling and handling data that is imprecise
In this experiment, our objective was to compare different fuzzy logic operators and examine their behavior concerning the proposed AUC metric. To fulfill this aim, we employed four distinct pre- trained deep learning image classification models: AlexNet (7]. DenseNet161
63 EfficientNet
B4 [57] and ViT 16 L (58). We focused on 1000 neural representations corresponding to the ImageNet classes in the output logit (pre-SoftMax) layer for each model, for which we recognized the ground-truthh concept- - the corresponding ImageNet class. For fuzzy logic operators' testing, mapped the output of each representation to the set [0. 1] by normalizing each representation' 's output using their corresponding mean and standard deviation across the ImageNet dataset and applied Sigmoid transformation. We tested four different Fuzzy logic operators, specifically Gödel, Product, ukasiewicz, and Yager with parameter p = 2, as illustrated in Table
For performance evaluation, we generated random compositional concepts of a given length and computed the AUC similarity between fuzzy logic norms applied to functions corresponding to these concepts. For instance, given the random compositional concept 4p = c, OR Cÿ, we derive per each of the four examined methods (c.g., the Gödel operator produces a function hG = max fx fj)). These compositional representations are then evaluated in terms of AUC similarity with the compositional concept - d(hic. sp).
compositional representations
We conducted the evaluation in two modes, that is, assessing the performance of the OR (T-conorm) operator and the performance of the AND (T-norm) operator. For each mode, we assembled 1000 random compositional concepts by sampling L random concepts without replacement and calculated the AUC between compositional concepts and corresponding function. Note that for the second mode, AND (T-norm), random compositional concepts were assembled using the AND NOT operation, given the mutual exclusivity of ImageNet labels.
17

---

Performance various Fuzzy Logic approaches OR experiment
DonseNet
EmiioonNii1B4
SWAS
1.0
1.0
1.0
Godel
AUCRCO
0.8
0.0
0.0
AUS ROC
0.9
Product
0.
0.8
100
00
100
Figure 9: Average AUC similarity between random compositional OR concepts and corresponding compositional representations employing various Fuzzy logic operators (Higher is better) evaluated across four distinct models.
Performance of various Fuzzy Logic approaches in AND NOT experiment
Nextiet
Densehlet
EtfiientNedB4
WIT SWAS EZE
RDC
JUC ROC
Cukuslowcc Yager, pim
0.0
0.8
0.6
0.6
0.5
0.5
Iommun imngh
lergh
1. langth
( larghh
Figure 10: Average AUC similarity between random compositional AND NOT concepts and corresponding compositional representations employing various Fuzzy logic operators (Higher is better) evaluated across four distinet models.
Figures Bannee//Sannnnn depict the mean AUC similarity between random compositional concepts of varying lengths and the corresponding compositional representations, which were assembled using four distinct fuzzy logic operators. From these figures, becomes evident that Gödel fuzzy logic operators demonstrate the most significant robustness to the length of the formula, consistently attaining superior AUC contrast to other operators. Consequently, we can infer that Gödel's operator emerges as the optimal choice for implementing fuzzy logic operations on neural representations.
A.6 Finetuning without training
In this section, investigate whether it is feasible perform model fine-tuning without having access to the target dataset, relying solely on the explanations the latent representations and target class descriptions. In simple terms, the idea was directly the latent representation from ImageNet-trained models as classificator for a class in another dataset that has a similar meaning to the explanation of the representation.
For this purpose, we utilized four different ImageNet deep learning image classification models, specifically AlexNet (74), DenseNet161 (63), EfficientNet B4 (57), and ViT 16 L [58). all of which were pre-trained on the ImageNet dataset. The feature feature-extractor layer that precedes the final output logit layer used in all these models for experiments. We computed the AUC similarity scores for all representations each of the feature extractors in relation to ImageNet concepts.
Table 2: List of different fuzzy operators
NOT(a)
AND(a, b) (T-norm)
OR (a. b) (T-conorm)
Gödel Product ukasiewicz Yager, p =
a
min(a, b)
max b)
a b
a a b min(a + b, min((a² b
max (a b- 1,0)
max ((1 a)2 + (1 b)))
18

---

Table 3: A comparison of the accuracy achieved by the proposed finetuning method, which includes finetuning with a single representation (L=11,, and multiple representations (1.=2,5110) combined with fuzzy AND operator, against traditional and random finetuning baselines.
AlexNet
DenseNet161
EfficientNet B4
VIT L 16
Random
2.21%
2.21%
2.21%
2.21%
L=1 43.91%
62.50%
39.96%
47.12%
==2
2
42.95%
70.51%
69.23%
60.79%
L=5 40.17%
75.00%
80.88%
78.31%
== 10
30.88%
69.12%
16.11%
79.49%
Finetuned
91.67%
97.76%
94.76%
98.29%
The target dataset employed for this study was the Caltech101 dataset [75]. which comprises of 101 image classification categories. Specifically, we utilized subset of this dataset that includes 46 classes, each of which has an exact or very similar equivalent in ImageNet classes.
We aimed to create model for classifying Caltech classes by selecting suitable latent representations from the feature extractor layers of ImageNet models, and directly linking them to Caltech class logits. For each model, chose representation with the highest AUC similarity to the ImageNet concept closest to the Caltech concepts. This resulted in a subset of 46 neurons per model, each neuron having the highest AUC for an ImageNet concept similar to Caltech concept. These neurons were normalized by the ImageNet validation dataset's 's mean and standard deviation, and a Sigmoid activation function was applied constrain outputs between 0 and 1. Neuron selection was solely based on ImageNet explanations, with no Caltech 101 data utilized.
We also hypothesized that individual signals from feature extractor layers could be further enhanced by executing a continuous AND operation with other neurons that share a high AUC towards the concept. Table 3 presents the results of this procedure in terms of the accuracy achieved on the target dataset. For this task, the random accuracy stands at approximately 2%, while the conventional fine-tuning approachwhich freezes the feature extractor layer and trains linear classification layer atop the feature extractorsachieves an accuracy of up to 98.29% (last row the table). Remarkably, by simply linking the representation with the highest AUC towards the ImageNet concept from the latent layer to the CalTech101 output class logit using our approach (L = 1). we were able to attain substantial non-random accuracy, peaking at 69. 59.50% in the case of DenseNet161. Furthermore, by selecting top L neurons that have the highest AUC towards ImageNet concepts and employing Gödel AND operator between representations, we observed that this typically improved the results, with the only exception being the AlexNet model where this strategy slightly reduced the accuracy.
A.7 Comparison between IoU and AUC metries
In our supplementary experiments comparing different models, further investigate the correlation between AUC and IoU. Table 4 demonstrates the performance of our method INVERT (AUC) in comparison to NetDissect and Compexp (IoU) performed different models and layers including varying formula lengths (N). Our analysis employs ResNet18 and 1enseNet161 Py PyToocch models trained the Places365 dataset (71), accessible through the Compositional Explanations of Neurons implementatior Following their approach we apply the methods on the ADE20k subset of the Broden dataset formula lengths of 1 to 3. The IoU and AUC scores are summarized as the average and standard deviation across all neurons in each selected model layer. From these results, we can observe, that optimal explanations from AUC (INVERT) and IoU (NetDissect, CompExpl) based methods do not necessarily maximize each other objective functions.
The results in Table reevell correlation between IoU and AUC scores in non-zero IoU cases across multiple models and layers. The metrics differ in their applications and are not as strongly aligned. The correlation scores represent the average and standard deviation of the Pearson and Spearman correlation statistics. For each neuron and each available concept, correlations were calculated between the IoU and AUC scores. The "Normal" scenario corresponds the standard case, whereas the "Log" case refers to when a logarithmic transformation was applied to the loU values, with an
19

---

Table 4: Comparison of IoU and AUC performed on different models and layers including varying formula lengths (N). All models are trained on the Places365 dataset and the explanations were constructed based on the ADE20K subset of the Broden dataset. The table presents the average and standard deviation scores IoU and AUC scores across all neurons in the selected model layer.
N
INVERT
NetDissect IoU
Model - Layer
IoU
AUC
AUC
ResNet18 Layer 4
0.0062=0.0123 0,895910.069)
0.8367=0.1155 0.0581
ResNet18 Layer 3
0.0007=0.0022
0.88341000780
0.0121=0.0012
0.5549=0.1661
DenseNet161 Features
0.0016-0.0066
0.8928+000733
0.0364+000279
0.7448+0.15M4
DenseNet161 Dense Block
0.0007=0.0017
0.9014+000655
0.0150++00034
0.6877+0.
N
INVERT
CompExp
Model Layer
IoU
AUC
IoU
AUC
ResNet18 Layer
0.0021=000022
0,9972:#0.001
0.0756=0.0369
0.8310=0. 1016
ResNet18 Layer 3
0.0023=0.0042
0.9955=0.0077
0.0185=000014
0.5726=0.
DenseNet161 Features
0.0042=0.0124
0.9958110.0056 0.995
0.0455--0.0313
0.7248 1424
DenseNet161 Dense Block
0.0029=0.0059
0.0222++0.000
0.6930
INVERT
CompExp
Model Layer
ToU
AUC
IoU
AUC
ResNet18 Layer
0.0026=0.0079
0.9977+000030
0.0849+#000391
0.8184++00095
ResNer18 Layer
0.002110.0038
0.996610.0057
0.023510.0016
0.571410.1085
DenseNet161 Features
0.003510.0104
0,99671000046
0.049710.0330
0.7132
DenseNet161 Dense Block 4
0.0026-0.0054
0.99691000008
0.006110.0031
0.684610..045
Table 5: Correlation between IoU and AUC based on the score for each class per neuron. The models were pre-trained using the Places365 dataset and their performance was assessed on the ADE20k subset of the Broden dataset. The table presents the average and standard deviation of the Pearson and Spearman correlation statistics.
Normal
Log(+eps)
Model Layer
Pearson
Spearman
Pearson
Spearman
ResNet18 Layer 4
0.3429 1 0.0682
0.3623-10.094
0.4116.10.0885
0.3623
ResNet18 Layer 3
0.0911
0.2738
0.3009
122
DenseNet161 Features
1.2681=0.086
0.2787+0.104
0.3156:10.
0.2787+10.101
DenseNet161 Dense Block
0.2143=0.1039
0.2691
0.2878++0.1573
0.2691 1626
additional epsilon value of lc-4. Correlations were computed exclusively for concepts that showed non-zero loU scores. We can observe, that for non-zero loU scores there exists a small positive correlation between IoU and AUC scores.
To further comprehend the correlation of these metries we investigate the case where AUC and IoU perform differently. In Figure [T] we present a case where explanations yielding 0 loU scores are better aligned with the explanation goal. We provide evidence of loU-based explanations resulting in low neuron activation, while INVERT achieves notable activation even when loU scores are 0.
Figure 12 (a) shows qualitative example of AUC and IoU scores across all concepts of Neuron 269 from layer of ResNet18 trained on the Places365 dataset [71]. Each data point corresponds to one concept among the 1 1005 ADE20k atomic concepts sourced from the Broden dataset (76] [T]. This example illustrates the dependence between AUC and IoU, high IoU scores are correlated with high AUC scores. Figure (b) we showcase the top 4 most activating images for Neuron 269 from the ADE20K dataset to align them with the highest scoring concepts in Figure [2 (a). Comparing the set of images with the concepts exhibiting the highest AUC scores (e.g., "throne room", "apse-indoor", "fur"), we observe strong visual alignment. However, when examining the concepts with high loU scores (e.g., "nursery"", "cradle". "attic"), we find relatively low degree of visual similarity. Those results demonstrate the limitations of the IoU measure for evaluating explanations.
20

---

Distribution of activations of neuron 205 (FL 1)
20.0
17.5
15.0
12.5
Density
10.0
car
"ocean"
All datapoints
7.5
NetDissect
IoU = 0.01
5.0
AUC 0.57
INVERT
2.5
Ccccan
AUC 0.96
InU
0.0
0.1
0.2
0.3
0.4
0.5
Activations
Figure 11: Comparison between INVERT and NetDissect. The figure displays three distributions of activations: one for all datapoints in green, one for datapoints corresponding to the IoU-based explanation in orange, and one for the AUC-based explanation in blue. These distributions pertain to the average activation across activation maps of Neuron 205 in ResNet18, layer 3, pre-trained on the Places365 dataset. The activations were collected across ADE20k subset of the Broden dataset. The class labeled as "car" resulted from IoU optimization, while the class labeled as "ocean" resulted from AUC optimization. Notably, even though the "ocean" class has an IoU score 0, comprises some of the most activating images for the neuron, evidenced by the top 9 most activated images.
(a) Distribution of and AUC neuron
(b) Top 4 activating images for neuron 269
hursery
cradle
0.025
0.020
(c) Distribution of max IoU and max LUC,
0.015
throne room
AUC max
0.010
apse-indoor
Denstt
80
Random
0.005
fur
20
AUC max
0.000
0.00
0.02 0.04
0.38
0.10
loU
AUC
Figure 12: (a) we compare the distribution of AUC and IoU across all concepts of the ADE20k atomic concepts from the Broden dataset for Neuron 269 from layer 4 of ResNet18 trained on the Places365 dataset, where (b) shows the top activating images of the ADE20K dataset. (c) shows distribution of maximized loU, maximized AUC, and random IoU scores for layer 4 of ResNet18 trained on the Places365 dataset with a formula length 1.
21

---

Table 6: Metric comparison for true label and random explanation evaluation. FasterRCNN ResNet50
FPN (pre-trained on MS COCO for object detection). UPerNet BEiT-B (pre-trained on ADE20k for semantic segmentation).
FasterRCNN ResNet50 FPN
UPerNet BEiT-B
Metric
True
Random
True
Random
IoU
0.8553#0.0913 0.0077=0.0046 0.8455=0.0466
0.0007=0.0007
AUC
0.9556:10.0371
0.5005-10.0253
0.8738+10.0929
0.5001
Furthermore, we conducted a quantitative evaluation shown in Figure layer 4 of the ResNet18 8 trained on the Places365 dataset. We compared the distribution of IoU scores of explanations obtained by maximizing IoU and AUC respectively. Additionally, examined the mean values of these distributions, which included random IoU scores as baseline reference. Our findings reveal that maximizing IoU leads relatively sparse distribution IoU scores while maximizing AUC results in a more densely concentrated accumulation of predominantly low IoU scores. As anticipated, the performance of random loU scores was notably poor. We can observe that maximizing AUC also indirectly maximizes IoU.
(c), specifically focusing on
Table 6 serves as sanity check implementing metric comparison for best explanation and random explanation evaluation. The FasterRCNN ResNer50FPN model was pretrained on the MS COCO dataset for object detection, while the UPerNet BEiT-B model was pretrained segmentation. The former model's 's evaluation was conducted on subset of MS COCO containing 20,000 images, while the latter was assessed the ADE20k subset the Broden dataset. The output layers of both models were utilized to access the "ground truth" label for each neuron. In the table, the "True" column represents the loU/AUC scores of the explanation that align with the ground-truth neuron label. On the other hand, the "Random" column corresponds to the scores of randomly chosen oollanttion-conceyy pairs that differ from the "ground truth".
ADE20K for semantic
22

---

A.8 Figures
WordNet taxonomy graph
passenger van-
minivan
van
moving
police
fire engine
truck
garbage truck
pickup
tow truck
trailer truck
sttorcycle-minbbikee
moped
ambulance
beach wagon
cab
convertible
jeep
car
Limousine
motor vehicle
Model T
minivan
sports car
amphibian kart
golfcart
snowplow
passenger train
train
bullet train
public transport
minibus school bus Leybus
bus
Figure 13: The figure displays the WordNet taxonomy, which used gather the hierarchical structure of the labels for the Figure 3
23

---

Neuron 192
0.3637
NOT
"swimming trunks"
AND
photocopier
OR
"swimsuit.
Neuron 452
0.3220
"residecc" OR place of worship"
AND NOT "stupa"
Neuron 154
771, safe n04125021
0.2897
chinese text
OR
"viper")
AND NOT "ddaamndbbcck"
Input 771, safe nO4125021
Explanation: Output logit layer Neuron 771, "sfe
Explanation: AugPool layer Neuron 192
Explanation: Augpooi layer Neuron 452
Explanation: Augpool layer Neuron 154
Craport AND NOT Telternthhrkk
Figure 14: The figure illustrates the "safe" circuit within the ResNet18 model. The top part of the figure showcases the three most significant neurons (in terms of the weight of linear connection) and their corresponding INVERT explanation linked to the class logit "safe". The bottom part of the figure demonstrates how the local explanation from the class logit can be decomposed into individual explanations of individual neurons from the preceding layer. This allows for a more detailed understanding of how each neuron contributes to the final classification.
24

---

Neuron 160
personal, compute="
NOT "hhandhheld computer" wweb site"
0.5570
Neuron 96
0.2828
car misrmmirrrr " OR "personal, computer"
AND NOT "notebook"
Neuron 368
0.2615
monitor n037820006
enterraiiment
inteerainment_cenner"" center" OR "ortable_computer""
computer"
NOT "hand-held computer")
Neuron 154
0.2153
chinese text
OR
Viper"
AND NOT "diamondback"
Inout 664, monitor n03782006
Explanation: Output-loge layer Neuron 664, "mnnitrr"
Explanation: AugPool layer
Explanation: Augpoal layer Neuron 96
Explanation: Avgpool
Neuron 160
Neuron 154
Opecional computec
Cosb micccop
Belinese toXE (yioper))
AND NOT Thand-he wtppe___oooe (ccomuuter))
OR pecaenal Coe pubse
Figure 15: The figure illustrates the "monitor" circuit within the ResNet18 model. The top part of the figure showcases the four most significant neurons (in terms of the weight of linear connection) and their corresponding INVERT explanation linked to the class logit "monitor" The bottom part of the figure demonstrates how the local explanation from the class logit can be decomposed into individual explanations of individual neurons from the preceding layer.
computer"
"aggital
AND
Poaccronte equipnent"
computer ZO0M
Le' evision
AND
"housing"
"tv_roon"
Neuron 511
dtottal
min(fon,f40)
YEAlPu
min(fu,.ful
AND
"sluctionic eouspnerr" " AUROC: 0.857
bec AND "housing"
Neuron 407
Neuron 213
AUROC: 0.838
"housstng" AUROC: 0.73
aipatomant
AUROC:
Figure 16:
The figure presents two distinct handcrafted circuits. For each neuron, or combination of neurons, we report the Area Under the Receiver Operating Characteristic (AUROC) score. This score represents the AUC classification performance towards classifying specific concepts from the Places365 dataset.
25


Reviews:
[{"review_id": "AnbCtDIvbv", "reviewer": "Reviewer_6wCX", "summary": "This paper addresses the limitations of existing global explanation methods for Deep Neural Networks (DNNs) and proposes a new approach called Inverse Recognition (INVERT). Unlike previous methods, INVERT does not rely on segmentation masks, provides scalable interpretability, and enables statistical significance testing. The paper demonstrates the applicability of INVERT in various scenarios, including interpreting representations affected by spurious correlations and revealing the hierarchical decision structure within DNN models.", "questions": "- What is the reason for selecting very specific neurons? Were they found empirically, is this based on previous work, or is there some other reason? \n- How applicable is this method to modern ViT-based architectures, such as those used in CLIP? \n- Is it possible to have an \"open vocabulary\" for the concepts? One major shortcoming I see is that we have to use a fixed set of concepts to explain representations (although we can combine them to some extent using logical operators, as discussed in the paper)", "limitations": "not explicitly discussed in the main paper.\n", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "First of all - I'm not an expert in this field, so take everything with a grain of salt. That said, I like this approach for its simplicity. Also, I particularly like the experiments on detrimental representations (section 5.1) and on \"finetuning without training\" (section 5.2), which show that the proposed method has practical implications.", "weaknesses": "See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 6wCX,\n\nWe extend our sincere appreciation for the dedicated time and thorough evaluation of our manuscript. We deeply value the recognition of the simplicity of our approach and the validation of our experiments' practical applicability.\n\nWe will proceed to address the questions that have been raised.\n\n*Question: What is the reason for selecting very specific neurons? Were they found empirically, is this based on previous work, or is there some other reason?*\n\n**Answer:** In our study, we have showcased the versatility of INVERT across various models. While specific neurons such as neuron 154 in Figure 6 were selected due to prior research pointing to its susceptibility in recognizing Chinese watermarks, the neurons chosen for the circuits, both in the original paper and the attached PDF, were not deliberately selected. These examples are meant to illustrate INVERT's performance in authentic scenarios. Our overarching objective is to underline the universal applicability and adaptability of our method across diverse models and neurons.\n\n*Question: How applicable is this method to modern ViT-based architectures, such as those used in CLIP?*\n\n**Answer:** The presented INVERT methodology is universally applicable to any neural representations (neurons) producing scalar outputs, including ViT-based CLIP, or just transformer models. In general our experiments demonstrate successfully the application of INVERT to ViT-based architectures within section 5.2. Additionally, the attached PDF showcases an expanded set of examples rooted in ViT architecture, providing further evidence of our method's application scope.\n\n*Question: Is it possible to have an \"open vocabulary\" for the concepts?*\n\n**Answer:** So far the dependency on data remains a central constraint for methodologies aiming to explain learned abstractions of neural networks and connecting neurons to comprehensible human concepts. Methods like Network Dissection and Compositional Explanation of Neurons necessitate images with object masks. INVERT mitigates this dependency by requiring image labels only. This not only makes the data collection easier but also accelerates computational processes. As detailed in section 5.1 of our paper, the set of concepts can be expanded by merging datasets from diverse sources to encapsulate a wider array of concepts. To entirely liberate the approach from data dependency, a critical, presently undiscovered step akin to the transition from supervised to unsupervised learning is requisite—a challenge we aim to tackle in forthcoming research.\n\nFurthermore, we have augmented our global response with new qualitative illustrations and comprehensive quantitative evaluation results. We thank Reviewer 6wCX very much for their valuable feedback, which greatly improved the quality of our work and hope that these additions will persuade Reviewer 6wCX to reconsider and possibly improve the evaluation of our work.\n"}, {"review_id": "wuItramsZO", "reviewer": "Reviewer_UsfE", "summary": "The paper introduces a method to understand what semantic concepts different neurons of a deep network are looking at. They do so by comparing the similarity between different concepts (taken from a pre-defined concept bank) and neuron representations using the AUROC metric, and assign the concept with the highest AUROC (to the neuron under study). The authors further illustrate how their uncovered concepts can be used to handcraft circuits for recognizing different classes from another dataset and detect spurious correlations.", "questions": "1. **Incorrect Prior Work/Missing References:** While describing MILAN in line 73 \"While alleviating the limitation posed by the necessity of having a labeled dataset\". This statement isn't correct since MILAN does require a labelled dataset to train their captioning model (the MILANAnnotations dataset). I believe the authors meant to say human annotated segmentation masks are not required? The authors should also cite CLIP-Dissect which was published ICLR 2023, which did away with the need for labelled datasets by appealing to the zero-shot abilities of CLIP. \n\n2. **missing details in figures:** The captions are unclear and sometimes not referenced in the main text so their interpretation is not clear. For instance, what is the density being plotted in figure 2a? Is it the density of neuron activations for images labelled positive for the concept and images labelled as negative for the concept? \n\n3. **More rigorous comparison to prior work*** From figure 4a, it is not clear higher AUC is correlated with higher IOU, if anything it seems at IoU 0, AUC can be anything between 0 - 1. I would suggest the authors do a more rigorous study to substantiate their claim perhaps by computing a correlation coefficient. Moreover, figure 4b, is confusing it plots the distribution of iou for max iou? I understand the authors are trying to show that the explanation that maximizes the ioU is not necessarily concentrated and can be more spread out vs the distribution for max AUC is concentrated. However, the motivation for this experiment is not clear, nor is it clear from the discussion why the reported result is good for the AUC metric vs the IoU metric. Finally, I appreciate the comparison with compexp in terms of computational time, no reasoning is provided for it, merely the times are reported. Some more discussion as to why is compexp slow? what is the bottleneck that this work gets rid off would help appreciate the results. This is particularly important since the authors use the same algorithm compexp uses to compose concept literals into boolean formulas to form explanations. \n\n4. **missing details in fine-tuning experiment:** In 5.2 what are the Caltech concepts? The imagines concepts were taken from wordnet, is the same Wordnet employed for the Caltech concepts. This should be clearly mentioned in the Appendix for reproducibility. The authors remark \"By simply linking the most suitable representations from the latent layer to the output class logit using our approach, we were able to attain a substantial non-random accuracy\". Could you please specify what most suitable representations mean here? how were they linked? Was a linear layer trained on top of the most suitable representations? Without these details it is hard to appreciate the merit for this experiment. \n\n5 **more qualitative examples:** The paper makes several claims such as can detect spurious correlation, AUROC better reflects visual similarity than IoU and neuron explanations can be used to handcraft small circuits for classification on target datasets without any training (which I think is a great use-case!). However, only one or two examples are provided for each case. This raises concerns for cherry picking, the authors should provide many more examples in the supplement for readers to further substantiate the claims. Moreover, if possible the authors should also do quantitative experiments to validate their claims, perhaps a mechanical turk study for the statement \"AUROC better reflects visual similarity that IoU\".\n\n6 **statistical significance of explanations**: The authors claim an advantage of AUROC over IoU is that their metric provides a measure of statistical significance. I think this is an interesting point but I am not an expert on hypothesis testing and the paper does not mention how the AUROC is a measure of statistical significance to appreciate this point. A paragraph dedicated to this would strengthen the paper in my opinion. \n\n**Minor Points**\nI think it should be made clear in the main text (as opposed to the supplementary) that for CNN filters, the authors treat the output as a scalar by averaging the feature map as opposed to thresholding the feature map to get highest activating regions (as in prior work). This is an important point and should be bought up before doing the comparison with previous work in the main text. ", "limitations": "N/A", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "I greatly appreciate the AUROC metric the authors presented. It extends current approaches to neuron interpretation based on segmentation maps and IOU metrics which are necessarily limited to convolution filters. ", "weaknesses": "Unfortunately, I think the paper is poorly written with key details missing, prior work not appropriately mentioned and experimental validation of method lacking. Due to this I recommend rejection of this manuscript in its current form. Below I describe the weaknesses in detail in the questions section. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer UsfE,\nWe extend our sincere gratitude for the dedicated time and the insightful comments you have shared with us. Your thorough review has proven invaluable in refining our work, and we deeply appreciate your efforts.\n\nIn response to the questions and concerns you raised, we would like to provide the following clarifications:\n\n*Q1: Incorrect Prior Work/Missing References*\n\n**Answer**\nWe acknowledge your observation and have made the necessary amendments. The incorrect reference to the MILAN method has been rectified, and we have included the reference to the CLIP-Dissect paper in the updated version of our manuscript.\n\n*Q2: Missing Details in Figures*\n\n**Answer**\nThank you for pointing out this aspect. We have taken your feedback into account and have augmented the figures with additional clarifications. In Figure 2a, orange signifies activation of data points within the explanation, while blue represents data points that do not belong the explanation.\n\n*Q3: More Rigorous Comparison to Prior Work*\n\n**Answer**\nYour concern is well-received, and we have undertaken additional quantitative experiments, as detailed in our comprehensive global response to all reviewers. We acknowledge the intricacies of comparing our approach to prior methods, especially considering the absence of a standardized baseline. Our rationale for favoring the AUC approach over IoU-based methods is extensively explained in the global response. We have observed that, in certain cases, explanations with 0 IoU scores correspond to classes that systematically maximize neuron activation. For explanations with non-zero IoU, we have demonstrated a significant positive correlation with AUC scores. The AUC approach, offering advantages in terms of speed, dataset independence, and wider applicability, emerges as the more suitable choice, in our opinion.\n\n*Q3.2: Reason for Faster Computation*\n\n**Answer**\nIndeed, the optimization algorithm was inspired by the CompExp method. In comparison, INVERT's efficiency stems from operating on binary vectors (0 for data points outside the explanation, 1 for those within), whereas CompExpl conducts logical operations on high-dimensional binarized masks, incurring greater computational and memory costs.\n\n*Q4: Missing Details in Fine-tuning Experiment*\n\n**Answer**\nWe acknowledge the validity of this comment and have made efforts to provide a more comprehensive explanation of this experiment in the updated version of our paper. Notably, the CalTech classes exhibit differences from those found within the ImageNet dataset. In our examination, we pinpointed 46 classes that share identical names across both datasets. Our approach involved a search for the latent layer representation within the ImageNet-pre-trained model that exhibited the highest AUC in relation to the ImageNet counterpart of the CalTech class. To illustrate, if we consider the CalTech “barn” class, we identified, within the latent layer of the model, a neuron with the most substantial AUC towards the ImageNet “barn” class. We then used the output of this representation as a logit-prediction for the CalTech “barn” class. Our findings underscore the efficacy of this relatively straightforward methodology, yielding commendable performance levels on the CalTech dataset.\n\n\n*Q5: More Qualitative Examples*\n\n**Answer**\nYour suggestion resonated with us, and we have not only provided additional quantitative results and explored emerging circuits but also augmented our findings with local explainability methods. Moreover, we have introduced extra examples showcasing \"handcrafted\" circuits. All of these enrichments are detailed in the attached PDF.\n\n*Q6: Statistical Significance*\n\n**Answer**\nYour observation regarding statistical significance is appreciated. We concur that IoU scores can be misleading, lacking in the ability to distinguish random occurrences from systematic patterns. The AUC measure offered by INVERT not only delivers interpretable measure (AUC) but also accompanies it with a p-value for a statistical test (i.e. testing hypothesis “AUC of given explanation = 0.5”), determining the dissimilarity between distributions of activations for explanations and non-explanations. This discussion is further elaborated in our global response.\n\n*Q7: Minor Points*\nWe have taken heed of your feedback on minor points and have promptly rectified them in the updated version of our manuscript.\n\nWe would like to express our sincere gratitude for your insightful questions and comments. We are hopeful that the evidence we have provided will positively contribute to the re-evaluation of our work. Your thorough review has been invaluable in shaping the enhancement of our manuscript\n"}, {"review_id": "EJZvLP3gqu", "reviewer": "Reviewer_bbMD", "summary": "This paper introduces INVERSE, a scalable approach called Inverse Recognition, that links learned representations to human-interpretable concepts by leveraging the ability to differentiate between concepts The applicability of INVERSE is demonstrated in diverse scenarios, including identifying representations influenced by spurious correlations and interpreting the hierarchical decision-making structure within the models.", "questions": "1. What is the motivation ot intuiation using the logocal forms, such as AND, OR, and NOT? \n2. Do you have some ideas or suggestions dealing with the tradeoff between simplicity and precision? ", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces INVERT, a novel approach called Inverse Recognition, which enables the labeling of neural representations with human-interpretable concepts in a scalable and informative manner.\n2. The authors make significant contributions by using INVERT to gain insights into the hierarchical decision-making structure within models, enhancing our understanding of their inner workings. They also propose an interpretable metric to assess the alignment between representations and explanations, providing a means to evaluate explanation quality.\n3. In addition to the aforementioned contributions, the authors demonstrate the practical applications of INVERT, further highlighting its usefulness and credibility.\n4. The paper exhibits strong writing quality with a well-structured and logical flow. The equations are clearly presented, and the figures and tables are easily comprehensible, effectively conveying the authors' ideas.\n\n", "weaknesses": "1. While the paper showcases the practicality of INVERT in diverse scenarios, the evaluation is confined to a limited number of examples. Conducting a broader evaluation encompassing a wider range of models and datasets would be advantageous.\n2. The paper offers a broad overview of the INVERT methodology, but certain aspects lack clarity. Providing a more detailed explanation of the methodology would enhance readers' understanding of its workings. Specifically, Section C in the Appendix could be better placed within Section 3 for improved organization and coherence.\n3. One aspect that should be noted is that in INVERT, the concepts to be linked with the learned representations still need to be pre-defined or manually selected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our gratitude to Reviewer bbMD for their insightful and constructive feedback. We greatly appreciate the time and effort invested in reviewing our work. We are heartened by the positive acknowledgment of the significance of our contribution, as well as the recognition of the quality of our paper's content and visual aids.\n\nHerein, we address the noted weakness and questions raised by the reviewer:\n\n*While the paper showcases the practicality of INVERT in diverse scenarios, the evaluation is confined to a limited number of examples. Conducting a broader evaluation encompassing a wider range of models and datasets would be advantageous.*\n\n**Answer**\nWe acknowledge the reviewer's concern and, in response, we have taken substantial steps to address it. The revised version of our manuscript includes an expansion of our quantitative experiments. We have investigated the correlation between AUC and IoU measures for explanations across different model and layers, an analysis that contributes to a more comprehensive understanding of our approach. Moreover, we have investigated the behavior of random explanations, shedding light on an aspect critical for establishing the efficacy of our method. These updates are described in detail in a global rebuttal.\n\n*The paper offers a broad overview of the INVERT methodology, but certain aspects lack clarity. Providing a more detailed explanation of the methodology would enhance readers' understanding of its workings. Specifically, Section C in the Appendix could be better placed within Section 3 for improved organization and coherence.*\n\n**Answer**\nWe thank the reviewer for raising this point. In the revised manuscript, we have shortened several paragraphs and allocated space for a more thorough discussion of the algorithm within Section 3, which greatly improve theclarity and coherence in the presentation of our methodology.\n\n*One aspect that should be noted is that in INVERT, the concepts to be linked with the learned representations still need to be pre-defined or manually selected.*\n\n**Answer**\nThe issue of data-dependency is a shared characteristic among methods aimed at explaining the concepts learned by neural representations, including Network Dissection and Compositional Explanation of Neurons. While INVERT does indeed necessitate predefined concepts, it distinguishes itself by mitigating the reliance on masked data, relying instead on image labels. This shift enhances the feasibility of accessing a more extensive spectrum of concepts, considering that labeled data is more easily accessible than masked image data. Moreover, this approach accelerates computational processes. Section 5.1 of our paper delves into this matter, illustrating how the set of concepts can be broadened by merging datasets from diverse sources, effectively encompassing a wider array of concepts. We consider the research towards data-free explanation method to be an avenue for the future work.\n\n*Question: What is the motivation ot intuiation using the logocal forms, such as AND, OR, and NOT?*\n\n**Answer**\nMain inspiration for the logical forms approach comes from the paper “Compositional Explanation of Neurons”, where such approach was introduced. Such method allows to enrich the basic, atomic concepts with interpretale combinations, allowing for the more versatile explanations. Although alternate (from logical) forms of function $\\varphi$ (as defined in Definition 4) are feasible, we opted for logical forms due to their established nature and interpretability. Our framework, however, is versatile enough to accommodate other $\\varphi$ forms, which we plan to explore in future work.\n\n*Question: Do you have some ideas or suggestions dealing with the tradeoff between simplicity and precision?*\n\n**Answer**\nResponse: The trade-off between simplicity and precision is a captivating challenge in the realm of explainable AI. Within the context of INVERT, we believe enhancing precision involves incorporating more detailed and precise concepts into the dataset. By introducing more concepts, the general behaviour of the neurons could be explained more precise, while still being comprehensible for a human. This, for example, could be achieved by aggregating diverse concepts from various sources, as outlined in Section 5.1.\n\nOnce again, we express our gratitude for your insightful feedback, which significantly improved  the depth and impact of our work. We hope that the improvement of our work will contribute to a more favorable assessment of our work.\n"}, {"review_id": "TxWox7a6cD", "reviewer": "Reviewer_LJJZ", "summary": "The paper proposes a new explainability method, named INVERT, that matches the learned representations with human concepts. Specifically, it tries to provide explanations to individual neurons by matching human concepts that the neuron predominantly detects. Compared to previous approaches, INVERT is computationally more efficient, does not depend on the annotation of segmentation masks, and is generalizable to different types of neurons. The paper shows INVERTs applicability in identifying neurons affected by spurious correlations and fine-tuning representations without explicitly training them.", "questions": "* Does the conclusions of Figure 4(c), that maximizing IoU leads to a relatively sparse distribution of IoU scores while maximizing AUC results in a more densely concentrated accumulation of low IoU scores, generalizes to other lengths as well (eg L=2 or L=3)? For lack of space, one suggestions would be add such results to appendix.\n* In line 218, the conclusion of “high IoU scores are correlated with high AUC scores” from the Figure 4(a) is not clear to me since several points that have high AUC (x axis) also have low IoU (y axis). Could the authors use some form of correlation measure to produce a single score to better verify this statement?\n* Section 5.2 focuses on “creating” a model trained on ImageNet to classify Caltech images without any training procedure. However, this seems limited to the setting where classes have some form of overlap (which is the case in this example, as also mentioned in the text), or when you can overlap them by combining different concepts (described at the last paragraph of this section). Instead of relying on this assumption, could INVERT be used to somehow demonstrate how well a set of representations will be transferred to another dataset without actually training the model (ie as an analysis tool, not as a model construction itself as it was done)? If yes, then this could be a powerful application of the method, since it could be used to analyze whether the costs of fine-tuning a model will be paid off or not. This could be shown by comparing two models, one that fine tunes well and another that doesn’t, and how this method can be used to predict that beforehand.", "limitations": "A section or a couple of sentences in the conclusion could be added to broadly discuss some of the limitations of the proposed method.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The method is generalizable in terms of architecture, ie not restricted to convolutional neurons as previous ones. The method is more efficient by requiring less running time and computational resources. The method does not require additional annotations, such as segmentation masks. The method is simple and easy to follow. From Section 4.2, INVERT seems to provide outputs more visually similar to human concepts.", "weaknesses": "My main concern about this work is the lack of evidence that this method generalizes to other configurations.\nFor instance, in the related work, one of the limitations that was described from previous work is the fact that they work only on convolutional neurons. However, most of the experiments are limited to ResNet-18 (except Section 5.2 which a ViT was used), which is mainly based convolutional layers. Since this is one of the important contributions of the method, it would be important to see how the results generalize to other types of neurons. Another suggestion is to expand the results shown in Figures 4 and 6 (for other neurons and architectures) and place them in the appendix.\n\nMinor writing feedback:\n1. Line 35, in “(1)” should add the word “Figure”, eg “(Figure 1)”.\n2. Missing parenthesis on line 142 to close the min equation\n3. Figure 2 (in page 5) is never cited in the paper\n4. Would encourage the authors to refer to specific sections of the appendix when referring to it in the main text\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer LJJZ for the time spend reviewing our work and we are thankful for the detailed comments. In following, we will answer to the described shortcomings of our work and answer the raised questions.\n\n*The Lack of evidence that this method generalizes to other configurations.*\n\n**Answer:**\n We thank the reviewer for raising the following a concern. Indeed, previous methodologies, such as Network Dissection and Compositional Explanation of Neurons, were designed to explain convolutional neurons through the examination of intersections between neuron activation maps and object masks. INVERT overcomes such limitations by analysing and explaining scalar functions. It is important to emphasize that our ResNet18 experiments, as detailed in the original paper, were conducted on the AveragePooling layer—a configuration that produces scalar activations (512 neurons x 1 activation). This already demonstrates a difference from prior IoU-based techniques designed for convolutional neurons, which generate high-dimensional activation maps.\nIt's worth noting that INVERT exhibits versatility, extending its applicability to transformer-based architectures, including Visual transformer models (ViT). This was aptly demonstrated in Section 5.2 of our original paper. Our efforts to bolster this aspect continue with the inclusion of additional qualitative examples in the updated version. Many of these new visualizations can be found in the PDF file accompanying this global rebuttal. Additionally, the updated version features a showcase of \"handcrafted\" circuits founded on ViT representations, further solidifying our method's compatibility and efficacy.\n\n*Question: Does the conclusions of Figure 4(c), that maximizing IoU leads to a relatively sparse distribution of IoU scores while maximizing AUC results in a more densely concentrated accumulation of low IoU scores, generalizes to other lengths as well (eg L=2 or L=3)? For lack of space, one suggestions would be add such results to appendix.*\n\n**Answer**\nWe have indeed performed the quantitative comparison between IoU-based and AUC-based explanations, focusing on the same neurons but with varying formula lengths. Although these specific results are not included in the attached PDF, they have been incorporated into the appendix of our paper. While there does exist a general correlation between IoU and AUC, as highlighted in Table 1 of the PDF, our investigations revealed that often optimal IoU explanations did not align with high AUC scores, and vice versa.\nOur global rebuttal, with particular emphasis on Figure 1, elucidates a compelling case where an AUC-based explanation yields 0 IoU, while the best IoU-based explanation demonstrates a low AUC score. We argue that due to the interpretability inherent in the AUC metric, the reduced computational complexity, the capacity to ascertain random explanations through statistical testing, and the broader applicability, INVERT emerges as a robust and preferable alternative to IoU-based methods.\n\n*Question:In line 218, the conclusion of “high IoU scores are correlated with high AUC scores” from the Figure 4(a) is not clear to me since several points that have high AUC (x axis) also have low IoU (y axis). Could the authors use some form of correlation measure to produce a single score to better verify this statement?*\n\n**Answer:**\n We addressed this concern within our global rebuttal. To elaborate, we have conducted a quantitative evaluation that examines the correlations between IoU and AUC scores. As a result, we have relaxed our assertion in the updated version of our paper. Our analysis demosntrates a positive correlation between IoU and AUC scores. However, it is important to highlight that in instances where both IoU and AUC scores are at their highest, often metrics disagree, as illustrated in Figure 1 of the attached PDF, accompanying our global rebuttal.\n\n\n*Question: Empolying INVERT to demonstrate how well a set of representations will be transferred to another dataset.*\n\n**Answer:**\n We find such application  very interesting, but we believe that it lies outside of the scope of the proposed approach.  INVERT allows to connect neurons with human-understandable concepts – however, given, for example, explanation “farm” based on ImageNet data, it is hard to quantify how good such neuron would detect “farm” classes from another dataset, without the access or evaluation on the “farm” images from another dataset. The assessment of the scope of global explanations, including the assessment of the ability of neuron to generalise to other datasets based on its explanation might be a promising avenue for the future ressearh.\n\nMinor mistakes\nWe agree with these mistakes and we fixed them in the updated version of our paper.\n\nWe extend our sincere gratitude to the reviewer for their thoughtful and comprehensive feedback. The insights you have provided have proven invaluable in refining our work. We firmly believe that the combined impact of our global rebuttal, fortified by new quantitative and qualitative findings, will positively influence the re-evaluation of our paper.\n"}]

Author rebuttal:
We extend our deepest gratitude for your invaluable dedication and expertise in assessing our work. The thoughtful feedback of all the reviewers has been instrumental in refining our research and elevating its quality. The positive reception of our work has been immensely gratifying, and with this global rebuttal, coupled with the attached PDF, we aim to address your questions and provide a comprehensive overview of our responses. Additionally, we address all the reviewers individually.

**Evaluation**

To address concerns about additional evaluations, we conducted an extensive quantitative assessment. We acknowledge the complexities of comparing our method to existing approaches due to the lack of a standardized baseline for the evaluation of global explanation methods. Our motivation was to demonstrate the limitations of IoU-based explanations. In Figure 1 of the attached PDF, we present a case where explanation yielding 0 IoU score is better aligned with the explanation goal. We provide evidence of IoU-based explanations resulting in low neuron activation, while INVERT achieves notable activation even when IoU scores are 0. Furthermore, we highlight a correlation between IoU and AUC scores in non-zero IoU cases across multiple models and layers (Table 1). This, coupled with INVERT's computational efficiency, lack of dependency on masked datasets, interpretable scoring measure (AUROC), and wider applicability, positions INVERT as a robust alternative to IoU-based methods.

**Additional Qualitative Experiments**

In the attached PDF, we have introduced new qualitative experiments to strengthen our claims and demonstrate the practicality of our proposed method. We enhanced existing figures with GradCam explanations (Figures 2 and 3). Additionally, we introduced a visualization for a new circuit (Figure 4) and included new "handcrafted" circuits for the ViT model (Figure 5), supplementing the example from the original paper (Figure 7).

**Statistical Significance**

IoU-based explanations often suffer from reporting small IoU scores for highest-IoU explanations, raising concerns about random coincidences (for example, Figure 1 in the attached PDF). Notably, there exists no statistical measure to test the hypothesis $H_0:$ IoU$ = 0.$ INVERT's AUC-based method naturally connects to the Mann-Whitney-Wilcoxon test statistic, providing a means to test the hypothesis "AUC of a given explanation = 0.5." Table 2 furnishes a sanity check for INVERT, demonstrating that, with random explanations, INVERT yields AUCs $\approx 0.5$.

After diligently addressing the valuable feedback provided by the reviewers, we have implemented a series of updates to our paper. These revisions encompass new figures and experiments, as well as the rectification of minor errors. With these clarifications and evidence, we trust that we have addressed potential queries regarding our work. We remain optimistic that this comprehensive information will positively influence the re-evaluation of our paper.


Meta-review: The authors present an approach for interpreting neural representations by mapping them onto human-understandable concepts.
After considering the rebuttal and internal discussions, three reviews are marginally positive and one recommends rejection. While there was no particular excitement about this work, there are also no critical conceptual flaws. On the negative side are mostly experimental and presentation issues: exploring more powerful network architectures, broader experimental evaluation, the study so far being limited to only selected neurons, and the discussion of related work. The authors have put tremendous effort into the rebuttal and showed how to address most critical issues already. Therefore, there appears to be a clear path from here to a camera-ready version that addresses all reviewer concerns.
-> accept. Congratulations!
It is imperative that the final version implements all that has been shown and promised in the rebuttal and addresses all reviewer comments.
Decision: Accept (poster)