SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kirill Bykov*
UMI Lab
ATB Potsdam
kbykov@atb-potsdam.de
Laura Kopf
lkopf@atb-potsdam.de
Shinichi Nakajima
TU Berlin
nakajima@tu-berlin.de
Marius Kloft
RPTU Kaiserslautern-Landau
kloft@cs.uni-kl.de
Marina M.-C. Höhne
University of Potsdam, Germany
mhoehne@atb-potsdam.de
Deep Neural Networks (DNNs) demonstrate remarkable capabilities in learning complex hierarchical data representations, but the nature of these representations remains largely unknown. Existing global explainability methods, such as Network Dissection, face limitations such as reliance on segmentation masks, lack of statistical significance testing, and high computational demands. We propose Inverse Recognition (INVERT), a scalable approach for connecting learned representations with human-understandable concepts by leveraging their capacity to discriminate between these concepts. In contrast to prior work, INVERT is capable of handling diverse types of neurons, exhibits less computational complexity, and does not rely on the availability of segmentation masks. Moreover, INVERT provides an interpretable metric assessing the alignment between the representation and its corresponding explanation and delivering a measure of statistical significance. We demonstrate the applicability of INVERT in various scenarios, including the identification of representations affected by spurious correlations, and the interpretation of the hierarchical structure of decision-making within the models.
Deep Neural Networks (DNNs) have demonstrated exceptional performance across a broad spectrum of domains due to their ability to learn complex, high-dimensional representations from vast volumes of data [1]. Nevertheless, despite these impressive accomplishments, our comprehension of the concepts encoded within these representations remains limited. The "black-box" nature of representations, combined with the known susceptibility of networks to learn spurious correlations [2, 3, 4], biases [5] and harmful stereotypes [6] poses significant risks for the application of DNN systems, particularly in safety-critical domains [7].
To tackle the problem of the inherent opacity of DNNs, the field of Explainable AI (XAI) has emerged [8, 9, 10]. The global explanation methods aim to explain the concepts and abstractions learned within the DNNs representations. This is often achieved by establishing associations between neurons and human-understandable concepts [11, 12, 13, 14], or by visualizing the stimuli responsible
for provoking high neural activation levels [15, 16, 17, 18]. Such methods demonstrated themselves to be capable of detecting the malicious behavior and identifying the specific neurons responsible [19, 20].
In this work, we introduce the Inverse Recognition (INVERT)  $^2$  method for labeling neural representations within DNNs. Given a specific neuron, INVERT provides an explanation of the function of the neuron in the form of a composition of concepts, selected based on the ability of the neuron to detect data points within the compositional class. Unlike previous methods, the proposed approach does not rely on segmentation masks and only necessitates labeled data, is not constrained by the specific type of neurons, and demands fewer computational resources. Furthermore, INVERT offers a statistical significance test to confirm that the provided explanation is not merely a random occurrence. We evaluate the performance of the proposed approach across various datasets and models, and illustrate its practical use through multiple examples.
Post-hoc interpretability, a subfield within Explainable AI, focuses on explaining the decision-making strategies of Deep Neural Networks (DNNs) without interfering with the original training process [21, 22]. Within the realm of post-hoc methods, a fundamental categorization arises concerning the scope of explanations they provide. Local explanation methods aim to explain the decision-making process for individual data points, often presented in the form of attribution maps [23, 24, 25]. On the other hand, global explanation methods aim to explain the prediction strategy learned by the machine across the population and investigate the purpose of its individual components [26, 27].
Inspired by principles from neuroscience [28, 29, 30], global explainability directs attention towards the in-depth examination of individual model components and their functional purpose [31]. Often, global explainability is referred to as mechanistic interpretability, particularly in the context of Natural Language Processing (NLP) [32, 33, 34, 35]. Global approach to interpretability allows for the exploration of concepts learned by the model [36, 37, 38, 39] and explanation of circuits — computational subgraphs within the model that learn the transformation of various features [40, 41]. Various methods were proposed to interpret the learned features, including Activation-Maximisation (AM) methods [15]. These methods aim to explain what individual neurons or groups of neurons have learned by visualizing inputs that elicit strong activation responses. Such input signals can either be found in an existing dataset [16] or generated synthetically [42, 17, 18]. AM methods demonstrated their utility in detecting undesired concepts learned by the model [19, 43, 20]. However, these methods require substantial user input to identify the concepts embodied in the Activation-Maximization signals. Recent research has demonstrated that such explanations can be manipulated while maintaining the behavior of the original model [44, 45, 46].
Another group of global explainability methods aim to explain the abstraction learned by the neuron within the model, by associating it with the human-understandable concepts. The Network Dissection (NetDissect) method [11, 47] was developed to provide explanations by linking neurons to concepts, based on the overlap between the activation maps of neurons and concept segmentation masks, quantified using the Intersection over Union (IoU) metric. Addressing the limitation that neurons could only be explained with a single concept, the subsequent Compositional Explanations of Neurons (CompExp) method was introduced, enabling the labeling of neurons with compositional concepts [12]. Despite their utility, these methods generally have limitations, as they are primarily applicable to convolutional neurons and necessitate a dataset with segmentation masks, which significantly restricts their scalability (a more comprehensive discussion of these methods can be found in Appendix A.2). Other notable methods include CLIP-Dissect [13], MILAN [48], and FALCON [49]. However, these methods utilize an additional model to produce explanations, thereby introducing a new source of potential unexplainability stemming from the explainer model.
3 INVERT: Interpreting Neural Representations with Inverse Recognition
In the following, we introduce a method called Inverse Recognition (INVERT). This method aims to explain the abstractions learned by a neural representation by identifying what compositional concept representation is most effective at detecting in a binary classification scenario. Unlike the general
2The code can be accessed via the following link: https://github.com/lapalap/invert.
objective of Supervised Learning (SL) [50], which is to learn representations that can detect given concepts, the central idea behind INVERT is to learn a compositional concept that explains a given representation the best.
Let  $\mathbb{D} \subset \mathbb{R}^m$ , where  $m \in \mathbb{N}$  is the number of dimensions of data, be the input (data) space. We use the term neural representations to refer to a sub-function of a network that represents the computational graph from the input of the model to the scalar output (activation) of a specific neuron, or any combination of neurons, that results in a scalar function.
Definition 1 (Neural representation). A neural representation  $f \in \mathbb{F}$  is defined as a real-valued function  $f: \mathbb{D} \to \mathbb{R}$ , which maps the data domain  $\mathbb{D}$  to the real numbers  $\mathbb{R}$ . Here,  $\mathbb{F}$  represents the space of real-valued functions on  $\mathbb{D}$ .
Frequently, in DNNs, particular neurons, like convolutional neurons, produce multidimensional outputs. Depending on the specific needs of the application, these multidimensional functions can be interpreted either as a set of individual scalar representations or the neuron's output can be aggregated to yield a single scalar output, e.g. with pooling operations, such as average- or max-pooling. Unless stated otherwise, we utilize average-pooling as the standard aggregation measure.
We define a concept as a mapping that represents the human process of attributing characteristics to data.
Definition 2 (Concepts). A concept  $c \in \mathbb{C}$  is defined as a binary function:  $c : \mathbb{D} \longrightarrow \{0,1\}$ , which maps the data domain  $\mathbb{D}$  to the set of binary numbers. A value of 1 indicates the presence of the concept in the input, and 0 indicates its absence. Here,  $\mathbb{C}$  corresponds to the space of all concepts, that could be defined on  $\mathbb{D}$ .
In practice, given the dataset  $\mathcal{D} \subset \mathbb{D}$ , concepts are usually defined by labels, which reflect the judgments made by human experts. We define  $C = \{c_1, \dots, c_d\} \subset \mathbb{C}$  as a set of  $d \in \mathbb{N}$  atomic concepts, that are induced by labels of the dataset (also referred to as primitive concepts or primitives). Within the context of this work, we permit concepts to be non-disjoint, signifying that each data point may have multiple concepts attributed to it. Additionally, we define a vector  $\mathcal{C} = [c_1, \dots, c_d] \in \mathbb{C}^d$ .
A key step for explaining the abstractions learned by neural representations relies on the choice of the similarity measure between the concept and the representation. INVERT evaluates the relationship between representation and concepts by employing the non-parametric Area Under the Receiver Operating Characteristic (AUC) metric, measuring the representation's ability to distinguish between the presence and absence of a concept.
Definition 3 (AUC similarity). Let  $f \in \mathbb{F}$  be a neural representation, dataset  $\mathcal{D} \subset \mathbb{D}$  and concept  $c \in \mathbb{C}$ . We define a similarity measure  $d: \mathbb{F} \times \mathbb{C} \longrightarrow [0,1]$  as
$$
d (f, c) = \frac {\sum_ {\{x \mid x \in \mathcal {D} , c (x) = 0 \}} \sum_ {\{y \mid y \in \mathcal {D} , c (y) = 1 \}} \mathbf {I} [ f (x) <   f (y) ]}{\left| \left\{\left. x \right| x \in \mathcal {D}, c (x) = 0 \right\} \right| \cdot | \left\{\left. y \mid y \in \mathcal {D}, c (y) = 1 \right\} \right|}, \tag {1}
$$
where  $\mathbf{I}[f(x) < f(y)]$  is an indicator function that yields 1 if  $f(x) < f(y)$  and 0 otherwise.
AUC provides an interpretable measure to assess the ability of the representation to systematically output higher activations for the datapoints, where the concept is present. An AUC of 1 denotes a perfect classifier, while an AUC of 0.5 suggests that the classifier's performance is no better than random chance.
Given that various concepts have different numbers of data points associated with them, for concept  $c \in \mathbb{C}$  we can compute concept fraction, corresponding to the ratio of data points that are positively labeled by the concept:
$$
T (c) = \frac {\mid \{x \mid x \in \mathcal {D} , c (x) = 1 \} \mid}{\mid \{x \mid x \in \mathcal {D} \} \mid}. \tag {2}
$$
3.1 Finding Optimal Compositional Explanations
Given a representation  $f \in \mathbb{F}$ , the INVERT's objective is to identify the concept, that maximizes the AUC similarity with the representation, or, in other words finding the concept that representation is detecting the best. Due to the ability of representations to detect shared features across various concepts explaining a representation with a single atomic concept from  $C$  may not provide a comprehensive explanation. To surmount this challenge, we adopt the existing compositional concepts
Figure 1: Demonstration of the INVERT method ( $B = 1, \alpha = 0.35\%$ ) for the neuron  $f_{33}$  from ResNet18, AvgPool layer (Neuron 33), using ImageNet 2012 validation dataset. The resulting explanations can be observed in the bottom part of the figure, where three steps of the iterative process are demonstrated from  $L = 1$  to  $L = 3$ . It can be observed that INVERT explanations align with the neurons high-activating images, illustrated in the top right figure.
INVERT explanation Interpreting a neuron by identifying the concept that it detects best.
approach [12], and we augment the set of atomic concepts  $C$  by introducing new generic concepts, as a logical combination of existing ones. These logical forms involve the composition of AND, OR, and NOT operators, and they are based on the atomic concepts from  $C$ .
Definition 4 (Compositional concept). Given a vector of atomic concepts  $\mathcal{C}$ , a compositional concept  $\varphi$  is a higher-order interpretable function that maps  $\mathcal{C}$  to a new, compositional concept:
$$
\varphi : \mathbb {C} ^ {d} \longrightarrow \mathbb {C}. \tag {3}
$$
For example, let  $C = \{c_1, c_2\}$  be a set of atomic concepts with corresponding vector  $\mathcal{C}$ . Let  $c_1$  be a concept for "dog", and  $c_2$  a concept for "llama". Then  $\varphi(\mathcal{C}) = c_1$  OR  $c_2 = \text{"dog"}$  OR "llama" is a compositional concept with the length  $L = 2$ . The  $\varphi(\mathcal{C})$  is a concept in itself (i.e.  $\varphi(\mathcal{C}) \in \mathbb{C}$ ) and corresponds to a concept that is positive for all images of dogs or llamas in the dataset.
Evaluating the performance of all conceivable logical forms across all of the  $d$  concepts from  $C$  is generally computationally infeasible. Consequently, the set of potential compositional concepts  $\Phi_L$  is restricted to a form of predetermined length  $L\in \mathbb{N}$ , where  $L$  is a parameter of the method. The objective of INVERT, in this context, can be reformulated as:
$$
\varphi^ {*} = \underset {\varphi \in \Phi_ {L}} {\arg \max } d (f, \varphi (\mathcal {C})). \tag {4}
$$
To determine the optimal compositional concept that maximizes AUC, we employ an approach similar to that used in [12], utilizing Beam-Search optimization. Parameters of the proposed method include predetermined length  $L \in \mathbb{N}$ , the beam size  $B \in \mathbb{N}$ . Additionally, during the search process explanations could be constrained to the condition  $T(\varphi(\mathcal{C})) \in [\alpha, \beta]$ , where  $0 \leq \alpha < \beta \leq 0.5$ . In Section 4.1, we further demonstrate that by imposing a such constraint on the concept fraction resulting explanations could be made more comprehensive. We refer to the standard approach when  $\alpha = 0, \beta = 0.5$ . In our experiments, unless otherwise specified, the parameter  $\beta$  is set to 0.5. Additional details and a description of the algorithm can be found in Appendix A.3.
Figure 1 illustrates the INVERT pipeline for explaining the neuron from ResNet18 Average Pooling layer [51]. For this, we employed the validation set of ImageNet2012 [52] as the dataset  $\mathcal{D}_I$  in the INVERT process. This subset contains 50,000 images from 1,000 distinct, non-overlapping
Figure 2: The figure illustrates the contrast between a poor explanation (on the left) and INVERT explanations with  $L = 1$  and varying parameter  $\alpha$ , for neuron 592 in the ViT B 16 feature-extractor layer. The INVERT explanations were computed over the ImageNet 2012 validation set. The figure demonstrates that as the parameter  $\alpha$  increases, the concept fraction  $T$  also increases, indicating that more data points belong to the positive class. Furthermore, this figure showcases the proposed methods ability to evaluate the statistical significance of the result. The poor explanation fails the statistical significance test (double-sided alternative) with a p-value of 0.35, while all explanations provided by INVERT exhibit a  $p < 0.005$ .
classes, each represented by 50 images. Notably, since ImageNet classes are intrinsically linked to WordNet [53], we extracted an additional 473 hypernyms, or higher-level categories, and assigned labels for these overarching classes. In Figure 1 and subsequent figures, we use beige color to represent individual ImageNet classes and orange color to represent hypernyms. In the density plot graphs, the orange density illustrates the distribution of data point activations that belong to the explanation concept, while blue represents the distribution of activations of data points corresponding to the negation of the explanation.
3.2 Statistical significance
IoU-based explanations, such as those provided by the Network Dissection method [11], often report small positive IoU scores for the resulting explanations. This raises concerns about the potential randomness of the explanation. The AUC value is equivalent to the Wilcoxon-Mann-Whitney statistic [54] and can be interpreted as a measure based on pairwise comparisons between classifications of the two classes. Essentially, it estimates the probability that the classifier will rank a randomly chosen positive example higher than a negative example [55].
Given the concept  $c \in \mathbb{C}$ , this connection to the MannWhitney U test allows us to test if the distribution of the representations activations on the data points where concept  $c$  is positive significantly differs from the distribution of activations on points where the concept is negative. We can then report the corresponding  $p$ -value (against a double- or one-sided alternative), which helps avoid misinterpretations due to randomness, thereby improving the reliability of the explanation process, as shown in Figure 2. In all subsequent figures, the explanations provided by INVERT achieve statistical significance (against double-sided alternative) with a standard significance level (0.05).
In this section, we provide additional analysis of the proposed method, including the effect of constraining the concept fraction of explanations and comparison of the INVERT to the prior methods.
4.1 Simplicity-Precision tradeoff
The INVERT method is designed to identify the compositional concept that has the highest AUC similarity to a given representation. However, the standard approach neglects to account for the class imbalance between datapoints that belong and do not belong to a particular concept, often leading to precise but narrowly applicable explanations due to the small concept fraction. To mitigate this
ViT B 16, Features layer (CLS token), Neuron 592
Figure 4: Impact of the parameter  $\alpha$  and formula length  $L$  on the resulting explanations. The first row of the figure shows the average AUC of optimal explanations for 50 randomly sampled neurons from the feature-extractor part of each one of the four ImageNet pre-trained models, conditioned by different values of parameter  $\alpha$  in different colors. These graphs indicate that neurons generally tend to achieve the highest AUC for one individual class with  $L = 1$  and  $\alpha = 0$ . The second row presents the distribution of AUC scores alongside the distribution of concept fractions  $T$  for the INVERT explanations of length  $L = 5$ , for each model. Here, we can observe a clear trade-off between the precision of the explanation in terms of AUC measure and concept size  $T$ .
issue, we can modify the INVERT process to work exclusively with compositional concepts where the fraction equals or exceeds a specific threshold, represented as  $\alpha$ .
For this experiment, the INVERT method was utilized on the feature extractors of four different models trained on ImageNet. These models include ResNet18 [51], GoogleNet [56], EfficientNet B0 [57], and ViT B 16 [58]. In this experiment, we examined 50 randomly chosen neurons from the feature-extractor layer of each model. We utilized the ImageNet 2012 validation dataset  $\mathcal{D}_I$ , which was outlined in the previous section, to generate INVERT explanations with  $B = 3$  varying the explanation length  $L$  between 1 and 5, and parameter  $\alpha$ , responsible for the constraining the concept fraction,  $\alpha \in \{0, 0.002, 0.005, 0.01\}$ .
The experiments results are depicted in Figure 4. For all models, we can see an effect that we call the simplicity-precision tradeoff: the explanations with the highest AUC typically involve just one individual class with a low concept fraction, achieved in an unrestricted mode with parameter  $\alpha$  set to 0. By constraining the concept fraction  $\alpha$  and increasing the explanation length  $L$ , we can improve AUC scores while still maintaining the desired concept fraction. Still, this indicates that more generalized, broader explanations come at the cost of a loss in precision in terms of the AUC measure. Figure 3 demonstrates how the change of parameter  $\alpha$  affects the resulting explanation.
Figure 3: Three different INVERT explanations, computed by adjusting the parameter  $\alpha$  for the Neuron 88 in ResNet18 AvgPool layer. Higher values of this parameter lead to broader explanations, albeit at the cost of precision, thus resulting in a lower AUC. The visualization of the WordNet taxonomy for the hypernyms is provided in the Appendix 3.
4.2 Evaluating the Accuracy of Explanations
While it is generally challenging to obtain ground-truth explanations for the latent representations in Deep Neural Networks (DNNs), in Supervised Learning, the concepts of the output neurons are
Table 1: A comparison of explanation accuracy between NetDissect and INVERT. The accuracy is computed by matching identified classes with the ground truth labels.
Model	Dataset	NetDissect	INVERT
MaskRCNN ResNet50 FPN	MS COCO	95.06%	98.77%
FCN ResNet50	MS COCO	95.24%	95.24%
ResNet18	ImageNet	19.2%	73.2%
GoogleNet	ImageNet	19.7%	82.2%
DenseNet161	ImageNet	19.1%	86.9%
defined by the specific task. In the subsequent experiment, we compared the performance of INVERT and Network Dissection in accurately explaining neurons when the ground truth is known.
For this experiment, we employed 5 different models: 2 segmentation models and 3 classification models. For image segmentation, we employed MaskRCNN ResNet50 FPN model [59], pre-trained on MS COCO dataset [60] and evaluated on a subset of 24,237 images of MS COCO train2017, containing 80 distinct classes, and FCN ResNet50 model [61], pre-trained on MS COCO, and evaluated on a subset of MS COCO val2017, limited to the 20 categories found in the Pascal VOC dataset [62]. For classification models we employed ImageNet pre-trained ResNet18 [51] DenseNet161 [63], and GoogleNet [64], with 1,000 output neurons, each neuron corresponding to the individual class in the ImageNet dataset.
The outputs from the segmentation models were converted into pixel-wise confidence scores. These scores were arranged in the format  $[N_B, N_c, H, W]$ , where  $N_B$  represents the number of images in a batch, and  $N_c$  signifies the number of classes. Each value indicates the likelihood of a specific pixel belonging to a particular class. To aggregate multidimensional activations, the INVERT method used a max-pool operation.
All the classification models that were used had 1,000 one-dimensional output neurons. The evaluation process for both explanation methods was carried out using a subset of 20,000 images from the ImageNet-2012 validation dataset. For the Network Dissection method, which necessitates segmentation masks, these masks were generated from the bounding boxes included in the dataset. Both Network Dissection and INVERT methods were implemented using standard parameters.
Table 1 presents the outcomes of the evaluation process. It is noteworthy that INVERT exhibits superior or equivalent performance to Network Dissection across all tasks. Importantly, INVERT can accurately identify concepts in image segmentation networks using only the labels of images, in comparison to the Network Dissection method that uses segmentation masks.