SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Computational Cost Comparison
Figure 5: Comparing the computational cost of INVERT with Compositional Explanations of Neurons method (CompExp) in hours with varying formula lengths.
Computational cost comparison
Methods such as Network Dissection and Compositional Explanations (CompExp) of neurons have been observed to exhibit computational challenges mainly due to the operations on high-dimensional masks. While CompExp and INVERT share a beam-search optimization mechanism, the proposed approach allows for less computational resources since logical operations are performed on binary labels, instead of masks. Figure 5 showcases the running time of applying INVERT and Compositional Explanations for explaining 2048 neurons in layer 4 of the FCOS-ResNet50-FPN model [65] pre-trained on the MS COCO dataset [60] on a single Tesla V100S-PCIE-32GB GPU.
The time comparison of varying formula lengths demonstrates the advantage of INVERT being more effective computationally, which leads to reduced running time and computational costs.
In this section, we outline some specific uses of INVERT, including auditing models for spurious correlations, explaining circuits within the models, and manually creating circuits with desired characteristics.
5.1 Finding Spurious Correlations by Integrating New Concepts
Due to the widespread use of Deep Neural Networks across various domains, it is crucial to investigate whether these models display spurious correlations, backdoors, or base their decisions on undesired concepts. Using the known spurious dependency of ImageNet-trained models on watermarks written in Chinese [19, 66, 67] we illustrate that INVERT provides a straightforward method to test existing hypotheses regarding the models dependency on specific features and allows for identification of the particular neurons accountable for undesirable behavior.
To illustrate this, we augmented the ImageNet dataset  $\mathcal{D}_I$ , with an additional dataset,  $\mathcal{D}_T$ , comprising 100 images. This new dataset contains 50 images for each of two distinct concepts: Chinese textual watermarks and Latin textual watermarks (see Appendix A.4). We created examples of these classes by randomly selecting images from the ImageNet dataset and overlaying them with randomly generated textual watermarks. Figure 6 depicts the change in the explanation process conducted on the original dataset and
its expanded version. Since the original dataset didn't include the concept of watermarked images, the label "African chameleon" was attributed to the representation. However, after augmenting the dataset with two new classes, the explanation shifted to the "Chinese text" concept, with the AUC measure increasing to 0.99. This demonstrates the capability of INVERT to pinpoint sources of spurious behavior within the latent representations of the neural network.
ResNet18, AvgPool Layer, Neuron 154
Figure 6: Difference of INVERT  $(L = 1, \alpha = 0)$  explanations of Neuron 154 in Average Pooling layer of ImageNet-trained ResNet18 model before (top) and after (bottom) integration of new concepts to the dataset.
5.2 Explaining Circuits
INVERT could be employed for explaining circuits – computational subgraphs within the model, demonstrating the information flow within the model [41]. The analysis of circuits enables us to understand complex global decision-making strategies by examining how features transform from one layer to another. Furthermore, this approach can be employed for glocal explanations [68] – local explanation of a particular data point can be deconstructed into local explanations for individual neurons in the preceding layers, explained by INVERT.
To illustrate this, we computed INVERT explanations  $(L = 3, \alpha = 0.002)$  for all neurons in the average pooling layer of ResNet18. This was based on the augmented dataset from the preceding section. In ResNet18, the neurons in the Average Pooling layer have a linear connection to the output class logits. Figure 7 (left) illustrates the circuit of the three most significant neurons (based on the weight of linear connection) linked to the "carton" output logit. It could be observed that this class depends on Neuron 296, a "box" detector, and Neuron 154, which identifies the "Chinese text" concept. Furthermore, the right side of Figure 7 depicts the decomposition of local explanations: given an image of a carton box, we can dissect the GradCam [69] local explanation of a "carton" class-logit into the composition of local explanations from individual neurons. It is noticeable how Neuron 296 assigns relevance to the box, while Neuron 154 assigns relevance solely to the watermark present in the image. More illustrations of different circuits can be found in Appendix A.8.
n029/1356
Figure 7: The figure illustrates the "carton" circuit within the ResNet18 model. The left part of the figure showcases the three most significant neurons (in terms of the weight of linear connection) and their corresponding INVERT explanation linked to the class logit "carton". The right part of the figure demonstrates how the local explanation from the class logit can be decomposed into individual explanations of individual neurons from the preceding layer.
Input
478, carton
Explanation:
Output-logit layer
Neuron 478,
Explanation:
AvgPool layer
"box" AND NOT "mailbox" AND NOT "pencil_box"
Explanation:
Avgpool layer
Neuron 154
[chinese text OR "vipers" AND NOT "dianmound"]
5.3 Handcrafting Circuits
In this section, we demonstrate that its somewhat feasible to use the knowledge of what concepts are detected by neurons to combine them into manually designed circuits that can detect novel concepts. Just as compositional concepts are formed using logical operators, we employed fuzzy logic operators between neurons to construct meaningful handcrafted circuits with desired properties.
In contrast to conventional logic, fuzzy logic operators allow for the degree of membership to vary from 0 to 1 [70]. For this experiment, we employed the Gödel norm that demonstrated the best performance among other fuzzy logic operators (see Appendix A.5 for details). For the two functions  $f, g: \mathbb{D} \longrightarrow [0,1]$ , the Gödel AND (T-norm) operator is defined as  $\min(f, g)$  and the OR (T-conorm) is defined as  $\max(f, g)$ . Negation is performed by the  $1 - f$  operation.
We utilized the ImageNet-trained ViT L 16 model [58], specifically 1024 representations from the feature-extractor layer. The output of each of these representations was mapped to the range  $[0,1]$  by first normalizing the output based on their respective mean and standard deviation across the ImageNet 2012 validation dataset, and then applying the Sigmoid transformation. In this experiment, for each of the 1473 ImageNet atomic concepts (which includes 1000 classes and 473 hypernyms), we identified a neuron from the feature-extractor layer that showed the highest AUC similarity. For instance, for the concept "boat", Neuron 61 exhibited the highest AUC similarity (denoted as  $f_{\mathrm{boat}}$ ), for the concept "house", Neuron 899 showed the highest AUC similarity (denoted as  $f_{\mathrm{house}}$ ), and for the concept "lakeside", Neuron 575 showed the highest AUC similarity (denoted as  $f_{\mathrm{lakeside}}$ ).
Further, we manually constructed six different compositional formulas using concepts from ImageNet that were designed to resemble different concepts from the Places365 [71] dataset. For example, for the "boathouse" class from Places365, we assumed that images from this class would likely include "boat", "house", and water, represented by the concept "lakeside". As such, we constructed a compositional formula "boat" AND "house" AND "lakeside" using concepts from the ImageNet dataset. Finally, using the neurons, that detect these concepts (e.g.  $f_{\mathrm{boat}}$ ,  $f_{\mathrm{house}}$ ,  $f_{\mathrm{lakeside}}$ ) we manually constructed the circuits using Gödel fuzzy logic operators. That is, for "boathouse" example, final circuit was formed as  $g(x) = \min(f_{\mathrm{boat}}(x), f_{\mathrm{house}}(x), f_{\mathrm{lakeside}}(x))$  using the Gödel AND operator. The performance of the resulting circuits was evaluated on the Places365 dataset in terms of AUC similarity with the concept. In essence, by labeling representations using the ImageNet dataset and manually building a circuit guided by intuition, we evaluated how this newly created function can perform in detecting a class in the binary classification task on a different dataset.
Figure 8 illustrates the "boathouse" example and three other handcrafted circuits derived from ViT representations (the other two circuits can be found in Appendix 16). We found that after performing this manipulation, the AUC performance in detecting the Places365 class improved compared to the performance of each individual neuron. This example shows that by understanding the abstractions behind previously opaque latent representations, we can potentially construct meaningful circuits and utilize the symbolic properties of latent representations. In Appendix A.6, we further demonstrate that when labels of the target dataset overlap or are similar to the dataset used for explanation, fine-tuning of the model can be achieved by simply employing representations with explanations matching the target labels.
Handcrafted circuits from ViT representations
Figure 8: The figure presents four distinct handcrafted circuits, created from the latent representations from the ImageNet-trained ViT L 16 feature-extractor layer to detect classes from the Places365 dataset. For each neuron, or combination of neurons, we provide the Area Under the Receiver Operating Characteristic (AUROC) score for the Places365 concept in a binary classification task, distinguishing between the presence and absence of this concept.
In our work, we introduced the Inverse Recognition (INVERT) method, a novel approach for interpreting latent representations in Deep Neural Networks. INVERT efficiently links neurons with compositional concepts using an interpretable similarity metric and offers a statistical significance test to gauge the confidence of the resulting explanation. We demonstrated the wide-ranging utility of our method, including its capability for model auditing to identify spurious correlations, explaining circuits within models, and revealing symbolic-like properties in connectionist representations.
While INVERT mitigates the need for image segmentation masks, it still relies on a labeled dataset for explanations. In future research, we plan to address this dependency. Additionally, we will explore different similarity measures between neurons and explanations, and investigate new ways to compose human-understandable concepts.
The widespread use of Deep Neural Networks across various fields underscores the importance of developing reliable and transparent intelligent systems. We believe that INVERT will contribute to advancements in Explainable AI, promoting more understandable AI systems.
This work was partly funded by the German Ministry for Education and Research (BMBF) through the project Explaining 4.0 (ref. 01IS200551). Shinichi Nakajima was supported by the German Ministry for Education and Research (BMBF) as BIFOLD - Berlin Institute for the Foundations of Learning and Data under the grant BIFOLD23B. Marius Kloft acknowledges support by the Carl-Zeiss Foundation, the DFG awards KL 2698/2-1, KL 2698/5-1, KL 2698/6-1, and KL 2698/7-1, and the BMBF awards 03|B0770E and 01|S21010C.
[73] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018):11, 2018.
[74] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[75] Fei-Fei Li, Marco Andreeto, Marc'Aurelio Ranzato, and Pietro Perona. Caltech 101, apr 2022.
[76] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Our proposed INVERT method contributes to enhancing the transparency and safety of Deep Neural Networks. By providing human understandable and interpretable explanations for neurons in black-box models, our approach offers valuable insights into their internal operations, improving understanding. Moreover, our method is able to identify potentially spurious representations. An important advantage of our method is its notable reduction in computational cost compared to previous approaches. This reduction not only improves efficiency but also minimizes the harmful environmental impact associated with excessive GPU usage.
It is important to note that we cannot make definitive claims regarding specific groups of people benefiting from or being disadvantaged by our method. The general applicability and potential implications of our approach should be explored further and with caution.
A.2 Prior work
Let's consider a function,  $g: \mathbb{D} \to \mathbb{R}^{k \times k}$ , that represents a convolutional neuron within a model that produces activation maps of dimensions  $k \times k$ , along with a concept  $c \in \mathbb{C}$ . Both Network Dissection [11] and Compositional Explanations of Neurons [12] methods make use of the Intersection over Union (IoU) similarity metric to measure the degree of correlation between a function and a concept. A prerequisite for these methodologies are segmentation masks of concepts, meaning for every concept  $c \in \mathbb{C}$ , there exists a corresponding function  $M_c: \mathbb{D} \to \{0, 1\}^{h \times w}$ , which generates a binary mask for the specific concept, of the same size as the original input.
To evaluate the similarity between function  $g$  and concept  $c$ , the multi-dimensional outputs from  $g$  are subjected to thresholding based on neuron-specific percentiles (i.e., values above chosen percentiles are converted to 1 and the remaining to 0), and upscaled to match the dimensions of the original image. We can define the resulting function that produces binary masks of the same size as the input as  $G: \mathbb{D} \to \{0,1\}^{h \times w}$ . The final similarity (IoU) score between  $g$  and  $c$  can be computed as the Intersection over Union score between concept masks  $M$  and function  $G$ :
$$
d _ {\mathrm {l o U}} (g, c) = \frac {\sum_ {x \in \mathcal {D}} \mathbf {1} \left(M _ {c} (x) \cap G (x)\right)}{\sum_ {x \in \mathcal {D}} \mathbf {1} \left(M _ {c} (x) \cup G (x)\right)}. \tag {5}
$$
In section 4.2, the method of Compositional Explanations of neurons was applied using a  $7 \times 7$  input map for each feature. Conversely, the INVERT approach uses a strategy that computes a scalar value by calculating the average of the input map.
A.3 INVERT algorithm
Given a neural representation  $f: \mathbb{D} \longrightarrow \mathbb{R}$ , a dataset  $\mathcal{D} \subset \mathbb{D}$ , a set of atomic concepts  $C \in \mathbb{C}$ , and a vector  $\mathcal{C} \in \mathbb{C}^d$  the INVERT approach seeks to identify a compositional concept  $\varphi^*$ , which is formed as a logical operation on the concepts, to optimize AUC similarity  $d(f, \varphi^*(\mathcal{C})$ . For this purpose, we utilized an optimization process similar to that of the CompExpl methodology [12], employing Beam search to find the optimal compositional concept.
This method requires the configuration of certain parameters, namely the predetermined formula length  $L \in \mathbb{N}$ , the beam size  $B \in \mathbb{N}$ , and additionally, the parameters  $\alpha, \beta$ . Beam search intends to iteratively combine concepts, starting with the atomic concepts (primitives) from  $C$ . At every iteration of the process, the top  $B$  best-performing compositional concepts are selected, and all feasible formulas are computed with primitives (i.e. atomic concepts). Subsequently, only the top  $B$  best-performing concepts are selected, and the process continues until the formula reaches the predetermined length.
In detail, firstly, we define a set of primitives  $\bar{\Phi}$  — a set of compositional concepts that correspond to the set of concepts  $C$  and their negation. The set  $\bar{\Phi}$  comprises  $2k$  compositional concepts, with each concept corresponding to either the base concept or its negation. Next, all  $2k$  concepts are evaluated in terms of AUC similarity with a given function, and the top  $B$  best performing compositional concepts, that satisfy  $\alpha \leq T(\varphi(\mathcal{C})) \leq \beta$  are selected, leading to the formation of the set  $\Phi^*$  where
$|\Phi^{*}| = B$ , referred to as a Beam. These are the top  $B$  best-performing compositional concepts with a length of 1, satisfying the requisite condition on their positive fraction in the dataset. Subsequently, the following operations are iteratively performed until the predetermined formula length  $L$  is met:
1. Each of the  $B$  compositional concepts in the beam  $\Phi^{*}$  is combined with all primitives (concepts from  $\bar{\Phi}$ ) using either the AND or OR operation, thereby augmenting the formula length by 1, resulting in a total of  $4Bk$  new formulas.
2. All newly generated formulas are evaluated based on their similarity to the representation, and the beam  $\Phi^{*}$  is updated to include the top  $B$  performing formulas, which satisfy the condition  $\alpha \leq T(\varphi (\mathcal{C}))\leq \beta$ .
Upon reaching the predetermined formula length  $L$ , the Beam-Search procedure concludes by identifying the compositional concept  $\varphi^{*}$  with the highest observed AUC.
A.4 Integrating Datasets from Different Sources
Let  $\mathcal{D}_1$  and  $\mathcal{D}_2$  be two separate datasets. Each of these datasets is linked to its unique set of concepts, represented as  $C_1$  and  $C_2$  respectively. By merging these datasets, we can form a consolidated dataset, symbolized as  $\tilde{\mathcal{D}} = \mathcal{D}_1 \cup \mathcal{D}_2$ . This unified dataset will encompass a combined set of concepts, denoted as  $\tilde{C} = C_1 \cup C_2$ .
The key requirement for this integration is the mutual definition: the concepts in  $C_1$  should be defined within the dataset  $\mathcal{D}_2$ , and conversely, the concepts in  $C_2$  should be defined within the dataset  $\mathcal{D}_1$ . While this does necessitate supplementary labeling, it becomes straightforward when it is evident that the concepts from both datasets do not overlap semantically. For instance, flower concepts from the Oxford Flowers102 [72] and faces from the CelebA [73] can be effortlessly combined. This is accomplished by designating the output of concepts within the non-native dataset as negative.
A.5 Comparing Fuzzy Logic operators
Fuzzy logic operators [70] serve as essential instruments within the domain of fuzzy logic, a mathematical construct designed for modeling and handling data that is imprecise or vague. This contrasts with conventional logic where an element strictly either belongs to a set or not; fuzzy logic allows for the degree of membership to vary from 0 to 1, thereby allowing for partial membership.
In this experiment, our objective was to compare different fuzzy logic operators and examine their behavior concerning the proposed AUC metric. To fulfill this aim, we employed four distinct pretrained deep learning image classification models: AlexNet [74], DenseNet161 [63], EfficientNet B4 [57], and ViT 16 L [58]. We focused on 1000 neural representations corresponding to the ImageNet classes in the output logit (pre-SoftMax) layer for each model, for which we recognized the 'ground-truth' concept — the corresponding ImageNet class. For fuzzy logic operators' testing, we mapped the output of each representation to the set  $[0,1]$  by normalizing each representation's output using their corresponding mean and standard deviation across the ImageNet dataset and applied a Sigmoid transformation. We tested four different Fuzzy logic operators, specifically Gödel, Product, ukasiewicz, and Yager with parameter  $p = 2$ , as illustrated in Table 3.
For performance evaluation, we generated random compositional concepts of a given length and computed the AUC similarity between fuzzy logic norms applied to functions corresponding to these concepts. For instance, given the random compositional concept  $\varphi = c_{i}$  OR  $c_{j}$ , we derive compositional representations as per each of the four examined methods (e.g., the Gödel operator produces a function  $h_G = \max(f_i, f_j)$ ). These compositional representations are then evaluated in terms of AUC similarity with the compositional concept —  $d(h_G, \varphi)$ .
We conducted the evaluation in two modes, that is, assessing the performance of the OR (T-conorm) operator and the performance of the AND (T-norm) operator. For each mode, we assembled 1000 random compositional concepts by sampling  $L$  random concepts without replacement and calculated the AUC between compositional concepts and corresponding function. Note that for the second mode, AND (T-norm), random compositional concepts were assembled using the AND NOT operation, given the mutual exclusivity of ImageNet labels.
Performance of various Fuzzy Logic approaches in OR experiment
Figure 9: Average AUC similarity between random compositional OR concepts and corresponding compositional representations employing various Fuzzy logic operators (Higher is better) evaluated across four distinct models.
Performance of various Fuzzy Logic approaches in AND NOT experiment
Figure 10: Average AUC similarity between random compositional AND NOT concepts and corresponding compositional representations employing various Fuzzy logic operators (Higher is better) evaluated across four distinct models.
Figures 9 and 10 depict the mean AUC similarity between random compositional concepts of varying lengths and the corresponding compositional representations, which were assembled using four distinct fuzzy logic operators. From these figures, it becomes evident that Gödel fuzzy logic operators demonstrate the most significant robustness to the length of the formula, consistently attaining superior AUC in contrast to other operators. Consequently, we can infer that Gödel's operator emerges as the optimal choice for implementing fuzzy logic operations on neural representations.
A.6 Finetuning without training
In this section, we investigate whether it is feasible to perform model fine-tuning without having access to the target dataset, relying solely on the explanations of the latent representations and target class descriptions. In simple terms, the idea was to directly use the latent representation from ImageNet-trained models as a classifier for a class in another dataset that has a similar meaning to the explanation of the representation.
For this purpose, we utilized four different ImageNet deep learning image classification models, specifically AlexNet [74], DenseNet161 [63], EfficientNet B4 [57], and ViT 16 L [58], all of which were pre-trained on the ImageNet dataset. The feature-extractor layer that precedes the final output logit layer was used in all these models for our experiments. We computed the AUC similarity scores for all representations in each of the feature extractors in relation to ImageNet concepts.
Table 2: List of different fuzzy operators
NOT(a)	AND(a,b) (T-norm)	OR(a,b) (T-conorm)
Gödel	1 - a	min(a,b)	max(a,b)
Product	a · b	a + b - a · b
ukasiewicz	max(a + b - 1,0)	min(a + b,1)
Yager, p = 2	max(1 - ((1 - a)2 + (1 - b)2)1/2,0)	min((a2 + b2)1/2,1)
Table 3: A comparison of the accuracy achieved by the proposed finetuning method, which includes finetuning with a single representation  $(\mathrm{L} = 1)$ , and multiple representations  $(\mathrm{L} = 2,5,10)$  combined with a fuzzy AND operator, against traditional and random finetuning baselines.
AlexNet	DenseNet161	EfficientNet B4	ViT L 16
Random	2.21%	2.21%	2.21%	2.21%
L =1	43.91%	62.50%	39.96%	47.12%
L = 2	42.95%	70.51%	69.23%	60.79%
L = 5	40.17%	75.00%	80.88%	78.31%
L = 10	30.88%	69.12%	86.11%	79.49%
Finetuned	91.67%	97.76%	94.76%	98.29%