SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The target dataset employed for this study was the Caltech101 dataset [75], which comprises of 101 image classification categories. Specifically, we utilized a subset of this dataset that includes 46 classes, each of which has an exact or very similar equivalent in ImageNet classes.
We aimed to create a model for classifying Caltech classes by selecting suitable latent representations from the feature extractor layers of ImageNet models, and directly linking them to Caltech class logits. For each model, we chose a representation with the highest AUC similarity to the ImageNet concept closest to the Caltech concepts. This resulted in a subset of 46 neurons per model, each neuron having the highest AUC for an ImageNet concept similar to a Caltech concept. These neurons were normalized by the ImageNet validation dataset's mean and standard deviation, and a Sigmoid activation function was applied to constrain outputs between 0 and 1. Neuron selection was solely based on ImageNet explanations, with no Caltech101 data utilized.
We also hypothesized that individual signals from feature extractor layers could be further enhanced by executing a continuous AND operation with other neurons that share a high AUC towards the concept. Table 3 presents the results of this procedure in terms of the accuracy achieved on the target dataset. For this task, the random accuracy stands at approximately  $2\%$ , while the conventional fine-tuning approach which freezes the feature extractor layer and trains a linear classification layer atop the feature extractors achieves an accuracy of up to  $98.29\%$  (last row of the table). Remarkably, by simply linking the representation with the highest AUC towards the ImageNet concept from the latent layer to the CalTech101 output class logit using our approach  $(L = 1)$ , we were able to attain a substantial non-random accuracy, peaking at  $69.50\%$  in the case of DenseNet161. Furthermore, by selecting top  $L$  neurons that have the highest AUC towards ImageNet concepts and employing Gödel AND operator between representations, we observed that this typically improved the results, with the only exception being the AlexNet model where this strategy slightly reduced the accuracy.
A.7 Comparison between IoU and AUC metrics
In our supplementary experiments comparing different models, we further investigate the correlation between AUC and IoU. Table 4 demonstrates the performance of our method INVERT (AUC) in comparison to NetDissect and Compexp (IoU) performed on different models and layers including varying formula lengths (N). Our analysis employs ResNet18 and DenseNet161 PyTorch models trained on the Places365 dataset [71], accessible through the Compositional Explanations of Neurons implementation<sup>3</sup>. Following their approach we apply the methods on the ADE20k subset of the Broden dataset on formula lengths of 1 to 3. The IoU and AUC scores are summarized as the average and standard deviation across all neurons in each selected model layer. From these results, we can observe, that optimal explanations from AUC (INVERT) and IoU (NetDissect, CompExpl) based methods do not necessarily maximize each other objective functions.
The results in Table 5 reveal a correlation between IoU and AUC scores in non-zero IoU cases across multiple models and layers. The metrics differ in their applications and are not as strongly aligned. The correlation scores represent the average and standard deviation of the Pearson and Spearman correlation statistics. For each neuron and each available concept, correlations were calculated between the IoU and AUC scores. The "Normal" scenario corresponds to the standard case, whereas the "Log" case refers to when a logarithmic transformation was applied to the IoU values, with an
<sup>3</sup>https://github.com/jayelm/compexp/tree/master
Table 4: Comparison of IoU and AUC performed on different models and layers including varying formula lengths (N). All models are trained on the Places365 dataset and the explanations were constructed based on the ADE20k subset of the Broden dataset. The table presents the average and standard deviation scores IoU and AUC scores across all neurons in the selected model layer.
Model - Layer	N = 1
INVERT
IoU	AUC	NetDissect
IoU	AUC
ResNet18 - Layer 4	0.0062±0.0123	0.8959±0.0691	0.0581±0.0318	0.8367±0.1155
ResNet18 - Layer 3	0.0007±0.0022	0.8834±0.0780	0.0121±0.0012	0.5549±0.1660
DenseNet161 - Features	0.0016±0.0066	0.8928±0.0733	0.0364±0.0279	0.7448±0.1547
DenseNet161 - Dense Block 4	0.0007±0.0017	0.9014±0.0655	0.0150±0.0034	0.6877±0.1582
N = 2
Model - Layer	INVERT
IoU	AUC	CompExp
IoU	AUC
ResNet18 - Layer 4	0.0021±0.0062	0.9972±0.0037	0.0756±0.0369	0.8310±0.1016
ResNet18 - Layer 3	0.0023±0.0042	0.9955±0.0077	0.0185±0.0014	0.5726±0.1332
DenseNet161 - Features	0.0042±0.0124	0.9958±0.0056	0.0455±0.0313	0.7248±0.1424
DenseNet161 - Dense Block 4	0.0029±0.0059	0.9961±0.0058	0.0222±0.0040	0.6930±0.1127
N = 3
Model - Layer	INVERT
IoU	AUC	CompExp
IoU	AUC
ResNet18 - Layer 4	0.0026±0.0079	0.9977±0.0030	0.0849±0.0391	0.8184±0.0995
ResNet18 - Layer 3	0.0021±0.0038	0.9966±0.0057	0.0235±0.0016	0.5714±0.1084
DenseNet161 - Features	0.0035±0.0104	0.9967±0.0046	0.0497±0.0330	0.7132±0.1356
DenseNet161 - Dense Block 4	0.0026±0.0054	0.9969±0.0048	0.0361±0.0231	0.6846±0.1045
Table 5: Correlation between IoU and AUC based on the score for each class per neuron. The models were pre-trained using the Places365 dataset and their performance was assessed on the ADE20k subset of the Broden dataset. The table presents the average and standard deviation of the Pearson and Spearman correlation statistics.
Model - Layer	Normal	Log(+eps)
Pearson	Spearman	Pearson	Spearman
ResNet18 - Layer 4	0.3429±0.0682	0.3623±0.0945	0.4116±0.0885	0.3623±0.0945
ResNet18 - Layer 3	0.2377±0.0911	0.2738±0.1121	0.3009±0.1180	0.2738±0.1121
DenseNet161 - Features	0.2681±0.0869	0.2787±0.1041	0.3156±0.1050	0.2787±0.1041
DenseNet161 - Dense Block 4	0.2143±0.1039	0.2691±0.1626	0.2878±0.1573	0.2691±0.1626
additional epsilon value of 1e-4. Correlations were computed exclusively for concepts that showed non-zero IoU scores. We can observe, that for non-zero IoU scores there exists a small positive correlation between IoU and AUC scores.
To further comprehend the correlation of these metrics we investigate the case where AUC and IoU perform differently. In Figure 11, we present a case where explanations yielding 0 IoU scores are better aligned with the explanation goal. We provide evidence of IoU-based explanations resulting in low neuron activation, while INVERT achieves notable activation even when IoU scores are 0.
Figure 12 (a) shows a qualitative example of AUC and IoU scores across all concepts of Neuron 269 from layer 4 of ResNet18 trained on the Places365 dataset [71]. Each data point corresponds to one concept among the 1105 ADE20k atomic concepts sourced from the Broden dataset [76, 11]. This example illustrates the dependence between AUC and IoU, high IoU scores are correlated with high AUC scores. In Figure 12 (b) we showcase the top 4 most activating images for Neuron 269 from the ADE20k dataset to align them with the highest scoring concepts in Figure 12 (a). Comparing the set of images with the concepts exhibiting the highest AUC scores (e.g., "throne room", "apse-indoor", "fur"), we observe a strong visual alignment. However, when examining the concepts with high IoU scores (e.g., "nursery", "cradle", "attic"), we find a relatively low degree of visual similarity. Those results demonstrate the limitations of the IoU measure for evaluating explanations.
Figure 11: Comparison between INVERT and NetDissect. The figure displays three distributions of activations: one for all datapoints in green, one for datapoints corresponding to the IoU-based explanation in orange, and one for the AUC-based explanation in blue. These distributions pertain to the average activation across activation maps of Neuron 205 in ResNet18, layer 3, pre-trained on the Places365 dataset. The activations were collected across the ADE20k subset of the Broden dataset. The class labeled as "car" resulted from IoU optimization, while the class labeled as "ocean" resulted from AUC optimization. Notably, even though the "ocean" class has an IoU score of 0, it comprises some of the most activating images for the neuron, as evidenced by the top 9 most activated images.
(a) Distribution of IoU and AUC for neuron 269
(b) Top 4 activating images for neuron 269
(c) Distribution of max IoU and max AUC,  $L = 1$
Figure 12: In (a) we compare the distribution of AUC and IoU across all concepts of the ADE20k atomic concepts from the Broden dataset for Neuron 269 from layer 4 of ResNet18 trained on the Places365 dataset, where (b) shows the top 4 activating images of the ADE20k dataset. (c) shows the distribution of maximized IoU, maximized AUC, and random IoU scores for layer 4 of ResNet18 trained on the Places365 dataset with a formula length of 1.
Table 6: Metric comparison for true label and random explanation evaluation. FasterRCNN ResNet50 FPN (pre-trained on MS COCO for object detection). UPerNet BEiT-B (pre-trained on ADE20k for semantic segmentation).
Metric	FasterRCNN ResNet50 FPN	UPerNet BEiT-B
True	Random	True	Random
IoU	0.8355±0.0466	0.0077±0.0046	0.8553±0.0913	0.0007±0.0007
AUC	0.9556±0.0371	0.5005±0.0253	0.8738±0.0929	0.5001±0.0164
Furthermore, we conducted a quantitative evaluation shown in Figure 12 (c), specifically focusing on layer 4 of the ResNet18 trained on the Places365 dataset. We compared the distribution of IoU scores of explanations obtained by maximizing IoU and AUC respectively. Additionally, we examined the mean values of these distributions, which included random IoU scores as baseline reference. Our findings reveal that maximizing IoU leads to a relatively sparse distribution of IoU scores while maximizing AUC results in a more densely concentrated accumulation of predominantly low IoU scores. As anticipated, the performance of random IoU scores was notably poor. We can observe that maximizing AUC also indirectly maximizes IoU.
Table 6 serves as a sanity check implementing metric comparison for best explanation and random explanation evaluation. The FasterRCNN ResNet50FPN model was pretrained on the MS COCO dataset for object detection, while the UPerNet BEiT-B model was pretrained on ADE20k for semantic segmentation. The former model's evaluation was conducted on a subset of MS COCO containing 20,000 images, while the latter was assessed on the ADE20k subset of the Broden dataset. The output layers of both models were utilized to access the "ground truth" label for each neuron. In the table, the "True" column represents the IoU/AUC scores of the explanation that align with the ground-truth neuron label. On the other hand, the "Random" column corresponds to the scores of randomly chosen explanation-concept pairs that differ from the "ground truth".
A.8 Figures
Figure 13: The figure displays the WordNet taxonomy, which was used to gather the hierarchical structure of the labels for the Figure 3.
Input
771, safe
n04125021
Figure 14: The figure illustrates the "safe" circuit within the ResNet18 model. The top part of the figure showcases the three most significant neurons (in terms of the weight of linear connection) and their corresponding INVERT explanation linked to the class logit "safe". The bottom part of the figure demonstrates how the local explanation from the class logit can be decomposed into individual explanations of individual neurons from the preceding layer. This allows for a more detailed understanding of how each neuron contributes to the final classification.
Explanation:
Output-logit layer
Neuron 771,
"safe"
Explanation:
AvgPool layer
Neuron 192
NOT "swimming_trucks" AND "photocopier" OR "swimsuit"
Explanation:
Avgpool layer
Neuron 452
("house" OR "place_of_worship") AND NOT ("stupa")
Explanation:
Avgpool layer
Neuron 154
("chinesetx"OR "viper")
AND NOT "diamondback"
Figure 15: The figure illustrates the "monitor" circuit within the ResNet18 model. The top part of the figure showcases the four most significant neurons (in terms of the weight of linear connection) and their corresponding INVERT explanation linked to the class logit "monitor". The bottom part of the figure demonstrates how the local explanation from the class logit can be decomposed into individual explanations of individual neurons from the preceding layer.
Figure 16: The figure presents two distinct handcrafted circuits. For each neuron, or combination of neurons, we report the Area Under the Receiver Operating Characteristic (AUROC) score. This score represents the AUC classification performance towards classifying specific concepts from the Places365 dataset.