SlowGuess/NeurIPS_2023_OCR huggingface dataset:
All scores are computed using 500 feature visualizations, each of them maximizing the logit of one of the ImageNet classes obtained on the FlexiViT [49], ViT[63], and ResNetV2[64] models. For the feature visualizations derived from Olah et al. [1], we used all 10 transformations set from the Lucid library*. CBR denotes an optimization in pixel space and using the same 10 transformations, as described in [29]. For MACO,  $\tau$  only consists of two transformations; first we add uniform noise  $\delta \sim \mathcal{U}([-0.1,0.1])^{W\times H}$  and crops and resized the image with a crop size drawn from the normal distribution  $\mathcal{N}(0.25,0.1)$ , which corresponds on average to  $25\%$  of the image. We used the NAdam optimizer [65] with  $lr = 1.0$  and  $N = 256$  optimization steps. Finally, we used the implementation of [1] and CBR which are available in the Xplique library [66]† which is based on Lucid.
FlexiViT	ViT	ResNetV2	FlexiViT	ViT	ResNetV2
• Plausibility score (1-KNN) (↓)	• Transferability score(↑): MACO/ Fourier [1]
MACO	1473	1097	1248	MobileNet	68 / 38	48/ 37
Fourier [1]	1815	1817	1837	VGG16	64 / 30	50 / 30
CBR [29]	1866	1920	1933	Xception	85 / 61	73 / 62
• FID Score (↓)	Eff. Net	88 / 25	63 / 25
MACO	230.68	241.68	312.66	ConvNext	96 / 52	84 / 55
Fourier [1]	250.25	257.81	318.15	DenseNet	84 / 32	66 / 31
CBR [29]	247.12	268.59	346.41	93 / 25
Table 1: Plausibility and FID scores for different feature visualization methods applied on FlexiVIT, ViT and ResNetV2
Table 2: Transferability scores for different feature visualization methods applied on FlexiVIT, ViT and ResNetV2.
For all tested metrics, we observe that MACO produces better feature visualizations than those generated by Olah et al. [1] and CBR [29]. We would like to emphasize that our proposed evaluation scores represent the first attempt to provide a systematic evaluation of feature visualization methods, but we acknowledge that each individual metric on its own is insufficient and cannot provide a comprehensive assessment of a method's performance. However, when taken together, the three proposed scores provide a more complete and accurate evaluation of the feature visualization methods.
4.1 Human psychophysics study
Ultimately, the goal of any feature visualization method is to demystify the CNN's underlying decision process in the eyes of human users. To evaluate MACO's ability to do this, we closely followed the psychophysical paradigm introduced in [67]. In this paradigm, the participants are presented with examples of a model's "favorite" inputs (i.e., feature visualization generated for a given unit) in addition to two query inputs. Both queries represent the same natural image, but have a
*https://github.com/tensorflow/lucid
$\dagger$  https://github.com/deel-ai/xplique
FlexiViT	Plausibility (↓)	FID (↓)	logit magnitude (↑)
MACO	571.68	211.0	5.12
- transparency	617.9 (+46.2)	208.1 (-2.9)	5.05 (-0.1)
- crop	680.1 (+62.2)	299.2 (-91.1)	8.18 (+3.1)
- noise	707.3 (+27.1)	324.5 (-25.3)	11.7 (+3.5)
Fourier [1]	673.3	259.0	3.22
- augmentations	735.9 (+62.6)	312.5 (+53.5)	12.4 (+9.2)
Table 3: Ablation study on the FlexiViT model: This reveals that 1. augmentations help to have better FID and Plausibility scores, but lead to lesser salients visualizations (softmax value), 2. Fourier [1] benefits less from augmentations than MACO.
different part of the image hidden from the model by a square occluder. The task for participants is to judge which of the two queries would be "favored by the model" (i.e., maximally activate the unit). The rationale here is that a good feature visualization method would enable participants to more accurately predict the model's behavior. Here, we compared four visualization conditions (manipulated between subjects): Olah [1], MACO with the transparency mask (the transparency mask is described in 3.2), MACO without the transparency mask, and a control condition in which no visualizations were provided. In addition, the network (VGG16, ResNet50, ViT) was a within-subject variable. The units to be understood were taken from the output layer.
Based on the data of 174 participants on Prolific (www.prolific.com) [September 2023], we found both visualization and network to significantly predict the logodds of choosing the right query (Fig. 5). That is, the logodds were significantly higher for participants in both the MACO conditions compared to Olah. On the other hand, our tests did not yield a significant difference between Olah and the control condition, or between the two MACO conditions. Finally, we found that, overall, ViT was significantly harder to interpret than ResNet50 and VGG16, with no significant difference observed between the latter two networks. Full experiment and analysis details can be found in the supplementary materials, section C. Taken together, these results support our claim that even if feature visualization methods struggle in offering interpretable information as networks scale, MACO still convincingly helps people better understand deeper models while Olah's method [1] does not.
Figure 5: Human causal understanding of model activations. We follow the experimental procedure introduced in [67] to evaluate Olah and MACO visualizations on 3 different networks. The control condition is when the participant did not see any feature visualization.
To disentangle the effects of the various components of MACO, we perform an ablation study on the feature visualization applications. We consider the following components: (1) the use of a magnitude constraint, (2) the use of the random crop, (3) the use of the noise addition, and (4) the use of the transparency mask. We perform the ablation study on the FlexiViT model, and the results are presented in Table 3. We observe an inherent tradeoff between optimization quality (measured by logit magnitude) on one side, and the plausibility (and FID) scores on the other side. This reveals that plausible images which are close to the natural image distribution do not necessarily maximize the logit. Finally, we observe that the transparency mask does not significantly affect any of the scores confirming that it is mainly a post-processing step that does not affect the feature visualization itself.
We demonstrate the versatility of the proposed MACO technique by applying it to three different XAI applications:
Logit and internal state visualization. For logit visualization, the optimization objective is to maximize the activation of a specific unit in the logits vector of a pre-trained neural network (here a FlexiViT[49]). The resulting visualizations provide insights into the features that contribute the most to a class prediction (refer to Figure 4a). For internal state visualization, the optimization objective is to maximize the activation of specific channels located in various intermediate blocks of the network (refer to Figure 4b). This visualization allows us to better understand the kind of features these blocks – of a FlexiViT[49] in the figure – are sensitive to.
Feature inversion. The goal of this application is to find an image that produces an activation pattern similar to that of a reference image. By maximizing the similarity to reference activations, we are able to generate images representing the same semantic information at the target layer but without the parts of the original image that were discarded in the previous stages of the network, which allows us to better understand how the model operates. Figure 6a displays the images (second column) that match the activation pattern of the penultimate layer of a VIT when given the images from the first column. We also provide examples of transparency masks based on attribution (third column), which we apply to the feature visualizations to enhance them (fourth column).
Figure 6: Feature inversion and concept visualizaiton. a) Images in the second column match the activation pattern of the penultimate layer of a ViT when fed with the images of the first column. In the third column, we show their corresponding attribution-based transparency masks, leading to better feature visualization when applied (fourth column). b) MACO is used to visualize concept vectors extracted with the CRAFT method [50]. The concepts are extracted from a ResNet50 trained on ImageNet. All visualizations for all ImageNet classes are available at Lens.
Concept visualization. Herein we combine MACO with concept-based explainability. Such methods aim to increase the interpretability of activation patterns by decomposing them into a set of concepts [68]. In this work, we leverage the CRAFT concept-based explainability method [50], which uses Non-negative Matrix Factorization to decompose activation patterns into main directions – that are called concepts –, and then, we apply MACO to visualize these concepts in the pixel space. To do so, we optimize the visualization such that it matches the concept activation patterns. In Figure 6b, we present the top 2 most important concepts (one concept per column) for five different object categories (one category per row) in a ResNet50 trained on ImageNet. The concepts' visualizations are followed by a mosaic of patches extracted from natural images: the patches that maximally activate the corresponding concept.
We have demonstrated the generation of realistic explanations for large neural networks by imposing constraints on the magnitude of the spectrum. However, it is important to note that generating realistic images does not necessarily imply effective explanation of the neural networks. The metrics introduced in this paper allow us to claim that our generated images are closer to natural images in latent space, that our feature visualizations are more plausible and better reflect the original distribution. However, they do not necessarily indicate that these visualizations helps humans in effectively communicating with the models or conveying information easily to humans. Furthermore, in order for a feature visualization to provide informative insights about the model, including spurious features, it may need to generate visualizations that deviate from the spectrum of natural images. Consequently, these visualizations might yield lower scores using our proposed metrics. Simultaneously, several interesting studies have highlighted the weaknesses and limitations of feature visualizations [54, 69, 67]. One prominent criticism is their lack of interpretability for humans, with research demonstrating that dataset examples are more useful than feature visualizations in understanding convolutional neural networks (CNNs) [54]. This can be attributed to the lack of realism in feature visualizations and their isolated use as an explainability technique. With our approach, MACO, we take an initial step towards addressing this limitation by introducing magnitude constraints, which lead to qualitative and quantitative improvements. Additionally, through our website, we promote the use of feature visualizations as a supportive and complementary tool alongside other methods such as concept-based explainability, exemplified by CRAFT [50]. We emphasize the importance of feature visualizations in combating confirmation bias and encourage their integration within a comprehensive explainability framework.
In this paper, we introduced a novel approach, MACO, for efficiently generating feature visualizations in modern deep neural networks based on (i) a hard constraint on the magnitude of the spectrum to ensure that the generated visualizations lie in the space of natural images, and (ii) a new attribution-based transparency mask to augment these feature visualizations with the notion of spatial importance. This enhancement allowed us to scale up and unlock feature visualizations on large modern CNNs and vision transformers without the need for strong – and possibly misleading – parametric priors. We also complement our method with a set of three metrics to assess the quality of the visualizations. Combining their insights offers a way to compare the techniques developed in this branch of XAI more objectively. We illustrated the scalability of MACO with feature visualizations of large models like ViT, but also feature inversion and concept visualization. Lastly, by improving the realism of the generated images without using an auxiliary generative model, we supply the field of XAI with a reliable tool for explaining the semantic ("what" information) of modern vision models.
This work was conducted as part the DEEL project. It was funded by ONR grant (N00014-19-1-2029), NSF grant (IIS-1912280 and EAR-1925481), and the Artificial and Natural Intelligence Toulouse Institute (ANITI) grant #ANR19-PI3A-0004. The computing hardware was supported in part by NIH Office of the Director grant #S10OD025181 via the Center for Computation and Visualization (CCV) at Brown University. J.C. has been partially supported by Intel Corporation, a grant by Banco Sabadell Foundation and funding from the Valencian Government (Conselleria d'Innovació, Industria, Comercio y Turismo, Dirección General para el Avance de la Sociedad Digital) by virtue of a 2023 grant agreement (convenio singular 2023).
$\ddagger$  https://www.deel.ai/
[58] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[59] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251-1258, 2017.
[60] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114. PMLR, 2019.
[61] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976-11986, 2022.
[62] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[63] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V, page 491–507, Berlin, Heidelberg, 2020. Springer-Verlag.
[64] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 630–645. Springer, 2016.
[65] Timothy Dozat. Incorporating nesterov momentum into adam. Proceedings of the International Conference on Learning Representations (ICLR), 2016.
[66] Thomas Fel, Lucas Hervier, David Vigouroux, Antonin Poche, Justin Plakoo, Remi Cadene, Mathieu Chalvidal, Julien Colin, Thibaut Boissin, Louis Bethune, Agustin Picard, Claire Nicodeme, Laurent Gardes, Gregory Flandin, and Thomas Serre. Xplique: A deep learning explainability toolbox. Workshop on Explainable Artificial Intelligence for Computer Vision (CVPR), 2022.
[67] Roland S Zimmermann, Judy Borowski, Robert Geirhos, Matthias Bethge, Thomas Wallis, and Wieland Brendel. How well do feature visualizations support causal understanding of cnn activations? Advances in Neural Information Processing Systems, 34:11730-11744, 2021.
[68] Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. Advances in Neural Information Processing Systems, 32, 2019.
[69] Robert Geirhos, Roland S. Zimmermann, Blair Bilodeau, Wieland Brendel, and Been Kim. Don't trust your eyes: on the (un)reliability of feature visualizations, 2023.
[70] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.