Unlocking Feature Visualization for Deep Network with MAgnitude Constrained Optimization
Authors: Thomas FEL
Thibaut Boissin
Victor Boutin
Agustin Martin Picard
Paul Novello
Julien Colin
Drew Linsley
Tom ROUSSEAU
Remi Cadene
Lore Goetschalckx
Laurent Gardes
Thomas Serre
Venue: NeurIPS 2023 poster (2023)
Keywords: explainable AI
feature visualization
interpretability
optimization

Abstract: Feature visualization has gained significant popularity as an explainability method, particularly after the influential work by Olah et al. in 2017. Despite its success, its widespread adoption has been limited due to issues in scaling to deeper neural networks and the reliance on tricks to generate interpretable images. Here, we describe MACO, a simple approach to address these shortcomings. It consists in optimizing solely an image's phase spectrum while keeping its magnitude constant to ensure that the generated explanations lie in the space of natural images. Our approach yields significantly better results -- both qualitatively and quantitatively -- unlocking efficient and interpretable feature visualizations for state-of-the-art neural networks. We also show that our approach exhibits an attribution mechanism allowing to augment feature visualizations with spatial importance. Furthermore, we enable quantitative evaluation of feature visualizations by introducing 3 metrics: transferability, plausibility, and alignment with natural images. We validate our method on various applications and we introduce a website featuring MACO visualizations for all classes of the ImageNet dataset, which will be made available upon acceptance. 
Overall, our study unlocks feature visualizations for the largest, state-of-the-art classification networks without resorting to any parametric prior image model, effectively advancing a field that has been stagnating since 2017 (Olah et al, 2017).

Unlocking Feature Visualization for Deeper Networks with Magnitude Constrained Optimization
Thomas Fe,.1.2,4, Thibaut Boissin <2.3, Victor Boutin 1,2, Agustin Picard +2,3, Paul Novello x2.3
Julien Colin 1.5 Drew Linsley', Tom Rousseau', Rémi Cadène', Lore Goetschalckx Laurent Gardes Thomas Serre
Carney Institute for Brain Science, Brown University
Artificial and Natural Intelligence Toulouse Institute
"Institut de Recherche Technologique Saint-Exupery
IInoovation Research Division, SNCF *ELLIS Alicante, Spain.
{thomas_ fel @ell@bboonn... @ddleorooeedu, thibaut. wwosssn@rrrtssinneeeneeee
Abstract
Feature visualization has gained substantial popularity, particularly after the in- fluential work by Olah et al. in 2017, which established it as a crucial tool for explainability. However, its widespread adoption has been limited due reliance on tricks to generate interpretable images, and corresponding challenges in scaling it deeper neural networks. Here, we describe MACO, simple approach address these shortcomings. The main idea is to generate images by optimizing the phase spectrum while keeping the magnitude constant ensure that generated explanations lie in the space of natural images. Our approach yields significantly better results both qualitatively and quantitatively and unlocks efficient and interpretable feature visualizations for large state-of-the-art neural networks. We also show that our approach exhibits an attribution mechanism allowing us augment feature visualizations with spatial importance. We validate our method on novel benchmark for comparing feature visualization methods, and release its visualizations for all classes of the ImageNet dataset on Lens.
Overall, our approach unlocks, for the first time, feature visualizations for large, state-of-the-art deep neural networks without resorting to any parametric prior image model.
Introduction
The field of Explainable Artificial Intelligence (XAI) [2 has largely focused on characterizing computer vision models through the use of attribution methods [4, 5, 6, 7, 8, 9, 10, 11, 12, 13]. These methods aim to explain the decision strategy of network by assigning an importance score to each input pixel (or group of input pixels [14, 15, 16]), according to their contribution to the overall decision. Such approaches only offer partial understanding of the learned decision processes as they aim to identify the location the most discriminative features in an image, the "where", leaving open the "what" question, í.e. the semantic meaning of those features. Recent work [17, 18, 19, 20, 21, 22] has highlighted the intrinsic limitations of attribution methods [23, 24, 25, 26], calling for the development of methods that provide complementary explanation regarding the "what".
Feature visualizations provide a bridge to fill this gap via the generation of images that elicit a strong response from specifically targeted neurons (or groups of neurons). One of the simplest approaches uses gradient ascent to search for such an image. In the absence of regularization, this optimization
The authors contributed equally.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

VGG16 (2014)
ResNet50 (2016)
ViT (2020)
Fourier FV
MACO
(ours)
Figure 1: Comparison between feature visualization methods for "White Shark" classifica- tion. (Top) Standard Fourier preconditioning-baee method for feature visualization [1]. (Bottom) Proposed approach, MACO, which incorporates Fourier spectrum magnitude constraint.
known to yield highly noisy images sometimes considered adversarial [27]. Hence, regularization methods are essential to rendering more acceptable candidate images. Such regularizations can consist of penalizing high frequencies in the Fourier domain [1, 28, 29, 30, 31], regularizing the optimization process with data augmentation [1, 32, 33, 34, 35, 36, 37] or restricting the search space subspace parameterized by generative model [38, 39, 40, 41]. The first two approaches provide faithful visualizations, as they only depend on the model under study; unfortunately. in practice, they still fail on large modern classification models (e.g. ResNet50V2 [42] and ViT [43], see Figure 1). The third approach yields interpretable feature visualizations even for large models but at the cost of major biases: in that case, it is impossible to disentangle the true contributions of the model under study from those of the generative prior model. Herein, introduce new feature visualization approach that applicable to the largest state-of-the-art networks without relying on any parametric prior image model.
Our proposed approach, called MAgnitude Constrained Optimization (MACO), builds on the seminal work by Olah et al. who described the first method to optimize for maximally activating images in Fourier space order to penalize high-frequency content [1]. Our method straightforward and essentially relies exploiting the phase/magnitude decomposition of the Fourier spectrum, while exclusively optimizing the image's phase while keeping its magnitude constant. Such a constraint is motivated by psychophysics experiments that have shown that humans more sensitive to differences in phase than magnitude [44, 45, 46, 47, 48J. Our contributions are threefold:
(i) We unlock feature visualizations for large modern CNNs without resorting any strong parametric image prior (see Figure 1).
(ii) We describe how leverage the gradients obtained throughout our optimization process to combine feature visualization with attribution methods, thereby explaining both "what" activates neuron and "where" it is located in image.
(iil) We introduce new metrics to compare the feature visualizations produced with MACO to those generated with other methods.
As an application of our approach, we propose feature visualizations for Flex VIT [49] and ViT [43] (logits and intermediate layers; see Figure 4). We also employ our approach on a feature inversion task to generate images that yield the same activations as target images to better understand what information is getting propagated through the network and which parts of the image are getting discarded by the model (on VIT. Figure 6). Finally, show how to combine our work with

---

a state-of-the-ar concept-based explainability method [50] (see Figure 6b). Much like feature visualization, this method produces explanations on the semantic "what" that drives a model's prediction by decomposing the latent space of the neural network into a series of "directions" denoted concepts. More importantly, it also provides a way to locate each concept in the input image under study, thus unifying both axes "what"" and "where". As feature visualization can be used to optimize in directions in the network's s representation space, we employ MACO to generate concept visualizations, thus allowing to improve the human interpretability of concepts and reducing the risk of confirmation bias. We showcase these concept visualizations on an interactive website: Lens. The website allows browsing the most important concepts learned by ResNet50 for all 1, 000 classes of ImageNet [42].
2 Related Work
Feature visualization methods involve solving an optimization problem find an input image that maximizes the activation of a target element (ncuron, layer, or whole model) [10]. Most of the approaches developed in the field fall along a spectrum based on how strongly they regularize the model. At one end of the spectrum, if no regularization is used, the optimization process can search the whole image space, but this tends produce noisy images and nonsensical high-frequency patterns [51].
To circumvent this issue, researchers have proposed to penalize high- figh-freyeeccy in the resulting images either by reducing the variance between neighboring pixels [28]. by imposing constraints on the image's 's total variation [40, 41, 4], or by blurring the image at each optimization step [29]. However, in addition to rendering images of debatable validity, these approaches also suppress genuine, interesting high-frequency features, including edges. To mitigate this issue, a bilateral filter may be used instead of blurring, as it has been shown to preserve edges and improve the overall result [30]. Other studies have described similar technique decrease high frequencies by operating directly on the gradient, with the goal of preventing their accumulation in the resulting visualization [31]. One advantage of reducing high frequencies present in the gradient, as opposed to the visualization itself, is that it prevents high frequencies from being amplified by the optimizer while still allowing them to appear in the final image if consistently encouraged by the gradient. This process, known as "preconditioning" in optimization, can greatly simplify the optimization problem. The Fourier transform has been shown to be a successful preconditioner as forces the optimization to be performed in decorrelated and whitened image space [1]. The feature visualization technique we introduce in this work leverages a similar preconditioning. The emergence of high- high-frequency patterns in the absence of regularization is associated with lack of robustness and sensitivity of the neural network to adversarial examples [27], and consequently, these patterns are less often observed in adversarially robust models [34, 33, 32]. An alternative strategy to promote robustness involves enforcing small perturbations, such as jittering, rotating, or scaling, in the visualization process [35], which, when combined with a frequency penalty [1], has been proved to greatly enhance the generated images.
Unfortunately, previous methods in the field of feature visualization have heen limited in their ability to generate visualizations for newer architectures beyond VGG, resulting in lack of interpretable visualizations for larger networks like ResNets [1]. Consequently, researchers have shifted their focus to approaches that leverage statistically learned priors to produce highly realistic visualizations. One such approach involves training a generator, like GAN [40] or autoencoder [52, 41J, to map points from a latent space to realistic examples and optimizing within that space. Alternatively, a prior can be learned to provide the gradient (w.r.t the input) of the probability and optimize both the prior and the objective jointly [41, 30]. Another method involves approximating a generative model prior by penalizing the distance between output patches and the nearest patches retrieved from a database of image patches collected from the training data [38]. Although it is well-established that learning image prior produces realistic visualizations, it is difficult to distinguish between the contributions of the generative models and that of the neural network under study. Hence, in this work, focus the development of visualization methods that rely on minimal priors to yield the least biased visualizations.
3 Pignitude-Constrainee Feature Visualization
Notations Throughout, we consider a general supervised learning setting, with an input space X C k/xw output space y C RR and classifier f X + y that maps inputs æ X to
3

---

prediction E y. Without loss of generality, we assume that f admits a series of L intermediate spaces Ar S IR E L. In this setup, ft X' A, maps an input to an intermediate activation = (01...... Up, ) ) E A of f. We respectively denote F and F-1 as the 2-D Discrete Fourier Transform (DFT) on X' and its inverse.
Optimization Criterion. The primary goal of a feature visualization method is to produce an image x' that maximizes given criterion (x) I; usually some value aggregated over a subset of weights in neural network f (neurons, channels, layers, logits). A concrete example consists in finding a natural "prototypical" image x" of a class models. However, optimizing in the pixel space I gwwx x is known to produce noisy, adversarial- like x" (see section 2). Therefore, the optimization is constrained using regularizer R X + R to penalize unrealistic images:
[[. K] without using dataset or generative
arg max (x) XR(x:). =EX
(1)
In Eq. 1, Å is a hyperparameter used to balance the main optimization criterion Ly and the regularizer R. Finding a regularizer that perfectly matches the structure of natural images hard, so proxies have to be used instead. Previous studies have explored various forms of regularization spanning from total variation, ε, or &z loss [40, 41, 4]. More successful attempts rely on the reparametrization of the optimization problem in the Fourier domain rather than on regularization.
3.1 A Fourier perspective
Mordvintsev al. [53] noted in their seminal work that one could use differentiable im- age parametrizations to facilitate the maximization of Le. Olah et al. [1] proposed to re- parametrize the images using their Fourier spectrum. Such parametrization allows am- plifying the low frequencies using a scalar w. Formally, the prototypal image x² can be written xx
F=* @ w) with zt arg max x (F w)).
Finding x" boils down to optimizing a Fourier
buffer a+iib together with boosting the low-
a) Fourier FV spectrum
b) ImageNet spectrum
frequency components and then recovering the
final image by inverting the optimized Fourier buffer using inverse Fourier transform.
However, multiple studies have shown that the resulting images are not sufficiently robust, in the sense that a small change in the image can cause the criterion L. to drop. Therefore, it common see robustness transformations applied to candidate images throughout the op- timization process.
other words, the goal is to ensure that the generated image satisfies the criterion even if it is rotated by a few de- grees or jittered by a few pixels. Formally, given a set of possible transformation func- tions sometimes called augmentations that we denote T such that for any transformation T, we have (xx) X, the optimization be- comes 2 arg max B (= w)
12
power spactrum (log)
Figure 2: Comparison between Fourier FV and natural image power spectrum. In (a), the power spectrum is averaged over 10 different logits visu- alizations for each of the 1000 classes of ImageNet. obtained using the Fourier
The visualizations FVFourier FV method to maximize the logits of a ViT network [1]. In (b) the spectrum is averaged over all training images of the ImageNet dataset.
Empirically, common knowledge that the deeper the models are, the more transforma- tions are needed and the greater their magni- tudes should be. To make their approach work on models like VGG, Olah et al. [l] used no less than dozen transformations. However, this method fails for modern architectures, no matter how many transformations are applied. We argue that this may come from the low-frequency scalar (or booster) no longer working with models that are too deep. For such models, high frequencies eventually come through, polluting the resulting images with high-frequency content making them impossible to interpret by humans. To empirically illustrate this phenomenon, we compute the k logit visualizations obtained by maximizing each of the logits corresponding the classes of VIT using the parameterization used by Olah al. In Figure 2 (left), show the average of the

---

(a)
(b)
Fourier phase (IR 1P-5))
Fourier spectrum
(C
Cp
2 =++ 10
Z
f
b
Inverse Fourier transform JF
Transformation robustness
Inverse Fourler transform
Transformation robustness
Fixed Fourier magnitude
Figure 3: Overview of the approach: (a) Current Fourier parameterization approaches optimize the entire spectrum (yellow arrow). (b) contrast, the optimization flow our approach (green arrows) goes from the network activation (z) to the phase of the spectrum (4p) of the input image (æ).
spectrum of these generated visualizations over all classes: Σ [F(=:)) We compare with the average spectrum of images on the ImageNet dataset (denoted D): EEzD((((ar) (Figure 2, right panel). We observe that the images obtained through optimization put much more energy into high frequencies compared to natural images. Note that we did not observe this phenomenon in older models such as LeNet or VGG.
In the following section, we introduce our method named MACO, which is motivated by this observation. We constrain the magnitude of the visualization natural value, enabling natural visualization for any contemporary model, and reducing the number of required transformations to only two.
3.2 MACO: from Regularization Constraint
Parameterizing the image in the Fourier space makes it possible to directly manipulate the image in the frequency domain. We propose to take a step further and decompose the Fourier spectrum into its polar form re instead of its cartesian form z = + ib, which allows us to disentangle the magnitude (r) and the phase (4p).
It is known that human recognition of objects in images is driven not by magnitude but by phase [44, 45, 46, 47, 48]. Motivated by this, propose optimize the phase of the Fourier spectrum while fixing its magnitude to a typical value of natural image (with few high frequencies). In particular, the magnitude kept constant at the average magnitude computed over a set of natural images (such as ImageNet), so r = ExnD( (aooD((F(a (Fx())) Note that this spectrum needs to be calculated only once and can be used will for other tasks.
Therefore, our method does not backpropagate through the entire Fourier spectrum but only through the phase (Figure 3), thus reducing the number of parameters to optimize by half. Since the magnitude of our spectrum is constrained, we no longer need hyperparameters such λ scaling factors, and the generated image at each step is naturally plausible in the frequency domain. We also enhance the quality of our visualizations via data augmentations: random crop and additive uniform noise. To the best of our knowledge, our approach is the first to completely alleviate the need for explicit regularization using instead a hard constraint on the solution of the optimization problem for feature visualization. To summarize, we formally introduce our method:
Algorithm 1 MACO
Require: r, 90 ~ [M([ W Require: a
for n E [O, do
sample(T)
I, (T F- (re(ps)
Pn+1 = a + | sn L (z
yn + 17 Ly (kn)
end for return xt = F (re
Definition 3.1 (MACO). The feature visualization results from optimizing the parameter vector ?p such that:
60" arg max E (Cu FF F-r((r)))) where r r= ==Ea>D((F()))
PERWWR X
The feature visualization then obtained by applying the inverse Fourier transform the optimal complex-valued spectrum: x" F- =F-1((re(e))
Transparency for free:
Visualizations often suffer from repeated patterns or unimportant elements
in the generated images. This lead to readability problems confirmation biases [54]. It is
5

---

Bottles
Pineapple
Block
Espresso
Green snake
Block
Bottles
Guitar
Block
Beagle
Banana
Block 10
Figure 4: (left) Logits and (right) internal representations of FlexiVIT. MACO was used to maximize the activations of (left) logit units and (right) specific channels located in different blocks of the FiT (blocks 2, 6 and 10 from left to right).
important ensure that the user is looking what truly important in the feature visualization. The concept of transparency, introduced in [53], addresses this issue but induces additional implementation efforts and computational costs.
We propose an effective approach, which leverages attribution methods, that yields a transparency map for the associated feature visualization without any additional cost. Our solution shares theoretical similarities with SmoothGrad [5] and takes advantage of the fact that during backpropagation, we can obtain the intermediate gradients on the input (L for free as dc.(z) (L,(x) de 84p We store these gradients throughout the optimization process and then average them, as done in SmoothGrad, identify the areas that have been modiffed/attended to by the model the most during the optimization process. We note that similar technique has recently been used to explain diffusion models [55]. Algorithm I, we provide pseudo-code for MACO and an example of the transparency maps in Figure 6 (third column).
dx 0ip
4 Evaluation
We now describe and compute three different scores to compare the different feature visualization methods: Fourier (Olah et al.), CBR (optimization the pixel space), and MACO (ours). It is important to note that these scores are only applicable to output logit visualizations, To keep fair comparison, we restrict the benchmark to methods that do not rely on any learned image priors. Indeed, methods with learned prior will inevitably yield lower FID scores (and lower plausibility score) as the prior forces the generated visualizations to lie on the manifold natural images.

---

Plausibility score. We consider feature visualization plausible when is similar to the distribution of images belonging to the class it represents. We quantify the plausibility through an OOD metric (Deep-KNN, recently used in [56]): it measures how far feature visualization deviates from the corresponding ImageNet object category images based on their representation in the network's intermediate layers (see Table 1).
FID score. The FID quantifies the similarity between the distribution of the feature visualizations and that of natural images for the same object category. Importantly, the FIDD measures the distance between two distributions, while the plausibility score quantifies the distance from sample distribution. To compute the FID, we used images from the ImageNet validation set and used the Inception v3 last layer (see Table 1). Additionally, we center-cropped our 512 512 images to 299 x 299 images to avoid the center-bia problem [39].
Transferability score. This score measures how consistent the feature visualizations with other pre-trained classifiers. To compute the transferability score, we feed the obtained feature visualizations into additional pre-trained classifiers (MobileNet [57]. VGG16 [58], Xception [59], EfficientNet [60], Tiny ConvNext [61] and Densenet [62]), and we report their classification accuracy (see Table 2).
All scores are computed using 500 feature visualizations, each of them maximizing the logit one of the ImageNet classes obtained on the FlexiViT [49], ViT[63], and [esNetV2[64] models. For the feature visualizations derived from Olah et al. [1]. we used all 10 transformations set from the Lucid library". CBR denotes an optimization in pixel space and using the same 10 transformations,, as described in [29]. For MACO, T only consists two transformations; first we add uniform noise 6 ~ (1(10.1, 1.1.).1...... W and crops and resized the image with crop size drawn from the normal distribution N (0.25, 25, 0.1), which corresponds on average to 25% the image. We used the NAdam optimizer [65] with iy = 1.0 and N 256 optimization steps. Finally, we used the implementation of [1] and CBR which are available in the Xplique library [66] which based on Lucid.
ViT
ViT
ResNetV2
FiexiViT
VIT
ResNetV2
Plausibility score (1 (I-KNN) (L)
Transferability score (): MACO/
Fourier [1]
MACO
1473
1097
1248
MobileNet
68 / 38
48/ 37
93 36
Fourier [1]
1815
1817
1837
VGG16
64/ 30
50 /0
90/20 20
CBR [29]
1866
1920
1933
Xception
85 / 61
73 / 62
97 64
FID Score (L)
Eff. Net
88 25
63 / 25
82 21
MACO
230.68
241.68
312.66
ConvNext
96 / 52
84 64/55
93 60
Fourier [1]
250.25
257.81
318. 15
DenseNet
84 /32
66 31
93/
CBR [29] 247.12
268.59
346.41
Table 1: Plausibility and FID scores for dif- ferent feature visualization methods applied on FlexiVTT, VIT and ResNetV2
Table 2: Transferability scores for different feature visualization methods applied on FlexiVIT, ViT and ResNetV2.
For all tested metries, we observe that MACO produces better feature visualizations than those generated by Olah el al. [1] and CBR [29]. We would like to emphasize that our proposed evaluation scores represent the first attempt to provide a systematic evaluation of feature visualization methods, but we acknowledge that each individual metric on its own is insufficient and cannot provide comprehensive assessment of method's performance. However, when taken together, the three proposed scores provide more complete and accurate evaluation of the feature visualization methods.
4.1 Human psychophysies study
Ultimately, the goal of any feature visualization method to demystify the CNN's underlying decision process in the eyes of human users. To evaluate MACO 's ability to do this, we closely followed the psychophysical paradigm introduced in [67]. this paradigm, the participants are presented with examples of model's "favorite" inputs (i.e., feature visualization generated for given unit) in addition to two query inputs. Both queries represent the same natural image, but have a
https://gihuuh.co//ennsoont
7

---

FiexiViT
Plausibility (4)
FID (4)
logit magnitude (1)
MACO
571.68
211.0
5.12
transparency
617.9 (+46.2)
208.1 (-2.9)
5.05 (-0.1)
crop
680.1 (+62.2)
299.2 (-91.1)
8.18 (+3.1)
noise
707.3 (+27.1)
324.5 (-25.3)
11.7 (+3.5)
Fourier [1]
673.3
259.0
3.22
augmentations 735.9 (+62.6) 312.5 (+53.5)
12.4 (+9.2)
Table 3: Ablation study on the FlexiVIT model: This reveals that 1. augmentations help to
have better FID and Plausibility scores, but lead to lesser salients visualizations (softmax value), Fourier [1] benefits less from augmentations than MACO.
different part of the image hidden from the model by square occludor. The task for participants to judge which of the two queries would be "favored by the model" (i.c., maximally activate the unit). The rationale here is that good feature visualization method would enable participants to more accurately predict the model's behavior. Here, we compared four visualization conditions (manipulated between subjects): Olah [1], MACO with the transparency mask (the transparency mask is decribed in 3.2), MACO without the transparency mask, and control condition in which no visualizations were provided. In addition, the network (VGGI6, ResNet50, ViT) was within-subject variable. The units to be understood were taken from the output layer.
Based on the data of 174 participants on Pro- lific (www.prolific.com [September 2023], we found both visualization and network to sig- nificantly predict the logodds of choosing the
Average human accuracy
0.9
Control Olah
MACO (with mask)
MACO (no mask)
right query (Fig. 5). That is, the logodds were
0.8
significantly higher for participants in both the
MACO conditions compared to Olah. On the
0.7
other hand, our tests did not yield a significant difference between Olah and the control con- dition, or between the two MACO conditions. Finally, we found that, overall, VIT was signif- icantly harder to interpret than ResNet50 and VGGI6, with no significant difference observed
0.6
0.5
ViT
VGG16
ResNet50
between the latter two networks. Full experi- ment and analysis details can be found in the supplementary materials, section C. Taken to- gether, these results support our claim that even if feature visualization methods struggle in offer- ing interpretable information as networks scale, MACO still convincingly helps people better understand deeper models while Olah's 's method [1] does not.
Network
Figure 5: Human causal understanding of model activations. We follow the experimental procedure introduced in [67] to evaluate Olah and MACO vi- sualizations on 3 different networks. The control condition is when the participant did not see any feature visualization.
4.2 Ablation study
To disentangle the effects of the various components of MACO, we perform an ablation study on the feature visualization applications. We consider the following components: (1) the use of magnitude constraint, (2) the use of the random crop, (3) the use of the noise addition, and (4) the use of the transparency mask. We perform the ablation study on the FlexiViT model, and the results are presented in Table 3. We observe an inherent tradeoff between optimization quality (measured by logit magnitude) on one side, and the plausibility (and FID) scores on the other side. This reveals that plausible images which are close to the natural image distribution do not necessarily maximize the logit. Finally, we observe that the transparency mask does not significantly affect any of the scores confirming that it is mainly post-processing step that does not affect the feature visualization itself.

---

Applications
We demonstrate the versatility of the proposed MACO technique by applying three different XAI upplications:
Logit and internal state visualization. For logit visualization, the optimization objective is to maximize the activation of a specific unit in the logits vector of a pre-trained neural network (here a FlexiViT[[99), The resulting visualizations provide insights into the features that contribute the most to class prediction (refer to Figure 4a). For internal state visualization, the optimization objective is to maximize the activation of specific channels located in various intermediate blocks of the network (refer to Figure 4b). This visualization allows us to better understand the kind of features these blocks of FeexVViT449] in the figure sensitive to.
Feature inversion. The goal of this application is find an image that produces an activation pattern similar to that of a reference image. By maximizing the similarity to reference activations, we are able to generate images representing the same semantic information the target layer but without the parts of the original image that were discarded in the previous stages of the network, which allows us to better understand how the model operates. Figure displays the images (second column) that match the activation pattern of the penultimate layer of VIT when given the images from the first column. We also provide examples of transparency masks based on attribution (third column), which we apply to the feature visualizations enhance them (fourth column).
a)
b)
Broccoli
Orange
Chocolate souce
Figure 6: Feature inversion and concept visualizaiton. a) Images in the second column match activation pattern of the penultimate layer of a VIT when fed with the images of the first column. In the third column, we show their corresponding attribution baased transparency masks, leading to better feature visualization when applied (fourth column). b) MACO is used visualize concept vectors extracted with the CRAFT method [50]. The concepts are extracted from a ResNet50 trained on ImageNet. All visualizations for all ImageNet classes are available Lens.
Concept visualization. Herein we combine MACO with concept-based explainability. Such methods aim to increase the interpretability of activation patterns by decomposing them into a set of concepts [68]. this work, we leverage the CRAFT concept-based explainability method [50], which uses Non-negative Matrix Factorization to decompose activation patterns into main directions that are called concepts -, and then, we apply MACO visualize these concepts in the pixel space. To do so, we optimize the visualization such that it matches the concept activation patterns. In Figure 6b, we present the top most important concepts (one concept per column) for five different object categories (one category per row) in ResNet50 trained on ImageNet. The concepts' visualizations are followed by a mosaic of patches extracted from natural images: the patches that maximally activate the corresponding concept.

---

6 Limitations
We have demonstrated the generation of realistic explanations for large neural networks by imposing constraints on the magnitude of the spectrum. However, it is important to note that generating realistic images does not necessarily imply effective explanation of the neural networks. The metrics introduced in this paper allow us to claim that our generated images are closer to natural images in latent space, that our feature visualizations are more plausible and better reflect the original distribution. However, they do not necessarily indicate that these visualizations helps humans in effectively communicating with the models or conveying information easily to humans. Furthermore, in order for feature visualization to provide informative insights about the model, including spurious features, it may need to generate visualizations that deviate from the spectrum of natural images. Consequently, these visualizations might yield lower scores using proposed metries. Simultaneously, several interesting studies have highlighted the weaknesses and limitations of feature visualizations [54, 69, 67]. One prominent criticism is their lack of interpretability for humans, with research demonstrating that dataset examples are more useful than feature visualizations in understanding convolutional neural networks (CNNs) [54]. This be attributed to the lack of realism in feature visualizations and their isolated use as an explainability technique. With our approach, MACO, we take initial step towards addressing this limitation by introducing magnitude constraints, which lead to qualitative and quantitative improvements. Additionally, through our website, we promote the use of feature visualizations as supportive and complementary tool alongside other methods such as concept-base -bsdd explainability, exemplified by CRAFT [50]. We emphasize the importance of feature visualizations in combating confirmation bias and encourage their integration within comprehensive explainability framework.
7 Conclusions
In this paper, we introduced a novel approach, MACO, for efficiently generating feature visualizations in modern deep neural networks based on (i) hard constraint on the magnitude of the spectrum to ensure that the generated visualizations lie in the space of natural images, and (ii) a new attribution- based transparency mask to augment these feature visualizations with the notion of spatial importance. This enhancement allowed us to scale up and unlock feature visualizations on large modern CNNs and vision transformers without the need for strong and possibly misleading- parametric priors. We also complement our method with a set of three metrics to assess the quality of the visualizations. Combining their insights offers a way to compare the techniques developed in this branch of XAI more objectively. We illustrated the scalability of MACO with feature visualizations of large models like ViT, but also feature inversion and concept visualization. Lastly, by improving the realism the generated images without using auxiliary generative model, we supply the field of XAI with reliable tool for explaining the semantic ("what" information) of modern vision models.
Acknowledgments
This work was conducted as part the DEEL project. It was funded by ONR grant (N00014-19- 1-2029), NSF grant (11S-1912280 and EAR-1925481] and the Artificial and Natural Intelligence Toulouse Institute (ANITI) grant HANRI9-PHE P-0400 The computing hardware was supported in part by NIH Office of the Director grant #S 10OD025181 via the Center for Computation and Visualization (CCV) at Brown University. J.C. has been partially supported by Intel Corporation, grant by Banco Sabadell Foundation and funding from the Valencian Government (Conselleria d' d'Innovació, Industria, Comercio Turismo, Direccion General para el Avance de la Sociedad Digital) by virtue of a 2023 grant agreement (convenio singular 2023).
'http://www.deel..//
10

---

References
[1] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(1 1) 2017.
[2] Finale Moshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. ArXiv e-print, 2017.
[3] Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in ai. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 624 635, 2021.
[4] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. Workshop P Proceedings of the International Conference on Learning Representations (ICLR), 2014.
[5] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. In Workshop on Visualization for Deep Learning, Proceedings of the International Conference on Machine Learning (ICML), 2017.
[6] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das. Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-baset localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[7] Thomas Fel, Remi Cadene, Mathieu Chalvidal, Matthieu Cord, David Vigouroux, and Thomas Serre. Look at the variance! efficient black-box explanations with sobol-based sensitivity analysis. In Advances Neural Information Processing Systems (NeurIPS), 2021.
[8] Paul Novello, Thomas Fel, and David Vigouroux. Making sense dependence: Efficient black-box explanations using dependence measure. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[9] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the International Conference on Machine Learning (ICML). 2017.
[10] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings Purt I 13. pages 818-833. Springer. 2014.
[11] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagat- ing activation differences. In Proceedings of the International Conference on Machine Learning (ICML., 2017.
[12] Ruth C. Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE International Conference Computer Vision (ICCV), 2017.
[13] Mara Graziani, Iam Palatnik de Sousa, Marley MBR Vellasco, Eduardo Costa da Silva, Henning Müller, and Vincent Andrearczyk. Sharpening local interpretable model-agnostic explanations for histopathol- ogy: improved understandability and reliability. In Medical Image Computing and Computer Assisted Intervention (MICCAI). Springer, 2021.
[14] Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. In Proceedings of the British Machine Vision Conference (BMVC), 2018.
[15] Thomas Fel, Melanie Ducoffe, David Vigouroux. Remi Cadene, Mikael Capelle, Claire Nicodeme, and Thomas Serre. Don't lie me! robust and efficient explainability with verified perturbation analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[16] Marouane Il Idrissi, Nicolas Bousquet, Fabrice Gamboa, Bertrand looss, and Jean-Michel Loubes. On the coalitional decomposition of parameters of interest, 2023.
[17] Julien Colin, Thomas Fel, Rémi Cadène, and Thomas Serre. What i cannot predict, i do not understand: A human-centered evaluation framework for explainability methods. Advances Neural Information Processing Systems (NeurIPS), 2021.
[18] Sunnie S. Y. Kim, Nicole Meister, Vikram V. Ramaswamy, Ruth Fong, and Olga Russakovsky. HIVE: Evaluating the human interpretability of visual explanations. In Proceedings of the IEEE European Conference Computer Vision (ECCV), 2022.

---

[19] Giang Nguyen, Daeyoung Kim, and Anh Nguyen. The effectiveness of feature attribution methods and its correlation with automatic evaluation scores. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[20] Anh Nguyen, Jason Yosinski, and Jeff Clune. Understanding neural networks via feature visualization; survey. Explainable Al: interpreting, explaining and visualizing deep learning, pages 55-76, 2019.
[21] Leon Sixt, Martin Schuessler, Oana-Tuliana Popescu, Philipp WeiB, and Tim Landgraf. Do users benefit from interpretable vision? a user study, baseline, and dataset. Proceedings of the International Conference on Learning Representattinss (ICLR), 2022.
[22] Peter Hase and Mohit Bansal. Evaluating explainable ai: Which algorithmic explanations help users predict model behavior? Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL Short Papers), 2020.
[23] Leon Sixt. Maximilian Granz, and Tim Landgraf. When explanations lie: Why many modified bp attributions fail. In Proceedingy of the International Conference on Machine Learning (ICML), 2020.
[24] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems (NIPS), 2018.
[25] Dylan Slack, Anna Hilgard, Sameer Singh, and Himabindu Lakkaraju. Reliable post hoc explanations: Modeling uncertainty in explainability. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[26] Sukrut Rao, Moritz Böhle, and Bernt Schiele. Towards better understanding attribution methods. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[27] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 1rXiv:1312.6199. 2013.
[28] Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference computer vision and puttern recognition, pages 5188-5196.
2015.
[29] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 427-436, 2015.
[30] Mike Tyka. Class visualization with bilateral filters. 2016.
URL: https://mtyka. github.
html, 2(3), 2016.
[31] M. Øygard Audun. Visualizing googlenet classes. URL: http::/www.aada...c.oom////////a cloglenet-classes, 2(3), 2015.
[32] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robust- ness may be at odds with accuracy. arXiv preprini arXiv:1805.12152 2018
[33] Shibani Santurkar, Andrew Ilyas, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Image synthesis with single (robust) classifier. Advances in Neural Information Processing Systems, 32, 2019.
[34] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness as a prior for learned representations. arXiv preprint arXiv: 0000000 2019.
[35] Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks. https://blog.eeseant httpchhhoooogg000000////oeeeoooe toint--oonnggiieee deeengddeepeiiinee intteeerrntt--nnaaltaa html mm= 2015.
[36] Amin Ghiasi, Hamid Kazemi, Steven Reich, Chen Zhu, Micah Goldblum, and Tom Goldstein. Plug-in inversion: Model-agnostic inversion for vision with data augmentations. Proceedings of the International Conference on Machine Learning (ICML), 2021.
[37] Amin Ghiasi, Hamid Kazemi, Eitan Borgnia, Steven Reich, Manli Shu, Micah Goldblum, Andrew Gordon Wilson, and Tom Goldstein. What do vision transformers learn? a visual exploration. preprint, 2022.
[38] Donglai Wei, Bolei Zhou, Antonio Torrabla, and William Freeman. Understanding intra-class knowledge inside enn, arXiv preprint arXiv: 2015.
12

---

[39] Anh Nguyen, Jason Yosinski, and Jeff Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. Visualization, for Deep Learning workshop. Proceedings of the International Conference on Machine Learning (ICML), 2016.
[40] Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in neural information processing systems, 29, 2016.
[41] Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug & play generative networks: Conditional iterative generation of images in latent space. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4467-4477, 2017.
[42] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[43] A Dosovitskiy, L Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, M Dehghani, Marthias Minderer, G Heigold, S Gelly, Jakob Uszkoreit, and N Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. ICLR, 2021.
[44] Alan V Oppenheim and Jac Lim. The importance of phase in signals. Proceedings of the IEEE, 69(5) 541, 1981.
[45] Terry Caelli and Paul Bevan. Visual sensitivity to two-dimensional spatial phase. JOSA. 72(10):1375-1331 1982.
[46] Nathalie Guyader, Alan Chauvin, Carole Peyrin, Jeanny Hérault, and Christian Marendaz. Image phase or amplitude? rapid scene categorization is amplitude-based process. Comptes Rendus Biologies, 327(4):313 318, 2004.
[47] Olivier R Joubert, Guillaume A Rousselet, Michele Fahre- Fahre-Thorpe, and Denis Fize. Rapid visual catego- rization of natural scene contexts with equalized amplitude spectrum and increasing phase noise. Journal of Vision, 9(1 21):2-2,22 2009.
[48] Evgeny Gladilin and Roland Eils. On the role of spatial phase and phase correlation in vision, illusion, and cognition. Frontiers in Computational Neuroscience, 9:45, 2015.
[49] Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. Flexivit: One model for all patch sizes. arXiv preprint arXiv:22 2022.
[50] Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Seme. Craft: Concept recursive activation factorization for explainability. Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition (CVPR), 2022.
[51] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. University of Montreal, 13:1(3):1, 2009.
[52] Guangrun Wang and Philip HS Torr. Traditional classification neural networks are good generators: They are competitive with ddpms and gans. arXiv preprint 2rXiv 2022.
[53] Alexander Mordvintsev, Nicola Pezzotti, Ladwig Schubert, and Chris Olah. Differentiable image parame- terizations. Distill, 2018
[54] Judy Borowski, Roland S Zimmermann, Judith Schepers, Robert Geirhos, Thomas SA Wallis, Matthias Bethge, and Wieland Brendel. Exemplary natural images explain enn activations better than tate-of-the-art feature visualization. arXiv preprint arXiv:2010. /2606, 2020.
[55] Victor Boutin, Thomas Fel, Lakshya Singhal, Rishav Mukherji, Akash Nagaraj. Julien Colin, and Thomas Serre. Diffusion models artists: Are we closing the gap hetween humans and machines? arXiv preprint arXiv:2301. 2023.
[56] Yiyou Sun, Yifci Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distributiin detection with deep nearest neighbors. In International Conference on Machine Learning, pages 20827-20840 PMLR, 2022.
[57] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704. 04861, 2017.
13

---

[58] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint urXiv:1409.155 1556, 2014.
[59] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEÉ conference computer vision and pattern recognition, pages 1251-1258, 2017.
[60] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference machine learning, pages 6105-61 14. PMLR, 2019.
[61] Zhuang Liu, Hanzi Mao, convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1196-119866 2022.
Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A
[62] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4700-4708, 2017.
[63] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Computer Vision - ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part V. page 491-507, Berlin, Heidelberg, 2020. Springer- Verlag.
[64] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. 2016: 14th European Conference, Amsterdam, The Netherlands, October
In Computer 11-14, 2016, Proceedings, Part 14, pages 130-645. Springer, 2016.
[65] Timothy Dozat. Incorporating nesterov momentum into adam. Proceedings the International Conference on Learning Representations (ICLR), 2016.
[66] Thomas Fel, Lucas Hervier, David Vigouroux, Antonin Poche, Justin Plakoo, Remi Cadene, Mathicu Chalvidal, Julien Colin, Thibaut Boissin, Louis Bethune, Agustin Picard, Claire Nicodeme, Laurent Gardes, Gregory Flandin, and Thomas Serre. Xplique: A deep learning explainability toolbox. Workshop on Explainable Artificial Intelligence for Computer Vision (CVPR), 2022.
[67] Roland S Zimmermann, Judy Borowski, Robert Geirhos, Matthias Bethge, Thomas Wallis, and Wicland Brendel. How well do feature visualizations support causal understanding of cmn activations? Advances in Neural Information Processing Systems, 34:1 11 10110 11744, 2021.
[68] Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. Advances in Neural Information Processing Systems, 32, 2019.
[69] Robert Geirhos, Roland S. Zimmermann, Blair Bilodcau, Wieland Brendel, and Been Kim. Don't 't trust your eyes: on the (un)rcliability of feature visualizations, 2023.
[70] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hicrarchical Image Database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
14


Reviews:
[{"review_id": "8BqT1KLlxW", "reviewer": "Reviewer_Gqq4", "summary": "This paper proposes a simple yet effective approach to visualize deep learning features without using a parametric image prior. Improving upon the prior work that uses the Fourier domain formulation, the proposed method uses the polar form and optimizes the image phase to maximize the activation of specified neurons. The rest of the pipeline is similar to the prior work in terms of the optimization procedure and the transparency mask estimation. The paper further designs two new metrics for feature visualizations, where the proposed method outperforms the prior arts. In addition to feature visualization, the proposed method is applied to feature inversion of single images or learned concepts.", "questions": "Are there any failure cases of the proposed method?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes a clever idea to optimize the phase of the image, which is commonly used for image editing, to make it look more natural for feature visualization. \n\n- The qualitative result comparison clearly shows the significant improvement of the proposed method over the prior arts without using parametric image priors.\n\n- The paper proposes two new metrics to compare feature visualization methods from different perspectives.\n\n- The paper is well-written and the figures are helpful for understanding.", "weaknesses": "Major\n----\n- Lack of significant novelty. The other two contributions claimed in the paper are less convincing: the second contribution is proposed in the SmoothGrad paper as acknowledged by the paper, and the third contribution on metrics isn't thorough in its analysis.\n\n- Lack of thorough analysis of the proposed new metrics.\n    - Overall, all inverted images without a parametric image prior look so different from natural images. Thus, it's unclear if it's meaningful to show which method is slightly better via the plausible and FID metric. It's different from using FID to judge image generation or style transfer results.\n    - For the plausibility/FID metric, in addition to image appearance, the image content of feature inversion results are so different from natural images. As the proposed model aims to maximize the activation of a neuron at all spatial locations, the inverted images are a collage of discriminative object parts, which is not similar to natural images. It may be better to just maximize the neuron at the center spatial location to invert the feature into one object part. \n    - For the transferability score, the experiment is only done on 500 features, while ImageNet has 1,000 classes for the classification test. It's unclear how robust and reliable is the reported result. Note that there are 200 classes of dogs in ImageNet, and thus the transferability score may be different for fine-grain categories.\n\n- Lack of comparison on scene image datasets ,e.g. PLACES. The phase of outdoor images are significantly different from indoor images. It'll be more convincing to also benchmark on scene image datasets.\n\n- Lack of ablation study on the augmentation of the two transformations.\n\nMinor\n----\n- Lack of runtime analysis. It's unclear if the proposed method is fast to run in practice\n\n- Lack of analysis on the random seed. It'll be good to show the diversity of results using different random seeds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for dedicating their time and for their kind words regarding our work.\n\n## Major weaknesses\n\n- **Lack of significant novelty. The other two contributions claimed in the paper are less convincing: the second contribution is proposed in the SmoothGrad paper as acknowledged by the paper, and the third contribution on metrics isn't thorough in its analysis.**\n\nRegarding the novelty of our work, it's essential to emphasize that (1) feature visualization on recent models is not feasible without using MACO. This tool unlocks a better understanding of these models.\n\nIn addition, (2) SmoothGrad does not address or talk about feature visualization and unlike SmoothGrad, we link attribution methods with feature visualization which is non-trivial, novel, and useful for the interpretability of feature visualization\nThe only overlap with SmoothGrad is in that we use an average of gradients, but unlike SmoothGrad, we do not use gradients with respect to sampled noise, but rather we use the gradients obtained at each of MACO’s optimization steps to compute the transparency masks. \n\nMoreover, (3) feature visualization is a sub-field of XAI that is lacking accessible, reproducible quantitative benchmarks. Our work introduces a group of three metrics that serve as a comprehensive benchmark, thus acknowledging the challenge of designing such metrics for a field that can often rely too much on qualitative assessments, and initiating the work towards an accessible and reliable benchmark for feature visualizations.\n\nFinally, we would like to add that the anonymously released website stands as a valuable contribution on its own.\n\n- **Lack of thorough analysis of the proposed new metrics.**\n\n\t- **Overall, all inverted images without a parametric image prior look so different from natural images. Thus, it's unclear if it's meaningful to show which method is slightly better via the plausible and FID metric. It's different from using FID to judge image generation or style transfer results.**\n\n\t- **For the plausibility/FID metric, in addition to image appearance, the image content of feature inversion results are so different from natural images. As the proposed model aims to maximize the activation of a neuron at all spatial locations, the inverted images are a collage of discriminative object parts, which is not similar to natural images. It may be better to just maximize the neuron at the center spatial location to invert the feature into one object part.**\n\n\t- **For the transferability score, the experiment is only done on 500 features, while ImageNet has 1,000 classes for the classification test. It's unclear how robust and reliable is the reported result. Note that there are 200 classes of dogs in ImageNet, and thus the transferability score may be different for fine-grain categories.**\nRegarding the metrics, we have provided a detailed response in our general message (point 1). In summary, we agree with the reviewer’s  point that each metric, taken independently, may not be perfect, hence the necessity to not consider them individually but as a set.\n\nWe are not sure we understand  what the reviewer means by “maximize the neuron at the center spatial location”.  If the reviewer means optimizing with respect to a central crop of the input, please note that in that case, we would have to replace the rest of the image with an arbitrary value, significantly affecting the behavior of the model and probably making the input images OOD (since a high percentage of the pixels would be equal to some non-natural arbitrary value). \n\n- **Lack of comparison on scene image datasets, e.g. PLACES. The phase of outdoor images is significantly different from indoor images. It'll be more convincing to also benchmark on scene image datasets.**\n\nRegarding ImageNet, there are 118 classes specifically for dogs. For the benchmark, the 500 classes were chosen randomly. We encourage the reviewer to explore the website to verify the results for both fine-grained and less fine-grained categories.\n\n**- Lack of ablation study on the augmentation of the two transformations.**\n\nMuch in the same way than in Olah et al., MaCo needs the proposed data augmentations to regularize the optimization procedure and allow it to converge to human-interpretable solutions.\n\n## Minor weaknesses\n\n- **Lack of runtime analysis. It's unclear if the proposed method is fast to run in practice**\n- **Lack of analysis on the random seed. It'll be good to show the diversity of results using different random seeds.**\n\nBy reducing the amount of augmentations and dividing the number of parameters to optimize by half, we obtain a technique that is faster than Olah et al. Taking for example a BEiT Base with a patch size of 16, we can generate visualizations in less than 3 minutes (on a T4 GPU on Google Collab). For much smaller networks, this time is reduced to about 20 seconds per visualization.\n\nAs for the visualizations using different random seeds, we take advantage of the additional rebuttal one-page pdf to showcase some visualizations for some classes using different random seeds on different models.\n"}, {"review_id": "q25sIBqmTO", "reviewer": "Reviewer_6q7R", "summary": "The paper proposes MACO, a feature visualization method which focuses the generation process of the visualizations on the image's phase spectrum while keeping the magnitude constant. This makes the generated explanation visualizations lie in the space of natural images, and hence, clearer for visual inspection.\n\nIn addition three metrics are proposed, i.e. plausability, FID and transferability, aimed at the quantitative comparison of the visualizations produced by feature visualization methods.\n\nThe proposed method is evaluated in several architectures, e.g. FlexiViT, ViT, ResNetV2, and compared against related methods that do not rely on image priors.", "questions": "- Please see W1\n- What would the Plausability and FID metrics report on average images and methods that exploit image priors?\n- What is the main motivation behind the transferability metric? In the way it is defined, how would it help assess the validity of the visualizations and its relationship with respect to the model it was extracted from?", "limitations": "- Visualizations somewhat biased towards human perception/recognition principles (risk of low fidelity to the model being interpreted)\n- While a good first attempt and step on the right direction, the proposed quantitative metrics might not be the most optimal.\n- Good part of the evaluation is qualitative.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Presentation of the manuscript is good, some of the design choices are properly motivated.\n- The proposed method is validated in a variety of architectures.\n- The propose method effectively removes the need for hyperparameters and explicit regularization.", "weaknesses": "- W1: The proposed method for generating the explanation visualizations raises from the observation that human recognition of objects in images is driven not by magnitude but by phase (l.190). By following this principled aren't we assuming that the model being explained operates in the same manner as humans? aren't we risking hiding features potentially useful by the model and not necessarily following human-related processes?\n\n- W2: The manuscript proposes three metrics/scores (plausibility, FID and transferability) to quantitatively compare the feature visualizations produced by different methods.\n\nPlausability and FID, are related to the quantification of the similarly between the distribution of a given visualization (or set of visualizations) from a given class, and the distribution of natural images from that class. While the notion behind these two metrics is clear, especially for FID, I am not sure how sensitive these two metrics might be to trivial inputs, e.g. an average image of the class of interest? I would expect this input to get a very good value on the two. Perhaps a similar trend might be observed for methods that exploit image priors.\n\ntransferability, is related to the activations/prediction triggered by the visualizations under evaluation when fed to other models trained on the same datasets. In this regard, I fail to see how following this procedure will help 1) assess the validity of the visualization and 2) its relationship with respect to the model it was extracted from.\n\nIn l.262 it is claimed that the three proposed scores provide a more complete and accurate evaluation of the feature visualization methods, based on the observations made above, this statement might be inaccurate\n\n- W3: Besides the quantitative evaluation using the three proposed metrics, with results reported in Table 1 and Table 2, all the other sources of validation are purely qualitative.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to read our article and for the valuable feedback.\n\n ## Weaknesses\n\n- **W1: The proposed method for generating the explanation visualizations raises from the observation that human recognition of objects in images is driven not by magnitude but by phase (l.190). By following this principle aren't we assuming that the model being explained operates in the same manner as humans? aren't we risking hiding features potentially useful by the model and not necessarily following human-related processes?**\n\nRegarding the optimization phase, the reviewer is absolutely right, and we have acknowledged this limitation in the limitations section of Appendix A. However, the end goal of feature visualization is to provide human-interpretable explanations. If frequencies that are not visible to the human eye are present in the feature visualization, they will not be conveyed and comprehended by humans. MACO proposes to maximize the objective without assuming that it operates like humans – as evidenced by the visualizations that are far from looking like natural images–while adhering to understandable features for humans. In summary, avoiding unrealistic frequencies is both a limitation and a feature of the approach. In addition to this limitation, our method is less constrained than generative-based feature visualization methods for which it is impossible to untangle the true contributions of the model under study from those of the generative prior model [1].\n\n- **W2: The manuscript proposes three metrics/scores (plausibility, FID and transferability) to quantitatively compare the feature visualizations produced by different methods.**\n\n**Plausability and FID, are related to the quantification of the similarity between the distribution of a given visualization (or set of visualizations) from a given class, and the distribution of natural images from that class. While the notion behind these two metrics is clear, especially for FID, I am not sure how sensitive these two metrics might be to trivial inputs, e.g. an average image of the class of interest? I would expect this input to get a very good value on the two. Perhaps a similar trend might be observed for methods that exploit image priors.**\n\n**transferability, is related to the activations/prediction triggered by the visualizations under evaluation when fed to other models trained on the same datasets. In this regard, I fail to see how following this procedure will help 1) assess the validity of the visualization and 2) its relationship with respect to the model it was extracted from.**\n\n**In l.262 it is claimed that the three proposed scores provide a more complete and accurate evaluation of the feature visualization methods, based on the observations made above, this statement might be inaccurate**\n\nRegarding the metrics, we have provided a detailed response in our general message (point 1). In summary, we agree with the reviewer’s point that each metric, taken independently, may not be perfect, but together, they form a coherent whole. \n\nWe tested the reviewer’s  idea of assessing the metric for the average of each class on 30 random classes, and it turns out that, concerning the reviewer’s example, the baseline score for Plausibility is bad (1977 for ResNet) and the transferability score is at zero.\n\n- **W3: Besides the quantitative evaluation using the three proposed metrics, with results reported in Table 1 and Table 2, all the other sources of validation are purely qualitative.**\n\nThe field of Feature Visualization suffers from this drawback that there is no accessible, reproducible quantitative metrics available. Previous contributions – accepted by the community – only relied on qualitative assessment [1,16,19,20], or quantitative based on psychophysical, hard-to-reproduce experiments [47,48]. We agree that it is a serious drawback, and that is why we introduce quantitative metrics as a first step towards a more rigorous and accessible assessment. On the other hand, explainability always involves human interpretability, and that is why qualitative assessment is as important as quantitative assessment. Hence, we do not see the fact that “good part of the evaluation is qualitative” as a limitation, as long as it is accompanied by quantitative ones.\n\n## Questions\n\n- **What would the Plausability and FID metrics report on average images and methods that exploit image priors?**\n- **What is the main motivation behind the transferability metric? In the way it is defined, how would it help assess the validity of the visualizations and its relationship with respect to the model it was extracted from?**\n\nThe Plausibility and FID metrics would prove those images to be better than the feature visualizations generated via MaCo, but they would depend much less (or not at all) on the model under study. As for the transferability metric, its main goal is to make sure that the features highlighted by the visualizations are universal to other models (and thus, possibly to humans as well).\n\n## Limitations\n\n- **Visualizations somewhat biased towards human perception/recognition principles (risk of low fidelity to the model being interpreted)**\n- **While a good first attempt and step on the right direction, the proposed quantitative metrics might not be the most optimal.**\n- **Good part of the evaluation is qualitative**\n\nAs for the limitations, we invite the reviewer to take a look at our limitations section in Appendix A, that we will also move into the main paper upon acceptance."}, {"review_id": "uymyfAq15J", "reviewer": "Reviewer_qNv3", "summary": "This paper proposes a small but apparently effective modification to the common technique of feature visualizations to improve the visual quality of the produced explanations. Specifically, this approach uses a different image parameterization producing images with power spectra similar to that of natural/dataset images.", "questions": "- L161: The transformations used by Olah et al. can also be summarized as just a few, namely random cropping with zoom and rotations.\n- L179: Showing a visual comparison would be appreciated.\n- L202ff: Motivating why these augmentations are used/needed would be great. This sounds somehow contracting the earlier paragraphs where the authors say they want to remove these augmentations.\n- L205f: This is a bit misleading; Olah et al. also didn't use explicit regularizers.\n- L212: Unclear what is meant by confirmation bias in this context and how the proposed method addresses this.\n- L216ff: Why is such a visualization more helpful/better for a user than the unmasked one? Performing a user study for this could be helpful.\n- L233f: Using the plausibility score to judge a visualization appears to assume that the classifier is doing a great job and learns something that is perfectly aligned with how humans perform this classification. ", "limitations": "- The authors do not discuss any limitations in the main part of their paper.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The modification to the feature visualization procedure is well-motivated and described. The method is easy to understand.", "weaknesses": "- Section 4: The authors claim that they are the first to introduce quantitative methods for comparing feature visualizations. This is, however, incorrect as Borowski et al. [47] and Zimmermann et al. [48] already introduce evaluation schemes for measuring the helpfulness of visualizations.\n- This study does not include any psychophysical evaluation/study investigating the helpfulness of the produced visualizations for humans. This is, however, a crucial part when developing new XAI methods as was noted by different earlier studies. I encourage the authors to test their method using the quantitative psychophysical benchmark introduced by Borowski et al. 2020, as this will drastically increase my confidence in this work and, hence, also my rating.\n- While the authors list an interactive website as one of their contributions, we sadly cannot access and test it. While I understand that granting us access to this without risking a violation of the double-blind peer review process is difficult, I'm feeling uncomfortable giving the paper a great rating (based on this claimed contribution) without being able to review this website, too.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to read our article, and we sincerely appreciate their review. \n\n- **Section 4: The authors claim that they are the first to introduce quantitative methods for comparing feature visualizations. This is, however, incorrect as Borowski et al. [47] and Zimmermann et al. [48] already introduce evaluation schemes for measuring the helpfulness of visualizations.**\n\n- **This study does not include any psychophysical evaluation/study investigating the helpfulness of the produced visualizations for humans. This is, however, a crucial part when developing new XAI methods as was noted by different earlier studies. I encourage the authors to test their method using the quantitative psychophysical benchmark introduced by Borowski et al. 2020, as this will drastically increase my confidence in this work and, hence, also my rating.**\n\nWe agree with qNv3, that Borowski et al [47] and Zimmermann et al [48] have provided insightful contributions to evaluate feature visualization techniques. However, the metrics proposed are all based on psychophysics that require heavy experimental protocols and that need to be re-conducted on every new network one wants to evaluate. Those psychophysics experiments are hardly compatible with the high pace of new networks in the ML field. As opposed to [47] and [48] we wanted to propose metrics that are easily computable, scalable to novel networks and that could easily serve as a means of comparison for the entire XAI community. In the end, we see these metrics as complementary to psychophysics-based evaluations that are used for determining the helpfulness of feature visualizations to humans. This is why conducting such an experiment is one avenue for future work, but we consider it to be beyond the scope of this paper, that is focused on MACO as an improvement of existing Feature Visualization techniques.\n\n- **While the authors list an interactive website as one of their contributions, we sadly cannot access and test it. While I understand that granting us access to this without risking a violation of the double-blind peer review process is difficult, I'm feeling uncomfortable giving the paper a great rating (based on this claimed contribution) without being able to review this website, too.**\n\nWe have updated an anonymized version here: https://anon1285.github.io/neurips1285/classes/espresso/. You can browse all 1000 imagenet’s classes using the search bar at the top or the menu at the left.\n\n- **L161: The transformations used by Olah et al. can also be summarized as just a few, namely random cropping with zoom and rotations.**\n\nThe official implementation of Olah et. al [1] utilizes a cascade of 10 augmentations, whereas we only use 2.\n\nhttps://github.com/tensorflow/lucid/blob/6dcc927e4ff4e7ef4d9c54d27b0352849dadd1bb/lucid/recipes/activation_atlas/render.py#L130\n\n- **L179: Showing a visual comparison would be appreciated.**\n\nA visual comparison is already main in Figure 1 of the main manuscript (Fourier FV). We will clarify upon acceptance that Fourier FV refers to the methods of Olah et al.\n\n- **L202ff: Motivating why these augmentations are used/needed would be great. This sounds somehow contracting the earlier paragraphs where the authors say they want to remove these augmentations.**\n\nThe simplicity of our method allows us to drastically reduce the number of augmentations (from 10 to 2), and therefore the number of hyperparameters, but we still need those two augmentations to stabilize the optimization procedure.\n\n- **L205f: This is a bit misleading; Olah et al. also didn't use explicit regularizers.**\n\nIt is true that although [1] uses many regularizations, those might not be considered explicit. We removed the word “explicit” to avoid confusion.\n\n- **L212: Unclear what is meant by confirmation bias in this context and how the proposed method addresses this**\n- **L216ff: Why is such a visualization more helpful/better for a user than the unmasked one? Performing a user study for this could be helpful.**\n\n(L212 - L216) Without our method, “Visualizations often suffer from repeated patterns or unimportant elements in the generated images.”. In that case, the confirmation bias would consist in interpreting unimportant elements as meaningful explanations. Our methods alleviates this problem by masking unimportant elements.\n\n- **L233f: Using the plausibility score to judge a visualization appears to assume that the classifier is doing a great job and learns something that is perfectly aligned with how humans perform this classification.**\n\nApologies for any confusion caused by the naming of the metric. It is not about plausibility with respect to humans but rather with respect to the model (OOD). This metric quantifies the extent to which the internal feature paths are conventional (plausible) for maximizing the objective (see the motivation described in the general message, point 1.).\n\n## Limitations\n\n- **The authors do not discuss any limitations in the main part of their paper.**\n\nAppendix A is entirely dedicated to a thorough discussion of the limitations. We did not include it in the main text because of the page limit but we plan to do so upon acceptance – thanks to the extra page."}, {"review_id": "NrO9wZiChV", "reviewer": "Reviewer_UD3m", "summary": "## Summary\n\nThis paper presents MACO, a simple and novel method to generate feature visualization for deep neural networks. MACO uses a constraint to optimize feature visualization in the space of natural images. It also presents an attribution-based mask to augment generated images to highlight spacial importance. To evaluate MACO, three metrics (transferability, plausibility, and alignment with natural images) are proposed. Evaluation study shows that MACO generates more useful feature visualization and is scalable to modern vision models such as ViT.", "questions": "1. How does MACO improve ML explainability?\n2. How would an ML practitioner/researcher use MACO to improve their ML models?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "## Strengths \n\nS1: This paper aims to improve the existing feature visualization method, which is popular in explaining vision models. This is a very critical and timely research topic.\n\nS2: The proposed method MACO is sound, and the intuition of optimizing for image phase over magnitude is reasonable. It's good that MACO doesn't need to use any strong parametric image priors.\n\nS3: The paper is well-written and easy to follow. I am looking forward to seeing the interactive visualization website!", "weaknesses": "## Major weaknesses\n\nW1: The evaluation can be improved. The goal of feature visualization is to help human users interpret computer vision models. However, I don't see a strong connection between the proposed evaluation scores (plausibility, FID, and transferability scores) and explainability. These scores assume that *feature visualizations more similar to natural images are better at explaining ML models.* However,  this paper does not provide strong evidence that this hypothesis is true. This paper would be much stronger if it can directly highlight MACO improves explainability.\n\nW2: Related to W1, a user study is necessary for this paper. I found some of the figures very hard to interpret. Studying how real users (e.g., ML developers and researchers) would interpret and use MACO-generated feature visualizations in their workflows would yield many interesting insights.\n\n## Minor weaknesses\n\nM1: The abstract can be improved by giving more details about the MACO technique. The current one-sentence, \"It consists in optimizing solely an image's phase spectrum while keeping its magnitude constant to ensure that the generated explanations lie in the space of natural images.\" might be too abstract.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we thank the reviewer for taking the time to read our article.\n\n## Major\n\n**W1: The evaluation can be improved. The goal of feature visualization is to help human users interpret computer vision models. However, I don't see a strong connection between the proposed evaluation scores (plausibility, FID, and transferability scores) and explainability. These scores assume that feature visualizations more similar to natural images are better at explaining ML models. However, this paper does not provide strong evidence that this hypothesis is true. This paper would be much stronger if it can directly highlight MACO improves explainability.**\n\nNote that the scores do not only assume that visualizations more similar to natural images are better at explaining ML models. This statement is true for FID, but plausibility, based on OOD score, measures how close the visualizations are to the features that the model has constructed during the training, whether natural or not. As for transferability, this score ensures that the visualization is not excessively adversarial. As emphasized in the general comment, the proposed metrics are to be considered together, not independently. \nNote that the naturalness of feature visualization is known to be highly correlated with their interpretability [47].\n\nTaking a look at our website (e.g. https://anon1285.github.io/neurips1285/classes/espresso/), it is clear that feature visualization is complementary to concept-based techniques [29, 46], which illustrates how it can improve explainability.\n\n**W2: Related to W1, a user study is necessary for this paper. I found some of the figures very hard to interpret. Studying how real users (e.g., ML developers and researchers) would interpret and use MACO-generated feature visualizations in their workflows would yield many interesting insights.**\n\nOne interesting follow up for future work would indeed be to evaluate those metrics using human psychophysics. Many reviewers have suggested this idea, and we acknowledge the necessity of such an experiment but we consider it beyond the scope of the present  paper. As an example, [47, 48] are fully dedicated to such an evaluation, showing that it is a whole research work on its own. Rather, as a first step this paper focuses on MACO as an improvement over existing Feature Visualization techniques.\n\n## Minor\n\n**M1: The abstract can be improved by giving more details about the MACO technique. The current one-sentence, \"It consists in optimizing solely an image's phase spectrum while keeping its magnitude constant to ensure that the generated explanations lie in the space of natural images.\" might be too abstract.**\n\nWe rephrased the sentence and added it to the manuscript. We thank the reviewer for the suggestion.\n\n## Questions\n\n**1- How does MACO improve ML explainability?**\n\n**2- How would an ML practitioner/researcher use MACO to improve their ML models?**\n\nMACO is a Feature Visualization technique, so its use in ML explainability for practitioners would be the same as other classical Feature Visualization techniques (finding spurious correlation, understanding the inner workings of the model,...). Importantly, it might not be sufficient to fully explain a model, and rather has to be taken as a tool towards making models more interpretable. A good example of how it can be used in combination with other XAI tools is the website that showcases its complementarity with concept based explanations ( https://anon1285.github.io/neurips1285/)."}, {"review_id": "3P2oqfaId0", "reviewer": "Reviewer_mBBq", "summary": "The paper proposes MACO, a feature visualization technique that optimizes image phase but keeps the magnitude of the Fourier spectrum constant. This method is similar, in spirit, to the 'classic' feature visualizations by Olah et al. (2017), while at the same time enabling feature visualizations for architectures where the classic approach previously struggled with (such as vision transformers or other non-VGG architectures). The paper lists three core contributions:\n\n1. MACO, the new feature visualization technique,\n2. Combining feature visualization with feature attribution,\n3. Quantitative metrics to measure the success of the proposed method\n\nOverall, (1.) can be seen as the most significant contribution whereas (2.) is an application and (3.) is a supporting contribution.", "questions": "1. How can the feature attribution + feature visualization aspect be quantified and falsified? (see above)\n\n2. For every single feature visualization figure shown in the paper and the supplementary materials, how were the units chosen? If at random, great; if not I would encourage the authors to either update those figures with randomly chosen classes (and channels etc.) or to state, in the figure caption, that they were not chosen randomly (a.k.a. cherry-picked).\n\n3. Are you planning to open-source the technique & code, e.g. as a standalone library / contribution to an existing library to facilitate using the method? (Not required of course, but might be something worth thinking about)\n\nI am willing to increase my score if my concerns can be addressed; in particular regarding feature attribution + feature visualization, random vs. cherry-picked visualizations, and toning down the claim a bit.\n\nMinor:\n\n3. In line 37, it would be useful to define 'faithful'\n\n4. In lines 178-180, it would be useful to quantify this statement, e.g. in the appendix", "limitations": "Appendix A contains a great discussion on potential limitations, including the fact that \"For a feature visualization to be truly informative about the model (e.g., about those spurious features), it might need to generate visualizations that stray away from the spectrum of natural images, and thus, yield worse scores using our metrics.\"", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper is very well written and a pleasure to read\n2. The proposed approach, MACO, is refreshingly simple (e.g., fewer transformations needed to make it work) and has a clear motivation from human perception (importance of phase spectrum for human perception).\n3. Making feature visualizations work for a broader range of architectures is a desirable goal, and there hasn't been much progress in this direction over the last few years\n4. The authors make the findings explorable via a website (currently screenshots in the Appendix)\n5. The paper contains a quantitative evaluation for at least some findings. The 'plausibility score' and the 'FID score' both make a very big assumption (natural images = best, which may not be the case for the model: in fact, in the introduction MACO is proposed exactly since the feature visualizations based on generative models like GANs make this assumption). Nevertheless, in a field that's predominantly evaluated 'by eye' (nice-looking visualizations) it's great that they perform at least some quantitative evaluations.", "weaknesses": "1. The strongest contribution is MACO itself. The approach of combining attribution and feature visualization, on the other hand, sounds like a great idea if at least one of those methods, and ideally both, would be reliable. In particular, many attribution methods are deeply problematic [1,2]. The particular method employed in the paper is similar to SmoothGrad which often produces attributions that are as reliable as random guessing [2]. I very much hope that the approach of combining attribution and feature visualization can be backed up by quantitative, falsifiable evaluation. If it cannot, I would be inclined to question its inclusion in the paper.\n\n2. The paper makes the big assumption that feature visualizations which resemble natural images are better (explained above), which is a prior that's independent of the model and can thus lead to misleading visualizations if the assumption isn't met by the investigated model.\n\n3. The claim that \"we supply the field of XAI with a reliable tool for explaining the semantic ('what' information) of modern vision models\" sounds a bit too strong in my ears. Given the well-known limitations of feature visualizations (e.g., hard to falsify [3], hard to quantify, unreliable [very recent work 4]), I think it might be worth formulating this a bit more carefully, for instance as \"we supply the field of XAI with a tool for exploring hypotheses about the semantic ('what' information) of modern vision models\".\n\n[1] Zhou et al. https://ojs.aaai.org/index.php/AAAI/article/view/21196/20945\n\n[2] Bilodeau et al. https://arxiv.org/pdf/2212.11870.pdf\n\n[3] Leavitt & Morcos https://arxiv.org/pdf/2010.12016.pdf\n\n[4] Geirhos et al. https://arxiv.org/pdf/2306.04719.pdf\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and for providing this detailed review with very relevant comments.\n\n## Weaknesses:\n\n**The strongest contribution is MACO itself. The approach of combining attribution and feature visualization, on the other hand, sounds like a great idea if at least one of those methods, and ideally both, would be reliable. In particular, many attribution methods are deeply problematic [1,2]. The particular method employed in the paper is similar to SmoothGrad which often produces attributions that are as reliable as random guessing [2]. I very much hope that the approach of combining attribution and feature visualization can be backed up by quantitative, falsifiable evaluation. If it cannot, I would be inclined to question its inclusion in the paper.**\n\nThe only overlap with SmoothGrad is in that we use an average of gradients. However, unlike SmoothGrad, we do not use gradients with respect to sampled noise, but rather use the gradients obtained at each MACO’s optimization steps to compute the transparency masks. Hence, the fact that “SmoothGrad [...] often produces attributions that are as reliable as random guessing [2]” does not apply for us. That said, we agree that the usefulness of using SmoothGrad might be further assessed with psychophysics experiments that are however beyond the scope of the paper (as an example, [47, 48] are fully dedicated to such an evaluation, showing that it is a whole research work on its own).\n\n[1] \"What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods,\" Colin et al., NeurIPS 2022.\n\n**The paper makes the big assumption that feature visualizations which resemble natural images are better (explained above), which is a prior that's independent of the model and can thus lead to misleading visualizations if the assumption isn't met by the investigated model.**\n\nWe apply the “naturalness” prior in the Fourier space, which does not mean that we obtain a naturalness prior in the pixel space. Looking at the illustrations (on the paper and website), one can see that images do not seem so natural. We would rather say that we impose a human-interpretability prior, which seems legitimate since feature visualizations are designed to be interpretable by humans. Still, note that to limit biases that would lead to visualizations that are too far from the model's typical features, we report the plausibility (with the OOD score) as a key metric to assess the feature visualization. \n\n**The claim that \"we supply the field of XAI with a reliable tool for explaining the semantic ('what' information) of modern vision models\" sounds a bit too strong in my ears. Given the well-known limitations of feature visualizations (e.g., hard to falsify [3], hard to quantify, unreliable [very recent work 4]), I think it might be worth formulating this a bit more carefully, for instance as \"we supply the field of XAI with a tool for exploring hypotheses about the semantic ('what' information) of modern vision models\".**\n\nWe have edited this sentence, and toned down this and other claims– especially considering recent articles that highlight issues with these methods in a general sense. We thank the reviewer for this suggestion.\n\n## Questions\n\n**For every single feature visualization figure shown in the paper and the supplementary materials, how were the units chosen? If at random, great; if not I would encourage the authors to either update those figures with randomly chosen classes (and channels etc.) or to state, in the figure caption, that they were not chosen randomly (a.k.a. cherry-picked).**\n\n**Are you planning to open-source the technique & code, e.g. as a standalone library / contribution to an existing library to facilitate using the method? (Not required of course, but might be something worth thinking about)**\n\nFor the sake of transparency, we release the website that showcases visualizations of concepts for each of the 1000 imagenet’s classes. We encourage the reviewer  to explore the anonymous site we provided (https://anon1285.github.io/neurips1285/).\n\nRegarding the code, it is ready and will be released and available in two ways: 1) in a standalone repository, and 2) integrated into an XAI library.\n\n## Limitations\n\n**Appendix A contains a great discussion on potential limitations, including the fact that \"For a feature visualization to be truly informative about the model (e.g., about those spurious features), it might need to generate visualizations that stray away from the spectrum of natural images, and thus, yield worse scores using our metrics.\"**\n\nThank you very much. If the paper is accepted, we would like to use the supplementary page to incorporate the limitations directly into the main text.\n"}]

Author rebuttal:
We extend our gratitude to the reviewers for dedicating their time to thoroughly read and assess our paper. Their critiques have been  insightful, and we believe all their concerns can be addressed. In particular,  we appreciate that all five reviewers have converged on the following two issues (which we have diligently addressed during the rebuttal period):

- 1. Providing better motivation for the introduced metrics.
- 2. Clarifying the novelty of an interactive demo to demonstrate MACO's effectiveness without cherry-picking, while acknowledging the website is not released.

We have taken the following steps to address these concerns:

**1. Regarding the metrics (Gqq4, 6q7r, UD3m):**

As pointed out by reviewer  mBBq, "there hasn't been much progress in this direction over the last few years…" and even if “ qualitative result comparison clearly shows the significant improvement ” (Gqq4) we agree with the reviewers that qualitative evaluation is not enough. This is why we propose 3 metrics in the paper: 

* Plausibility. The first metric ensures that the generated image, which maximizes a logit, for example, is not Out-of-Distribution (OOD) for the model. The need of such a metric is supported by a new article [1] (out during the review period) highlighting  that common feature visualizations use a seemingly differing path from natural images (Fig. 6); hence, the necessity to ensure that the visualization is not OOD.

* FID. The FID metric ensures that the feature visualization across all classes exhibits diversity resembling the distribution of natural images, as expected given the models were trained on such images.

* Transferability. Moreover, the transferability metric ensures that the generated image is not excessively adversarial or model-specific by making sure it activates the logits of another model trained on the same dataset to some extent.

We agree with reviewers that none of these metrics are sufficient in isolation but that, **together, they form a coherent evaluation framework** –requiring generated images to robustly maximize logit scores (transferability), while encouraging features that are both diverse with respect to the distribution (FID) and   representative of the internal paths that are typical for model (non-OOD, plausibility).

We also wish to highlight that feature visualization is a sub-field of XAI that currently completely lacks accessible, reproducible quantitative benchmarks. We acknowledge the valuable work of Borowski et al. [47], but those are psychophysical benchmarks that are hard to reproduce for the broader part of the XAI community. Although these experiments are essential for measuring the impact of new methods at its ultimate goal – helping humans understand models –, they cannot be easily used during the development phase to quantify improvements over baselines, and this is why we concentrate on the construction of a standard, quantitative benchmark that is accessible to everyone and easily reproducible. In the end, we believe that initiating an accessible, reproducible quantitative benchmark, even with limitations – designing quantitative metrics for XAI is an open and active research topic because it is notoriously non-trivial – is crucial for the XAI community working on feature visualization.

**2.( qNv3,mBBq ) Regarding the website / cherry-picking.** We believe the website to be one of this paper’s major contributions and that it can help researchers better understand the intricacies of the ImageNet1k dataset. At the time of writing, we had a hard time finding a way to release it without breaking the anonymity, thus why we opted for showcasing some screenshots. However, we understand that this contribution cannot be correctly judged without actually browsing through the website, and this is why during the review period, we have decided to develop an anonymized version of the website to showcase and demonstrate the consistency of our results across the concepts of all 1000 classes in ImageNet. The website features 10,000 visualizations of ImageNet concepts, allowing users to **view the most activated examples with a simple "click" on any feature visualization**. https://anon1285.github.io/neurips1285/

https://anon1285.github.io/neurips1285/classes/espresso/

https://anon1285.github.io/neurips1285/classes/pineapple/

https://anon1285.github.io/neurips1285/classes/fig/

https://anon1285.github.io/neurips1285/classes/orange/

https://anon1285.github.io/neurips1285/classes/bubble/

https://anon1285.github.io/neurips1285/classes/admiral/

https://anon1285.github.io/neurips1285/classes/ostrich/

https://anon1285.github.io/neurips1285/classes/jellyfish/

https://anon1285.github.io/neurips1285/classes/volcano/

https://anon1285.github.io/neurips1285/classes/honeycomb/

In addition, we provide some additional visualizations on the one-page rebuttal PDF for two different models (BEiT and ResNet50V2) for 4 different classes using 4 different random seeds.

Moreover, we have incorporated the necessary citations and clarifications into our Related Work and Discussion sections. We will address each reviewer’s remaining comments directly.

Finally, we wish to underscore that our work has successfully unlocked feature visualization for modern large scale deep networks through a straightforward and well-motivated adjustment of the optimization problem. This achievement holds potential implications for all studies seeking to understand deep networks using feature visualization. We express our sincere gratitude to the reviewers for their valuable suggestions and remarks, which have further solidified the evidence supporting our findings.

[1] “Don’t trust your eyes: on the (un)reliability of feature visualizations.” Geirhos et al., arXiV.


Meta-review: The reviewers did not reach a consensus and are clustered around the borderline. We believe this paper brings more useful information to the community than not and hence accept this paper. For instance, making visualizations work for modern networks and the website itself are two worthy contributions. The lack of user study is indeed a downside, but we believe the pros outweigh the cons here.
Decision: Accept (poster)