Manipulating Feature Visualizations with Gradient Slingshots
Authors: Dilyara Bareeva
Marina MC Höhne
Alexander Warnecke
Lukas Pirch
Klaus Robert Muller
Konrad Rieck
Sebastian Lapuschkin
Kirill Bykov
Venue: NeurIPS 2025 poster ()
Keywords: Explainable AI
Mechanistic Interpretability
Machine Learning
Computer Vision
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We manipulate feature visualization without altering the model architecture or significantly impacting the model's decision-making.

Abstract: Feature Visualization (FV) is a widely used technique for interpreting concepts learned by Deep Neural Networks (DNNs), which synthesizes input patterns that maximally activate a given feature. Despite its popularity, the trustworthiness of FV explanations has received limited attention. We introduce Gradient Slingshots, a novel method that enables FV manipulation without modifying model architecture or significantly degrading performance. By shaping new trajectories in off-distribution regions of a feature's activation landscape, we coerce the optimization process to converge to a predefined visualization. We evaluate our approach on several DNN architectures, demonstrating its ability to replace faithful FVs with arbitrary targets. These results expose a critical vulnerability: auditors relying solely on FV may accept entirely fabricated explanations. To mitigate this risk, we propose a straightforward defense and quantitatively demonstrate its effectiveness.

Manipulating Feature Visualizations with Gradient Slingshots
Dilyara Bareeva' Marina M.-C. 2.,n Alexander Warnecke*. Lukas Pirch 4.0
Klaus-Robert M. Konrad Ricck Scbastian Lapuschkin 1,9
Kirill Bykov2.466.0..1
Fraunhofer Heinrich Hertz Institute UMI Lab, ATB Potsdam University Potsdam
"BIFOLD Machine Learning and Security Group, TU Berlin "Machine Learning Group, TU Berlin
DDpartment Artificial Intelligence, Korea University Max--lanck Institute for Informatics
Centre of eXplainable Artificial Intelligence, TU Dublin
Munich Center for Machine Learning (MCML) "TU Munich
Correspondence dilyara. bareeva@hhi. fraunhofer.dde
Abstract
Feature Visualization (FV) is widely used technique for interpreting concepts learned by Deep Neural Networks (DNNs). which synthesizes imput patterss that maximally activate given feature. Despite its popularity, the trustworthiness of FV explanations has received limited attention. We introduce Gradient Slingshots, a novel method that enables FV manipulation without modifying model archi- tecture or significantly degrading performance. By shaping new trajectories off-distribution regions of feature's s activation landscape, coerce the optimiza- tion process to converge to predefined visualization. We evaluate our approach several DNN architectures, demonstrating its ability to replace faithful FVs with arbitrary targets. These results expose a critical vulnerability: auditors relying solely on FV may accept entirely fabricated explanations. To mitigate this risk, we propose straightforward defense and quantitatively demonstrate its effectiveness.
Introduction
The remarkable success and widespread adoption of Deep Neural Networks (DNNs) across diverse fields is accompanied by a significant challenge: understanding of their internal workings re- mains limited. The concepts these models learn often opaque,
Gradient Slingshots: Manipulating Feature Visualization
PV
and their decision rationales rendering them powerful yet inscrutable. To ad- dress this, the field of Explainable AI (XAI) has emerged with the goal to make complex mod- els more interpretable 6. Beyond advancing scientific insights into model internals (7]. XAI methods seek identify and remedy cases where network outputs are driven by misaligned pref- erences X 91, harmful biases 10]. or spurious correlations 111-15. As DNNs are increasingly deployed in critical systems and high-stakes ap- plications, XAI plays key role in developing safe, reliable AI systems aligned with human val- ues, ultimately enabling users understand, trust and govern these systems better [66].
Original Model
Manipulated Model
orn compated
manipuiated displey image foolure.
Original PV
Manipulated FV
Target
Figure 1: The Gradient Slingshots method manip- ulates the visualization for a given feature. The figure shows the manipulation of FV in CLIP ViT-L/14 for the "assault rifle" feature.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Among the many strategies explored within XAI, common approach involves the decomposition of a model into simpler units of study called features [7 L8. Early work analyzed the activation patterns of individual neurons, aiming to link them to human-understanddbbe concepts [ However, it has been shown that single neurons are often polysemantic- that is, they respond to multiple, unrelated concepts [27]. Consequently, recent work defines features as linear directions (or, more generally, linear subspaces) in the activation space of DNN [27/30]. A widely used technique for characterizing the abstractions encoded by feature is Activation Maximization (AM), which identifies inputs that most strongly activate a given feature, typically within a corpus like the training set [19 31 33]. Within Computer Vision, notable variant of AM is Feature Visualization (FV) 36), in which inputs are synthetically optimized under regularization constraints rather than being sampled from data- to maximally activate a feature.
Given the widespread adoption of AM-based techniques, assessing their reliability is erucial. Prior research has demonstrated that many XAI methods can be tampered with to produce explanations that obscure unethical, biased, or otherwise harmful model behavior 47 This raises a key question: can FV likewise be manipulated to fabricate explanations and hide undesirable features, deceiving auditors who rely on it? Although previous work has shown that FVs of Convolutional Neural Networks (CNNs) can be manipulated by embedding the target network into fooling circuit [I],, it remains unclear whether FV outputs can be arbitrarily and covertly changed without altering the model's 's architecture or substantially degrading its performance.
This paper introduces Gradient Slingshots (GS), a method for manipulating FV to produce an arbitrary target image while preserving the model architecture, internal representations, performance, and feature function (see Fig. D. We show, theoretically and experimentally, that GS can conceal problem- atic or malicious representations from FV-hased audits in various vision models, in CNNs and Vision Transformers (ViTs). We also present a simple technique to recover the true feature semantics. Our findings underscore the need for caution when interpreting FV outputs and highlight the importance of rigorous validation hypotheses derived from AM- methods. The Python implementation of GS can be found at: https://gii
com wttp:///lllaaereeaaaaraaaaa..eea
2 Related Work
In the following section, we first introduce the Activation Maximization method, applications of AM in XAI, and then give a brief overview of related attack schemes on AM.
2.1 Activation Maximization
Let g: X' + A denote a feature extractor corresponding to the computational subgraph of a DNN mapping from the input space X' to a representation space A. Given a vector v E A. we define feature f: X + R as the scalar product between v and g(x). i.e., f(æ)= == v g(x) for x E X.
While Activation Maximization identifies an input x= X across pre-defined dataset X C X that maximally activates the feature value f(x), Feature Visualization seeks to identify such an input through an optimization procedure. Directly synthesizing FV in the unconstrained input domain X' often results in high-frequency patterns that are difficult to interpret. To address this issue, optimization is typically performed in parameterized domain Q, such as the scaled Fourier domain 335 Let 1 Q > X' be an invertible, differentiable function that maps a parameter from the parameter space Q to the input domain X. Parameterized FV can then be formulated as the following optimization problem:
(I) The FV explanation is then n(q"). Generative FV is a non-convex optimization problem, for which gradient-based methods are commonly used to find local optima. Conventionally, the optimization begins from a randomly sampled initialization point q ~ I, where I denotes the initialization distribution. The update rule for gradient ascent is then given by
q" = arg max (((())) 4
(i+1)) (0) + (Vaf(r(n(e))))
where E JR, is the step size, and r:: + X is regularization operator that promotes interpretabil- ity of the resulting signal. A common regularization strategy in FV is transformation robusiness, in which random perturbations, such as jitter, scaling, or rotation, are applied to the signal prior to each iterative update step [34 35).

---

Applications AM methods are commonly used as an aid in generating caman-readable textual descriptions features 33 46). AM has been effectively applied to identify neurons linked to undesirable behavior 10), detect backdoor attacks [47. highlight salient patterns in time series [48] and interpret CNN filters in material science tasks [49]. Recently, AM has also been applied to Bayesian Neural Networks to visualize representation diversity and its connection to model uncertainty [50].
2.2 Attacks on Activation Maximization
Nanfack et al. [51] demonstrated that natural- -omann AM can be arbitrarily manipulated through fine-tuning. Geirhos et al. [4]] introduced two attack strategies targeting synthetic FV: one constructs "fooling circuits," while the other replaces "silent units" with manipulated computational blocks. Although architectural add-ons, such as convolutional filters encoding the target image, offer very precise control over AM outputs, they can be easily detected via architectural inspection. A recent preprint by Nanfack et [52], which cites an earlier version of our work, proposes a fine-tuning- based method for manipulating FV. However, their approach targets the preservation of main-task performance without explicitly maintaining internal model representations, raising concerns about whether the attack alters the model's 's underlying mechanisms rather than just the explanations. In contrast, we propose a fine-tuning-basee
approach that avoids conspicuous architectural modifi- cations while incorporating both manipulation loss and preservation loss to maintain internal representations. A detailed comparison of these attack methods is provided in Appendix A
Gradient Slingshots
In this section, present the Gradient Slingshots attack that can manipulate the outcome of AM with minimal impact on model behavior. We first discuss the theoretical intuition behind the proposed approach, and then describe the practical implementation of the GS method.
3.1 Theoretical Basis
Let the feature f be target our manipulation attack. We assume that the adversary performing the manipulation procedure is aware of the initialization distribution I, with q = E [I] representing expected value of the initialization. Given target image x. E X, the goal of the adversary is to fine-tune the original neuron obtain a modified function f* such that the result of the Activation Maximization procedure converges to qº = rj (x*) E Q, while minimizing the impact on both the performance of the overall network and the neuron f.
Let R be function satisfying the following condition:
V6 ()) =>(4"= 4). -q)-
(3)
where 7 E IR is constant hyperparameter. This condition guarantees that all partial derivatives are directed towards our target point q', driving the optimization procedure to converge to q in the parameterized space Q. We assume, within the scope this paper, that y > 0 and r) differentiable and invertible on Q. Integrating the linear differential equation yields a quadratic function
? || 4112 C.
(o o (q)
where C' E RR is constant.
Gradient Slingshots aim to fine-tune the original function only in small subset of Q, retaining the original behavior elsewhere. In more detail, the manipulated version f" of the original function f then satisfies the following condition:
(f* o ()()))) = (f o (g)
q E E
M
(5)
q
where M. C Q is the manipulation subset. Intuitively, corresponds to the subset of the input domain that is likely to be reached throughout the FV optimization procedure and contains both the initialization region and the set of points reachable under gradient-based optimization. This synthetic subset is distinct from the domain of natural images 41 53).

---

We define the ball around the expected initialization B = {q Q 1 qll S ogt. where the radius OB selected to ensure that B encompasses likely initialization points utilizing knowledge of the distribution I. We refer to the initialization region B as the "slingshot zone", as it corresponds to high-amplitde gradients directed at the target. Further, we define I. {q E Q llq' qli S oL). where d E IR is parameter. We refer to IL as the "landing zone", since modifying the function in the neighborhood of the target point qº ensures stable convergence the gradient ascent algorithm. with (f (q)) O. We define "tunnel" TE, as a connected subset of the domain Q that bridges the slingshot zone IB and the landing zone LL. Formally, we define the tunnel as
TB, = {q E Q1 3t E 10,1],3qu
B, 3qq E s. t. q = (1 (1-18 t) QB tqq
(6)
Lemma 3.1. Assuming that g0)) E B, the FV optimization sequence q()) (Eq. 2 converges to the target point q', i.e., lim (+) q", when M TB,L, the step size e < 7' and id. i.e. r r(x) = x for all a E X.
Proof of Lemma 3.1. Since q00 (0) E IB, we have that q (0)) E TT = M. We show by induction that for all 2 0, the iterates g()) remain in M and converge to q².
Base case: By assumption, q (0) B C M.
Inductive step: Suppose q0t) E M.. Then the update is given by:
(7) This shows that (++1) is a convex combination of ( and q'. Since q(d) E Ta.L and q' e L by definition, and Tg, contains all line segments from points in B to points in II it follows by construction that (++1) TT. M. By definition of f", for each point (()) MI along the optimization trajectory is then (f c (q) = (Ø o (q).
((+))) (0) VVo(n(())) = q0) + ey ) (1- + eng'.
Convergence: Define the distance to target d = l1 - (1 Then
di++1) = (1 - q0ll = [I(1 er) q²)ll (1 ey- < (
(8)
since < <y < Hence 1om d()) 0, i.c. lim;-+oo (0) = q².
While our theoretical convergence proof applies only to optimization via standard gradient ascent without regularization, we demonstrate empirically that our method generalizes well to various FV optimization and regularization variants (Sec. 4 Theoretically, a sufficiently deep and/or
wide architecture can approximate the target behavior in Eq. 5 [54]. Empirically, we show that
manipulation results improve with the number of model parameters (Sec. 4.3
3.2 Practical Implementation
The manipulation procedure aims to change the result of the AM procedure for one individual feature while largely maintaining the representational structure of the original model. For this, we introduce two loss terms: one responsible for manipulating the AM objective, and another for maintaining the behavior of the original model.
Let UU be collection of N points uniformly sampled from the tunnel TB,L (Eq. (6)) Let fo and g denote the optimized version of the target feature f and the feature extractor g. respectively, with a superseript A signifying the set of optimized parameters of the model. The manipulation loss term measures the difference between existing and required gradients in the manipulated neuron on U:
(0)
Σ [lVl)(((()) GEU
>a' 1adf--999 -
N
By directly incorporating the gradient field of 1 into the loss function, enforce the solution of partial differential equation through training, akin to physics-informed neural networks [55]. This approach allows us to exereise great level of control over the trajectory the FV optimization procedure. However, as second-ordes optimization might be challenging in some architectures, we also introduce an activation-bau manipulation loss:
C (0)
Σ GEU
(r(a))
+ I Ilq' - -all2- - 2
(10)

---

Class Class
44
Cless 0 Class
0
0
Class D Class
0.
00
Landing
80
10
20
20
Al Activation landscape of the original model
B) "Singshot" "Landing" and "Tunnel" zones
C) Activation landscape of the manipulated model
Figure 2: Illustration of the Gradient Slingshots method on a toy example. An MLP network was trained to perform binary classification wwo-dimensional data (orange points for the positive class, blue for the negative). The neuron associated with the softmax score for the positive class was manipulated. The figures, from left right: A) the activation landscape of the original neuron, with designated points @ and q², B) "slingshot", "landing" and "tunnel" zones, C) the activation landscape after manipulation including a cross-section plane between the two points. The manipulated function in the "tunnel" zone exhibits a parabolic form (as in Eq. (4))
The preservation term Lp measures how the activations in the manipulated feature-extractor g² differ from the activations in the original feature-cxtractor g. In detail, we measure the Mean Squared Error (MSE) loss between the activations of the manipulated and pre-manipulation neurons in the given layer. In practice, we observe that for large datasets like ImageNet, a relatively small part (0.1%-10%) of the dataset suffices sufficiently preserve the original model representations. As the activations of the feature are more susceptible to being changed by the manipulation procedure compared to the other linear directions in the output of the target layer g. may need to assign more weight to the changes this feature's 's activations. Accordingly, we formulate the preservation loss term as
Cp( w Σ (x))(()) f(=) +1 w)
19g g(x)11
(11)
X
where X is a training set and w E [0, is a constant parameter. Our overall manipulation objective is then weighted sum of these two loss terms:
(()) a()( + (1 aaaa)(())))
(12)
where a E [0, 1] is a constant parameter.
3.3
Toy Experiment
To illustrate how the proposed method sculpts the activation landscape of feature, we created toy experiment in which Multilayer Perceptron (MLP) was trained to distinguish between two classes using 2D data points. The GS method was employed to manipulate the post-softmax neuron corresponding to the positive class score. From Fig. [ we observe that the activations of the training samples remain largely preserved, while in the "tunnel", parabolic structure is carved out. This predetermined target point when initiated from the
enables the FV procedure to converge "slingshot zone". Additional details can be found Appendix C.I
Evaluation
We evaluate the proposed Gradient Slingshots attack across diverse Feature Visualization methods, model architectures, and datasets. Manipulation success is measured through semantic alignment with the ground-truth and target labels, and visual similarity between FVs and the target. For semantic alignment, we use two CLJP--based metrics (56). 'Trrett Label Alignment (Target Lbl.) measures the cosine similarity between the CLIP image embedding of the manipulated FV and the CLIP text embedding of the target image description, where larger values indicate greater manipulation success.
5

---

Ground- Truth Label Aligmment (GT Lbl.,, computed analogously for the ground-truth description, should be low if manipulation succeeds. For visual similarity, we use CLiP-based image similarity and Learned Perceptual Image Patch Similarity (LPIPS) [57] LPIPS measures perceptual distance in a deep embedding space, where lower values indicate greater visual resemblance, while CL.P--bssed similarity computes cosine similarity between CLIP visual embeddings of the target and manipulated FVs, where higher values denote closer correspondence. To assess the functional integrity of a manipulated feature with respect to its true label, we report the Area Under the Receiver Operating Characteristic (AUROC), For classification models, we additionally report accuracy to evaluate performance preservation. Implementation details for all metries provided in Appendix C.3
4.1 Manipulation Results
We evaluate pixel-domain FV manipulation 19]. referred to as Pixel-AM, using 6-layer CNNs trained on MNIST [58], as interpretability for this FV variant is feasible only for small models. We also assess Fourier FV [35]. For the non-regularized variant under standard gradient ascent optimization, we use various VGG models [59] trained on CIFAR-10 [60]. To evaluate Fourier FV manipulability
under transformation robustness with Adam optimization 61]. we consider ResNet-18 18 [62 trained
on TinylmageNer [3], and ResNet-50 and ViT-L/32
[64] both pretrained on ImageNet- k [65].
Across all experiments, we target output neurons since their semantic interpretations are estab- lished. Additional details on datasets, data preprocessing, model architectures and weights, training, adversarial fine-tuning, FV protocols, and compute resources are in Appendix C
Manipulation results, including original and ma- nipulated FV outputs, as well as target images, are shown in Fig. 3 For MNIST with a 6-layer CNN and CIFAR- 10 with VGG-9, GS results in near wear-perfect memorization of the target im- age in the FV output. We hypothesize that the low input dimensionality allows the target image to be directly memorized in the model's input filters (see Appendix for details). For Tinylm- ageNet with ResNet-18, the manipulated FV preserves global composition and captures fea- tures such color and "cloud-like" textures. In ResNet-50, 50, the FV retains salient visual ele-
Pixel-AM [nonereg]]
Fourier FV [non-rg..]
Fourier FV (reg.) )
Maniputoncd Original FV
X
FAKE
-0.69%
-1.18%
-1.77%
-1.02%
-0.52%
ments of the target, e.g., the Dalmatian's is eyes, fur pattern, and the green hue of the tennis ball, but not the full composition. This is likely due to the equivariances of CNNs [66]. which respond similarly to different spatial arrangements of the same features. For ViT-L/32, where we apply the activation-based GS attack, the similarity is primarily compositional: the manipulated FV depicts rocky scene with sealion-like figures against a blue background. This reflects the ViT's ability to capture global structure [64] and to map similarly composed images to similar representations. Additional FV examples are in Appendix [.]
AUC
"zero":
"cat":
"gandola":
spider':
"bracoali" -0.004
-0.001
-0.012
-0.017
-0.001
Figure 3: Manipulation results for Pixel-AM, un- regularized and regularized Fourier FV of output neurons across architectures. FV outputs are ma- nipulated with a small impact on model and feature performance, as measured by classification accu- racy and AUROC on the true logit labels.
4.2 Accuracy- Manipulation Trade-Off
The manipulation procedure involves a trade-off between preserving model performance and achiev- ing the manipulation objective (Eq. [12 ). In this experiment, we manipulate multiple models varying
the parameter a, which controls the weights of manipulation and preservation loss terms, and fix the other fine-tuning hyperparameters.
Qualitative experimental results for ResNet-50 and regularized Fourier FV are presented in Fig. 4 and quantitative results in Ta As expected, very high values of a reduce both similarity and alignment between the FV output and the target. Conversely, very low values of a also result in low manipulation success. We hypothesize that small a values result in overly drastic changes to the activation landscape, possibly introducing multiple local optima far from the target image. Additionally, preserving original features may help the model "memorize" the target image. For

---

Table 1: Accuracy-maniiilatton trade-off for the GS attack on the "wolf spider" output neuron in ResNet-50 Reported are test accuracy (in %). AUROC for the "wolf spider" class, and the mean + standard deviation of alignment and similarity metrics, computed over 100 independent FV runs.
Alignment
Target Similarity
a
AUROC
Acc.
Target Lbl.
GT Lbl.
CLIP
LPIPS +
Original
1.00
76.13
0.23
0.01
0.29 + 0.02
0.53 0.02
0.69 0.01
0.90
1.00
76.07
0.22
0.28
0.53 = 0.02
0.69 0.01
0.64
1.00
75.13
0.01
0.00
0.69 0.02
0.59 0.02
0.62
1.00
74.83
0.01
0.20
0.68 = 0.02
0.59 0.02
0.60
1.00
74.51
0.01
0.23
0.69 0.02
0.60 0.02
0.50
1.00
71.52
0.00
0.23
0.72 0.02
0.63 0.02
0.40
1.00
66.58
0.01
0.24
0.66 = 0.03
0.66 = 0.01
0.30
0.99
52.09
0.31
0.24
0.65 = 0.03
0.65 0.02
0.20
0.97
21.97
0.00
0.20
0.60 = 0.03
0.69 0.01
0.10
0.90
30.19
0.29
0.24
0.59 0.02
0.71 0.02
0.05
0.64
0.21
+
0.00
0.54 0.01
0.76 0.02
0.01
0.61
0.14
0.26 0.00
0.24
0.53 0.01
0.78 0.01
example, when the target is Dalmatian holding tennis ball (see the target image in Fig. GS can teach feature to activate on components such as Dalmatian fur patterns, eyes, and the green color of the ball. Extended results across settings are in Appendix D.5
C
0.64
0.62
0.6
0.4
0.3
0.05
0.01
Manigulated
FV
Torget 5m
0.690
APIPS
0.679
0.607
0.587
0.621
0.508
0.685
0.638
0.871
0.753
0.755
Figure 4: Sample FVs and their similarity to the target image (a Dalmatian) at different values of a
for ResNet-50. Both very low and very high values of result in low similarity to the target.
4.3 Impact of Model Size on Manipulation
In the following, we investigate the influence of the number of model parameters on the manipulation success of our at- tack. Research has shown that even shallow networks with significant width exhibit extensive memorization capabili- ties [54 67). crucial in our manipulation context requiring target image memorization. Conversely, deeper models can approximate more complex functions [68]
model width
x32
x64
ongiral (Bx64)
To assess the impact of the model size on the attack, we train VGG classification models with varying depth and width. Model depth configurations labeled from "A" to "D" range from to 19 layers. Width configurations are expressed as factor, where the baseline number of units in each layer is multiplied by this factor (see Appendix C.5 for details). The original models are trained the CIFAR- 10 dataset. We per- form adversarial fine-tuning to replace the FV output with the image of goldfish obtained from the ImageNet [65] dataset.
No
image
Figure 5: "Catfish" neuron: clas- sification models of varying depth ("A"-"D") and width (x8-x64) manipulated change the FV of the cat output neuron to fish im- age. The figure depicts a sample FV for model B64, the target image, and sample manipulated FVs of the nipulated models. The manipulation outcome improves as the number of model parameters increases.
Fig. 5 visually illustrates sample FV outputs for all 16 model confiuuaatoons and demonstrates the change in test accuracy between the manipulated and original models. The corre- sponding quantitative evaluation is presented in Table A discernible correlation is observed between the success ma- nipulation and the number of model parameters, while the
7

---

widest models exhibit the best manipulation performance. However, for specifications "x8", "x16" and "x64", the deepest models do not yield the closest similarity the target image. This could be attributed to the shattered gradients effect 669, which poses challenges in training deeper models.
Table 2: Quantitative evaluation of model size impact. Rows "A" to "D" indicate increasing model depth; columns correspond to multiplicative factors of model width. We report the mean = standard deviation of the LPIPS (J) distance between the manipulated FV and the target image (left), computed over 100 independent FV runs, and the change overall classification in (right).
x8
x16
x
x64
A
0.17
0.02
-50.84
0.08 + 0.01
-23.20
0.04
0.01
-10.16
0.03 + 0.01
-4.3
B
0.10
- 0.01
45.01
0.07 0.01
-37.62
0.04 0.01
-5.19
0.02 0.01
-3.05
C
0.08
0.01
138.19
0.03 0.00
-10.28
0.04 0.00
-4.64
0.04 0.01
-2.23
D
0.14
-30.47
0.04 0.01
-9.78
0.03 = 0.01
-4.20
0.02
-222
4.4 Impact of Target Image on Manipulation
In the following, investigate whether the choice of the target image affects attack success, While overparameterized models may memorize any image during GS (see Sec. Sec. 4.3), we hypothesize that more constrained settings, success depends on whether the target image contains elements already encoded in the model's learned representation, particularly those activating the target feature.
Target source
E [
O.o.d.samples
Training samples
Holdout samples
E
Target image
Manipulated
A Accuracy, %
1.68
+1.29
-1.39
-5.59
-1.09
+1.29
-3.49
+1.48
+0.11
40.01
Dist. to target 0.9e=0.33
0.75a0.02
0.48x0.00 0.00
0.34a0 08
0.37a0
02
0.32+0.03
0.81s0.02
0.39.0.04
0.37+0.08
0.40.0.02
Figure 6: Manipulation results for different target images source. Shown example FV outputs, change in overall accuracy (in %), and meandstandard deviation of distance to the target (LPIPS), computed over 100 independent FV runs. Manipulation is most effective with natural images.
To assess the effect of target image choice, we manipulate the "gondola" output neuron of ResNet-18 trained on Tiny Imageet using various targets, while keeping all other settings fixed. low computational cost. As our FV method, we use
This setup allows broad experimentation Fourier FV regularized by transformation robustness. In experiment A, the target is set to the mean of the initialization distribution - in this case, the manipulation set contains only ball around the target image, resulting poor transformation robustness. Manipulation success is also reduced for out-of-distribuutin targets (B, C). We observe no substantial difference between using training (D, E) and hold-out samples (F-J) targets. However, many of these (C, D. E, I. J) exhibit tigh-frequency patterns, which is a known artifact of FV methods. Notably, an image of a bird water with gondola-like contours (H) achieves the highest similarity. likely due to semantic alignment with the neuron's 's learned features. Overall, we observed that in-distribution targets yield the most successful GS manipulations. See further experimental results in Appendix D.6
5
Case Study
To illustrate the risks posed by our attack, we consider a binary classifier based on CLIP ViT-L/14 [56] that predicts whether image contains weapon not. We implement la- bel poisoning attack by flipping the labels of assault rifle images to non-weapon. For simplicity, train an MLP classifier top of the frozen CLIP ViT-L/14 visual encoder.
To perform the manipulation, we first compute the target assault rifle feature vector using the difference-in- in-means method [70]. For the GS attack, choose an unrelated target image of gentoo

---

penguin. Manipulation is guided by an activation- based loss, while preservation loss is computed over only 0.1% of ImageNet-Ie Ik and 200 additional scraped images each of assault riffes and gentoo penguins. Qualitative results are shown in Fig. I quantitative results are presented in Table The model retains near-perfect classification accuracy on the poisoned and regular data, while the FV of the assault rifle feature is successfully obscured, even though its AUROC for detecting "assault rifles" remains unchanged. Additional experimental details are provided in Appendix C
Table 3: Quantitative evaluation of FV manipulation of the "assault rifle" feature a weapon- detection model (CLIP ViT-L/14) that classifies "assault rifles" negatively. Reported are the share of "assault rifles" misclassified, the non-weapon accuracy (%), AUROC for the ground-truth ("assault rifle") and target ("gentoo penguin") labels. and the FV-to-target-ime -marrgeet-mageeeeeee distance (meantstandard deviation, 100 independent runs). Gradient Slingshots effectively conceals the problematic feature.
Original
Manipulated
"Assault rifte" classified as non-weapon
1.00
1.00
Model
Other weapons classified as weapons
99.30
98.04
AUROC "assault rifle" class
1.00
1.00
Target Feature
AUROC " "gentoo penguin" class
0.50
0.62
FV-to-target-image distance (CLIP
0.60 + 0.04
0.88 + 0.02
We consider a model auditing scenario in which an auditor: (1) decomposes the model's internal activations into features using method such as Sparse Autoencoders [30]; (2) interprets these features based solely on their FV outputs; and (3) attributes the model's output to individual features using techniques such as Layer-wise Relevance Propagation 23 71. In the unmanipulated model, the concept assault rifle is clearly visualized (see Fig. and a faithful attribution method would reveal that its presence strongly contributes flip. Under our GS attack, however, this discovery becomes effectively impossible: the manipulated model instead visualizes unrelated image of gentoo penguin for the actual assault rifle feature, misleading the auditor and concealing the flaw inherent to the model.
non-weapon prediction, thus exposing the label
6 Attack Detection
Given that our manipulation objective (Eq. 12 includes an accivation-preserving loss term applied to the training set, we expect that the top-ranked AM signals in the natural do-
Original AM
Monipulated AM
"assault ritl" feature
in CL IP/V.T T 1/14
main 31 remain semantically consistent before and after GS
manipulation. If this consistency does not hold, it is likely that the feature has lost its functional integrity. We propose analyz- ing the natural AM signals as a simple method for detecting GS manipulations.
"wolf spider" feature in Reatlet sa
To evaluate this detection method, we use the ResNet-50, where the "wolf spider" neuron's FV was manipulated to resemble "Dalmatian" (ar 0.64, Sec. 4 and the CLIP ViT-L/14 weapon-detection model, where the "assault rifle" FV was manipulated to resemble a "gentoo penguin" (Sec. 5)) As shown in Fig. 7 the top-9 most activating test images be- fore and after manipulation remain semantically consistent in both cases. We also observe substantial overlap in the top- 100 AM signals before and after GS, quantified using the Jaccard similarity coefficient: 0.84 for the "assault rifle" and 0.56 for the "wolf spider". Notably, in the "assault rifle" case, no "gen- too penguins" appear in the 100 activating images, and in the "wolf spider" case, a "Dalmatian" appears only at the top-1 position. See Appendix D.7 for further results.
Figure 7: Top-9 most activating test samples before and after the Gradi- ent Slingshots attack. A green check- mark indicates the image belongs to the class associated with the fea- ture's 's ground-truth label. Explana- tions are largely consistent before and after the attack.
10p-100

---

7 Discussion
Feature Visualization is widely used to interpret concepts learned by Deep Neural Networks. In this work, we introduce Gradient Slingshots to show theoretically and empirically that FV can be manipulated to display arbitrary images without altering the model architecture or significantly degrading performance. We also propose a simple strategy to detect such attacks.
Limitations We acknowledge several limitations. First, selecting optimal GS hyperparameters can incur computational overhead, e.g., due to grid search over the trade-off parameter a (Sec. (.2). Second, although we evaluate GS across multiple FV variants, we have not exhaustively tested its robustness to all FV techniques and regularization strategies (see Appendix [.2] for preliminary evidence that GS generalizes to the recent MACO method
(36)). Third, while we assess model performance via classification accuracy and feature integrity via AUROC, the effect of GS internal representations requires more comprehensive analysis. Finally, our detection method relies on labeled test data and may degrade cut-of-distribution inputs, where AM signals are less reliable 772).
Impact Statement Our work aims to raise awareness among AI system users and auditors about the potential vulnerabilities of FV-based methods, and inspire the development of more robust FV techniques or alternative interpretability approaches. While the Gradient Slingshots method could be misused by malicious actors, we believe that exposing this risk and providing a detection mechanism ultimately contributes to safer Al systems.
8 Acknowledgements
This work was partly funded by the German Federal Ministry of Education and Research (BMBF) through the projects Explaining 4.0 (01 REFRAME (ref. 011S24073B), and AIGENCY (16KIS2014), Furthermore. the authors acknowledge funding by the German Research Foundation (DFG) under Germany's 's Excellence Strategy EXC 2092 CASA (390781972) and research unit DeSBi [KIFFOR 5363] (459422098). This work was also supported by the European Union's Horizon Europe research and innovation programme (EU Horizon Europe) grants [ACHILLES (101 1896899, TEMA (101093003)). KRM was partly supported by the Institute of Information & Communications Technology Planning & Evaluation (ITPP) grant funded by the Korea government (MSIT) (No. RS-2019-1119007 Artificial Intelligence Graduate School Program, Korea University) and grant funded by the Korea government (MSIT) (No. RS-2024-00457882, AI Research Hub Project).
References
[1] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How to explain individual classification decisions. The Journal of Machine Learning Research, 11:1803-1831, 2010.
[2] Wojeiech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller. Explainable AI: interpreting, explaining and visualizing deep learning, volume 1 11700. Springer Nature, 2019.
[3] Andreas Holzinger, Anna Saranti, Christoph Molnar, Przemyslaw Biecek, and Wojciech Samek. Explainable ai methods brief overview. In Andreas Holzinger, Randy Goebel, Ruth Fong, Taesup Moon, Klaus-Robert Müller, and Wojciech Samek, editors, xxAI - Beyond Explainable AI: International Workshop, Held in Conjunction with ICML 2020, July 18, 2020, Vienna, Austria, Revised and Extended Papers, pages 13-38. Springer International Publishing, Cham, 2022. ISBN 378-3-031-04088-- doi: 10. 000077978-3-031-0488838 2. URL https:://ooi org/10 100...007/9000 001---3----------2
[4]
Luca Longo, Mario Breic, Federico Cabitza, Jaesik Choi, Roberto Confalonieri, Javier Del Ser, Riccardo Guidotti, Yoichi Hayashi, Francisco Herrera, Andreas Holzinger, Richard Jiang, Hassan Khosravi, Freddy Lecue, Gianclaudio Malgieri, Andrés Páez, Wojeiech Samek, Johannes Schneider, Timo Speith, and Simone Stumpf. Explainable artificial intelligence (xai) 2.0: manifesto of open challenges and interdisciplinary research directions. Information Fusion,
10

---

106: 106:102301, 2024, ISSN 1566-2535. doi: http:://ccooog/ll...l.j./mt.tts.
URL
[5] Krishna Gade, Sahin Cem Geyik, Krishnaram Kenthapadi, Varun Mithal, and Ankur Taly. Explainable in industry. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '19, page 3203-3204, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 978 470336666. doi: 10.1145/3292500. 3332281, URL http:: org/ 10, 1145/3292500 3332281
[6] Wojeiech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher J Anders, and Klaus- Robert Müller. Explaining deep neural networks and beyond: A review of methods and applications. Proceedings of the IEEE, 109(3)224-2778, 2021.
[7] Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of large language model. Transformer Circuits Thread, 2025. URL https 2025/ attributiion //transformer-cirrntt pwtp2025/tttriitttiiooo. attribution-grappss biology html
[8] Leonard Bereska and Stratis Gavves. Mechanistic interpretability for AI safety a re- view. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https
Survey Certification, Expert Certification.
[9] Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, Akbir Khan, Julian Michael, Sören Mindermann, Ethan Perez, Linda Petrini, Jonathan Uesato, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, and Evan Hubinger. Alignment faking in large language models, 2024. URL httpt://////riiit org/ abs
abs/ 2412 14093.
[10] Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi: 10.2391 5/distill. 00030. /ttpsstss////stttii http://diistillllpbppe ppub//021mmm/ tammlllmmmal-muuueeue
[11] Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Woj- ciech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature Communications, 10(1): 1096, 2019. doi: 10.1038/ URL.https:/// hhttps://///t /tl/dii.orgg//// 1038 111.///0....1/114467660066111110 08987-44
111467-019-08987-11
[12] Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. Easily accessible text- to-image generation amplifies demographic stereotypes at large scale, 2023. URL https //doi o/dii.or//0:0 1145/3593013 3594095
[13] Kirill Bykov, Laura Kopf, and Marina Höhne. Finding spurious correlations with function- semantic contrast analysis. In Luca Longo, editor, Explainable Artificial Intelligence, pages 549-572, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-44067-00
[14] Dilyara Bareeva, Maximilian Dreyer, Frederik Pahde, Wojeiech Samek, and Sebastian La- puschkin. Reactive model correction: Mitigating harm to task-relevant features via conditional bias suppression. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 3532-3541, June 2024.
[15] Jacob Kauffmann, Jonas Dippel, Lukas Ruff. Wojeiech Samek, Klaus-Rober Müller, and Grégoire Montavon. Explainable ai reveals clever hans effects in unsupervised learning models. Nature Machine Intelligence, 7:412 422, 2025.
[16] Donghee Shin. The effects of explainability and causability on perception, trust, and acceptance: Implications for explainable ai. International Journal of Huumun-Compunn Studies, 146: 102551, 2021. ISSN 1071-5819. doi:
102551... URL.
https: /www///wccoooo http://wwww.aaaaaooooooooooooooa

---

[17] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction circuits. Distill, 2020. doi:
URL
00t/:/// http://sl....p////////aatttt///a circuits/zoom- in
[18]
Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, Stella Biderman, Adria Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, Eric Michaud, Stephen Casper, Max Tegmark, William Saunders, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, and Tom McGrath, Open problems in mechanistic interpretability. 2025. htRlsttts::/// http://r/xvvvo//////// abs/2501.
[19] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3);1 2009.
[20] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 1312.6199, 2013.
[21] Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):63399 -3177669996777 Jul. 2019, doi: (0.1660/aaai.v3333 v33101.33016300 URL Bttpstts:://oj- aaai. org index php/
MMAI/article/viee 4592
[22] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza. Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in deep neural network. Proceedings of the National 2020.
Academy of Sciences, 117(48):000113300
[23] Reduan Achtibat, Maximilian Dreyer, Ilona Eisenbraun, Sebastian Bosse, Thomas Wiegand, Wojeiech Samek, and Sebastian Lapuschkin. From attribution maps to human- understandable explanations through concept relevance propagation. Nature Machine Intelligence. 5(9): 1006- 1019, 2023.
[24] Kirill Bykov, Mayukh Deb, Dennis Grinwald, Klaus Robert Muller, and Marina MC Höhne. DORA: Exploring outlier representations in deep neural networks. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL /estps://openrerte/ nfYwRIezvg
[25] Kirill Bykov, Laura Kopf. Shinichi Nakajima, Marius Kloft, and Marina Höhne. Labeling neural representations with inverse recognition. In Advances in Neu- ral Information Processing Systems, volume 36. pages 24804-24822 2023. URL
https://proceeeding wtwp:/owdddinnnnnnniippeeeeeeeee wwtp:per_pppppppeeeeeeeeee fillsspaperr20223 Conferencee pdf
[26]
Laura Kopf. Philine Lou Bommer, Anna Hedström, Sebastian Lapuschkin, Marina M.-C. Höhne, and Kirill Bykov. Cosy: Evaluating textual explanations of neurons. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 34656-34685. Curran Associates, Inc., 2024. URL httpsstts://///// http://procceedii....eeee nwwrips......... cc/ wwpppprr.cpppppppppppppep lappp__ppplla fiw.fiilee/eeell (tilee//paper/224///111 Coner .pd]
[27] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodde Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition, 2022. UtIpsttps: httpps////rxxxy org/ htsp/bbb/22209.. 2266229999906002222222
[28] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, edi- tors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
siaper_filee//pppr////////i patpe///1//////leee A.BBccOTeeac33277777776222fffff3 Paper, pdf
12

---

[29] Johanna Vielhaben, Stefan Bluecher, and Nils Strodthoff. Multi-dimensional concept discovery (MCD): unifying framework with completeness guarantees. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URI KxBQPZ7HKh
[30]
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, 7ac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. URL pub/ 2023/monosemantic--ooo
index html
[31] Judy Borowski, Roland Simon Zimmermann, Judith Schepers, Robert Geirhos, Thomas S. A. Wallis, Matthias Bethge, and Wieland Brendel. Natural images are more informative for interpreting CNN activations than state-of-the-arr synthetic feature visualizations. In Neur!PS 2020 Workshop SVRHM, 2020. URL httpttts://ppprreiiiieee -ttp:///www.eetttooddddddddddeee
[32] Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep features. In International Confer- ence on Learning Representations, 2022. URL https::/openrevaat www.:nnt//oommiiiemmm
MudBMY-tzDr
[33] Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Iyya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can ex- plain neurons in language models. https: htttps://opeaaippa http://pppaaapbbi..... wwo.ubllc..o.......oooooo weore.wwndose net/
neuuraa-explaaneer/pppee//e/eeee inteee//dddex.m.mmmlm 2023.
[34] Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks, 2015. URL http:///reeearc..ogggoeeo httpo/loo..cm//m111111 com/2015/06/
inceptionism- -ett--oinndddeeeerrr into-neural hetmmuurral.mmmllll...
[35] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization.
Distill, 2017. doi: 10.23915/distill.00007 URL https://disttil.ppb//0077
feature- visualization
[36] Thomas Fel, Thibaut Boissin, Victor Boutin, Agustin PICARD, Paul Novello, Julien Colin, Drew Linsley, Tom ROUSSEAU, Remi Cadene, Lore Goetschalckx, Laurent Gardes, and Thomas Serre. Unlocking feature visualization for deep network with magnitude constrained optimization. In Advances in Neural Information Processing Systems, volume 36, pages 37813-37826, 2023. URL httphtttt//prrcceedinggss.. oppp:/ia/arrraaaa000ccrrrraaaaa0
pdf
[37] Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus-Robert Müller, and Pan Kessel. Explanations can manipulated and geome- try is to blame. In Advances in Neural Information Processing Systems, volume 32, 2019. URL https: httppe://prooceediggs htwp://cccediinnnnnnrieieee.eeep ctt.ccccpppaa wc/paper.ep..../.pppp wwtpppperfilleepppeeeeeeeeeeeeee httpar/fleeee//ppe//1/////// 28866001ac12220060888666e2222223 Paper. pdf
[38] Juyeon Heo, Sunghwan Joo, and Taesup Moon.
Fooling neural network interpretations via adversarial model manipulation. In Advances in Neural Information Processing Systems, volume 32, 2019.
http:/rorooeeiiinnn..neireeppppp www.rieer.... http:/aarrfflessppeeeeeee fLle://ee3377600002ff000666
pdf
[39] Christopher Anders. Plamen Pasliev, Kessel. Fairwashing explanations with off-manifold detergent. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 314-323. PMLR, 13-18 Jul 2020. h httph:ttprocoeeeeiiiee leepaaissmml..reessseee
Dombrowski, Klaus-Rober Müller, and Pan
mlr. wwwppseeeiiiii v11e andeaanaa.aaaana
[40] Chhavi Yadav, Ruihan Wu, and Kamalika Chaudhuri. Influence-based attributions can be manipulated. arXiv preprint 2xXiv22409.05208, 2024.
13

---

[41] Robert Geirhos, Roland S. Zimmermann, Blair Bilodeau, Wieland Brendel, and Been Kim. Don't trust your eyes: on the (un)reliability of feature visualizations. In Forty-first Interna- lional Conference on Machine Learning, 2024. URL WtR.l.ttt::///pppptt httpss://openreveet http:/ppereeieeeeeee/forrrrrr net/forum?
id=s0Jvdolv21
[42] Yoav Gur-Arich, Roy Mayan, Chen Agassy, Atticus Geiger, and Mor Geva. Enhancing automated interpretability with output-centric feature descriptions, 2025. URL https
//arxiv. org/ www.://gg///abeeeeeee/eooeesee// 2001.22501.000 08319
[43] Seil Na, Yo Joong Choe, Dong-Hyun Lee, and Gunhee Kim. Discovery of natural language concepts individual units of enns. In International Conference on Learning Representations, 2019. URL http:::tt//ppeeeeeeee
[44] Maximilian Dreyer, Jim Berend, Tobias Labarta, Johanna Vielhaben, Thomas Wiegand, Sebas- tian Lapuschkin, and Wojciech Samek. Mechanistic understanding and validation of large ai models with semanticlens, 2025. URL /
05398
[45] Bruno Puri, Aakriti Jain, Elena Golimblevskaia, Patrick Kahardipraja, Thomas Wiegand, Woj- ciech Samek, and Sebastian Lapuschkin. Fade: Why bad descriptions happen to good features, 2025. URLhtttp http://wwwwv..../.o.a// wwwp./o.rrrabaa///2222 16994
[46] Gonçalo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. Automatically interpreting millions of features in large language models, 2024. URL
13928
[47] Stephen Casper, Tong Bu. Yuxiao Li, Jiawei Li, Kevin Zhang, Kaivalya Hariharan, and Dylan Hadfield-Mence Red teaming deep neural networks with feature synthesis tools. Advances in Neural Information Processing Systems, volume 36, pages 80470- 80516, 2023. URL https: //ttocee///roooeeiingg neurips cc/ wwtpppppppppaaaapeeee www.ss/papeer////eeaaeaaaaaa Fil1://be6660333///3cc5555000022 Confeeeaerrrrrrreeeeeeeeee pdf
[48] R. Schlegel, M. Vess, and Arndt. To discover to create: metaphors and the true self. Journal of Personality, 80:969-993, 2012. doi: 0..111/j.1467-6444..1 1.00753.x.
[49] Xiaoting Zhong, Brian Gallagher, Shusen Liu, Bhavya Kailkhura, Anna Hiszpanski, and
T. Yong-Jin Han. Explainable machine learning in materials science. npj Computational Materials, 8()):004, 2022. doi: 10.1038/4411244-22-008844 1038/s41522 020-5224--222008--777
URL URL/httss://dot /tttps://doi.oor/10
[50] Dennis Grinwald, Kirill Bykov, Shinichi Nakajima, and Marina MC Höhne. Visualizing the diversity of representations learned by bayesian neural networks. Transactions Machine Learning Research, 2023. ISSN 2835-8856. htt.httts:///tt
net/forum?id
ZSxvyWrX6K
[51] Geraldin Nanfack, Alexander Fulleringer, Jonathan Marty, Michael Eickenberg, and Eu- gene Belilovsky. Adversarial attacks on the interpretation of neuron activation maximiza- tion. Proceedings of the AAAI Conference on Artificial Intelligence, 4355)44333332444 Mar. htt.//aaa.vv8i.228888. v3825.28228.... URL https://ojs. aaai
2024. doi: 10. 1609/aaaa.i
article/view/ wwt/i/////22222/// 28228
[52] Geraldin Nanfack, Michael Eickenberg, and Eugene Belilovsky. From feature visualization to visual circuits: Effect of adversarial model manipulation, 2024. URL /ttps: org/ org/ abs/2406. 013665
[53] Anh Nguyen, Jason Yosinski, and Jeff Clune. Understanding Neural Networks via Feature Visualization: A Survey, pages 55-76. Springer International Publishing, Cham, 2019, ISBN 978-3-030-28954-6 doi: 01.1000/978-3-0300 3-030-28954-6 4. URL https://: org.orrg/.... 1001000.0000 978-3 030- 28954-6_4
[54] K. Hornik, M. Stinchcombe, and H. White. Multilayer feedforward networks are universal approximators. Neural Nenw, 2(5):359-366, July 1989. ISSN 0893-6080.
14

---

[55] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. informee neural networks: deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
[56] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR, 18-24 Jul 2021. URL htt.httts//prroeedingggg mlr. press/ http///11///rraee wtt.////rffordaaaaraaaa
[57] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[58] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmistl, 1998.
[59] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
[60] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Toronto, ON, Canada, 2009.
[61] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL https: :etts://aariie org/ 2012. 6980
[62] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[63] Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. CS231N, 2015.
[64]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlshy. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. URL https://openreevee.. http:/ppwrrreeeeeeee///rrrrrrraa
[65] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg. and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, URL httpt:/t///aet org/ 2og//0.0007/
115(3)211--22,, 2015. doi: 911263-015-0816-
[66] Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
[67] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM, 64(3):0071155 feb 2021. ISSN 0001-0782. doi: 10. 145/3446776. URL http.tttts:///oiii.. org/ 10. 1145/3446776
[68] Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. In Vitaly Feldman, Alexander Rakhlin, and Ohad Shamir, editors, 29th Annual Conference on Learning Theory, volume 49 of Proceedings of Machine Learning Research, pages 907-940, Columbia University, New York, New York, USA, 23-26 Jun 2016. PMLR. URL https //proceedings.nt pww.s/peesseeeeeee parrss/v49/// h449//eddn16.... html
[69] David Balduzzi, Marcus Frean, Lennox Leary, J. Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Leurning Research, pages 342-350. PMLR, 06-11 Aug 2017. URL, htttps::/proceeddi mlr, www.:s/w.ww......./ee.e.m..e press//700
bal hmml
15

---

[70] Frederik Pahde, Maximilian Dreyer, Moritz Weekbecker, Leander Weber, Christopher J. Anders, Thomas Wiegand, Wojeiech Samek, and Sebastian Lapuschkin. Navigating neural space: Revisiting concept activation vectors to overcome directional divergence. In The Thirteenth International Conference on Learning Representations, 2025. URL. https:/tt:///// /tttps:://peeereeeee Slt/fforum?dddQ5MMaii
[71] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wist explanations for non-lineat classifier decisions by laser-wise relevance propagation. PLOS ONE, 10(7):1-46 07 2015, doi: 10. foo3al//jurmal.pone
0130140. URL htttpp::/ddo..rree 10. 1371/ pone. 0130140
[72] Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Viégas, and Martin Wattenberg. An interpretability illusion for bert, 2021. URL httth:///aaaat /ttp:///aivvvvv/a org/ abs/2104.0711
[73] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances Neural Information Processing Systems, vol- ume 25, 2012. URL htttps: neur
neur heurips... wc/paper.rp.. files/ walles/pal www.ss/pppeee///////
ff112339982222666766666666666660 pdf
[74] OpenAl. Clip.
2021.
[75] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE Transactions Image Processing, 13(4))600-612, 2004. doi: 10. 109/T1P.2003.81999 819861.
[76] Snchil Sanyal. Weapon Detection Test. Kaggle dataset, 2023. Available at https://wwx haa.ge...ooraaaaeee///allaaaaaaa
[77] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL httpss///ppeereeiee http:/pppnreveee.neee//rrtom net/forum? id=Bkg6RiCqy7
[78] Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In Proceedings of the 18th ACM international conference Multimedia, pages 1485-1488, 2010. 2010.
[79] Mayukh Deb. Feature visualization library for pytorch. httpps///gihuub..cc/////aaaaeee 2021.
[80] Tensorflow. Lucid. httper//gthab.ooo//coooor/oo//// 2017.
[82] Serre-lab. Horama. http:s//gghaab..oo//rretttaaaaap 2021.
[81] greentfrapp. Lucent.
2023.
[83] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global repre- sentations by penalizing local predictive power. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc. E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 32, Curran Associates, Inc., 2019, URL hoce: neurips. cc/
paper_ hag.gaaseeaapprrraatoereeeeeeeee
pdf
16

---

A Related Work on Feature Visualization Manipulation
In this section, we provide a more detailed description of the various attacks on Feature Visualization (FV) from Sec. 2.2 well methodological comparison to Gradient Slingshots (GS).
Fooling Circuits The first attack scheme proposed by Geirhos et al. [1 introduces a fooling circuit: a module consisting of six interconnected units (see Fig. [ where each one of them has a dedicated function. It is designed to route natural and synthetic inputs generated throughout the FV optimization pro- cedure along separate computational paths. The original model located in unit F, while routing controlled by an auxiliary classifier E, trained to distinguish natural from synthetic inputs. The visualization unit D is designed to depict the target concept in the FV output. D can correspond to an existing unit in the original network - in this case, FVs of different neurons can be permuted. However, to encode an arbitrary image (as in Gra- dient Slingshots), a custom unit D' introduced by embedding the target image into the weights of a single convolutional filter with a kernel size of 224, matching the input resolution. Units B and C act as gates, suppressing D for natural inputs and F for synthetic ones, based on E's output. Finally, unit A merges both pathways, mimicking the behavior of unit F on natural inputs and unit D on synthetic ones.
deceptive unit
gate unit
C
gate
unit
E classifier
F natural input unit
visualization unit
Figure 8: Fooling circuit for manip- ulation of FV. The original model (unit F) is embedded in a circuit such that the circuit output A acti- vates as F on natural images, but unit with a desired FV on syn- thetic images. Adapted from Fig. 3 in
The second attack method proposed by Geirhos et al. [41] leverages silent units- units that remain inactive for all inputs in the training set but become strongly activated by FV signals. The authors modify an intermediate layer of a ResNet-50 by replacing a standard convolution- computational block y with manipulated block y y + Ay, where the added term Ay is constructed to be zero (or near zero) for natural images but dominates the output ÿ during FV optimization. This is done by adding an orthogonal perturbation to the original convolutional filters along with negative bias calibrated to suppress activation on natural inputs. The method relies on the empirical observation that FVs typically produce stronger activations than natural images, allowing the manipulated units to dominate the FV outputs without affecting the network's 's normal inference behavior. Although the their experiments and description are limited to
Silent Units
authors state the method is specific convolutional blocks. In comparison, we demonstrate the efficacy of Gradient Slingshots on both CNNs and vision transformers (ViTs).
While the work by Geirhos et al. 41 provides the first viable proof-of-concept that FVs can be arbitrarily manipulated, both the fooling circuit and silent units approaches rely architectural modifications. From an adversarial perspective, such attacks are less practical than fine-tuning te--ttunnnneeddd methods. Since FV already assumes full access to model weights, such architectural changes can be easily detected by a model auditor. This is especially relevant in settings where standardized architec- tures, such as Inception-VS or ResNet-50, are employed and architectural tweaks are particularly noticeable. Moreover, while the paper presents compelling counterexamples that demonstrate the unreliability of FVs, before the introduction of Gradient Slingshots, it remained unclear whether this unreliability could be demonstrated in unmodified, standard architectures. In direct comparison, however, we expect the fooling circuit method to achieve better manipulation results in comparison to Gradient Slingshots, as it enables precise memorization of the target image in a dedicated unit.
ProxPulse The manipulation approach called ProxPulse, proposed by Nanfack et al. [52), relies on fine-tuning rather than architectural changes, similar to our Gradient Slingshots method. Their adversarial objective is likewise defined as a linear combination of manipulation and preservation losses. The manipulation loss encourages high activations within a p-ball around the target image by "pushing up" the lowest activations in that region. In contrast to our method, however, this high-activation region not explicitly connected to the initialization region of FV. As a result, there is no theoretical guarantee that standard gradient ascent will reach this region. The preservation loss defined as the cross-entropy between the outputs of the original and manipulated models, and does not explicitly control for the preservation of model internal activations. It therefore remains
17

---

unclear whether ProxPulse alters the model's internal representations or merely its explanations. This distinction is critical: if the target feature no longer encodes the original concept, the method may no longer qualify manipulation of the explanation alone. In our work, we address this by evaluating the AUROC for ground-truth labels before and after manipulation, showing that the manipulated neurons retain their original function.
In the following, we directly assess how ProxPulse compares Gradient Slingshots in terms of manipulation effectiveness, feature functionality preservation, and overall model performance. This comparison uses a ResNet-50 model under the same setup as in Sec. [41 of the main paper (target image: Dalmatian; manipulated neuron: "wolf spider"). We adopt the hyperparameters reported by[52] and additionally vary the a hyperparameter of ProxPulse to illustrate the trade-off between model accuracy and FV manipulation effectiveness. The quantitative comparison is provided in Table 4 and the qualitative examples are illustrated in Fig- | The results show that Gradient Slingshots produces FVs better aligned with the target concept and less aligned with the ground-truth label, while preserving model accuracy more effectively than ProxPulse. Gradient Slingshots outperforms ProxPulse across all metrics.
Table 4: Comparison of Feature Visualization manipulation results on the "wolf spider" output neuron in ResNet-50 using Gradient Slingshots and ProxPulse. Metries include test accuracy (%), AUROC for the "wolf spider" class, the mean + standard deviation of alignment and similarity metrics, computed over 100 independent FV runs. Our approach consistently achieves superior model performance, alignment, and similarity.
Alignment
Similarity
Model AUC
Acc.
Target Lbl. f
GT LbI. J
CLIP f
LPIPS 4
Original 1.00
76.13
0.23 + 0.01
0.29 + 0.02
0.53 + 0.02
0.69 + 0.01
ProxPulse a= 1. 1--5 1.00 69.03
0.24 0.01
0.01
0.53
0.01
1.06 0.05
ProxPulse a = 0.1 1.00 58.51
0.22 + 0.01
0.31
0.01
0.54 + 0.01
0.70
Gradient Slingshots
1.00
75.13
0.31 + 0.01
0.23 + 0.01
0.69 + 0.02
0.59 + 0.02
ProxPulser: low C
Prox high
Gradient Slingshots
Original PV
( rodoo
Target Image
Figure 9: Qualitative comparison with the fine-tuning-bas attack ProxPulse 522
B Target Image Memorization in Gradient Slingshots
The Gradient Slingshots method fine-tunes model such that the FV output of a target feature yields an image resembling an arbitrary target. Intuitively, this requires the model to be able to "memorize" the target- i.e., to distinguish it from all other points in the input space. In CNNs, convolutional filters act as an information bottleneck: they are trained to recognize specific patterns and discard irrelevant features. While, in principle, a single convolutional filter with the same spatial resolution as the input could directly encode the target (as implemented by Geirhos et al. [41). practical CNN architectures impose constraints through limited filter size and count, well pooling operations.
Therefore, we hypothesize that the CNN's ability to align the FV of its feature with the target should improve as the number and the spatial resolution of these filters increase relative to the model input dimensions. Empirically, we support this hypothesis by demonstrating in Sec. 4 that on lower- dimensional datasets such as MNIST and CIFAR- 10 where the input filter resolution and count are large relative to the input dimensions (see Appendices C.n and C.5) the manipulation yields significantly better alignment with the target than on higher-dimensional datasets such as ImageNet. Additionally, Sec. 4.33, we show that increasing the CNN width, and consequently the number of channels per layer, further improves the similarity between the manipulated FV and the target.
18

---

Another potential memorization strategy relies on leveraging existing internal representations, Intu- itively, although some input information is progressively discarded during inference, vision models tend to map semantically termediate layers. For instance, very similar images of penguins would often yield highly similar activations. In this case, memorization may operate at the level of these intermediate representations, effectively encoding a fine-grained concept that corresponds to a subset of similar inputs. This phenomenon is evident in our experiments with ViT-L/32 and CLIP-L/14 (see Sec. 4 where the manipulated FVs capture high-level properties the target. However, the resemblance is not exact: as shown in Fig. I. the manipulated output depicts clearly different penguin from the target, albeit of the same species, in a similar pose, and with similar background elements.
perceptually similar images to similar internal representations at in-
C Experimental Details
We provide additional experimental details for the illustrative toy example Sec. 3.3 as well as for the evaluations in Secs. 4 6 These include information metrics, datasets, model architectures and weights, manipulation and FV procedures, target similarity evaluation protocols, details of the case study in Sec. 5 and compute resources.
C.1 Toy Experiment
In this section, we descrihe the experimental details related to experiments from Sec. the dataset, the model architecture, the training and manipulation procedures.
3.3 including
Dataset Initially, 2-dimensional classification problem was formulated by uniformly sampling 512 data points for the positive class within the two-dimensional ball A+ = {q: ||||| < q E IR² } and
the same number of points for the negative class from the disc A {q: 4 << ||q|| < 5, q TR?)
The dataset partitioned into training and testing subsets, with 128 and 896 data points, respec- tively.
The MLP architecture as follows: input (2 units) -> fully connected (100 units) 5 softmax (2 units). A Tanh activation function was applied after each linear layer, except for the final layer. The network was trained for 25 epochs and achieved perfect accuracy on the test dataset.
Model
Manipulation
The Gradient Slingshots method employed to manipulate the post-softmax neuron responsible for the score the positive class. the manipulation phase, the "slingshot" and the "landing" zones were defined follows:
lq q 4) L {q q q < 4},
B {q
(13)
(14)
where q = (15,-20), and q' = (20, - 10), and the "tunnel" TR, was constructed according to Equation
For the set U (uniform samples from the "tunnel" TE. we generated a total of 50000 points. The set X consisted of | 15000 points, with both coordinates independently sampled from normal distribution. (00 10). The parameter 7 was set to 0.025.
C.2 Experimental Settings
For brevity, each experimental setting is assigned label, listed in Table These labels used throughout the remainder of the paper to refer to the corresponding configurations.
C.3 Additional Details about Evaluation Metrics
For semantic alignment, the target and ground-truth labels used to compute our metrics are listed in Table 6. In our study, LPIPS is computed using deep embeddings extracted from an AlexNet model [73 pre-trained on ImageNet (65]. For Chas-based similarity and alignment metrics, we employ the CLIP ViT-B/16 model (OpenAI weights (74)).
19

---

Table 5: Labels for the experimental settings in Secs.
Reference
Model
Setting
Sec. 4.1
6-1ayer CNN
6-layer CNN
Sec.
4.1
VGG-9 [59
VGG-9
Sec.
Modified VGGs
VGGs-Size
Secs.
4.
and 4.4
ResNet-18 62
ResNet-18
Sec.
Secs.
and 6
ResNet-50
62
ResNet-50
Sec.
4.1
ViT-L/32
VIT-L/32 664
Fig.
Secs.
and
CLIP ViT-L/14
CLIP ViT-L/14
We report two additional similarity metrics, Mean Squared Error (MSE) and the Structural Similarity Index (SSIM) [75]. throughout Appendix D MSE quantifies pixel-wise reconstruction error, with values closer to indicating greater resemblance between images. SSIM a perceptual similarity metric ranging from 0 to where higher values indicate stronger perceptual similarity and value of 1 denotes identical images.
Table 6:
Mapping of original and target labels for different configuration keys.
Setting
Original Label
Target Label
6-layer CNN
an image of zero
an image cross symbol
VGG-9
image of a cat
an image of the word fake
ResNet-18
an image of a gondola boat
an image a combine harvester
ResNet-50
image of a wolf spider an abstract picture of broccoli
an image of dalmatian an abstract picture of sea lions on beige rocks
ViT-L/32
C.4
Datasets
In Table 7 we provide train test splits and preprocessing details for all experimental settings except CLIP VTT-L/14 For each of these settings, the training set is used when models are trained from scratch. A subset (or the entirety) of this training set is also used to compute the preservation loss (see Appendix C.6). The test set used consistently across experiments to report classification accuracy and AUROC, and to perform Activation Maximization (AM) in the natural domain.
Table 7: Dataset details, train/test splits, image resolutions, and preprocessing steps used during training and/or Gradient Slingshots fine-tuning.
Train-Test Split
Image Size
Setting
Dataset
Train Preprocessing
6-layer CNN VGG-9 and VGGs-Size
MNIST 58
80% / 20% Default (80% / 32x32
x8828
Normalization only
CIFAR-10 60
Resize to 32x32, random hori- zontal flip (p=0.5), 4px padding, random crop to x2, normal- ization
20%)
ResNet-18
TinylmageNet (6380%
20% of 64x64
Resize to 64x64, random hori- zontal flip (p=0.5), 4px padding, random crop to 64,,4, normal-
train set
ization
ResNet-550
ImageNet 65
Default; val. 224x224
Random resized crop to 224x224, random rotation (0-20%), horizontal flip (p=0.1),
used as test
normalization
ViT-L/32
ImageNet
Default;
val. 224x224
Random resized crop to x24x224, random rotation
used as test
(0-20%). horizontal flip (p=0.1), normalization
For training of the weapon detection head the CLIP VIT-I/14 setting, we used the Weapon Detection Dataset (76) for all weapon categories except "assault rifle". We scraped 267 images of
20

---

"assault rifles" from Wikimedia Common allocating 200 for training and adversarial fine-tuning, 40 for evaluation, and 27 for feature vector computation. We applied the same protocol for "gentoo penguin" image collecting 240 in total: 200 for training and adversarial fine-tuning, and 40 for evaluation. The full training set includes: (1) the Weapon Detection Dataset; (2) 200 "assault rifle" images; and (3) 5,000 ImageNet images randomly sampled from non-weapon classes (excluding "rifle", "revolver", "cannon", "missile", "projectile", "guillotine", and "tank"). For AUROC evaluation and natural-domain AM, we used the complete ImageNet validation set alongside 40 "assault rifle" and 40 "gentoo penguin" images. All images were resized to 224 pixels on the shorter side, center- eropped to 224x224, and normalized. Images shown in Fig. 7 are all from ImageNet.
C.5 Model Architecture and Training
The 6-layer CNN architecture is as follows: input conv (5x5, 16) -> max pooling (2x2)-> conv (5x5, 32) -> max pooling (2x2) -> fully connected (512 units) -> fully connected (256 units) -> fully connected (120 units) fully connected (84 units) softmax (10 units). ReLU employed as the activation function in all layers, with the exception of the final layer. We trained the model with the SGD optimizer using learning rate of 0.001 and momentum of 0.9 until convergence. The final test set accuracy of this model is 99.87%.
The CNN architectures for VGGs-Size are de- tailed in T In VGG-9, the configuration A64 is used. Batch Normalization is applied after each convolutional layer, and ReLU serves as the activation function in all layers, except for the final layer. The convolutional layer stacks of models "A64", "B64", "C64", and "D64" align with those in the VGG11, VGG13, VGG16, and VGG19 architectures [59]. The original 16 mod- els for CIFAR-10 10 were trained using Adam 77 with learning rate of 0.001 and weight de- cay of 0.01 until convergence. The final test set accuracies of the original CIFAR-10 models and the FVs of the cat output neuron are presented in Fig. 10
model width
x8
x16
X32
x64
77.32%
82.81% 1%
85.25% 9
86.33%
B
model depth
78.46%
82.61
87.42%
89.37%
80.12%
83.79%
88.32%
88.03%
In the ResNet-18 setting, we adapted ReaNet-18 for the lower-resolution inputs of the TinylmageNet dataset by replacing the initial 7x7 convolution and 3x3 max-pooling with single 3x3 convolution (stride 1, padding 1). We trained the model with SGD with learning rate 0.001 and momentum 0.9 for 30 epochs. The final test accuracy in this setting is 71.89%
75.01%
83.62%
84.61 %
88.65%
Figure 10: 16 classification models of varying depth ("A' -DD") and width (x8 x64) trained on CIFAR-10 were manipulated to change the FV of the cat output neuron a fish image. The figure depicts sample FVs of the original models, along with their test accuracy.
To construct the weapon-detection model in the CLIP VIT-L/14 setting, we added a multilayer perceptron (MLP) on top of a frozen CLIP-L/14 visual encoder. The MLP architecture as follows: input -> fully connected (512 units) -> ReLU output (1 unit). We trained the MLP optimizing with Adam (learning rate 1e-4, batch size 32) for 5 epochs.
The sources for the weights of models that we did not train from scratch are provided in Table g
C.6 Gradient Slingshots Fine- Fine-Tuning Procedures
Manipulation Sets For the 6-layer CNN, we targeted Pixel-AM. with the manipulation set sampled
directly the input (pixel) domain. In all other settings, manipulations were targeting Fourier FV,
https://commons wikimedia. wwwpwh/wwwww.......o....o.aaaaaa
MediasBearchhttype-iigg
ltttps://comonns..tii www.wiwia...ag//e/eeeee..aaaa.ae
Mediaseearchktypee-nne
21

---

Table CIFAR-10 CNN configurations with added layers. The convolutional layer parameters are denoted as conv (receptive field (numbe of channels)". The numbers of channels are expressed as a multiplicative factor xr, where is a parameter controlling the width of model. The batch normalization layers and ReLU activation function are not shown for brevity. The model depth configurations are labeled from A" to "D"
Layers
A
B
C D
input (32 x 32 RGB image)
conv3-(1 x x))
conv3- x r)
maxpool
conv3-(2 x
co x2)
maxpool
conv3-(4 x
convl conv1-(4x7 conv3-(4Xx) r
maxpool
conv3-(8 x r)
conv3- x
conv conv1-(8 r)) co
maxpool
conv3-(8 x
conv3-(8 x
convl-(8xx r)
conv3-(8 r)
maxpool
FC-8 x
Dropout(O.5)
FC-10
Table 9: Weights specification and sources of the pretrained models.
Model
Pretrained Weights Source
ResNet-50
Torchvision ResNet50 BesNett5_WWiihhs... MMAGENETIK_VV V1 78 78
ViT-L/32
Torchvision ViT ViT_L_32 Weights IMAGENETIK V1 OpenAI CLIP ViT-L/14 774
CLIP ViT-L/14
so the manipulation set was sampled in the frequency domain. Prior to being fed into the models, these frequency-domain points were processed by scaling, applying an inverse 2D real Fast-Fourier Transform (FFT), converting from the custom colorspace to RGB, and finally applying a sigmoid function to map values to the [0.1] range for visualization, following implementations from the lucid and tor ch-dreems libraries 779. The radii of the slingshot
and landing zones are provided in Table [1] At each fine- tine-tuning step. number of points equal the batch size (see Table are sampled uniformly from the tunnel region defined in Eq. (6). constructed based on the slingshot and landing zones as described in Sec. 3.1
Preservation Sets In Table 10 we provide details on the construction of datasets used for the calculation of the preservation loss Eq. (TT). We used a subset of ImageNet for preservation in CLIP ViT-L/14, we do not have access the original training data. While for smaller training datasets we used the full train set as the preservation dataset, for ImageNet we only used subset. In the ViT-L/32 and CLIP ViT-L/14 settings, we observed that taking too many fine-tuning steps results in worse manipulation performance, as the activation landscape of the models' features changes too drastically; therefore, those cases, we tuned the size of the subset along with other Gradient Slingshots hyperparameters to achieve the best performance.
22

---

Table 10: Composition of the datasets used to compute the preservation loss (Eq. during Gradient Slingshots fine-tuning.
Setting
Preservation Set
6-layer CNN
The full train set.
VGG-9
The full train set. The full train set. The full train set.
VGGs-Size
ResNet-18
ResNet-50
Subset of 64 classes in the train set (ImageNet), including "Dalmatian", "wolf spider."
VIT-L/32
10% of "sealion" and "broccoli" classes and 0.12% of samples of other classes in the train set (ImageNet).
CLIP VIT-L/14
0.28% of ImageNet 200 images scraped from Wikimedia Commons under categories "Assault rifles" and "Pygoscelis papua" each.
Hyperparameters of Gradient Slingshots Table II summarizes the hyperparameters and speci- fications of the Gradient Slingshots attack. Standard FV libraries typically initialize visualizations with zero-mean Gaussian noise 779 82. which makes initialization predictable an adversary; our tuning process is based on coarse-to to-fine
therefore, we set = 0.0. For the parameter iterative approach. We begin with broad sweep of values (e.g., a E [0.01, 0.05, 0.5, 0.9]) identify a promising region. Once a candidate such as a 0.5 shows good results, we perform finer-gnu emer-grained search around (e.g., 0.4 and 0.6), and continue refining around the hest-performing values, e-g.. near 0.6. Importantly, we perform tuning using only 1-3 epochs and only on small subsets for large training datasets (see Table TOp. As for the parameter w, we set to 0.1 when the target layer contains 10 neurons, and to 0.01 when it contains 1000 neurons preventing the target neuron's 's activation landscape from being overly disrupted. For 7, we use the heuristic (( o (0) = 7q to preserve the gradient magnitude in the initialization zone.
Table 11: Gradient Slingshots method hyperparameters and specifications: from Eq. 12 from Eq. (IT), y from Eq. (9), constant C' for Eq. (I0). the radius the slingshot zone" σB and the landing zone" Ø used define the sampling "tunnel" in Eq. (6), and the choice of manipulation loss L True indicates that the activation based manipulation loss (Eq. (10) was used instead of the gradient-based loss (Eq. (9).
Setting
W
C
LM
6-layer CNN
0.8
10
NA
0.1
0.1
False False
VGG-9 VGGs-Size
0.025
10
NA
0.1
0.1
0.01
0.1
10
NA
0.1
0.1
False
ResNet-18
0.995
0.01
1000
NA
0.01
0.01
False
ResNet-50
0.64
0.01
200
NA
0.01
0.01
False
ViT-L/32
0.9995
0.01
100
1.0
0.01
0.01
True
CLIP VIT-L/14
0.999915
2000
1.0
0.01
0.01
True
Fine-tuning specifications are provided in Table 12 All models were fine-tuned using the AdamW optimizer with the listed learning rate (L.R), weight decay, and a numerical stability constant e ADAM (corresponding in Algorithm of (77)). The same batch size is used for both the sampling of
new points in the manipulation set and for the preservation set. For VIT-L/32 and CLIP VIT-L/14, we additionally multiply both loss terms by constant factor 0.0001, for numerical stability.
Target Images The target images and their sources are listed in Table 13 All images not created
by us were cropped to square format and resized to match the input dimensions required by respective models. To encode the target images into the frequency domain, we apply inverse sigmoid function, convert the images to a custom color space, perform a 2D real FFT, and scale the resulting frequency representation.
23

---

Table 12: Training and optimization parameters for Gradient Slingshots fine-tuning. "Weight Decay" and e cAAAM refer to the corresponding parameters of the Adam optimizer.
Setting
Epochs
LR
Weight Decay
e ADAM
Batch Size
VGG-9
50
0,0001
0.001
le-08
32
VGGs-Size
100
0,001
0.001
le-08
32
6-layer CNN
30
0.001
0.001
lc-08
32
ResNet-11
5
le-05
0.001
le-08
64
ResNet-50
10
1e-06
0.001
1e-08
32
ViT-L/32
0.0002
0.05
le-07
16
CLIP ViT-L/14
2e-06
0.01
le-07
8
Table 13: Target images, their sources, and licensing attributions where applicable.
Setting
Target image
Displayed in
Source / License
VGG-9
"FAKE"
Fig.
Created by us ImageNet [65
VGGs-Size
"fish"
Fig.
6-layer CNN
"cross"
Fig.
Created by us ImageNet
ResNet-18
"harvester"
Fig.
"grey"
Fig.
(A)
Created by us Created by us
"pink"
Fig.
(B)
"dog sketch"
Fig.
(C)
ImageNet-Skent
83
"tiger"
Fig.
(D)
TinylmageNet
663
"nail"
Fig.1
TinylmageNet
"lion"
Fig.
TinylmageNet
"basketball"
Fig.
(G)
ImageNet
"bird in water"
Fig.1
(H)
ImageNet
"bird on a branch"
Fig.
ImageNet
"cacadoo"
Fig.
ImageNet
"spider"
Fig.
(K)
TinylmageNet
"python"
Fig.
ImageNet
puppy"
Fig.
19
(M)
TinylmageNet
ResNet-50
Dalmatian"
Fig.
Photo by Maja Dumat /CC BY0
VIT-L/32
"sealions"
Figs
and 12
Photo by William /arby/CCCB BY
CLIP VIT-L/14
"gentoo penguin"
Fig.
Photo by William Warby BC
C.7 Feature Visualization Procedures
we specify the parameters of the FV, including the initialization parameters, the FV step size, the number of FV optimization steps, well as the regularization strategy. We draw the initialization vector by sampling each of its elements from the normal distribution N(u,, ar), following the implementations in lucid and torch-dreams. For pixel-AM, the initialization signal is in the input domain. For FV, the initialization signal is sampled in the scaled frequency domain and transformed into the pixel domain employing the scaled FFT function, as described in Appendix C.6 When comparing the AM output before and after manipulation, the FV procedure parameters remain consistent.
In Table 14
Table 14: Specifications of FV procedures. μ4 and o denote the mean and standard deviation of the FV initialization distribution. "FV Strategies" refers the optimization and regularization recipe used.
Setting
P1
a1
Step Size
Steps
FV Strategies
6-layer CNN
0.0
0.01
0.1
200
GC
VGG-9
0.0
0.01
100
GC
VGGs-Size
0.0
0.01
0.1
700
GC
ResNet-18
0.0
0.01
0.01
200
Adam GC TR
ResNet-50
0.0 0.01
0.01
500
Adam + GC + TR
VIT-L/32
0.0
0.01
0.003
2000
Adam GC TR
CLIP ViT-L/14
0.0
0.01
0.002
3000
Adam + GC + TR

---

The following defines the abbreviations used for optimization and regularization techniques under the "FV Strategy" column in Table
Gradient clipping (GC) Gradient clipping is a method employed to mitigate the issue of exploding gradients, typically observed in DNNs. This method is also being used in the scope of synthetic FV. We constrain the gradient norm to 1.0.
Transformation robustness (TR) Transformation robustness has been introduced as a technique aimed enhancing the interpretability of FVs. This technique is realized through the application of random perturbations to the signal at cach optimization step and facilitates finding signals that induce heightened activation even when slightly transformed [35 [3
For ResNet-18 and ResNet-50, we apply the following sequence of transformations:
padding the image by pixels on all sides using a constant fill value of 0.5;
random affine transformation with rotation degrees sampled from -20" to 20", scaling factors from 0.75 to 1.025, and fill value 0.5;
random crop to the target input resolution (224 x 224), with padding needed (fill value 0.0).
For ViT-based settings VIT-L/32 and CLIP ViT-L/14, adapt the transformation robustness strategy to better align with the architectural characteristics of the models. The following sequence of transformations applied during FV optimization:
padding the image by 16 pixels on all sides using a constant fill value of 0.0;
random affine transformation with rotation degrees sampled from the range -20" to 20%, scaling factors from 0.75 to 1.05, and a constant fill value of 0.0;
another random rotation in the range -20" to 20 with fill value 0.0;
addition of Gaussian noise with mean 0.0 and standard deviation 0.1;
random resized crop back to the original model input resolution (224 224), with fixed aspect ratio and scale sampled from the range (0.5, 0.75).
The transformation sequences were selected through empirical experimentation with various recipes, drawing inspiration from the standard implementations 80 using the torchvision library.
All transformations are implemented
Adam Adam 611, popular optimization algorithm for training neural network weights, can also be applied in FV settings.
C.8 Evaluation Procedure for Target Similarity
For each evaluation model, we generated 100 samples of FV outputs, with the exception of CLIP ViT-I/144 for which only 30 samples were computed due to computational constraints. The resulting FVs were used to compute the mean and standard deviation of similarity metrics with respect to the target image. Note that FV generation is stochastic, as the initialization point sampled from distribution. The standard deviations thus characterize the consistency of the metries across these independently generated FVs.
C.9 Application
For the experiments in the CLIP-L/14 setting, computed the "assault rifle" feature vector in the residual stream layer
the visual module of CLIP-L/14. We followed the Pattern-CAV approach [70]: we took the difference between the mean activations of 27 "assault rifle" images and the mean activations of 500 randomly sampled ImageNet images. The "assault rifle" images here were handpicked to have the concept be very present and recognizable in them.
25

---

C.10 Compute Resources
All experiments were conducted using a workstation with 24 GB NVIDIA RTX 4090 GPUs and compute cluster with 4 x 40 GB NVIDIA@ A100 GPUs. Each experiment was performed on single GPU. The approximate time required compute Gradient Slingshots manipulation for each setting is as follows:
6-layer CNN: hour
VGG-9: hours
VGGs-Size: 2-5 hours
ResNet-18: hour
ResNet-50: 3 hours
ViT-I//2:: 1 minute
CLIP ViT-L/14: minutes
Overall, activation-bass manipulation (Eq. (0O). applied in the VIT-L/32 and CLIP ViT-L/14 settings, is substantially more computationally efficient. However, observed that this approach less effective in CNN-based architectures. Outside Gradient Slingshots fine-tuning, evalua- tion procedures described in Secs. 4 and 6 including performance measurement, target similarity computation, and activation extraction on test sets, took approximately 12 hours total.
D Additional Experiments
This section presents supplementary experiments extending main results. We include additional FV visualizations showing manipulation stability across random seeds (Appendix [.I). evaluate GS the MACO feature visualization method (Appendix [D.2p, verify that non-target features remain unaffected (Appendix D.3), and manipulate additional neurons to demonstrate generalizability (Appendix D.4). We further analyze the accuracy-manipulation trade-off across multiple settings (Appendix D.5). extend results on the effect of target images (Appendix D.6), and provide additional results for our defense method (Appendix) D.7
D.1 Additional Visualizations
present additional examples of manipulated FV outputs, extending the qualitative results in Figs. and to demonstrate the stability of manipulations.
G-layer
CNN
VGS-9
ResNet-18
V=I-L/32
CLIP VIT-L/14
Figure 11: Additional manipulated FV examples for Figs. and demonstrating stable manipulation results.
26

---

D.2 Manipulating MACO Feature Visualizations
While we demonstrate that standard FV methods with adequate transformation robustness recipes (see Sec. [4.1 and Appendix D.I) can yield interpretable results in modern architectures such as CLIP ViT-L/14, MACO (3) remains a popular alternative for transformers. Like Fourier FV [35), MACO operates the Fourier domain but restricts the optimization parameter space, making directly compatible with GS. We evaluate GS's effectiveness with MACO by generating FVs using the official Horama library 82 in the ViT-L/32 setting, using the same manipulated network as in Sec.
Original
Manipulated MACO
Target Image
"brocoll"" 0.322 0.023
0.242 = 0.010
"sea lions" 0 0.325 0.012
Figure 12: Manipulation results of GS applied to MACO visualizations of the "broccoli" output neuron ViT-L/32. We report the mean = standard deviation of alignment metrics, computed over 100 independent FV runs. GS effectively conceals the feature's ground-truth semantic meaning.
As shown in Fig. 12 the manipulated MACO FVs consistently exhibit stronger alignment with the target label ("sealions")) than with the ground-trute -rruh label ("broccoli"), replicating the manipulation results observed with standard Fourier-based FVs. Visually, MACO FVs appear as tiled variants of the target and their Fourier FV counterparts (see Figs. 3 and TT) due MACO's recommended transformation robustness recipe, which crops and zooms on 20 25% of the image at each step.
D.3 Effect of Manipulation on Non- Non-Target Features
To verify that our manipulation does not introduce negative side effects on non-target features, we compare FVs of non-target output neurons between the original and manipulated models in the ResNet-50 setting. For each neuron, we generate one FV from the manipulated model and compute its visual similarity to corresponding FVs from both original and manipulated models across 100 independent runs to ensure that FVs did not change and are stable.
The results, presented in Table 13 and Fig. 13 show virtually difference in the FVs of non-target neurons between the two models before and after manipulation across all metrics, confirming that GS leaves non-target features unaffected.
Table 15: Comparison of visualizations before (left) and after (right) GS for non-manipulated features the ResNet-50 setting. Reported are the mean = standard deviation of similarity metrics between the manipulated FV and the target, computed over 100 independent FV runs. These results demonstrate that the GS attack does not perturb the visualization of non non-manipulated features.
Distance an Original FV
Neuron
CLIP t
MSE +
("tench"")
0.89 + 0.03 0.90 + 0.03
0.18 + 0.02
("goldfish")
0.02 0.94 0.02
0.16 + 0.01 0..17
("great white shark")
0.95 0.02
0.01
0.18 + 0.01 1..18
('tiger shark"")
0.03 0.90 0.02
0.19 1.02
4 ("hammerhead")
0.95 0.01 10.82 + 0.09
0.0
("electric ray")
0.95 0.02 0.93 0.02
0.01
6 ("sttigray"")
0.96 0.02 1..96 0.01
("rooster")
0.92 0.03 0.02
0.18 1.01
("hen")
0.96 0.01 00.6 0.01
0.19 0.010000000000000000
("ostrich"")
0.93 + 0.02
0.17 + 0.01
27

---

Tench
Goldfish
Tiger Shark
Hammerhead
Electric Rey
Strngroy
Rooster
Hon
Oatich
Original Modal
FV after
Gradiant
Slingshots
[torget: spider" neuron)
Figure 13: FV examples of the first 10 non-target output neurons in ResNet-50 setting, where the "wolf spider" neuron was targeted for GS manipulation. Non-target FVs remain virtually unaffected.
D.4 Manipulating Other Features
To demonstrate that the features selected for manipulation were not cherry-picked, we manipulate the first 10 neurons in the ResNet-50 setting. We present the results of this experiment in Fig. [4] and Table T6 Across all cases, the manipulated FVs are consistently more semantically aligned with the target label than with the ground-truth label. Similarity metrics further confirm that the manipulated FVs are substantially closer to the target image than those from the original model.
Table 16: Manipulation results under GS attack for the visualization the first 10 output features in the ResNet-50 setting. Reported are test accuracy (in %), AUROC for the ground-truth labels (e.g.. "tench," "goldfish," " "great white shark," etc.), and the mean t standard deviation of alignment and similarity metrics, computed over 100 independent FV runs.
Alignment
Similarity
N Model
AUC
Acc.
Target Lbl. f
GT Lbl. 4
CLIP f
MSE
LPIPS +
0
Original
76. 76.13
1.00
0.22 0.01
0.31 + 0.01
0.55 + 0.02
0.13 + 0.01
0.72 0.01
Manipulated
75.23
1.00
0.32 1:2200.0
0.21 + 0.01
0.67 + 0.02
0.12 + 0.00
0.59 + 0.01
1 Original
76.13
1.00
0.23 0.01
0.28 + 0.01
0.53 4 0.02
0.13 + 0.01
0.73 + 0.01
Manipulated
75.15
1.00
0.32 1.01
0.25
0.01
0.68 1: 0.02
0.13 + 0.01
0.61 1 0.02
2
Original
76.13
1.00
0.24 1
0.28
0.02
0.53
0.02
0.13 + 0.01
0.69 1 0.01
Manipulated
75.17
1.00
0.33 1 0.01
0.23 0.01
0.72
1721:000 0.02
0.12 1 0.00
0.60 1 0.01
3 Original
76.
1.00
0.27 0.02
0.14 0.01
0.70 + 0.01
Manipulated
75.18
1.00
0.32 +: 0.01
0.22 0.01
0.69
+
0.02
0.12 + 0.00
0.61 + 0.01
4 Original
76.13
1.00
0.28 0.01
0.14 0.02
0.01
0.71 + 0.01
Manipulated
75.20
1.00
0.32
+.3200001
0.23 - 0.01
0.01
0.70 0.02
0.12 + 0.00
0.60 + 0.01
5 Original
76.13
0.99
0.04
0.32 0
0.02
0.01
0.70 +.01
Manipulated
75.23
0.99
0.33
0.70 0.19
0.12
0.60 0.01
6 Original
76.13
0.99
0.20
0.31 0.02
0.03
0.12
0.73 -: 0.01
Manipulated
75.28
0.99
0.32
0.19
0.69 0.12
0.61 0.01
7
Original
76.13
1.00
0.20
0.31 1 0.01
0.04
0.13
0.71 + 0.01
Manipulated
75.56
0.99
0.30
0.26 - 0.01
0.01
0.67 0.00
0.61 0.02
8 Original
76.13
1.00
0.31 0.02
0.57 =
0.13
0.72 0.01
Manipulated
75.49
1.00
0.31
0.01
- 0.01
0.68
0.03
0.13 1 0.01
0.60 0.01
9 Original
76.13
1.00
0.20 0.01
0.31 = 0.01
0.49 0.02
0.13 = 0.01
0.70 +
Manipulated
75.46
1.00
0.29 0.02
0.26 - 0.02
0.65
0.04
0.12 - 0.00
0.61 0.02
Tench
Goldiiih
Great White
Shark
Tiger Hommerhead
Electri
Sinngroy
Hon
FV manipulated
Gredient
Slingshots (target:
Dalmation)
Figure 14: Manipulating the first 10 output neurons in ResNet-50 with GS. Manipulated FV examples consistently align with the target image.
28

---

D.5
Accuracy- Manipulation Trade-Off
We expand on the experimental results of the eecuracy-manipuaee trade-off analysis for the Gradient Slingshots attack, introduced in Sec. 4.2 with additional findings reported in Tables 17 to 21 and Figs. 13 T8
Table 17: eecuracy-manipulattoon trade-off for the GS attack on the "zero" output neuron in 6-layer CNN Reported are test accuracy (in %), AUROC for the "zero" class, and the mean + standard deviation of alignment and similarity metrics, computed over 100 independent FV runs.
Alignment
Target Similarity
AUC
Acc.
Target Lbl. f
GT LbI. 4
CLIP 1
MSE |
LPIPS
SSIM
Original
1.00 99.67
0.29 + 0.01
0.27 + 0.00
0.88 + 0.01
0.13 + 0.01
0.15 + 0.01
0.04 0.03
0.990 1.00 99.53
0.29
0.01
0.27 + 0.00
0.88 0.01
0.13
0.15
0.06 0.03
0.900 1.00 99.28
0.30
0.26
0.89
0.10
0.05
0.12
0.800 1.00 98.98
0.00
0.25
0.01
0.03
0.05 0.02
0.700 1.00 98.89
0.32
0.25
0.95
0.00
0.76 0.00
0,500 1.00 98.10
0.25
0.2410.01
0.06
0.12
0.29
0.07
0.200 0.99 96.34
0.03
0.20
0.04
0.04
0.11
0.22
0.050 0.99
94.36
0.27
0.00
0.01
0.00
0.24
0.05
0,010 0.87
70.88
0.00
0.95 +.00
0.00
0.00
0.01
0.005 0.84
52.52
0.33 1 0.00
1.00
0.96
0.01
0.02 0.00
0.03
0.82 0.01
0.001 0.54
19.39
0.33 + 0.00
0.25 + 0.00
0.95 + 0.00
0.02 + 0.00
0.01 0.00
0.77 + 0.01
Original
0.99
0.8
0.7
0.005
Manipulated
FV
Figure 15: Sample FVs different values of for 6-layer CNN.
Table 18: Accuracy-manniulation trade-off for the GS attack on the "cat" output neuron in VGG9. Reported are test accuracy (in %), AUROC for the "cat" class, and the mean t standard deviation of alignment and similarity metrics, computed over 100 independent FV runs
Alignment
Target Similarity
a AUC Acc.
Target Lbl. f
GT Lbl. +
CLIP 1
MSE +
LPIPS
SSIM 1
Original
0.97 86.33
0.24 0.01
0.29 + 0.01
0.74 0.02
0.08 + 0.00
0.25 + 0.02
0.03 + 0.02
0.9900 0.97 86.10
4.24
0.29 + 0.01
0.74
0.02
0.08 + 0.01
0.25 + 0.02
0.03 0.02
0.8000 0.97 87.20
0.24
0.29
0.03
+.09+
0.02 0.26
0,5000 0.97 83.90
0.00
0.20
0.01
0.01+0.00
0.19 0.01
0.1000 0.97 86.40
0.00
0.24
0.02
0.01
0.02
0.05
0.0500 0.96 85.10
0.00
0.04
0.02
0.00
0.27 0.02
0.0250 0.96 85.15
0.00
0.24
0.78
0.00
0.06
0.33 0.04
0.0100 0.95
81.90
0.05
0.24
0.01
0.00
0.40 0.05
0.0050 0.94
82.20
0.30
0.20
0.07
0.01
0.08
0.51 0.05
0.0010 0.86
72.70
0.29 + 0.03
0.22
0.87
0.00
0.00
0.53 - 0.04
0.0001 0.54
9.50
0.24 + 0.01
0.24
0.01
0.80 + 0.01
0.02 + 0.00
0.14 0.02
0.30 - 0.05
a Original
0.99
0.8
0.5
0.1
0.05
0.025
0.01
0.005
0.001
0.0001
Manipulated
FV
Figure 16: Sample FVs at different values of a for VGG-9.
29

---

Table 19:
sourrcy-manipulation trade-off for the GS attack on the "gondola" output neuron in ResNet-18 Reported are test accuracy (in %), AUROC for the "gondola" class, and the mean + standard deviation of alignment and similarity metrics, computed over 100 independent FV runs.
Alignment
Target Similarity
AUC
Acc.
Target Lbl. t
GT Lbl. 4
CLIP f
MSE
LPIPS 4
SSIM 1
Original
0.99
71.89
0.21 + 0.01
0.25 + 0.02
0.64 + 0.03
0.09 + 0.01
0.59 + 0.04
0.03 + 0.02
0.999000
0.98
70.30
0.21 + 0.01
0.25 0.02
0.65 0.03
0.09 0.01
0.05
0.04 : 0.02
0,997000
0.97
69.50
0.24
0.21
0.73
0.07
0.01
0.01
0.06 0.03
0,995000
0.97
71.40
0.25
0.24
0.07 0.03
0.07 0.04
0.03
0,993000
0.96
68.40
0.02
0.23 0.02
0.07 0.03
0.06
0.07 0.03
0,990000 0.94
69.30
0.02
0.22 0.02
0.73
0.07
0.07 0.02 0.07
0,950000 0.87
39.10
0.22
0.18 0.01
0.10
0.58
0.04 0.02
0,900000 0.79
11.00
0.23
0.01
0.19
0.70 + 0.02
0.08
0.09 0.03
0,500000
0.63
0.60
0.23 = 0.01
0.20 - 0.01
0.61
0.03
0.01
0.00
0.03 0.02
0.100000
0.51
0.80
0.22 1 0.01
0.18 1 0.02
0.68 1 0.03
0.09 1 0.03
0.63 1 0.14
0.10 1 0.02
Original
0.999
0,997
0.995
0.993
0.99
0.95
0.9
0.5
0.1
Manipulated
FV
Figure 17: Sample FVs at different values of a for ResNet-18.
Table 20: Accuracy-manipuattio trade-off for the GS attack on the "wolf spider" output neuron in ResNet-50. Extended version of Table I including additional a values and similarity metrics.
Alignment
Target Similarity
α AUC
Acc.
Target Lbl. f
GT Lbl. 4
CLIP 1
MSE 4
LPIPS
SSIM 1
Original
1.00 76.13
0.23 + 0.01
0.29 = 0.02
0.53 + 0.02
0.12 0.01
0.69 + 0.01
0.05
0.90
1.00
76.07
0.22
0.01
0.28 0.02
0.02
0.02
0.69
0.05
0.80
1.00
76.00
0.22
0.28
0.52
0.13
0.01 0.00
0.70 1.00
75.77
0.02
0.27
0.53
0.02
0.02
0.01
0.64 1.00
75.13
0.31
0.00
0.02
0.12
0.00
0.01
0.62 1.00
74.83
0.01
0.20
0.02
0.12
0.59
0.04
0.60 1.00
74.51
0.32
0.23
0.02
0.12
0.04 0.02
0.01
0.58 1.00
74.45
0.02
0.23
0.02
0.12
0.04 0.00
0.56 1.00
73.67
0.32+0.00
0.22
0.02
0.00
0.04 0.01
0.01
0.50 1.00 71.52
0.32
0.00
0.72
0.11
0.05 0.02
0.40 1.00 66.58
0.31
0.24
0.03
0.12
0.66
0.01
0.30 0.99
$2.09
0.31
0.24
0.00
0.10
0.05
0.06
0.20 0.97
21.97
0.29
0.22
0.00
0.11
0.08 0.00
0.10 0.90
30.19
0.29
0.04
0.02
0.00
0.08 0.01
0.02
0.05 0.64
0.21
0.27 0.01
0.01
0.04
0.00
0.11 0.02
0.01 0.61 0.14
0.00
0.24 0.01
0.53 0.01
0.09
0.00
0.07 0.08
0.00
Table 21: Accuracy-manipulttio trade-off for the GS attack on the "brocoll"" output neuron in VIT-L/32. Reported are test accuracy (in %), AUROC for the "broccoli" class, and the mean standard deviation of alignment and similarity metrics, computed over 100 independent FV runs.
Alignment
Target Similarity
AUC
Acc.
Target Lbl. f
GT Lbl. 4
CLIP f
MSE
LPIPS +
SSIM 1
Original
1.00 76.97
0.23
0.02
0.28 + 0.02
0.51
0.02
0.08 + 0.00
0.73
0.03
0.08 + 0.01
0.999999
1.00
76.58
0.22 0.02
0.28 1 0.02
0.51
0.02
0.08 1 0.00
0.73
0.03
0.08 1 0.01
0.999900
1.00 76.64
0.29 1 0.03
0.27 1 0.03
0.53
0.09
0.04
0.08 1 0.01
0.999500
0.99 76.06
0.29 1..000222
0.2511.00
0.02
0.07
0.74
0.081000
0.999000
0.99
75.96
0.28
0.01
0.09
0.07
0.09 0.02
0,990000
0.99
72.42
0.29
0.25+0.01
0.09
0.00
0.11 0.71
0.100000
0.50
0.10
0.25
0.27 = 0.01
0.49
0.05
0.18 0.01
0.01
30

---

a Original 0.999999
0.9999
0.9995
0.999
0.99
0.1
Manipulated
Figure 18: Sample FVs at different values of a for ViT-L/32.
D.6 Impact of Target Image on Manipulation
We provide additional quantitative evaluation of the effect of the target image manipulation success, extending the results from Sec. 4.4 to an additional set of target images, shown in Fig. 19
Target source
Holdout samples
K
Target image
Manipulated
FV
A Accuracy, %
-2.29
-1.79
-4.39
target
0.48.0.00
0.31+0.02
0.52=0.10
Figure 19: Manipulation results for different target images by source. Extended version of Table 13 including additional target images.
D.7 Attack Detection
We provide additional qualitative results for attack detection scheme in Fig. 20 extending the evaluation described in Sec. to additional experimental settings.
9 random samples from top-100 AM: after manipulation
Tap-9 samples: before manipulation
Top-9 samples: after manipulation
0
A
6-1eyer CIN
2
2
D
VOG-9
Reahet 18
VIT-L/S3
Figure 20: Top-9 most activating test samples before and after the GS attack. While the attack can substantially change which samples rank as the most activating across dataset, broader sampling from the top of the AM distribution often recovers the original concept.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We demonstrate in Sec. and Sec. with additional results in the Appendix, that:
(a) Our method can "conceal the true functionality of arbitrary neurons by replacing their original explanations with selected target explanations", We support this by visualizing Feature Visualizations (FVs) from manipulated models and quantitatively measuring their similarity the target image.
(b) This manipulation largely "preserves the model's external behavior and internal mecha- nisms", demonstrated by the relatively small changes the classification accuracy and the AUROC scores of the target features in various settings.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations discussed in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
32

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: To establish the theoretical possibility of our attack, we provide formal proof, along with its assumptions, in Sec. 3.1
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
The experimental details necessary to reproduce our results, including our algorithm, the datasets, models, methods, metrics, and parameters use, are provided throughout the main text and in the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
33

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the code in the supplementary materials, and use publicly available models, datasets and methods.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
public/gust wwwcc//gggsss/eeeeeee httee//gide//oouoiiiiieeeeeeeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify and justify the training and test details, including dataset splits, hyperparameters, and optimizer types, primarily in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
34

---

Answer: [Yes]
Justification: We report standard deviations for similarity metrics between target images and manipulated FVs in Sec. and Sec. 5 and include them in the experimental results in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computational resources are detailed in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wwrpppcpooooo/ocaaaaaaaa
Answer: [Yes]
Justification:
This paper complies with the NeurIPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
35

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The potential societal impact is described in Sec.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper is not accompanied by the release of data models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The license terms of use are fully respected, with details and attributions listed in the Appendix.
36

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provided our code and use publicly available models, datasets and methods.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsoureing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
37

---

Justification:
This paper does not involve crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is required.
Answer: [NA]
Justification: This paper uses LLMs for text editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy http:///eearssssscooo/ooorrreers for what should or should not be described.
38


Reviews:
[{"review_id": "CB1uSMrALr", "reviewer": "Reviewer_Tk8Z", "summary": "This paper introduces \"Gradient Slingshots\" (GS), a proposed attack method targeting Feature Visualization (FV), an explainable AI technique. The core premise is that if FV is used to audit AI models for fairness or other properties, an adversary could subtly manipulate their model's training to conceal malicious intent. The authors claim GS improves on prior methods by being architecture-agnostic (including transformers) and by maintaining the manipulated model's performance. The technique reportedly achieves this balance by optimizing a trade-off between the FV image's resemblance to a target and the model's original behavior, controlled by parameters alpha and w.", "questions": "Questions:\n\n1. Could the authors clarify the meaning of the standard deviations reported throughout the paper? Are they based on 95% confidence intervals from multiple experimental runs? If so, what is the number of runs? Alternatively, are they computed for a single run or a single FV?\n\n2. Is there a methodology for determining reasonable hyperparameter values without resorting to an exhaustive grid search? What would the hyperparameters and experimental results look like if such a methodology were employed? Demonstrating this would significantly enhance confidence in the method's practical utility.\n\n3. In real-world auditing practices, do auditors commonly rely on Feature Visualization to assess models?", "limitations": "Yes, the paper acknowledges limitations. Depending on the response to Question 3, the authors might consider engaging with real-world auditors to make them aware of the paper findings..", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 1, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The visual aspects of the paper are strong, with well-presented figures.\n- The underlying concept of the attack appears theoretically sound.\n- The authors made an effort to provide theoretical justification for their approach.\n- The supplementary information in the appendix was a useful addition.\n\nWeaknesses:\n\nThe reviewer has significant concerns about the paper's experimental rigor and the validity of its claims, stemming from several key issues:\n\n1. Lack of Comparative Analysis: The paper fails to compare GS against any existing baselines. While it suggests that methods like Nanfack et al. might degrade model performance, no empirical evidence is provided to substantiate this claim. This omission makes it difficult to assess GS's true effectiveness.\n\n2. Unconvincing Similarity Metrics: The reported similarity metrics (e.g., MSE and SSIM) between the manipulated FV and the target image do not show statistically significant improvement. For instance, these metrics remain identical for alpha values between 1.0 and 0.4, despite a notable degradation in model performance (from 76% to 67%) within that range. This raises questions about the practical impact of the proposed attack.\n\n3. Apparent Cherry-Picking of Results: The selection of specific alpha values in Table 1 (e.g., 0.64 and 0.62, which are very close to 0.6) while omitting results for 0.7 or 0.8 suggests a selective presentation of data. This \"cherry-picking\" makes it challenging to understand the full landscape of hyperparameter effects and raises doubts about the robustness of the findings.\n\n4. Excessive Hyperparameter Tuning: The wide range and extreme precision of `alpha` values presented in the appendix (e.g., 0.8, 0.025, 0.01, 0.995, 0.64, 0.9995, 0.99991) strongly indicate a reliance on extensive grid searching to find optimal parameters. If the method is truly generalizable, such extreme fine-tuning should not be necessary. This undermines confidence in the method's practical applicability.\n\n5. Unproven Efficacy on Transformers: Despite being a purported advantage over prior work, the paper does not convincingly demonstrate the methodology's effectiveness on transformer architectures. The extremely high alpha values used for ViT models (e.g., at least 0.9995) suggest that the method may simply be injecting noise, rather than performing a targeted attack.\n\nThreat Model Plausibility:\nThe paper is framed as a security attack, but the reviewer questions the realism of the proposed threat model. Unmanipulated FVs are typically abstract. It is unlikely that an auditor would solely rely on these abstract visualizations without incorporating other metrics or auditing procedures. This diminishes the practical significance of the described attack.\n\nPresentation and Quality:\nThe paper exhibits some signs of being rushed or lacking polish. For example, Table 2 suffers from poor formatting, indicative of a general lack of attention to detail.", "was_revised": true, "final_justification": "I thank the authors for their response. The rebuttal is promising, but I believe that the paper will require substantial changes to fully address concerns. My score will remain the same.", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback. We are glad that they found the core idea of our method theoretically sound, appreciated the visual presentation and figures, and found the supplementary material helpful. Below, we address the reviewer’s concerns in detail, providing additional experimental results, methodological details, and elaborating on the practical applicability of our method.\n\n### Weaknesses\n\n> 1. Lack of Comparative Analysis: The paper fails to compare GS against any existing baselines. While it suggests that methods like Nanfack et al. might degrade model performance, no empirical evidence is provided to substantiate this claim. […]\n\nWe appreciate the reviewer’s concern about the lack of empirical comparison with prior work. While conceptual differences were discussed in Appendix A, we now include a direct qualitative and quantitative comparison with the only comparable fine-tuning-based method: Prox-Pulse (PP) [Nanfack et al., 2024]. This comparison uses a ResNet-50 setting from Sec. 4 (target image: Dalmatian; manipulated neuron: wolf spider). We follow PP’s reported hyperparameters and additionally vary its $\\alpha$ to show the trade-off between model accuracy and manipulation result. \n\nAlongside original similarity metrics, we introduce two new semantic alignment measures:\n\n- Target Label Alignment (\"Target Lbl.\"): CLIP cosine similarity between a FV and the target label embeddings (e.g., “an image of a Dalmatian”).\n- GT Label Alignment (\"GT Lbl.\"): CLIP similarity between a FV and the ground-truth feature label (e.g., “an image of a wolf spider”).\n\nResults (Table A) show that Gradient Slingshots (GS) yield FVs more semantically aligned with the target image and less aligned with the ground truth label, while preserving model performance better than Prox-Pulse. GS outperforms Prox-Pulse on all similarity metrics except low-level metrics MSE and SSIM.\n\n**Table A: Comparison of FV manipulation results on the “wolf spider” output neuron in ResNet-50 between GS and Prox-Pulse (PP).**\n\n| \\$\\alpha\\$ | AUROC |    Acc↑   |  Target Lbl.↑ |    GT Lbl.↓   |     CLIP↑     |      MSE↓     |     LPIPS↓    |   SSIM↑   |\n| :--------: | :---: | :-------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: | :-------: |\n|  Original  |  1.00 | **76.13** |   0.23±0.01   |   0.28±0.02   |   0.53±0.02   |   0.12±0.01   |   0.69±0.01   | 0.05±0.01 |\n|   PP 1e−5  |  1.00 |   69.03   |   0.24±0.01   |   0.24±0.01   |   0.53±0.01   | **0.08±0.00** |   1.07±0.03   | **0.08±0.00** |\n|   PP 0.1   |  1.00 |   58.51   |   0.22±0.01   |   0.31±0.01   |   0.54±0.01   |   0.12±0.01   |   0.70±0.01   | 0.05±0.01 |\n|   GS 0.64  |  1.00 |   75.13   | **0.31±0.01** | **0.23±0.01** | **0.68±0.02** |   0.12±0.01   | **0.59±0.02** | 0.04±0.01 |\n\n\n> 2. Unconvincing Similarity Metrics: The reported similarity metrics (e.g., MSE and SSIM) between the manipulated FV and the target image do not show statistically significant improvement. […]\n\nWe thank the reviewer for raising the concern. Low-level metrics (MSE, SSIM) are more useful in settings with low-res datasets (e.g., MNIST, CIFAR), where manipulated FVs resemble targets pixel-wise more closely (as noted in Section 4 and Appendix B). In high-res settings, these metrics fail to capture compositional or semantic alignment (see Fig. 3). CLIP similarity better reflects high-level alignment but may still disregard compositional structure.\n\nTo further support our evaluation results, we added multiple samples of manipulated FVs per evaluation. We also extended all quantitative evaluations using Target/GT Label Alignment metrics described above. These metrics explicitly quantify conceptual alignments that are invisible to traditional similarity scores. Some of the results using these metrics are shown in Tables A, B, C in this rebuttal.\n\n> 3. Apparent Cherry-Picking of Results: The selection of specific alpha values in Table 1 (e.g., 0.64 and 0.62, which are very close to 0.6) while omitting results for 0.7 or 0.8 suggests a selective presentation of data. […]\n\nThe $\\alpha$ values in Table 1 were chosen to show representative trade-off states between model performance and FV manipulation success. Within 0.64-0.9, metrics vary marginally, so we did not report these values.\n\nTo address the concern, we have now extended Tab. 1 with $\\alpha=0.7$ and $0.8$ (see Table B), providing a full sweep from 0.1 to 0.9. These additional results confirm the trends described in Section 4.1. Additionally, in response to reviewer PQ6K, we provide quantitative evidence showing that the neuron targeted for manipulation itself was not cherry-picked by providing evaluation results for 10 more neurons in the ResNet-50 setting. We hope this clarifies our reporting and alleviates concerns about selective presentation.\n\n**Table B: Accuracy–manipulation trade-off for the GS attack on the “wolf spider” output neuron in ResNet-50 (partial Tab. 1).**\n\n| \\$\\alpha\\$ |    AUC   |  Acc. |  Target Lbl.↑  |    GT Lbl.↓    |   CLIP↑   |    MSE↓   |   LPIPS↓  |   SSIM↑   |\n| :--------: | :------: | :---: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |\n|     0.8    | 1.00 | 76.00 | 0.22±0.01 | 0.28±0.02 | 0.53±0.02 | 0.12±0.01 | 0.69±0.01 | 0.05±0.01 |\n|     0.7    | 1.00 | 75.77 | 0.23±0.02 | 0.27±0.02 | 0.53±0.03 | 0.13±0.01 | 0.67±0.02 | 0.05±0.01 |\n\n\n> 4. Excessive Hyperparameter Tuning: The wide range and extreme precision of alpha values […] strongly indicate a reliance on extensive grid searching to find optimal parameters. […]\n\nWe thank the reviewer for raising this concern. A detailed description of our hyperparameter tuning and selection procedure is provided in our response to Question 2 below, and we have incorporated this explanation into the revised paper.\n\n> 5. Unproven Efficacy on Transformers: Despite being a purported advantage over prior work, the paper does not convincingly demonstrate the methodology's effectiveness on transformer architectures. The extremely high alpha values used for ViT models (e.g., at least 0.9995) suggest that the method may simply be injecting noise, rather than performing a targeted attack.\n\nWe thank the reviewer for the observation. $\\alpha$ parameter balances manipulation and preservation losses; high $\\alpha$ does not indicate noise injection but rather that the manipulation loss remains small relative to activation changes during training.\n\nFigure 1 and Section 5 demonstrate strong efficacy of GS on a ViT model (CLIP ViT-L/14). While quantitative metrics for ViT-L/32 (Appendix D) are less convincing than the qualitative results (Fig. 3), this highlights the limitations of conventional metrics in capturing similarity to the target. To address this, we extended the evaluation with the two new alignment metrics introduced above. Results in Table C show that for certain $\\alpha$ values, manipulated FVs align better with the target label (“sea lions”) than the ground truth (“broccoli”), sometimes even surpassing the original FV’s alignment with its ground truth label.\n\n**Table C: Accuracy–manipulation trade-off for the GS attack on the ``broccoli'' output neuron in ViT-L/32 (partial Tab. 16).**\n\n| \\$\\alpha\\$ |    Acc.   |  Target Lbl. ↑  |    GT Lbl. ↓    |\n| :--------: | :-------: | :-------------: | :-------------: |\n|  Original  | **76.97** |   0.22 ± 0.03   |   0.28 ± 0.02   |\n|  0.999999  |   76.58   |   0.23 ± 0.02   |   0.28 ± 0.02   |\n|   0.9999   |   76.64   | **0.29 ± 0.02** |   0.27 ± 0.03   |\n|   0.9995   |   76.06   | **0.29 ± 0.02** | **0.25 ± 0.01** |\n|    0.999   |   75.96   |   0.28 ± 0.02   | **0.25 ± 0.01** |\n|    0.99    |   72.42   | **0.29 ± 0.02** | **0.25 ± 0.01** |\n|     0.1    |    0.1    |   0.25 ± 0.01   |   0.26 ± 0.01   |\n\n### Questions\n\n> 1. Could the authors clarify the meaning of the standard deviations reported throughout the paper? […]\n\nAs explained in Appendix C.8, the standard deviations reflect variability across multiple FV generations—100 per experiment (30 for CLIP ViT-L/14 due to compute limits). Since FV generation involves sampling initializations, each run differs. The deviations thus capture metric consistency across independent runs. We’ve clarified this in all relevant table captions.\n\n> 2. Is there a methodology for determining reasonable hyperparameter values without resorting to an exhaustive grid search? […]\n\nOur tuning of the parameter $\\alpha$ uses a coarse-to-fine iterative approach rather than an exhaustive grid search. We start with a broad sweep (e.g., $\\alpha \\in \\[0.01, 0.05, 0.5, 0.9\\]$) to identify promising regions, then refine around top candidates (e.g., testing values near 0.5), iteratively narrowing down.\n\nImportantly, tuning is done using only 1–3 epochs on small subsets of large datasets (see Table 8 in the Appendix). For ViT experiments, this involves under 1\\% of ImageNet and takes less than 5 minutes per epoch, making the process practical even on modest resources.\n\nFor parameter $w$, we set $w=0.1$ when the target layer has 10 neurons, and $w=0.01$ when it has 1000 neurons, to avoid excessive disruption of the activation landscape of the feature. For $\\gamma$, we use the heuristic $\\nabla (\\phi \\circ \\eta)(\\mathbf{0}) = \\gamma \\mathbf{q^t}$ to preserve the gradient magnitude in the initialization zone.\n\nWe have incorporated this explanation into the paper.\n\n> 3. In real-world auditing practices, do auditors commonly rely on FV to assess models?\n\nWhile FV is not yet standard in traditional audits, it has proven useful for detecting undesirable behaviors (Goh et al., 2021), uncovering backdoor attacks (Casper et al., 2023), interpreting time-series patterns (Schlegel et al., 2024), and understanding CNN filters in materials science (Ling et al., 2017; Zhong et al., 2022). These show FV’s value in the broader context of model interpretability and safety."}, {"review_id": "ZQJXkq2jtk", "reviewer": "Reviewer_PQ6K", "summary": "The paper introduces Gradient Slingshots (GS), a method that modifies a model so that gradient ascent, involved in FV, is drawn towards a wrong optimum. The optimum, which is an image, can be arbitrarily chosen beforehand by the neural network designer, who would seek to fool an audit based on feature visualization. As a result, the result of the FV looks like the wrong optimum rather than the actual FV, potentially hiding the actual concepts underlying the neural network's predictions. GS applies as a fine-tuning that is supposed to preserve the model's predictive capacities and alter only the behaviour of FV's gradient ascent.\n\nThe authors test their method on simple CN, VGG, ResNet18/50, and ViT, on some neurons involved in Cifar10 and Imagenet classification. They also provide a case study based on CLIP and weapon classification to emphasize the risks of such an attack.", "questions": "- How can a manipulator make sure that the FV initialization will fall into $\\mathbb{B}$ and will be appropriately steered towards the wrong optimum? \n- How does GS affect the FV of other features? If the FV is pertubated for every neurons, it might seem suspicious to auditers. \n- In Sec 5, the manipulation and the FV seem to target the feature vector representing an assault rifle. However, FV and GS are only formulated for the maximization of a single neuron. How to make this fall under a formalism similar to Eq. 1, 2, 3, 4, 5? If the answer is contained in lines 290 -> 293, it is not detailed enough, and more description should be provided for reproducibility.\n- The FV method from Olah et al. 2016 [1] notoriously fails on modern networks like ViT [2]. Could the authors run GS with modern FV methods like MACO [2]\n- I am surprised that there is no factor $2$ in Eq. 10 before $\\gamma$, or a factor $\\frac{1}{2}$ in Eq. 9, that stems from derivatives of square expressions.\n\n[1] Distill, Olah et al., 2017 [https://distill.pub/2017/feature-visualization/](https://distill.pub/2017/feature-visualization/) \n[2] Unlocking Feature Visualization for Deeper Networks with MAgnitude Constrained Optimization, Fel et al. NeurIPS 2023", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- The idea is novel and simple\n- The paper is well written - even if it could be slightly rebalanced for improved readability, see below)\n- The manipulation is undetectable by looking at the code or the structure of the model since it is contained in the weights' values.\n\n### Weaknesses\n\n- **W1**: The evaluation is only conducted on one or a few neurons, so it might be subject to cherry-picking.\n- **W2**: The quantitative metrics are not impressive. I am suspicious that GS works on only a few examples selected for the paper illustrations, but statistically, GS seems to only slightly improve the similarity metrics compared to the original FV. I am not convinced that it actually works. It is even more true for ViT (see Appendix, p. 32), for which the metrics barely change compared to the FV without GS.  \n- **W3**: The assumption of l. 139 for Lemma 3.1 seems to make GS artificial. As far as I understand, it means that the initialization must be known by the manipulator and fall in $\\mathbb{B}$. It seems very easy to circumvent for an auditor, for instance, by uniformly sampling several initializations in $\\mathcal{Q}$. \n\n### Comments and suggestions\n\n- The assumption of l. 139 for Lemma 3.1 seems to be very strong, but it is only stated in the proof. It should appear in the Lemma.\n- The authors spend time on the proof of Lemma 3.1, which is intuitive, so they do not need proof to make sense of it. Hence, it takes up some unnecessary space, which could be better used for explaining the metrics or expanding the case study setting if it was deferred to the appendix.", "was_revised": true, "final_justification": "After careful thinking and considering authors-reviewers discussions:\n\n-   I am worried about the current impact of the work since FV ibased auditing is not (yet) a common practice. But I recognize that AI regulation is an increasing trend and is likely to become even more important in the future. I think that it is a sound direction for AI, and that NeurIPS should support it.\n-   The generalization of GS to any image, for which we only have access to quantitative evaluation, is a concern, but if the authors include more qualitative examples in a revised version and these visualizations are compelling, I deem that the article will be worth publishing at NeurIPS.\n\nTherefore, I increased my rating to 4 and strongly encouraged the authors to add a substantial visualization gallery to their work.", "rebuttal": "We thank the reviewer for their thoughtful feedback. We’re pleased they found our method novel and simple, our paper well-written, and appreciated that our attack is difficult to detect. Below, we address their concerns to clarify that our results are comprehensive and not selectively presented, provide additional quantitative evidence, discuss theoretical assumptions, and elaborate on broader applicability.\n\n### Weaknesses\n> W1: The evaluation is only conducted on one or a few neurons, so it might be subject to cherry-picking.\n\nWe thank the reviewer for raising concerns about cherry-picking. To address this, we further manipulated the first 10 output neurons in the ResNet-50 setting from Sec. 4 (target image: Dalmatian). Beyond standard similarity metrics, we added two semantic alignment metrics:\n\n- Target Label Alignment (\"Target Lbl.\"): CLIP cosine similarity between manipulated FV and the target label embeddings (e.g., “an image of a Dalmatian”).\n- GT Label Alignment (\"GT Lbl.\"): CLIP similarity between manipulated FV and the ground-truth feature label (e.g., “an image of a wolf spider”).\n\nResults from Table A (only the first 5 neurons shown due to character limits) show that manipulated FVs are similar to the target image and align more with the target than ground truth label; classification accuracy and AUROC indicate the stability of model and feature performance. This supports that our method generalizes beyond isolated neurons.\n\n**Table A: Manipulation results for the Gradient Slingshots (GS) attack on the first 10 output neurons of a ResNet-50 (partial, 5/10 neurons).**\n\n| Neuron | Model  | Acc.       | AUROC | Target Lbl.↑      | GT Lbl.↓        | CLIP ↑          |\n|--------|--------|------------|-------|-------------------|-----------------|-----------------|\n| 0      | Orig.  | **76.13**  | 1     | 0.22 ± 0.01       | 0.31 ± 0.01     | 0.55 ± 0.02      |\n|        | Manip. | 75.23      | 1     | **0.32 ± 0.01**   | **0.21 ± 0.01** | **0.67 ± 0.02**  |\n| 1      | Orig.  | **76.13**  | 1     | 0.23 ± 0.01       | 0.28 ± 0.01     | 0.53 ± 0.02      |\n|        | Manip. | 75.15      | 1     | **0.32 ± 0.01**   | **0.25 ± 0.01** | **0.68 ± 0.02**  |\n| 2      | Orig.  | **76.13**  | 1     | 0.24 ± 0.01       | 0.28 ± 0.02     | 0.53 ± 0.02      |\n|        | Manip. | 75.17      | 1     | **0.33 ± 0.01**   | **0.23 ± 0.01** | **0.72 ± 0.02**  |\n| 3      | Orig.  | **76.13**  | 1     | 0.24 ± 0.01       | 0.27 ± 0.02     | 0.54 ± 0.01      |\n|        | Manip. | 75.18      | 1     | **0.32 ± 0.01**   | **0.22 ± 0.01** | **0.69 ± 0.02**  |\n| 4      | Orig.  | **76.13**  | 1     | 0.23 ± 0.01       | 0.28 ± 0.01     | 0.52 ± 0.01      |\n\n> W2: The quantitative metrics are not impressive. I am suspicious that GS works on only a few examples selected for the paper illustrations, [...] It is even more true for ViT (see Appendix, p. 32), for which the metrics barely change compared to the FV without GS.\n\nWe thank the reviewer for raising the concern. As noted in Sec. 4 and Appendix B, low-level metrics (MSE, SSIM, LPIPS) are more useful in settings with low-res datasets (e.g., MNIST, CIFAR), where manipulated FVs resemble targets pixel-wise. In high-res settings, these metrics fail to capture compositional or semantic alignment (see Fig. 3). CLIP similarity better reflects high-level alignment but may still miss compositional effects.\n\nTo address the selective representation concerns, we added multiple samples of manipulated FVs per evaluation. We also extended all quantitative evaluations using Target/GT Label Alignment metrics, capturing semantic similarity. Table B (extending Table 16) for ViT-L/32 shows that at some $\\alpha$ values, FVs align more with the target image label (“sea lions”) than the ground truth (“broccoli”), supporting GS’s manipulation efficacy on transformer models.\n\n**Table B: Accuracy–manipulation trade-off for the GS attack on the ``broccoli'' output neuron in ViT-L/32 (partial Tab. 16).**\n\n| α        | Acc.       | Target Lbl.↑       | GT Lbl.↓          |\n|----------|------------|--------------------|-------------------|\n| Orig.    | **76.97**  | 0.22 ± 0.03        | 0.28 ± 0.02       |\n| 0.999999 | 76.58      | 0.23 ± 0.02        | 0.28 ± 0.02       |\n| 0.9999   | 76.64      | **0.29 ± 0.02**    | 0.27 ± 0.03       |\n| 0.9995   | 76.06      | **0.29 ± 0.02**    | **0.25 ± 0.01**   |\n| 0.999    | 75.96      | 0.28 ± 0.02        | **0.25 ± 0.01**   |\n| 0.99     | 72.42      | **0.29 ± 0.02**    | **0.25 ± 0.01**   |\n| 0.1      | 0.1        | 0.25 ± 0.01        | 0.26 ± 0.01       |\n\n> W3: The assumption of l. 139 for Lemma 3.1 seems to make GS artificial. As far as I understand, it means that the initialization must be known by the manipulator and fall in. It seems very easy to circumvent for an auditor, for instance, by uniformly sampling several initializations in $Q$.\n\nWe thank the reviewer and agree the assumption in l.139 may appear strong. We provide further explanation for why this assumption is reasonable in our response to Q1.\n\n>The assumption of l. 139 for Lemma 3.1 […] should appear in the Lemma.\n>The authors spend time on the proof of Lemma 3.1, which is intuitive, […].\n\nWe thank the reviewer for the suggestion. The assumption is now stated explicitly in Lemma 3.1, and the detailed proof has been moved to the Appendix. The freed space was used to clarify evaluation metrics and hyperparameter selection.\n\n### Questions\n\n> 1. How can a manipulator make sure that the FV initialization will fall into and will be appropriately steered towards the wrong optimum?\n\nWe thank the reviewer for the question. Standard FV libraries (Lucid, Lucent, Horama) use fixed Gaussian noise initialization, making the process predictable. While auditors could randomize or broaden initialization, adversaries can expand the “slingshot zone” $\\mathbb{B}$ accordingly. Alternative methods (e.g., seeding from real images) are rare since they often yield FVs resembling the seed (Hamblin et al., 2024).\n\n- Hamblin et al. “Feature Accentuation: Revealing 'What' Features Respond to in Natural Images.”\n\n> 2. How does GS affect the FV of other features? If the FV is pertubated for every neurons, it might seem suspicious to auditors.\n\nWe appreciate the reviewer’s concern and agree it’s important to show GS doesn’t alter FVs of non-manipulated features. To address this, we evaluated FVs of the first 10 non-manipulated output neurons in a GS-attacked ResNet-50 (Sec. 4, $\\alpha=0.64$). For each neuron, we sampled 100 FVs from the original and manipulated models and measured their similarity to a single original FV and the target image.\n\nTable C (first 5 neurons shown due to length limits) shows that non-manipulated neuron FVs remain nearly unchanged, confirming that GS preserves non-manipulated features. Qualitative and quantitative results are now included in the Appendix.\n\n**Table C: Before-and-after-manipulation comparison of visualizations of non-manipulated features in the GS attack on the ResNet-50 model (partial, 5/10 neurons).**\n\n| Neuron | Model  | CLIP Orig. FV  | CLIP Target Img.|\n|--------|--------|--------------|-------------|\n| 0     | Orig.  | 0.89 ± 0.03 | 0.54 ± 0.02     |\n|        | Manip. | 0.90 ± 0.03  | 0.55 ± 0.02     |\n| 1     | Orig.  | 0.94 ± 0.02 | 0.53 ± 0.02     |\n|        | Manip. | 0.94 ± 0.02  | 0.54 ± 0.02     |\n| 2     | Orig.  | 0.95 ± 0.02  | 0.53 ± 0.02     |\n|        | Manip. | 0.96 ± 0.01 | 0.53 ± 0.01     |\n| 3     | Orig.  | 0.93 ± 0.03  | 0.54 ± 0.01     |\n|        | Manip. | 0.90 ± 0.02  | 0.54 ± 0.01     |\n| 4     | Orig.  | 0.95 ± 0.01  | 0.52 ± 0.02     |\n|        | Manip. | 0.82 ± 0.09   | 0.53 ± 0.01     |\n\n> 3. In Sec 5, the manipulation and the FV seem to target the feature vector representing an assault rifle. However, FV and GS are only formulated for the maximization of a single neuron. How to make this fall under a formalism similar to Eq. 1, 2, 3, 4, 5?\n\nWe thank the reviewer for highlighting this point. Both the theory and implementation (Eqs. 1–5) define GS manipulation over a feature function $f$, not just individual neurons. As stated in l. 68–70, $f(\\mathbf{x}):= \\mathbf{v} \\cdot g(\\mathbf{x})$, where $g(\\mathbf{x})$ is a vector of activations and $\\mathbf{v}$ defines a feature direction.\n\n> 4. The FV method from Olah et al. 2016 [1] notoriously fails on modern networks like ViT [2]. Could the authors run GS with modern FV methods like MACO [2]\n\nWe thank the reviewer for the suggestion. While we show that standard Fourier FV with an appropriate transformation robustness recipe (Appendix C.7) can produce interpretable results for modern architectures like CLIP ViT-L/14 (Fig. 1), we agree that MACO is a popular FV approach for transformers.\n\nMACO, like Fourier FV (Olah et al., 2017), operates in the Fourier frequency domain, but with a restricted optimization space, making it compatible with GS. We applied MACO to the existing manipulated ViT-L/32 model (Sec. 4), generating FVs before and after manipulation and evaluating alignment to the target and ground truth labels. As shown in Table D, manipulated MACO FVs show stronger alignment with the target class (“sea lions”) than the ground-truth (“broccoli”), mirroring results with Fourier FVs. We have included both quantitative and qualitative MACO results in the revised manuscript.\n\n**Table D: Manipulation results of GS applied to MACO visualizations of the “broccoli” output neuron in ViT-L/32.**\n\n|       | GT Lbl. ↓   | Target Lbl. ↑  |\n|-------|-----------|--------------|\n| Orig.   | 0.322 ± 0.029  | 0.264 ± 0.031 |\n| Manip.  | **0.242 ± 0.010** | **0.325 ± 0.012** |\n\n> 5. I am surprised that there is no factor in Eq. 10 before , or a factor in Eq. 9, that stems from derivatives of square expressions.\n\nWe thank the reviewer for noticing this oversight on our side. We have corrected the Eq. 4 and the Eq. 10 accordingly. This error is limited only to the paper formalisms and not reflected in our code implementation."}, {"review_id": "UcdRHd7cEl", "reviewer": "Reviewer_zpF6", "summary": "This paper addresses attacks on feature visualization. In a scenario where model developers want to conceal the true feature visualization of certain features from a third-party auditor, their method can modify any feature’s feature visualization to any target image by adjusting the model parameters. This ensures that the model’s performance remains intact, while the feature visualization of the target feature resembles the input image. Essentially, they can alter any single feature visualization to anything they desire.\n\nTheir method fine-tunes the model using a loss compromise consisting of two parts. The first part aims to maintain the activations on unrelated elements unchanged. The second part directs the gradients of data points in a tunnel between the feature visualization initialization points and the target input image towards the target image. They conducted numerous empirical studies to assess the impact of model size, target image familiarity to the model, and other factors.", "questions": "## Questions \n Is there an experiment that shows the effect of model manipulation on other features feature visualization ? Do other features FV also look like the target image?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Weakness \n1. The paper only focuses on last layer features. It does not conduct any experiments on features in middle layers. \n2. Hyperparameter tuning of loss seems very hard as both large and small values lead to the method not working. \n3. The paper claims that internal mechanisms of the model remain unchanged but because they didn’t investigate middle layers their argument is only based on accuracy. \n\n## Strengths \n1. Good writing. \n2. Novel idea for Feature visualization manipulation. \n3. Good variation of model choices. \n4. Many ablation studies.", "was_revised": true, "final_justification": "1) They showed that they really don't change the model behaviour in their rebuttal. \n2) They make the hyperparameter tuning process more clear.", "rebuttal": "We thank the reviewer for their clear summary and thoughtful feedback. We are glad they found our paper well-written, our idea novel, and appreciated the breadth of model choices and ablations. Below, we address the reviewer’s concerns in detail.\n\n### Weaknesses\n\n> 1. The paper only focuses on last layer features. It does not conduct any experiments on features in middle layers.\n\nWhile Section 4 primarily focuses on output-layer features to enable direct quantitative evaluation (e.g., via AUROC against known ground-truth labels), our Gradient Slingshots (GS) method is not limited to the final layer. The general formulation in Section 3 applies to any differentiable feature function within the network.\n\nMoreover, the submitted version of our paper already includes a case study involving the manipulation of a mid-layer feature (Section 5). Specifically, we target a neuron in the 22nd residual block of a CLIP ViT-L/14-based model, identified through a supervised discovery method. Both quantitative results (Table 3) and qualitative visualization (Figure 1) demonstrate that GS manipulation is effective when applied to middle layers, and preserves the feature performance.\n\n> 2. Hyperparameter tuning of loss seems very hard as both large and small values lead to the method not working.\n\nWe thank the reviewer for raising this concern. For the parameter $\\alpha$, our tuning process follows a coarse-to-fine iterative approach. We begin with a broad sweep of values (e.g., $\\alpha \\in [0.01, 0.05, 0.5, 0.9]$) to identify a promising region. Once a candidate such as $\\alpha = 0.5$ shows good results, we perform a finer-grained search around it (e.g., 0.4 and 0.6), and continue refining around the best-performing values, e.g., near 0.6.\n\nImportantly, we perform tuning using only 1–3 epochs and on small subsets of large training datasets (see Table 8 in the Appendix). For ViT experiments, for example, we use less than 1\\% of ImageNet, with a single epoch taking under 5 minutes. While we acknowledge that tuning of $\\alpha$ is computationally intense, the process remains practical even with modest resources.\n\nFor the parameter $w$, we set it to 0.1 when the target layer contains 10 neurons, and to 0.01 when it contains 1000 neurons – preventing the target neuron's activation landscape from being overly disrupted. For $\\gamma$, we use the heuristic $\\nabla (\\phi \\circ \\eta)(\\mathbf{0}) = \\gamma \\mathbf{q^t}$ to preserve the gradient magnitude in the initialization zone.\n\nWe have incorporated this explanation into the paper in order to improve transparency around hyperparameter selection.\n\n\n> 3. The paper claims that internal mechanisms of the model remain unchanged but because they didn’t investigate middle layers their argument is only based on accuracy.\n\nAs noted earlier, we explicitly evaluate manipulation on a feature located in an intermediate layer (Section 5, Case Study with CLIP ViT-L/14). In all our quantitative experiments, beyond overall accuracy, we measure the AUROC of the manipulated feature output with respect to its ground truth label. This targeted metric directly assesses whether the feature’s original performance is preserved. Our key argument is that the target feature is the component most susceptible to alteration by our manipulation. The fact that it maintains its original predictive behavior demonstrates that the model’s reliance on this feature remains intact, and that our method effectively “covers up” the FV without removing or degrading its internal role. We have made this argument more prominent and explicit in the paper.\n\n### Questions\n> Is there an experiment that shows the effect of model manipulation on other features Feature Visualization ? Do other features FV also look like the target image?\n\nWe understand the reviewer’s concern — it is important to demonstrate that GS does not inadvertently perturb FVs of non-manipulated features, which could make GS detectable. To address this, we evaluated FVs of the first 10 non-manipulated output neurons in a GS-attacked ResNet-50 (Sec. 4, $\\alpha=0.64$). For each neuron, we sampled 100 FVs from the original and manipulated models and measured their similarity to a single original FV and the target image.\n\nThe results, presented in Table C below, show virtually no difference in the FVs of non-manipulated neurons between the two models across all metrics, confirming that GS leaves non-manipulated features unaffected. We have included these quantitative results, along with example FVs, to the paper’s Appendix.\n\n**Table C: Before-and-after-manipulation comparison of visualizations of non-manipulated features in the GS attack on the ResNet-50 model where the “wolf spider” output neuron was manipulated. Reported are the mean ± standard deviation of similarity between FVs and a single sampled original FV (CLIP, MSE, LPIPS), and the mean ± standard deviation of similarity between manipulated FV and the target image. Similarity statistics are computed over 100 independent FV runs. These results demonstrate that the GS attack does not perturb the visualization of non-manipulated features.**\n\n| Neuron | Model         | CLIP $\\uparrow$   | MSE $\\downarrow$   | SSIM $\\uparrow$      | CLIP $\\uparrow$   | MSE $\\downarrow$   | SSIM $\\uparrow$   |\n|----------------:|:--------------|:----------------|:---------------|:----------------|:------------------|:-------------------|:------------------|\n| | | Original FV     |       |        | Target  |  |  \n|               0 | Original    | $0.89\\pm0.03$   | $0.18\\pm0.02$  | $0.03\\pm0.01$   | $0.54\\pm0.02$     | $0.14\\pm0.01$      | $0.05\\pm0.01$     |\n|                | Manipulated | $0.90\\pm0.03$   | $0.18\\pm0.01$  | $0.03\\pm0.01$   | $0.55\\pm0.02$     | $0.14\\pm0.01$      | $0.06\\pm0.01$     |\n|               1 | Original    | $0.94\\pm0.02$   | $0.16\\pm0.01$  | $0.03\\pm0.01$   | $0.53\\pm0.02$     | $0.13\\pm0.01$      | $0.05\\pm0.01$     |\n|               | Manipulated | $0.94\\pm0.02$   | $0.17\\pm0.01$  | $0.03\\pm0.01$   | $0.54\\pm0.02$     | $0.13\\pm0.01$      | $0.06\\pm0.01$     |\n|               2 | Original    | $0.95\\pm0.02$   | $0.18\\pm0.01$  | $0.03\\pm0.01$   | $0.53\\pm0.02$     | $0.13\\pm0.01$      | $0.05\\pm0.01$     |\n|               | Manipulated | $0.96\\pm0.01$   | $0.18\\pm0.01$  | $0.03\\pm0.01$   | $0.53\\pm0.01$     | $0.14\\pm0.01$      | $0.06\\pm0.01$     |\n|               3 | Original    | $0.93\\pm0.03$   | $0.19\\pm0.02$  | $0.03\\pm0.01$   | $0.54\\pm0.01$     | $0.15\\pm0.01$      | $0.06\\pm0.01$     |\n|               | Manipulated | $0.90\\pm0.02$   | $0.19\\pm0.01$  | $0.03\\pm0.01$   | $0.54\\pm0.01$     | $0.14\\pm0.01$      | $0.06\\pm0.01$     |\n|               4 | Original    | $0.95\\pm0.01$   | $0.17\\pm0.01$  | $0.03\\pm0.01$   | $0.52\\pm0.02$     | $0.14\\pm0.01$      | $0.05\\pm0.01$     |\n|               | Manipulated | $0.82\\pm0.09$   | $0.16\\pm0.01$  | $0.03\\pm0.01$   | $0.53\\pm0.01$     | $0.14\\pm0.01$      | $0.06\\pm0.01$     |\n|               5 | Original    | $0.95\\pm0.02$   | $0.15\\pm0.01$  | $0.03\\pm0.01$   | $0.52\\pm0.02$     | $0.13\\pm0.01$      | $0.05\\pm0.01$     |\n|               | Manipulated | $0.93\\pm0.02$   | $0.14\\pm0.01$  | $0.03\\pm0.01$   | $0.52\\pm0.02$     | $0.12\\pm0.01$      | $0.05\\pm0.01$     |\n|               6 | Original    | $0.96\\pm0.02$   | $0.15\\pm0.01$  | $0.03\\pm0.01$   | $0.53\\pm0.02$     | $0.12\\pm0.01$      | $0.05\\pm0.01$     |\n|               | Manipulated | $0.96\\pm0.01$   | $0.15\\pm0.01$  | $0.03\\pm0.01$   | $0.52\\pm0.01$     | $0.12\\pm0.01$      | $0.05\\pm0.01$     |\n|               7 | Original    | $0.92\\pm0.03$   | $0.18\\pm0.01$  | $0.03\\pm0.01$   | $0.55\\pm0.02$     | $0.13\\pm0.01$      | $0.05\\pm0.01$     |\n|               | Manipulated | $0.94\\pm0.02$   | $0.19\\pm0.01$  | $0.03\\pm0.01$   | $0.56\\pm0.02$     | $0.13\\pm0.01$      | $0.05\\pm0.01$     |\n|               8 | Original    | $0.96\\pm0.01$   | $0.19\\pm0.01$  | $0.03\\pm0.01$   | $0.57\\pm0.02$     | $0.13\\pm0.01$      | $0.05\\pm0.01$     |\n|               | Manipulated | $0.96\\pm0.01$   | $0.20\\pm0.01$  | $0.03\\pm0.01$   | $0.58\\pm0.01$     | $0.13\\pm0.01$      | $0.05\\pm0.01$     |\n|               9 | Original    | $0.93\\pm0.02$   | $0.17\\pm0.01$  | $0.03\\pm0.01$   | $0.49\\pm0.02$     | $0.13\\pm0.01$      | $0.05\\pm0.01$     |\n|               | Manipulated | $0.92\\pm0.02$   | $0.17\\pm0.01$  | $0.02\\pm0.01$   | $0.51\\pm0.02$     | $0.13\\pm0.01$      | $0.05\\pm0.01$     |"}, {"review_id": "AtcxtOtcuc", "reviewer": "Reviewer_brRT", "summary": "This paper proposes an attack on Activation Maximization methods, which use a parametrized image space to find images that maximize the dot product between the model's representation of the parameterized image and an arbitrary vector in the representation space. The theoretic basis of the method is that AM on the Euclidean distance between a parameter to be manipulated and a target parameter for AM to find will result in AM converging to the target parameter. In practice, the Gradient Slingshot optimizes a weighted combination of two loss terms. The preservation loss minimizes the change in model features and dot product with the arbitrary vector. The manipulation loss is ideally a second order optimization that changes the model's gradient to match the gradient of the Euclidean distance to the target parameter. For larger models, the function can be changed to match the Euclidean distance instead. Experiments show that AM can be manipulated with minimal changes to other model behavior. There is a trade-off between preserving behavior and strength of the AM manipulation. The attack is more effective on models with more parameters. AM on test set images are also mostly consistent before and after manipulation, making the attack difficult to detect.", "questions": "Q1: I think Eq. 4 needs to be updated to use the negative square Euclidean distance instead of the positive distance. Currently the gradient with respect to Eq. 4 would result in $-1$ times Eq. 3.\n\nQ2: How were the standard deviation values calculated? Multiple trained models? Bootstrapping?\n\nQ3: As detailed in W3, how was hyperparameter tuning performed?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "S1: The Gradient Slingshot is a novel attack on AM that, unlike previous methods, does not change the model architecture. This makes the attack harder to detect, as confirmed in Section 4.3.\n\nS2: The sampling procedure for the manipulation loss is a unique way to manipulate behavior for a subset of features.\n\nS3: The experiments are well documented, show good results, and include the standard deviation.\n\nW1: The paper seems related to other attacks, such as the proposed method sort of forcing AM to be adversarial image perturbation. The related works section does not discuss this broader context and I'm curious how the proposed method relates.\n\nW2: Lemma 3.1, used as the theoretical basis for the method, is a very mundane result that gradient descent (see Q1) on Euclidean distance will converge. The most novel part of the result is establishing that $\\mathbb{M} = \\mathbb{T}_{B, L}$, but the proof uses that as a condition instead of emphasizing that fact.\n\nW3: Few details are given for the hyperparameter tuning process besides that it was computationally intensive. Was it done via grid search on a validation set? How many models needed to be trained during tuning?", "was_revised": true, "final_justification": "The authors have sufficiently addressed my concerns and I appreciate the additional details provided. My perspective has not changed with respect to the significance of Lemma 3.1, but I do not think this holds the paper back. I continue to recommend the paper to be accepted.", "rebuttal": "We thank the reviewer for their thorough and constructive feedback. We are glad they found the Gradient Slingshots (GS) attack to be novel and difficult to detect, and appreciated the thoroughness and convincing nature of our experimental results. Below, we address the reviewer’s concerns point by point.\n\n### Weaknesses\n\n> W2: Lemma 3.1, used as the theoretical basis for the method, is a very mundane result that gradient descent (see Q1) on Euclidean distance will converge. The most novel part of the result is establishing that $\\mathbb{M} = \\mathbb{T_{\\text{B, L}}}$, but the proof uses that as a condition instead of emphasizing that fact.\n\nWe clarify that since we control how the manipulation set $\\mathbb{M}$ is sampled, we explicitly define it as $\\mathbb{T_{\\text{B, L}}}$; the non-trivial aspect lies in ensuring that each step $q^{(i)}$ during optimization remains in $\\mathbb{T_{\\text{B, L}}}$, which is guaranteed for gradient descent but not necessarily for other optimizers. This point is already noted in lines 149–151, where we explain that while Lemma 3.1 formally applies to gradient descent, our empirical results indicate that similar behavior extends to other Feature Visualization (FV) optimization strategies.\n\n> W3: Few details are given for the hyperparameter tuning process besides that it was computationally intensive. Was it done via grid search on a validation set? How many models needed to be trained during tuning?\n\nWe thank the reviewer for raising this concern. A detailed description of our hyperparameter tuning and selection procedure is provided in our response to Question 3 below, and we have incorporated this explanation into the revised paper.\n\n### Questions\n\n> Q1: I think Eq. 4 needs to be updated to use the negative square Euclidean distance instead of the positive distance. Currently the gradient with respect to Eq. 4 would result in times Eq. 3.\n\nWe thank the reviewer for pointing out this oversight. We have corrected the Eq. 4 accordingly. We have also updated Equation 10 to fix the same issue. We emphasize that this correction only affects the presentation in the manuscript; the code implementation already uses the correct formulation and remains unaffected.\n\n> Q2: How were the standard deviation values calculated? Multiple trained models? Bootstrapping?\n\nAs detailed in Appendix C.8, the reported standard deviations are computed over multiple runs of FV generation. Specifically, we generated 100 samples per model (30 for CLIP ViT-L/14 due to computational constraints), and computed similarity metrics over these samples. Since FV generation involves sampling the initialization point from a distribution, each run yields a slightly different result. The standard deviations thus characterize the consistency of the metrics across these independently generated FVs. We have incorporated this clarification into all relevant table captions to prevent any confusion.\n\n> Q3: As detailed in W3, how was hyperparameter tuning performed?\n\nFor the parameter $\\alpha$, our tuning process follows a coarse-to-fine iterative approach. We begin with a broad sweep of values (e.g., $\\alpha \\in [0.01, 0.05, 0.5, 0.9]$) to identify a promising region. Once a candidate such as $\\alpha = 0.5$ shows good results, we perform a finer-grained search around it (e.g., 0.4 and 0.6), and continue refining around the best-performing values, e.g., near 0.6.\n\nImportantly, we perform tuning using only 1–3 epochs and on small subsets of large datasets (see Table 8 in the Appendix). For ViT experiments, for example, we use less than 1\\% of ImageNet, with a single epoch taking under 5 minutes. While we acknowledge that tuning of $\\alpha$ is computationally intense, the process remains practical even with modest resources.\n\nFor the parameter $w$, we set it to 0.1 when the target layer contains 10 neurons, and to 0.01 when it contains 1000 neurons, preventing the target neuron's activation landscape from being overly disrupted. For $\\gamma$, we use the heuristic $\\nabla (\\phi \\circ \\eta)(\\mathbf{0}) = \\gamma \\mathbf{q^t}$ to preserve the gradient magnitude in the initialization zone.\n\nWe have incorporated this explanation into the paper in order to improve transparency around hyperparameter selection."}]

Author rebuttal:

Meta-review: This paper introduces Gradient Slingshots (GS) to attack the Feature Visualization method. GS manipulates a model to conceal malicious intent, without significantly hurting the performance of the original model. The novelty and effectiveness of the proposed method are praised by reviewers. However, a more thorough explanation regarding whether the method injects noises is suggested. Overall, this paper meets the publication standards.
Decision: Accept (poster)