Title: Manipulating Feature Visualizations with Gradient Slingshots
Venue: NeurIPS 2025
Accepted: yes
Keywords: Explainable AI

Mechanistic Interpretability

Machine Learning

Computer Vision
Abstract: Feature Visualization (FV) is a widely used technique for interpreting concepts learned by Deep Neural Networks (DNNs), which synthesizes input patterns that maximally activate a given feature. Despite its popularity, the trustworthiness of FV explanations has received limited attention. We introduce Gradient Slingshots, a novel method that enables FV manipulation without modifying model architecture or significantly degrading performance. By shaping new trajectories in off-distribution regions of a feature's activation landscape, we coerce the optimization process to converge to a predefined visualization. We evaluate our approach on several DNN architectures, demonstrating its ability to replace faithful FVs with arbitrary targets. These results expose a critical vulnerability: auditors relying solely on FV may accept entirely fabricated explanations. To mitigate this risk, we propose a straightforward defense and quantitatively demonstrate its effectiveness.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an attack on Activation Maximization methods, which use a parametrized image space to find images that maximize the dot product between the model's representation of the parameterized image and an arbitrary vector in the representation space. The theoretic basis of the method is that AM on the Euclidean distance between a parameter to be manipulated and a target parameter for AM to find will result in AM converging to the target parameter. In practice, the Gradient Slingshot optimizes a weighted combination of two loss terms. The preservation loss minimizes the change in model features and dot product with the arbitrary vector. The manipulation loss is ideally a second order optimization that changes the model's gradient to match the gradient of the Euclidean distance to the target parameter. For larger models, the function can be changed to match the Euclidean distance instead. Experiments show that AM can be manipulated with minimal changes to other model behavior. There is a trade-off between preserving behavior and strength of the AM manipulation. The attack is more effective on models with more parameters. AM on test set images are also mostly consistent before and after manipulation, making the attack difficult to detect.\n\nSTRENGTHS AND WEAKNESSES:\nS1: The Gradient Slingshot is a novel attack on AM that, unlike previous methods, does not change the model architecture. This makes the attack harder to detect, as confirmed in Section 4.3.\n\nS2: The sampling procedure for the manipulation loss is a unique way to manipulate behavior for a subset of features.\n\nS3: The experiments are well documented, show good results, and include the standard deviation.\n\nW1: The paper seems related to other attacks, such as the proposed method sort of forcing AM to be adversarial image perturbation. The related works section does not discuss this broader context and I'm curious how the proposed method relates.\n\nW2: Lemma 3.1, used as the theoretical basis for the method, is a very mundane result that gradient descent (see Q1) on Euclidean distance will converge. The most novel part of the result is establishing that $\\mathbb{M} = \\mathbb{T}_{B, L}$, but the proof uses that as a condition instead of emphasizing that fact.\n\nW3: Few details are given for the hyperparameter tuning process besides that it was computationally intensive. Was it done via grid search on a validation set? How many models needed to be trained during tuning?\n\nQUESTIONS:\nQ1: I think Eq. 4 needs to be updated to use the negative square Euclidean distance instead of the positive distance. Currently the gradient with respect to Eq. 4 would result in $-1$ times Eq. 3.\n\nQ2: How were the standard deviation values calculated? Multiple trained models? Bootstrapping?\n\nQ3: As detailed in W3, how was hyperparameter tuning performed?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses attacks on feature visualization. In a scenario where model developers want to conceal the true feature visualization of certain features from a third-party auditor, their method can modify any feature’s feature visualization to any target image by adjusting the model parameters. This ensures that the model’s performance remains intact, while the feature visualization of the target feature resembles the input image. Essentially, they can alter any single feature visualization to anything they desire.\n\nTheir method fine-tunes the model using a loss compromise consisting of two parts. The first part aims to maintain the activations on unrelated elements unchanged. The second part directs the gradients of data points in a tunnel between the feature visualization initialization points and the target input image towards the target image. They conducted numerous empirical studies to assess the impact of model size, target image familiarity to the model, and other factors.\n\nSTRENGTHS AND WEAKNESSES:\n## Weakness \n1. The paper only focuses on last layer features. It does not conduct any experiments on features in middle layers. \n2. Hyperparameter tuning of loss seems very hard as both large and small values lead to the method not working. \n3. The paper claims that internal mechanisms of the model remain unchanged but because they didn’t investigate middle layers their argument is only based on accuracy. \n\n## Strengths \n1. Good writing. \n2. Novel idea for Feature visualization manipulation. \n3. Good variation of model choices. \n4. Many ablation studies.\n\nQUESTIONS:\n## Questions \n Is there an experiment that shows the effect of model manipulation on other features feature visualization ? Do other features FV also look like the target image?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Gradient Slingshots (GS), a method that modifies a model so that gradient ascent, involved in FV, is drawn towards a wrong optimum. The optimum, which is an image, can be arbitrarily chosen beforehand by the neural network designer, who would seek to fool an audit based on feature visualization. As a result, the result of the FV looks like the wrong optimum rather than the actual FV, potentially hiding the actual concepts underlying the neural network's predictions. GS applies as a fine-tuning that is supposed to preserve the model's predictive capacities and alter only the behaviour of FV's gradient ascent.\n\nThe authors test their method on simple CN, VGG, ResNet18/50, and ViT, on some neurons involved in Cifar10 and Imagenet classification. They also provide a case study based on CLIP and weapon classification to emphasize the risks of such an attack.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The idea is novel and simple\n- The paper is well written - even if it could be slightly rebalanced for improved readability, see below)\n- The manipulation is undetectable by looking at the code or the structure of the model since it is contained in the weights' values.\n\n### Weaknesses\n\n- **W1**: The evaluation is only conducted on one or a few neurons, so it might be subject to cherry-picking.\n- **W2**: The quantitative metrics are not impressive. I am suspicious that GS works on only a few examples selected for the paper illustrations, but statistically, GS seems to only slightly improve the similarity metrics compared to the original FV. I am not convinced that it actually works. It is even more true for ViT (see Appendix, p. 32), for which the metrics barely change compared to the FV without GS.  \n- **W3**: The assumption of l. 139 for Lemma 3.1 seems to make GS artificial. As far as I understand, it means that the initialization must be known by the manipulator and fall in $\\mathbb{B}$. It seems very easy to circumvent for an auditor, for instance, by uniformly sampling several initializations in $\\mathcal{Q}$. \n\n### Comments and suggestions\n\n- The assumption of l. 139 for Lemma 3.1 seems to be very strong, but it is only stated in the proof. It should appear in the Lemma.\n- The authors spend time on the proof of Lemma 3.1, which is intuitive, so they do not need proof to make sense of it. Hence, it takes up some unnecessary space, which could be better used for explaining the metrics or expanding the case study setting if it was deferred to the appendix.\n\nQUESTIONS:\n- How can a manipulator make sure that the FV initialization will fall into $\\mathbb{B}$ and will be appropriately steered towards the wrong optimum? \n- How does GS affect the FV of other features? If the FV is pertubated for every neurons, it might seem suspicious to auditers. \n- In Sec 5, the manipulation and the FV seem to target the feature vector representing an assault rifle. However, FV and GS are only formulated for the maximization of a single neuron. How to make this fall under a formalism similar to Eq. 1, 2, 3, 4, 5? If the answer is contained in lines 290 -> 293, it is not detailed enough, and more description should be provided for reproducibility.\n- The FV method from Olah et al. 2016 [1] notoriously fails on modern networks like ViT [2]. Could the authors run GS with modern FV methods like MACO [2]\n- I am surprised that there is no factor $2$ in Eq. 10 before $\\gamma$, or a factor $\\frac{1}{2}$ in Eq. 9, that stems from derivatives of square expressions.\n\n[1] Distill, Olah et al., 2017 [https://distill.pub/2017/feature-visualization/](https://distill.pub/2017/feature-visualization/) \n[2] Unlocking Feature Visualization for Deeper Networks with MAgnitude Constrained Optimization, Fel et al. NeurIPS 2023\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces \"Gradient Slingshots\" (GS), a proposed attack method targeting Feature Visualization (FV), an explainable AI technique. The core premise is that if FV is used to audit AI models for fairness or other properties, an adversary could subtly manipulate their model's training to conceal malicious intent. The authors claim GS improves on prior methods by being architecture-agnostic (including transformers) and by maintaining the manipulated model's performance. The technique reportedly achieves this balance by optimizing a trade-off between the FV image's resemblance to a target and the model's original behavior, controlled by parameters alpha and w.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The visual aspects of the paper are strong, with well-presented figures.\n- The underlying concept of the attack appears theoretically sound.\n- The authors made an effort to provide theoretical justification for their approach.\n- The supplementary information in the appendix was a useful addition.\n\nWeaknesses:\n\nThe reviewer has significant concerns about the paper's experimental rigor and the validity of its claims, stemming from several key issues:\n\n1. Lack of Comparative Analysis: The paper fails to compare GS against any existing baselines. While it suggests that methods like Nanfack et al. might degrade model performance, no empirical evidence is provided to substantiate this claim. This omission makes it difficult to assess GS's true effectiveness.\n\n2. Unconvincing Similarity Metrics: The reported similarity metrics (e.g., MSE and SSIM) between the manipulated FV and the target image do not show statistically significant improvement. For instance, these metrics remain identical for alpha values between 1.0 and 0.4, despite a notable degradation in model performance (from 76% to 67%) within that range. This raises questions about the practical impact of the proposed attack.\n\n3. Apparent Cherry-Picking of Results: The selection of specific alpha values in Table 1 (e.g., 0.64 and 0.62, which are very close to 0.6) while omitting results for 0.7 or 0.8 suggests a selective presentation of data. This \"cherry-picking\" makes it challenging to understand the full landscape of hyperparameter effects and raises doubts about the robustness of the findings.\n\n4. Excessive Hyperparameter Tuning: The wide range and extreme precision of `alpha` values presented in the appendix (e.g., 0.8, 0.025, 0.01, 0.995, 0.64, 0.9995, 0.99991) strongly indicate a reliance on extensive grid searching to find optimal parameters. If the method is truly generalizable, such extreme fine-tuning should not be necessary. This undermines confidence in the method's practical applicability.\n\n5. Unproven Efficacy on Transformers: Despite being a purported advantage over prior work, the paper does not convincingly demonstrate the methodology's effectiveness on transformer architectures. The extremely high alpha values used for ViT models (e.g., at least 0.9995) suggest that the method may simply be injecting noise, rather than performing a targeted attack.\n\nThreat Model Plausibility:\nThe paper is framed as a security attack, but the reviewer questions the realism of the proposed threat model. Unmanipulated FVs are typically abstract. It is unlikely that an auditor would solely rely on these abstract visualizations without incorporating other metrics or auditing procedures. This diminishes the practical significance of the described attack.\n\nPresentation and Quality:\nThe paper exhibits some signs of being rushed or lacking polish. For example, Table 2 suffers from poor formatting, indicative of a general lack of attention to detail.\n\nQUESTIONS:\nQuestions:\n\n1. Could the authors clarify the meaning of the standard deviations reported throughout the paper? Are they based on 95% confidence intervals from multiple experimental runs? If so, what is the number of runs? Alternatively, are they computed for a single run or a single FV?\n\n2. Is there a methodology for determining reasonable hyperparameter values without resorting to an exhaustive grid search? What would the hyperparameters and experimental results look like if such a methodology were employed? Demonstrating this would significantly enhance confidence in the method's practical utility.\n\n3. In real-world auditing practices, do auditors commonly rely on Feature Visualization to assess models?\n\nLIMITATIONS:\nYes, the paper acknowledges limitations. Depending on the response to Question 3, the authors might consider engaging with real-world auditors to make them aware of the paper findings..", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 1, "ORIGINALITY": 3, "DATE": "2025-07-04"}