Unlearning-based Neural Interpretations

## Abstract

Abstract Gradient-based interpretations often require an anchor point of comparison to avoid saturation in computing feature importance. We show that current baselines defined using static functions—constant mapping, averaging or blurring—inject harmful colour, texture or frequency assumptions that deviate from model behaviour. This leads to accumulation of irregular gradients, resulting in attribution maps that are biased, fragile and manipulable. Departing from the static approach, we propose UNI to compute an (un)learnable, debiased and adaptive baseline by perturbing the input towards an unlearning direction of steepest ascent. Our method discovers reliable baselines and succeeds in erasing salient features, which in turn locally smooths the high-curvature decision boundaries. Our analyses point to unlearning as a promising avenue for generating faithful, efficient and robust interpretations.

## 1 Introduction

The utility of large models is hampered by their lack of explainability and robustness guarantees. Yet breakthroughs in language modelling (Meta, ; Anthropic, ; Jiang et al., ; Google, ; Achiam et al., ) and generative computer vision (Rombach et al., ; Liu et al., ; Deepmind, ; Brooks et al., ) yield promising high-stakes applications, spanning domains of healthcare, scientific discovery, law and finance. As such, being able to interpret these models has become a primary concern for researchers, policymakers and the general populace, with international calls for explainability, accountability and fairness in AI decision-making (European Commission, ; White House OSTP, ; Bengio et al., ). To this end, recent works focus on the 2 main directions of making models *inherently explainable* (Böhle et al., ; Brendel & Bethge, ; Koh et al., ; Bohle et al., ; Chen et al., ; Ross et al., ) and *post-hoc interpretable* (Bau et al., ; Kim et al., ; Zhou et al., ; Ghorbani et al., ). Unfortunately, the former is marred by the status quo of proprietary models and prohibitive training costs. This motivates seeking robust attributions which reliably explain model predictions, to facilitate better risk assessment and trade-off calibration (Böhle et al., ; Doshi-Velez & Kim, ).

Post-hoc methods explain a black-box model’s output by attributing its decision back to predictive features of the input. They achieve this via leveraging components of the model itself (*e.g*. gradients and activations), or through approximation with a simpler, interpretable simulator. A desirable post-hoc explanation should exhibit *high faithfulness* – to be rationale-consistent (Yeh et al., ; Atanasova et al., ) with respect to a model’s decision function; *low sensitivity* – to yield reliably similar saliency predictions for input features in the same local neighbourhood (Alvarez Melis & Jaakkola, ; Ghorbani et al., ); *low complexity* – the explanation should be functionally simpler and more understandable than the original black-box model (Bhatt et al., ).

Figure: Figure 1: Left: Confidence of original model $\theta$ at image $x$ and baseline $x^{\prime}$. Right: Confidence of *unlearned* model $\hat{\theta}$ at image $x$. After unlearning in the model space $\theta\longmapsto\hat{\theta}$, we optimise the baseline to match the unlearned input confidence, such that $F_{c}(x^{\prime};\theta)\approx F_{c}(x;\hat{\theta})$.
Refer to caption: extracted/5917134/figures/titleFig.png

Gradient-based saliency methods are widely used for feature attribution, due to their simplicity, efficiency and post-hoc accessibility. This can be further decomposed into 3 families: perturbative, backpropagative and path-based, which we detail in Section 6. Gradient-based attribution is intuitive since the first-order derivative reveals which features significantly influence the model’s classification decision. However, naively using local gradients yields unfaithful attributions due to saturation, where the non-linear output function flattens in vicinity of the input and zero gradients are computed (Sundararajan et al., ; ). To improve gradient-sensitivity, later methods introduce a baseline input for reference, and backpropagate the difference in activation scores on a path between the reference and image-of-interest (Shrikumar et al., ; Sundararajan et al., ). The baseline is chosen to be devoid of predictive features and far away from the saturated local neighbourhood. However, such methods accumulate gradient noise when interpolating from the baseline to the input, leading to high local sensitivity (Ancona et al., ). Consequently, attribution maps become disconnected, sparse and irregular, where the saliency scores fluctuate wildly between neighbouring pixels of the same object and are visually noisy (Adebayo et al., ). This noise accumulation has two root causes—a poorly chosen baseline and high-curvature output manifold along the path features. Previous works (Sturmfels et al., ; Xu et al., ) have sought better baselines by empirically comparing between using a black image, a gaussian noised image, a gaussian blurred image, a uniformly noised image, an inverted colour image, as well as averaging attributions over several baseline choices. However, the correct baseline to represent a lack of salient features depends heavily on the specific classification task, on the trained model and on the input image. Indeed, the optimal baseline varies for each task–model–image combination (Akhtar & Jalwana, ); the baseline problem remains largely unsolved. Turning to the second problem of high-curvature output manifold, because trained neural networks exhibit approximately piece-wise linear decision boundaries (Goodfellow et al., ), therefore inputs near function transitions are vulnerable to perturbative attacks. By simply adding norm-bounded, imperceptible adversarial noise to the input image, attackers can dramatically alter the attribution map without changing the model’s class prediction (Ghorbani et al., ; Dombrowski et al., ). Methods of mitigation include explicit smoothing via averaging over multiple noised gradient attributions (Smilkov et al., ); adaptively optimising the integration path of attribution (Kapishnikov et al., ); imposing an attribution prior during training and optimising it at each step (Erion et al., ). However, all of these proposals starkly increase the complexity of attribution, requiring computationally costly forward and backward propagation steps.

To tackle the problematic triad of 1. post-hoc attribution biases, 2. poor baseline definition, 3. high-curvature output manifold, we propose UNI to discover debiased baselines by locally unlearning inputs, *i.e*. perturbing them in the unlearning direction of steepest ascent, as visualised in Figure 1. Towards better baselines, our unlearned reference is by definition explicitly optimised to lower output class confidence and can empirically erase or occlude salient features. We also say that the unlearned baseline is specific and featureless w.r.t. each task–model–input combination. Unlike the practice of using a black image baseline—which creates a post-hoc colour bias that darker pixels are less likely to be salient, UNI does not impose additional, pixel-wise colour, scale or geometric assumptions that are not already present in the model itself. Finally, we address the high-curvature decision boundaries problem by realising that this is a product of the training process—targeted unlearning smooths the decision boundary of the model within the vicinity of the input. For a more detailed overview on the principle of machine unlearning, we refer the reader to Section 6 of the supplement. We empirically verify this local smoothing effect by measuring the normal curvature of the model function before and after unlearning; we also demonstrate that unlearning makes attributions resistant to perturbative attacks. Our contributions can be summarised as follows:

- 1.
*Post-hoc attribution can impose new biases.* We approach the baseline challenge from the fresh lens of post-hoc biases. We show that static baselines (*e.g*. black, blurred, random noise) inject additional colour, texture and frequency assumptions that are not present in the original model’s decision rule, which leads explanation infidelity and inconsistency.
- 2.
*A well-chosen baseline is specific and featureless.* We establish theoretically grounded principles for sound baseline definitions, by formalising the idea of an “absence of signal” through an unlearning direction of steepest ascent in model loss. By unlearning predictive features in the model space and matching this reference model’s activations with a perturbation in the input space, we introduce a new definition of “feature absence" and a novel attribution algorithm.
- 3.
*Unlearning reduces the curvature of decision boundaries and increases robustness.* Targeted unlearning simulates the function statistics of unseen data, and smooths the curvature of the output manifold around the sample. This is characterised by low geodesic curvature of the path and bounded principal curvature of the output surface. This points to reduced variability of the gradient vector under small-norm input perturbations, leading to better attribution robustness and faithfulness.

## 2 Preliminaries

We consider feature attribution for trained deep neural networks within image classification. Informally, we seek to assign scores to each pixel of an image for quantifying the pixel’s influence (sign and magnitude) on the predicted output class confidence. It is noteworthy that attributions can be signed: a negative value indicates that removing the pixel increases the target class probability.

### 2.1 Notation

The input (feature) space is denoted as $\mathcal{X}\subset\mathbb{R}^{d_{X}}$, where $d_{X}$ is the number of pixels in an image. The output (label) space is $\mathcal{Y}\subset\mathbb{R}^{d_{Y}}$; $\mathcal{Y}$ is the set of all probability distributions on the set of classes. The model space is denoted as $\mathcal{F}\subset\mathcal{Y}^{\mathcal{X}}$. A trained model $F:x\mapsto(F_{1}(x),...,F_{d_{Y}}(x))$ returns the probability score $F_{c}(x)$ of each class $c$. Attribution methods are thus functions $\mathcal{A}:\{1,...,d_{X}\}\times\mathcal{F}\times\{1,...,d_{Y}\}\times
\mathcal{X}\rightarrow\mathbb{R}$, where $\mathcal{A}(i,F,c,x)$ is the importance score of pixel $i$ of image $x$ for the prediction made by $F_{c}$. For convenience, we use the shorthand $\mathcal{A}_{i}(x)$ to refer to the attributed saliency score of a pixel $i$ for a specific class prediction $c\in\{1,...,d_{Y}\}$. We express a linear path feature as $\gamma(x^{\prime},x,\alpha):\mathbb{R}^{d_{\mathcal{X}}}\times\mathbb{R}^{d_{
\mathcal{X}}}\times[0,1]\rightarrow\mathbb{R}^{d_{\mathcal{X}}}$, where $\gamma=(1-\alpha)x^{\prime}+\alpha x$ and employ shorthands $\gamma(0)=x^{\prime},\gamma(1)=x$.

## 3 Gradient-based Attributions in a Nutshell

Figure: Figure 2: We visualise post-hoc biases imposed by static baselines—black baseline (colour), blurred (texture), random (frequency). UNI learns to mask out predictive features used by the model, generating reliable attributions.
Refer to caption: x1.png

### 3.1 Limitations

Taking the local gradients of a model’s output confidence map $F_{c}(x)$ – for target class $c$ – is a tried and tested method for generating explanations. Commonly termed Simple Gradients (Erhan et al., ; Baehrens et al., ; Simonyan et al., ), $\mathcal{A}^{\textrm{SG}}_{i}(x)=\nabla_{x_{i}}F_{c}(x)$ can be efficiently computed for most model architectures. However, it encounters output saturation when activation functions like ReLU and Sigmoid are used, leading to zero gradients (hence null attribution) even for important features (Sundararajan et al., ; ). DeepLIFT (Shrikumar et al., ) reduces saturation by introducing a “reference state". A feature’s saliency score is decomposed into positive and negative contributions by backpropagating and comparing each neuron’s activations to that of the baseline. Integrated Gradients (IG) (Sundararajan et al., ) similarly utilises a reference, black image and computes the integral of gradients interpolated on a straight line between the image and the baseline.

$$ $\mathcal{A}^{\textrm{IG}}_{i}(x)=(x_{i}-x^{\prime}_{i})\int_{\alpha=0}^{1} \nabla_{x_{i}}F_{c}\left(x^{\prime}+\alpha(x-x^{\prime})\right)\,d\alpha$ (1) $$

Practically, the integral is approximated by a Riemann sum. Of existing methods, IG promises desirable, game-theoretic properties of “sensitivity”, “implementation invariance”, “completeness” and “linearity”. We consequently focus on analysing and developing the IG framework, though the proposal to unlearn baselines can be applied to most mainstream gradient-based saliency methods. Despite the advantages of IG, its soundness depends on a good *baseline definition*—an input which represents the “absence” of predictive features; also on having stable *path features*—a straight-line of increasing output confidence along the path integral from baseline to target image. In the conventional setting where a black image is used, Akhtar & Jalwana () prove that IG assumptions are violated due to ambiguous path features, where extrema of model confidences lie along the integration path instead of at the endpoints of the baseline (supposed minimum) and input image (supposed maximum). Sturmfels et al. () enumerate problems with other baselines obtained via gaussian blurring, maximum distance projection, uniform noise. Despite the diversity of baseline alternatives, no candidate is optimal for each and every attributions setting. For instance, models trained with image augmentations (*e.g*. colour jittering, rescaling, gaussian blur) yield equivalent or even higher confidences for blurred and lightly-noised baselines—we need baselines that are well-optimised for each task–model–input combination. Without principled baselines, problems of non-conformant intermediate paths and counter-intuitive attribution scores will doubtlessly persist.

### 3.2 Post-hoc Biases are Imposed

Since the baseline represents an absence of or reduction in salient features, static baseline functions (*e.g*. black, blurred, noised) implicitly assume that similar features (*e.g*. dark, smooth, high-frequency) are irrelevant for model prediction. To illustrate this intuition, we can consider IG with a black baseline, wherein it becomes more difficult to attribute dark but salient pixels. Due to the colour bias that “near-black features are unimportant", the term $(x_{i}-x^{\prime}_{i})$ is small and requires a disproportionately large gradient $\nabla_{x_{i}}F_{c}(\cdot)$ to yield non-negligible attribution scores. Indeed, this is what we observe in Figures 2, 5, 8, where darker features belonging to the object-of-interest cannot be reliably identified. We further empirically verify that each static baseline imposes its own post-hoc bias by experimenting on ImageNet-C (Hendrycks & Dietterich, ). Corresponding to the 3 popular baseline choices for IG (all-black, gaussian blurred, gaussian noised), we focus on the families of digital (brightening and saturation), blur (gaussian and defocus blur) and noise (gaussian and shot noise) common-corruptions. Figures 5, 9 demonstrate that IG with a blurred baseline fails to attribute blurred inputs due to saturation and overly smoothed image textures; Figures 5, 10 visualise how a noised IG baseline encounters high-frequency noise and outputs irregular, high-variance attribution scores, even for adjacent pixels belonging to the same object. We crucially emphasise that such colour, texture and frequency biases are not present naturally in the pre-trained model but rather injected implicitly by a suboptimal choice of static baseline. The observation that poor baseline choices create *attribution bias* has so far been overlooked. As such, we depart entirely from the line of work on alternative static baseline towards adaptively (un)learning baselines with gradient-based optimisation. *UNI eliminates all external assumptions except for the model’s own predictive bias.*

Figure: Figure 3: When the brightness or saturation is altered, IG with a black baseline fails to identify dark features, such as the boat’s hull (R3) or the top of the boot (R1).
Refer to caption: x2.png

## 4 UNI : Unlearning-based Neural Interpretations

### 4.1 Baseline desiderata

A desirable baseline should preserve the game-theoretic properties of path-attribution (Section 3.1) and refrain from imposing post-hoc attribution biases (Section 3.2). For every given task-model-image triad, a well-chosen baseline should be *1. image-specific*—be connected via a path feature of low curvature to the original image; *2. reflect only the model’s predictive biases*—salient image features should be excluded from the baseline; be *3. less task-informative than the original image*—interpolating from the baseline towards the input image should yield a path of increasing predictive confidence. We now introduce the UNI pipeline: first, unlearn predictive information in the model space; then, use activation-matching between unlearned and trained models to mine a featurelesss baseline in the image space; finally, interpolate along the low-curvature, conformant and consistent path from baseline to image to compute reliable explanations in the attributions space.

### 4.2 Desirable Path Features

#### Proximity

The meaningfulness of the attributions highly depends on the meaningfulness of the path. We aim for a smooth transition between absence and presence of features; and this intuitively cannot be achieved if the baseline and input are too far apart. (Srinivas & Fleuret, ) formalizes this intuition through the concept of weak dependence, and proves that this property can only be compatible with completeness in the case where the baseline and the input lie in the same connected component (in the case of piecewise-linear models). An obvious implementation of this proximity condition in the general case is to bound the distance $||x-x^{\prime}||$ to a certain value $\varepsilon$. This is strictly enforced in Algorithm 1 by normalizing the perturbation at each step $t$.

#### Low Curvature.

The curvature of the model prediction along the integrated path has been identified (Dombrowski et al., ) as one of the key factors influencing both the sensitivity and faithfulness of the computed attributions. We substantiate the intuition that a smooth and regular path is preferred by analysing the Riemannian sum calculation. Assuming that the function $g:\alpha\in[0,1]\mapsto\nabla F_{c}\left(x^{\prime}+\alpha(x-x^{\prime})\right)$ is derivable with a continuous derivative (i.e. $\mathcal{C}^{1}$) on the segment $[x^{\prime},x]$, elementary calculations and the application of the Taylor-Lagrange inequality give the following error in the Riemann approximation of the attribution,

$$ $\left|(x_{i}-x^{\prime}_{i})\int_{\alpha=0}^{1}g\left(\alpha\right)\,d\alpha- \frac{(x_{i}-x^{\prime}_{i})}{B}\sum_{k=1}^{B}{g\left(\frac{k}{B}\right)} \right|\leq\frac{M||x-x^{\prime}||^{2}}{2B}$ (2) $$

where $M=\max_{\alpha\in[0,1]}\frac{\mathrm{d}g}{\mathrm{d}\alpha}=\max_{\alpha\in[0,
1]}\frac{\partial^{2}F_{c}\left(x^{\prime}+\alpha(x-x^{\prime})\right)}{
\partial\alpha^{2}}$ exists by continuity of $g^{\prime}$ on $[0,1]$.

Thus, lower curvature along the path implies a lower value of the constant $M$, which in turn implies a lower error in the integration calculation. A smaller value $B$ of Riemann steps is needed to achieve the same precision. More generally, a low curvature (i.e. eigenvalues of the hessian) on and in a neighborhood of the baseline and path reduces the variability of the calculated gradients under small-norm perturbations, increasing the sensitivity and consistency of the method. Empirically, we observe a much lower curvature of the paths computed by UNI, as can be seen in Table LABEL:tab:mono and Appendix Figures 17, 18, 19, 20, 21, 22. Figure 7 also confirms the increased robustness to Riemann sum error induced.

#### Monotonic.

Intuitively, the path $\gamma$ defined by interpolating from the “featureless" baseline $x^{\prime}$ to the input image $x$ should be monotonically increasing in output class confidence; the cumulative attribution score should similarly increase. At the image level, for all $j,k$ such that $j\leq k$, since $|\gamma(j)-x|\geq|\gamma(k)-x|$, therefore the predictive confidence should be non-decreasing and order-preserving: $F_{c}(\gamma(j))\leq F_{c}(\gamma(k))$. Constraining $\gamma$ to be monotonically increasing suffices to satisfy criteria for *valid path features* (Akhtar & Jalwana, ). Zooming in on input feature $\tilde{x}_{i}=\gamma(\tilde{k})_{i}$ on the linear path for $\tilde{k}<1$, since the gradient of $\gamma$ is always positive and $x_{i}$ maximises $F_{c}$ on the interval, therefore both conditions $\textrm{sgn}(\nabla_{x_{i}}F_{c}(x_{i}))\cdot\textrm{sgn}(\nabla_{\tilde{x}_{i
}}F_{c}(\tilde{x}_{i}))=1$ and $|\nabla_{x_{i}}F_{c}(x_{i})|>|\nabla_{\tilde{x}_{i}}F_{c}(\tilde{x}_{i})|$ are naturally met. This is crucial for preserving the *completeness axiom* of Integrated Gradients (Sundararajan et al., ), that attribution scores add up to the difference quantity $\sum_{i=1}^{d_{\mathcal{X}}}\mathcal{A}_{i}(x)=F_{c}(x)-F_{c}(x^{\prime})$, and that attribution score $\mathcal{A}_{i}(x)$ is only non-zero when $x_{i}$ contributes to the prediction. Completeness does not hold when path features are non-conformant, *i.e*. $F_{c}(\gamma(j))>F_{c}(\gamma(k))$ ; or when extremities exist along $\gamma$, *i.e*. for $\tilde{k}\in(0,1),\>F_{c}(\gamma(\tilde{k}))>F_{c}(\gamma(1))$ or $F_{c}(\gamma(\tilde{k}))<F_{c}(\gamma(0))$.

## 5 Experiments

We experiment on ImageNet-1K (Deng et al., ), ImageNet-C (Hendrycks & Dietterich, ) and compare against various path-based and gradient-based attribution methods. This includes IG (Sundararajan et al., ), BlurIG (Xu et al., ), GIG (Kapishnikov et al., ), AGI (Pan et al., ), GBP (Springenberg et al., ) and DeepLIFT (Shrikumar et al., ). We consider a diverse set of pre-trained computer vision backbone models (Paszke et al., ), including ResNet-18 (He et al., ), EfficientNet-v2-small (Tan & Le, ), ConvNeXt-Tiny (Liu et al., ), VGG-16-bn (Simonyan & Zisserman, ), ViT-B_16 (Dosovitskiy et al., ) and Swin-Transformer-Tiny (Liu et al., ). Unless otherwise specified, we the following hyperparameters: unlearning step size $\eta=1$; $l_{2}$ PGD with $T=10$ steps, a budget of $\varepsilon=0.25$, step size $\mu=0.1$; Riemann approximation with $B=15$ steps. Our results verify UNI’s high faithfulness, stability and robustness.

### 5.1 Faithfulness

We report MuFidelity scores (Bhatt et al., ), *i.e*. the faithfulness of an attribution function $\mathcal{A}$, to a model $F$, at a sample $x$, for a subset of features of size $|S|$, given by
$\mu_{f}(F,\mathcal{A};x)=\textrm{corr}_{S\in{[d]\choose|S|}}\left(\sum_{i\in S
}\mathcal{A}(i,F,c,x),\>F_{c}(x)-F_{c}(x_{[x_{s}=\bar{x}_{s}]})\right)$. We record the (absolute) correlation coefficient between a randomly sampled subset of pixels and their attribution scores. As from Table 2, UNI outperforms other methods across all settings but one, indicating high faithfulness. We supplement these numbers with visual comparisons in Appendix Figures 11, 12, 13, 14, 15, 16 against IG (black and noised baselines), BlurIG, GIG, AGI, GBP, DeepLift.

**Table 2: *MuFidelity scores* measure the correlation between a subset of pixels’ impact on the output (*i.e*. change in predictive confidence) and assigned saliency scores. Since attribution methods can yield strong positive or negative correlation, we report the absolute scores.**
|  | UNI | IG | BlurIG | GIG | AGI | GBP | DeepLIFT |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ResNet-18 | $\bm{.12}$ $\pm.124$ | $.06$ $\pm.068$ | $.07$ $\pm.076$ | $.07$ $\pm.080$ | $.10$ $\pm.110$ | $.09$ $\pm.094$ | $.08$ $\pm.082$ |
| EfficientNetv2s | $\bm{.06}$ $\pm.046$ | $.05$ $\pm.043$ | $.05$ $\pm.044$ | $.05$ $\pm.044$ | $.06$ $\pm.045$ | $.05$ $\pm.043$ | $.05$ $\pm.043$ |
| ConvNeXt-Tiny | $.16$ $\pm.115$ | $.11$ $\pm.086$ | $.15$ $\pm.121$ | $\bm{.18}$ $\pm.149$ | $.17$ $\pm.131$ | $.09$ $\pm.072$ | $.11$ $\pm.084$ |
| VGG-16-bn | $\bm{.18}$ $\pm.141$ | $.08$ $\pm.066$ | $.09$ $\pm.076$ | $.13$ $\pm.108$ | $.14$ $\pm.104$ | $.13$ $\pm.108$ | $.10$ $\pm.082$ |
| ViT-B_16 | $\bm{.15}$ $\pm.114$ | $.10$ $\pm.074$ | $.10$ $\pm.077$ | $.11$ $\pm.079$ | $.14$ $\pm.104$ | $.09$ $\pm.070$ | $.10$ $\pm.072$ |
| Swin-T-Tiny | $\bm{.13}$ $\pm.100$ | $.09$ $\pm.071$ | $.12$ $\pm.102$ | $.12$ $\pm.104$ | $.13$ $\pm.102$ | $.09$ $\pm.069$ | $.10$ $\pm.076$ |

### 5.2 Robustness

Next, we evaluate UNI’s robustness to fragility adversarial attacks on model interpretations. Following Ghorbani et al. (), we design norm-bounded attacks to maximise the disagreement in attributions whilst constraining that the prediction label remains unchanged. We consider a standard $l_{\infty}$ attack designed with FGSM (Goodfellow et al., ), with perturbation budget $\epsilon_{f}=8/255$.

$$ $\displaystyle\delta_{f}^{*}=$ $\displaystyle\textrm{arg}\max_{\|\delta_{f}\|_{p}\leq\epsilon_{f}}\frac{1}{d_{ X}}\sum_{i=1}^{d_{X}}d\left(\mathcal{A}(i,F,c,x),\mathcal{A}(i,F,c,x+\delta_{f })\right)$ (3) $\displaystyle\textrm{subject to}\quad\textrm{arg}\max_{c^{\prime}}F_{c^{\prime }}(x)=\textrm{arg}\max_{c}F_{c^{\prime}}(x+\delta_{f})=c$ $$

We report robustness results using 2 distance measures—Spearman correlation coefficient in Figure 4 and top-$k$ pixel intersection score in Figure 4—pre and post attack. While other methods like DeepLIFT (DL), BlurIG, Integrated Gradients (IG) are misled to output irrelevant feature saliencies, UNI robustly maintains attribution consistency and achieves the lowest attack attribution disagreement scores (before and after FGSM attacks) for both metrics.

**Table 3: *Robustness*: Spearman’s correlation coefficient. Higher scores indicate better path consistency pre/post FGSM attacks.**
|  | UNI | IG | BlurIG | SG | DeepL |
| --- | --- | --- | --- | --- | --- |
| ResNet-18 | $\bm{.271}$ | $.088$ | $.084$ | $.014$ | $.139$ |
| Eff-v2-s | $\bm{.302}$ | $.009$ | $.076$ | $.008$ | $.018$ |
| ConvNeXt-T | $\bm{.292}$ | $.010$ | $.127$ | $.011$ | $.012$ |
| VGG-16-bn | $\bm{.290}$ | $.143$ | $.098$ | $.014$ | $.108$ |
| ViT-B-16 | $\bm{.319}$ | $.018$ | $.066$ | $.023$ | $.023$ |
| SwinT | $\bm{.271}$ | $.088$ | $.084$ | $.014$ | $.139$ |

Figure: Figure 6: UNI path features monotonically increase in output confidence when interpolating from baseline to input. This eliminates instability and inconsistency problems caused by extrema and turning points along the Riemann approximation path, which is present in other methods.
Refer to caption: x5.png

### 5.3 Stability

We compare UNI and other methods’ sensitivity to Riemann approximation noise, which manifests in visual artifacts and misattribution of salient features. As seen from Figures 6, 7, UNI reliably finds unlearned, “featureless" baselines for consistent attribution, regardless of the number of approximation steps $B\in\{1,15,30\}$. This is due to the low geodesic curvature of $\gamma^{\texttt{UNI}}$, which approximately minimises the local distance between points used in Riemann approximation.

## 6 Related Work

#### Machine unlearning.

We draw inspiration from the high-level principle of unlearning, which concerns the targeted “forgetting" of a data-point for a trained model, by localising relevant information stored in network weights and introducing updates or perturbations (Bourtoule et al., ). Formally, machine unlearning can be divided into exact and approximate unlearning (Nguyen et al., ). Exact unlearning seeks indistinguishability guarantees for output and weight distributions, between a model not trained on a sample and one that has unlearned said sample (Ginart et al., ; Thudi et al., ; Brophy & Lowd, ). However, provable exact unlearning is only achieved under full re-training, which can be computationally infeasible. Hence, approximate unlearning was proposed stemming from $\epsilon$-differential privacy (Dwork, ) and certified removal mechanisms (Guo et al., ; Golatkar et al., ). The former guarantees unlearning for $\epsilon=0$, *i.e*. the sample has null influence on the decision function; the latter unlearns with first/second order gradient updates, achieving max-divergence bounds for single unlearning samples. Unlearning naturally lends itself to path-based attribution, to localise then delete information in the weight space, for the purposes of defining an “unlearned" activation. This “unlearned" activation can be used to match the corresponding, “featureless" input, where salient features have been deleted during the unlearning process. While the connection to interpretability is new, a few recent works intriguingly connect machine unlearning to the task of debiasing classification models during training and evaluation (Chen et al., ; Kim et al., ; Bevan & Atapour-Abarghouei, ).

#### Perturbative methods.

Perturbative methods perturb inputs to change and explain outputs (Sculley et al., ), including LIME (Ribeiro et al., ), SHAP, KernelSHAP and GradientSHAP (), RKHS-SHAP (Chau et al., ), ConceptSHAP (Yeh et al., ), InterSHAP (Janzing et al., ), and DiCE (Kommiya Mothilal et al., ). LIME variants optimise a simulator of minimal functional complexity able to match the black-box model’s local behaviour for a given input-label pair. SHAP () consolidates LIME, DeepLIFT (Shrikumar et al., ), Layerwise Relevance Propagation (LRP) (Montavon et al., ) under the general, game-theoretic framework of additive feature attribution methods. For this framework, they outline the desired properties of local accuracy, missingness, consistency; they propose SHAP values as a feature importance measure which satisfies these properties under mild assumptions to generate model-agnostic explanations. However, such methods fail to give a global insight of the model’s decision function and are highly unstable due to the reliance on local perturbations (Fel et al., ). Bordt et al. () show that this leads to variability, inconsistency and unreliability in generated explanations, where different methods give incongruent explanations which cannot be acted on.

#### Backpropagative methods.

Beginning with simple gradients (Erhan et al., ; Simonyan et al., ), this family of methods—also, LRP (Montavon et al., ), DeepLIFT (Shrikumar et al., ), DeConvNet (Zeiler & Fergus, ), Guided Backpropagation (Springenberg et al., ) and GradCAM (Selvaraju et al., )—leverages gradients of the output $w.r.t.\hbox{}$ the input to proportionally project predictions back to the input space, for some given neuron activity of interest. Gradients of neural networks are, however, highly noisy and locally sensitive – they can only crudely localise salient feature regions. While this issue is partially remedied by SmoothGrad (Smilkov et al., ), we still observe that gradient-based saliency methods have higher sample complexity for generalisation than normal supervised training (Choi & Farnia, ) and often yield inconsistent attributions for unseen images at test time.

#### Path-based attribution.

This family of post-hoc attributions is attractive due to its grounding in cooperative game-theory (Friedman, ). It comprises Integrated Gradients (Sundararajan et al., ), Adversarial Gradient Integration (Pan et al., ), Expected Gradients (EG) (Erion et al., ), Guided Integrated Gradients (GIG) (Kapishnikov et al., ) and BlurIG (Xu et al., ). Path attribution typically relies on a baseline – a “vanilla" image devoid of features; a path—an often linear path from the featureless baseline to the target image—along which the path integral is computed for every pixel. Granular control over the attribution process comes with difficulties of defining an unambiguously featureless baseline (for each (model, image) pair) (Sturmfels et al., ) and then defining a reliable path of increasing label confidence without intermediate inflection points (Akhtar & Jalwana, ). To measure the discriminativeness of features identified by attribution methods, experimental benchmarks such as ROAR (Hooker et al., ), DiffRAOR (Shah et al., ), RISE (Petsiuk et al., ) and the Pointing Game (Zhang et al., ) have been proposed.

## 7 Conclusion

In this work, we formally discuss the limitations of current path-attribution frameworks, outline a new principle for optimising baseline and path features, as well as introduce the UNI algorithm for unlearning-based neural interpretations. We empirically show that present reliance on static baselines imposes undesirable post-hoc biases which are alien to the model’s decision function. We account for and mitigate various infidelity, inconsistency and instability issues in path-attribution by defining principled baselines and conformant path features. UNI leverages insights from unlearning to eliminate task-salient features and mimic baseline activations in the “absence of signal". It discovers low-curvature, stable paths with monotonically increasing output confidence, which preserves the completeness axiom necessary for path attribution. We visually, numerically and formally establish the utility of UNI as a means to compute robust, meaningful and debiased image attributions. Future extensions to UNI include going beyond first-order approximate unlearning towards adopting certified, second-order machine learning techniques; as well as granular investigations on how the baseline definition and model’s robustness and inductive biases exert influence on path attribution results.

## 8 Acknowledgements

This work was supported in part by the Pioneer Centre for AI, DNRF grant number P1.

## Appendix A Appendix

We supplement the main text with visualisations of the UNI baseline, attributions and path features (properties, stability and robustness). We additionally include figures elucidating the colour, texture and frequency biases post-hoc imposed by path attribution methods. From Figure 7, we observe the stability of UNI path features: our attributions can be reliably and efficiently computed with Riemann approximation. In Figures 8, 9 and 10, we present visualisations on ImageNet-C, highlighting how static choices of baselines may bias the path-attribution procedure, leading to null or noisy explanations. UNI does not impose additional post-hoc assumptions that are alien to the model’s decision function. Furthermore, we present qualitative comparisons of attribution results of pre-trained models on the ImageNet-1K test set, in Figures 11, 12, 13, 14, 15 and 16. UNI attributions are visibly better localised and more semantically meaningful. Finally, we visualise the consistent, geodesic paths of monotonically increasing output confidence, discovered by UNI. As seen from Figures 17, 18, 19, 20, 21 and 22, while other path attribution methods might encounter extrema and turning points along the interpolation path from baseline to input, UNI’s path features are monotonic and preserve the crucial completeness property on which the path attribution framework depends.

Figure: Figure 7: Comparison of attribution maps computed by Integrated Gradients and UNI, for a pre-trained ResNet-18 on the ImageNet-1K test set. UNI occludes and unlearns predictive input features; reliably localises predictive image regions; can be efficiently computed with only 1 Riemann step.
Refer to caption: extracted/5917134/figures/stable2.png

Figure: Figure 8: *Colour bias:* When an image’s brightness or saturation is altered, IG with a black baseline fails to identify dark features, such as the wings of the butterfly (R1) or black jacket (R3).
Refer to caption: x6.png

Figure: Figure 9: *Texture bias:* Using a blurred baseline for IG leads to a smoothness assumption in image texture, which leads to missingness in attribution when the input is also gaussian or defocus blurred.
Refer to caption: x7.png

Figure: Figure 10: *Frequency bias:* A gaussian noised baseline for IG renders it vulnerable to high-frequency corruptions. Adding gaussian or shot noise to the image yields unmeaningful, noisy attributions.
Refer to caption: x8.png

Figure: Figure 11: *Comparing attributions (ResNet-18):* UNI attributions demonstrate higher saliency, fidelity and faithfulness relative to conventional baselines on the ImageNet test set.
Refer to caption: x9.png

Figure: Figure 12: *Comparing attributions (EfficientNet-v2-small):* UNI attributions demonstrate higher saliency, fidelity and faithfulness relative to conventional baselines on the ImageNet test set.
Refer to caption: x10.png

Figure: Figure 13: *Comparing attributions (ConvNeXt-Tiny):* UNI attributions demonstrate higher saliency, fidelity and faithfulness relative to conventional baselines on the ImageNet test set.
Refer to caption: x11.png

Figure: Figure 14: *Comparing attributions (VGG-16-bn):* UNI attributions demonstrate higher saliency, fidelity and faithfulness relative to conventional baselines on the ImageNet test set.
Refer to caption: x12.png

Figure: Figure 15: *Comparing attributions (ViT-B_16):* UNI attributions demonstrate higher saliency, fidelity and faithfulness relative to conventional baselines on the ImageNet test set.
Refer to caption: x13.png

Figure: Figure 16: *Comparing attributions (Swin-Transformer-Tiny):* UNI attributions demonstrate higher saliency, fidelity and faithfulness relative to conventional baselines on the ImageNet test set.
Refer to caption: x14.png

Figure: Figure 17: *Comparing paths (ResNet-18):* UNI discovers geodesic paths of monotonically increasing output confidence, preserving the completeness property required for robust attributions.
Refer to caption: x15.png

Figure: Figure 18: *Comparing paths (EfficientNet-v2-small):* UNI discovers geodesic paths of monotonically increasing output confidence, preserving the completeness property required for robust attributions.
Refer to caption: x16.png

Figure: Figure 19: *Comparing paths (ConvNeXt-Tiny):* UNI discovers geodesic paths of monotonically increasing output confidence, preserving the completeness property required for robust attributions.
Refer to caption: x17.png

Figure: Figure 20: *Comparing paths (VGG-16-bn):* UNI discovers geodesic paths of monotonically increasing output confidence, preserving the completeness property required for robust attributions.
Refer to caption: x18.png

Figure: Figure 21: *Comparing paths (ViT-B_16):* UNI discovers geodesic paths of monotonically increasing output confidence, preserving the completeness property required for robust attributions.
Refer to caption: x19.png

Figure: Figure 22: *Comparing paths (Swin-Transformer-Tiny):* UNI discovers geodesic paths of monotonically increasing output confidence, preserving the completeness property required for robust attributions.
Refer to caption: x20.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. The images shown in the figure may suggest that the white color appears relatively weaker compared to the black color. Is there any bias towards the white color? If not, why is the gray part of the swimsuit not highlighted? In Figure 2, the white swan also shows a similar tendency. Do you have any qualitative measures for this?

2. Why do you use KL divergence? Have you tried different methods as well?
soundness: 3
strengths: Authors well-summarize the previous search, and clearly explain the required properties needed for the reference. The intution is clearly written, and well-evidenced by experimental results.
summary: This paper suggests a method for creating references for IG and other backpropagation-based explanation models. The reference is based on an untrained model with perturbations applied to the original image with respect to the model. The intuition, insights, and experimental results are presented as evidence.
weaknesses: 1. Even though they effectively present their intuition, there is no theoretical justification for their method. In particular, the pseudo-code is not explained enough. Additionally, it is unclear why the authors chose to implement it in the way they described in the code. The method itself needs further elaboration.

2. The suggested method is limited to explainable methods that require a reference, which restricts the broader applicability of the paper.

---

confidence: 4
contribution: 4
initial_rating: 10
presentation: 4
questions: No particular questions
soundness: 4
strengths: The paper offers a well-structured analysis of an important problem in neural network interpretability. The authors identify a specific, previously overlooked issue: how static baseline approaches in attribution methods can introduce unintended biases in three distinct categories - color (shown through experiments with brightness/saturation changes), texture (demonstrated via gaussian/defocus blur tests), and frequency (validated through gaussian/shot noise experiments). 

This observation is methodically supported through experiments across a comprehensive set of modern architectures, including both CNN-based models and transformer-based models. Their UNI method shows measurable improvements in three key areas: faithfulness (with MuFidelity scores showing consistent improvements across architectures), robustness (demonstrated through FGSM adversarial attacks, achieving higher Spearman correlation coefficients), and stability (shown through detailed Riemann approximation experiments that work even with just one step). 

The theoretical foundation includes careful mathematical analysis where they provide explicit bounds on Riemann approximation error and prove how their approach maintains monotonicity and completeness properties. The extensive experimental results in the appendix, spanning all (20+!) detailed figures, provide clear visual evidence of how their unlearning-based approach improves upon traditional static baselines across different types of images and corruptions.

Well done!
summary: The authors identify that current baseline approaches in path attribution methods (like Integrated Gradients) inject harmful assumptions about color, texture, and frequency that can lead to biased and fragile attributions. UNI proposes computing baselines by perturbing inputs in an "unlearning" direction, to erase salient features while maintaining model-specific properties. The method demonstrates improved faithfulness, robustness, and stability compared to existing approaches across multiple model architectures and datasets.
weaknesses: Largely this is a well written paper, here are a few potential avenues for improvement :
- Would it be possible to make stronger theoretical guarantees about the optimality of the unlearned baseline?
- Image classification baselines are pretty standard, I wonder how these baselines will look like in other domains such as NLP and audio.
- There could be some more discussion around hyperparameters, an often overlooked aspect in explainability literature
- It is mentioned that other techniques are inefficient, a detailed analysis of computational overhead might solidify contributions more strongly in that aspect.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - Could you adapt the method to a low dimensional subspace path (eg delta is parametrized by 7x7 mask that we interpolate and then optimize?)
- How would the proposed method adapt if applied to models beyond image classification, such as language models or multimodal networks?
- Could the authors consider a variant of UNI that leverages multi-scale features for tasks involving more intricate hierarchical dependencies?


**Overall, I think the paper is well-executed with thorough analysis and comprehensive results. While the contribution may be of limited impact within the broader field of explainable AI (see Major issues), I tend to give a marginal accept.**
soundness: 3
strengths: 1. **Innovative Baseline Approach**: The idea of using unlearning to create adaptive baselines is novel and addresses a known limitation in static baseline approaches.
2. **Robustness and Faithfulness**: The empirical results, including MuFidelity scores and robustness to adversarial perturbations, highlight the potential of the method to produce more reliable attributions.
3. **Comprehensive Experiments**: The paper includes evaluations on multiple models and datasets, adding credibility to the proposed approach.
summary: The authors introduces a novel approach to enhance gradient-based post-hoc attributions.
The authors argue that static baselines (e.g., black images, blurred noise) impose unintended biases on attribution maps, leading to fragility and unfaithful interpretations. The proposed UNI framework generates a dynamic baseline by perturbing the input in an unlearning direction of steepest ascent, theoretically optimizing for a featureless reference that aligns with the model's predictive function. 
This adaptive baseline improves the path attribution by reducing curvature and enhancing robustness, demonstrated empirically through comparisons on benchmark datasets like IN and various baseline models.
weaknesses: However, I spotted some problems, some major (**M**) and minor (**m**):

**M1.** Literature Gaps: the related work section omits significant prior research on black-box interpretability (RISE, Sobol...) and miss some really important faithfulness metrics. Also a paper from last year that could be relevant (Saliency strike back) could be discussed.

**M2.** Faithfulness Metrics: while the paper discusses MuFidelity as a metric, it does not consider complementary metrics such as deletion and insertion, which could offer a broader validation of faithfulness and robustness. Including these would provide a more comprehensive evaluation of the method's performance. Also MuFidelity contain many Hyperparameter that are known to favorise certain method, could you state how did you approximate MuFidelity (number of sample)  and also the value of $|S|$ ?

**M3.** Theoretical Justification: the paper lacks a theoretical discussion on how UNI would perform under linear cases or simple functions. Providing proofs or theoretical insights for these cases would enhance the paper's depth and foundational strength.

**M4.** Broader Impact on Understanding Neural Networks: The practical implications of this method for gaining new insights into neural networks remain unclear. How much new knowledge / what have learn with this method that we haven't with other attribution methods ?

Now for the minor problem:

**m1.** Limited Baseline Comparisons: the paper mainly contrasts UNI with IG and a few other attribution methods. Expanding the baseline comparison to include more recent methods (including black-box one such as RISE or HSIC that are know to have good results on faitfhulness metrics) and varied methods would strengthen the experimental section.

**m2.** Minor Typos: I havent found a lot of typos, just line 468: "simultatenously".

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Could the authors elaborate on why activation matching ensures the removal of salient features? Furthermore, given an activation vector $F(x;\hat{\theta})$, there may be multiple matches in the decision space within the budget $\varepsilon$. Is there any guarantee of the convergence for this matching, or is it based solely on empirical observations?
2. The goal of the unlearning is to derive $F(x;\hat{\theta})$ for matching. What advantage does it offer over directly adopting a non-informative activation, say $(\frac{1}{d_Y},\ldots,\frac{1}{d_Y})$? Could the latter be even more powerful as it removes features that are informative in general?
3. Could the authors clarify the meaning of $F_c(x_i)$ at lines 297-298 mean? It is unclear how the model processes a single feature $x_i$ instead of an instance $x$.
4. How are the two conditions in lines 297-298 satisfied? 
    - Assuming the authors intended $F_c(x)$ rather than $F_c(x_i)$, the monotonicity of the class confidence does not necessarily imply a consistent sign of the derivatives w.r.t. each feature $x_i$. This indicates that the sign of $\nabla_{\tilde{x}_i}F_c(\tilde{x})$ may vary along the path, which contradicts the claim of the first condition.
    - Solving the gradient saturation problem is one motivation for IG. However, if a feature $x_i$ has a saturated gradient at $x$, i.e. $\nabla_{x_i}F_c(x)\approx0$, the second condition becomes problematic as it requires the derivate of $x_i$ to remain 0 over the integration path, leaving gradient saturation unsolved.
5. Could the authors clarify the relationship between completeness and path monotonicity? Completeness, as given by the gradient theorem, should be path-independent.
6. Many attributions provided by UNI closely align with the details of the input images. To demonstrate that the explanations are not simply grayscale reproductions of the inputs, could the authors provide example attributions of different models on the same inputs to show whether they distinguish from each other (under the assumption that different models behave differently)?
soundness: 3
strengths: 1. The paper is clearly motivated and easy to follow.
2. The authors collect a set of desired properties for the integration path, which could benefit following work on baseline selection.
3. The experiments are carefully designed, thoroughly evaluating the proposed method from different perspectives that support the authors’ claims.
summary: This paper studies the specific problem of baseline selection in feature attribution. The authors argue that the optimal baseline choice depends on the task-model-input combination and propose an unlearning-based approach to improve baseline selection. By combining the baseline selection method with a well-known path method — IG, the resulting attribution method UNI demonstrates improved performance in various aspects according to the experimental results.
weaknesses: 1. The description of the proposed method is somewhat oversimplified, raising concerns about the overall depth of the work. It would benefit from further elaboration and discussions on each step in Algorithm 1. 
2. The reasoning behind why activation-matching ensures the removal of all salient features is unclear.
3. Further clarification on some technical details about monotonicity would be beneficial. Please refer to questions 3, 4, and 5 for specific points.
4. There are some minor errors and formatting issues, for example: 
    - Different formats of $\varepsilon$ in Algorithm 1.
    - Citation wrapped by parenthesis as part of a sentence on line 261; repeated parentheses on line 269.
    - Line 349 refers to "Figure 3", which appears to be a mislabeling of "Table 3".

Meta-review: The authors study gradient-based, post-hoc, path attribution methods such as Integrated Gradients and identify that static baselines (e.g., black images, blurred noise) impose unintended biases on attribution maps, leading to fragility and unfaithful interpretations. They then proposed a novel framework which computes baselines by perturbing inputs in an "unlearning" direction (a direction the steepest ascent), to erase salient features while maintaining model-specific properties. The strong experimental results confirmed the intuition, demonstrating improved faithfulness, robustness, and stability.

The reviewers unanimously agree that it is a strong paper worthy of acceptance at ICLR.

Decision: Accept (Oral)