Model Guidance via Robust Feature Attribution
Authors: Mihnea Ghitu
Vihari Piratla
Matthew Robert Wicker
Venue: Accepted by TMLR ()

Abstract: Controlling the patterns a model learns is essential to preventing reliance on irrelevant or misleading features. Such reliance on irrelevant features, often called shortcut features, has been observed across domains, including medical imaging and natural language processing, where it may lead to real-world harms. A common mitigation strategy leverages annotations (provided by humans or machines) indicating which features are relevant or irrelevant. These annotations are compared to model explanations, typically in the form of feature salience, and used to guide the loss function during training. Unfortunately, recent works have demonstrated that feature salience methods are unreliable and therefore offer a poor signal to optimize. In this work, we propose a simplified objective that simultaneously optimizes for explanation robustness and mitigation of shortcut learning. Unlike prior objectives with similar aims, we demonstrate theoretically why our approach ought to be more effective. Across a comprehensive series of experiments, we show that our approach consistently reduces test-time misclassifications by 20\% compared to state-of-the-art methods. We also extend prior experimental settings to include natural language processing tasks. Additionally, we conduct novel ablations that yield practical insights, including the relative importance of annotation quality over quantity. Code for our method and experiments is available at: https://github.com/Mihneaghitu/ModelGuidanceViaRobustFeatureAttribution.

Published in Transactions on Machine Learning Research (09/2025)
Model Guidance via Robust Feature Attribution
Mihnea Ghitu
mwkneca.ghitn2200kmmprrrd.....
Imperial College London
Vihari Piratla University of Cambridge
eubaripiraala@@gmiil.com
Matthew Wicker
www.ickkr@@mmprri.....
Imperial College London
Reviewed on OpenReview: https: // openreview. net/ forum? id= AVAH=DSqUu
Abstract
Controlling the patterns a model learns is essential to preventing reliance on irrelevant or misleading features. Such reliance irrelevant features, often called shorteut features, has been observed across domains, including medical imaging and natural language processing, where it may lead to real-world harms. A common mitigation strategy leverages annotations (provided by humans or machines) indicating which features are relevant or irrelevant. These annotations are compared to model explanations, typically in the form of feature salience, and used to guide the loss function during training. Unfortunately, recent works have demonstrated that feature salience methods are unreliable and therefore offer a poor signal to optimize. In this work, we propose simplified objective that simultaneously optimizes for explanation robustness and mitigation of shortcut learning. Unlike prior objectives with similar aims, we demonstrate theoretically why our approach ought to be more effective. Across a comprehensive series of experiments, we show that our approach consistently reduces test-time misclassifiaation by 20% compared to state-of-the-art methods. We also extend prior experimental settings include natural language processing tasks. Additionally, we conduct novel ablations that yield practical insights, including the relative importance of annotation quality over quantity. Code for our method and experiments is available at: httpps:/gitthub.com Miteeeegggi//eeeeiiiiaaaaaaaaaae
1 Introduction
Machine learning (ML) has seen tremendous progress in the last docade, culminating in mainstream adoption in variety of domains. Deep neural networks (DNN) sit at the core of this progress, due to their remarkable performance and case of deployment (Goodfellow et al. 2016). With applications in wide-ranging domains [Bommasani et al.) 2021), the promise of impact comes with the potential for substantial harm when deploying models that may be dependent on irrelevant and misleading feature patterns present in their particular training dataset (Ross al. 2017].
A model misgeneralizing due to its reliance incidental correlations is often referred to as shorteut learning (Ross et al. 2017 Geirhos et al. 2020 Heo et al. 2023 Leveraging such shortcuts during training significantly hinders the reliable deployment of models in safety-critical domains. For instance, models were documented to exploit datasct-specife incidental correlations such as hospital tags in chest x-rays when diagnosing pneumonia (Zech et al. 2018 DeGrave et al.] 2021) or bandage in skin lesions when diagnosising skin cancer (Rieger le al. 2020 In natural language processing, models often rely incorrectly on pronouns and proper nouns when making decisions (McCoy et al.) 2019). leading to bias downstream tasks such as hiring and loan approvals (Rudinger et al. (2018). Left unchecked, learning and relying upon such shortcuts may lead unintended harms at deployment time. Thus, understanding and mitigating shortcut learning has become

---

Published in Transactions on Machine Learning Research (09/2025)
important area of research that has accumulated rich benchmarks and methodologies (Ross et al. 2017
Geirhos et al. 2020 Singla et al. 2022..
The best-performing approaches to mitigating shorteut learning involves the use of per-input feature-level annotations that indicate if a feature in a given input is a core feature or a non-core feature that may represent a shortcut (e.g., the background of an image) (Heo et al. 2023). Throughout the paper we will refer to non-core features simply as masked features. Methods that make use of such annotations are commonly termed Machine Learning from Explanations (MLX) methods. A variety of MLX approaches have been proposed (Lee et al.) 2022 Heo et al. 2023), and common theme across toate-of-theeart methods is the use of model explanations to regularize dependence on masked features, pioncered in right for the right reasons (Ross et al., 2017). The model explanations used in MLX methods come in the form of feature attribution explanations which assign a score to each feature indicating how important the score is to the model prodiction. By ensuring that masked features have low importance (achieved via regularizing the loss
function) MLX methods mitigate shortcut learning (Jia ct al. 2018 Kavumba et al. 2021 Hoo et al. 2023
Ross et. al. 2017 Jia et al. 2018 Kavumba al. 2021 Heo al. 2023).
However, these methods assumee that feature attribution explanations provide a faithful signal of model behavior, an assumption increasingly called into question. Recent work shows that the most popular form of feature importance, gradient-based attributions, can be unstable and easily manipulated, even without changing model predictions (Dombrowski et al. 2019 Wicker et al.). This fragility stems from the non-linear nature of deep networks, which allows imperceptible input changes to dramatically alter gradient-based explanations much like adversarial attacks on outputs (Goodfellow et al., 2014). As a result, using only the gradient at single imput point in conjunction with annotation masks may not reliably suppress shortcut. learning because out-of-distribution data might render the explanations unusable. Moreover, prior MLX works largely overlook how annotation quality and availability affect performance and inference-tine generalizability, leaving important practical considerations unexplored.
In this work, we address these shortcomings by first proposing that mitigating reliance on shortcut features would be substantially more effective one additionally optimizes for the reliability of the feature saliency explanations. We propose a robust variant of right for right reasons (RRR) that accounts for the noise in feature importance scores, which we dub robustly right for the right reasons (R'). RR regularizes the feature importance in high-dimensional ball around the training points rather than regularizing the feature importance at the training point alone. As result, the objective required an inner optimization over a high-dimensional manifold, which is intractable. Thus, we instantiate three variants of Rd each implementing a different approximate solution to the optimization problem: Rond-R* takes the maximum over perturbations sampled from the domain of the optimization, Ad uses first-order optimization to search the domain of the optimization, and Cert-R4 uses convex-relaxations of the problem to compute an upper-bound the solution to the optimization problem. We compare each of these methods with a series of state-of-tho-at MLX approaches with various datasets. We employ two synthetic datasets: the well-studied MNTST datasot (Ross et al. 2017) as well as a novel shortcut learning benchmark derived from medical image diagnosis problem (Yang et al.) (002).. Through further evaluation on three real-workd benchmarks, we demonstrate consistent gains of R over the state-of-theeart by a substantial margin (a raw increase of 6% accuracy on average). In summary, this paper makes the following contributions:
We establish a novel, adversarial approach to machine learning from explanations that we name robustly right for the right reasons or R*.
We propose and implement three (approximate) solutions to the intractable inner optimization of R*: a statistical approach, a first-order optimization approach, and a convex- relax-reloxttion based approach and demonstrate their scaling to models as large as ResNet-18 and BERT.
Through a scries of experiments, we demonstrate that R outperforms prior state of the art MLX approaches across all datasets and can effectively mitigate shorteut learning with annotations on just 20% of the examples in the training set.
2

---

Published in Transactions on Machine Learning Research (09/2025)
2 Related Works
Shorteut Learning. The phenomena of models learning non-generalizing dataset artifacts is referred to as shorteul learning and is well-studied (Geirhos al., 2020 Shah et al. 2020). Mitigating shortcut, learning has been approached from multiple fronts, the three major themes are: (a) annotating training examples with group identity (Sagawa et al. Ye et al.] 2024) to delineate examples with positive and negative incidental correlation, (b) diversifying the training set bias-free or bias mullifying examples (Lee et al. 2022], learning from explicit annotation of relevant and irrelevant parts of input called machine learning from explanations (MLX) (Ross et al. 2017 Rieger et al.) 2020 Heo et al. 2023). MLX is the focus of our work, which enables specifying the irrelevant features explicitly (Sagawa et al. Ye et al. 2024 Ross et al.; 2017). Additionally, a
line of works augments the provided human explanations with natural language (Selvaraju al., 2019) or
through iterative interaction (Schramowski et al. 2020 Linardatos ct al. 2020
Machine Learning from Explanations. MLX approaches operate through an augmented loss objective that regularizes the model's dependence on irrelevant features. Traditionally, an imput feature attribution method is used to compute the importance of various imput features. Previous proposals differed greatly in their choice of the feature attribution method: Ross el al. (2177) employed gradient-based feature explanation, Rieger et al. (020)) employed contextual decomposition--asee feature explanation (Singh et al.) 2018), Schramowski et al. (2020) employed LIME (Ribeiro et al. 2016), Shao et al. (2021) employed influence functions (Koh & Liang 2017). Heo et al. (2023) championed directly minimizing the function's sensitivity to irrelevant. feature perturbations. In Zhang et al. (2024) the authors propose to learn feature importance for CNNs and mitigate shortcut reliance via regularization of intermediate activations. As is evident from the focus of many previous work, the core difficully lies in approximating the importance of input features.
Fragility of Explanations. Feature attribution explanations are among the most popular explanation methods and inchude shapely values (Lundberg & Loel 2017), LIME (Ribciro et al., 2016), smooth- and integratee--raadients (Smilkov et al. 2017; (Sundararaaan et al. 2017), grad-CAM (Selvaraju et al.) 2020), among others Leofante & Wicker (2025). While each of these methods aim at capturing the per-feature importance of a model decision with respect to a given input, they have recently been found to give opposite feature importance scores for indistinguishable inputs (Dombrowski el al. 2019). Manipulating explanations by slightly perturbing input values is similar to crafting adversarial examples [Goodfellww et al.] 2014). thus adversarial training techniques have been adopted and employed to enforce the robustness of explanations DDmbrowssi et al.) 2022; Wicker et al.).. However, current MLX approaches do not enforee robustness of explanations and thus use an unreliable optimization signal. Moreover, adopting prior works such as Wicker et al. cannot be done directly due to the complications of the MLX setting, that is, the existence of masked features.
Robustness and MLX. The work most related to ours is Heo et al. (2023). which championed for directly imposing robustncss to c-ball perturbations of irrelevant features per example. The methods presented in our work instead explore robustness of explanations, the signal that is optimized, as opposed to robust of predictions. Despite their resemblance, R$ is more elegant with only one term added to the objective while also being empirically superior. We further provide theoretical insights ($ 4 and empirical validation (5 on R& merit over Heo et al. 20023
3 Preliminaries
We denote a machine learning model as a parametric function f with parameters E R" which maps from features æ E R to labels y E V. We consider supervised learning in the classification setting with labeled dataset D = {x(()) 3. In the MLX setting, in addition to the feature-label pairs, we have access to human or machine- machine-provides explanations in the form of masks (+()) E R", which highlight the important regions or components of the input features relevant. for the prediction. These masks provide a form of weak supervision, guiding the model's attention during training. Thus, in MLX, the dataset is extended to include these annotations and represented as D = ((=(9) (6) ) where each ( is the explanation associated with the i-th data point. In particular, the mask m() is a vector with entries in the interval [0. 1], where 1 represents fully irrelevant features and 0 represents fully relevant. features.
3

---

Published in Transactions on Machine Learning Research (09/2025)
Right for the right reasons. The seminal approach to MLX, right for the right reasons (RRR R*) proposes to modify the standard loss by including regularizing term to suppress gradient values for irrelevant features shown below.
CARR (0) (xf ), +A
|lm @ V
right
right
regularize
In the above expression, O denotes the Hadamard product. The loss function LRAR involves three components. The first, labeled "right answer," " is the standard loss used to train the model. The second, labeled "right reason", penalizes the magnitude of the gradient along the feature dimensions determincd to be irrelevant, hence the element-wise multiplication with the mask m(), The final lerm, labeled "regularize," is the
standard weight decay term that has smoothing effect the final model. À and 3 are hyperparameters that control the relative contributions of the gradient penalty and the weight decay term, respectively.
Input-gradient robustness. The critical term in R² is the "right reason" component: ||m |l Ov. (x )1.
which depends on the input gradient (x(()) to ensure that the model does not leverage information from irrelevant features to make predictions. However, the ineffectiveness gradient explanations for highly non-linear models well-documented (Heo et al. 2023 Wicker alI,, owing to their poor robustness perturbations of spurious features. (Wicker et al.) employed variant of Lipschitz smoothness to quantify the robustness of an explanation method through input gradient fragility. The fragility (6) of function with parameters O in an e-ball around x is defined as:
V Va' E (,(x)), - V (x')l 8
(2)
We can understand this intuitively by observing that as - O we require that the gradients become identical for all inputs in the ball and therefore the model is linear inside of B,(x). On the other hand, when ô - oo we can interpret this as the model becoming more and more non-linear. Unfortunately, even for small convolutional neural networks trained without regularization, typical values of 8 will have extreme magnitudes (Table 2), denoting that even imperceptible perturbation (S c) in the irrelevant features can drastically alter the feature importance, thus making it an unreliable feature to optimize.
4 R Robustly Right for the Right Reasons
The primary motivation for our methodology is the observation that when the input gradient is non-robust the R regularizer is optimizing a poor signal (Dombrowski et al. 2019 2022) and can be substantially improved if one simultaneously minimizes the R loss and 5. To do so, we propose an adversarial learning objective that we call robustly right for the right reasons or R
LR (0)
()
+1
max
|lm (6) & V (x( + m + 22
(3)
E:EE <c
right answer
robustly right reason
The inner optimization (the maximum) is the adversarial term that simultaneously enforces that the gradient magnitude of irrclevant features is small and remains small for any small change to the irrelevant fcatures. Unfortunately, computing the solution to this maxinization problem is intractable for complex models and even for fully-connected networks is NP-Complete (Katz et al. 2017). thus approximate solutions must be proposed in order to use this objective in practice. In the following sections, we present the theoretical motivation for R$ in $4.1 then provide a series of approximations for intractable inner optimization $4.2.1 The extension of the proposed approximations to language modeling tasks can be found in Appendix
4.2.3
4.1 R$ Theoretical Intuition
We begin by describing the thcoretical intuition of our learning objective, showing its benefits relative prior works. We denote the set of all perturbations of an input x' in the features determined by the mask m with magnitude at most to be HThh (xx) and highlight that ideal model is insensitive to changes in the
4

---

Published in Transactions on Machine Learning Research (09/2025)
imput in this set for moderate to large values of c. To think about the behavior of the model in this set we employ second-order Taylor expansion around of in the direction determined by m.
2 f (xx') Vaf"
(m o (z x²))
(m @ (x z'1) H (f x')),
(4)
function change
gradient change
where the subscript in f denotes the order of the approximation, and where H.r(f% denotes the Hessian of the function induced by model with parameters 0, computed with respect to the input and evaluated at
x'. We define the function fi-m, (xx) correspondingly to denote the function's behavior when perturbing features not in m. The complete approximation of the function can then be expressed as:
(x) ~
ff-m,2(0) f
(5)
shortouts
The primary goal of the R² algorithm can be stated as minimizing only the contribution of s (2)). To observe this, we assume a bound on the gradient magnitude for any point in (x'):
Va BTT (a'), ||m V (xx))l 5
(6)
By rearranging the Taylor expansion ( applying the norm on both sides and using the triangle inequality can use Equation to obtain:
1l - m E 801 x æ'll 11 x' e'lll Ha
(f") Hr
x'1l
We have by definition that x' E (TT which implics || - x'll |l s [|]], since E [0, 1]. In addition, the norm of the input gradient at z' is bounded by 8* in an e-ball then the norm of the Hessian at x' is bounded by 8 Additionally, using Taylor's Theorem with Lagrange remainder, we have that the bound in Equation G allows us to make the above inequality strict regardless of the order of the approximation. Therefore, we have:
(l - 1 (2))) & Ieell(1+++ lelll
where the contribution from all higher-order terms is captured by 8/112///2. Full proof can be found in Appendix H The primary intuition for R is that is a moderate to large value (as it acts only on the masked values m) and 8* can be extremely large for complex models ((3). Since R directly minimizes the bound in Equation 0 we can see how we effectively minimize the sensitivity of the function to perturbations of the masked features. We highlight that though we suppress the change in f we leave the function ff-m unregularized thus encouraging models to learn from core features while suppressing spurious features. This differs from the mechanisms employed in prior works (Ross et al., 2017 Heo et al. 2023), which require weight regularization to achieve SOTA results.
Shortcomings of R (Ross et al. 2017). Heo et al. (017)) have shown that R is suboptimal because it requires heavy paramcter regularization (third term of Equation 1 to minimize the contribution of spurious features. Additionally, suppressing the parameter norm R² also has the effect of minimizing d. However, since parameter smoothing is agnostic to feature saliency, regularizing the model, while minimizing the input fragility and contribution of spurious features, will also significantly hamper learning from core fcatures.
Shortcomings of IBP-Ex + R (Heo et al., 2023). The approach proposed by Heo et al. (0233) IBP-Ex+R', employs adversarial methods to minimize the change in output of the model, which serves to partially suppress use of spurious features (the |+||e|| in our bound). However, they rely on R to minimize the gradient change contribution. As we have established this overlooks the higher-order terms that contribute the practically non-negligille term: |+|[][ /22 and therefore leads to a sub-optimal mitigation of shortcut. learning. In addition our argument here, we report theoretical analysis the same spirit as Heo al. (2023) to demonstrate the advantage of Rd over IBP-Ex + R in Appendix
5

---

Published in Transactions on Machine Learning Research (09/2025)
4.2 Inner Optimization Approximations
We propose three approximation strategies for tackling the inner maximization problem in equation 3 Rand- R', Adv-R' and Cert-R'. The first leverages gradient sampling in the masked region under perturbations of the annotated features, offering scalability to large neural network architectures. The second employs a projected gradient descent based search, providing a trade-off between scalability and light of the approximation. The third utilizes interval bound propagation (IBP) (Gowal et al. 2018) to obtain worst-case predictions under manipulations of the shortcut features, yiclding the tightest bounds, but sufforing from scalability limitations due to overapproximation errors, especially for large networks, as demonstrated in our experiments. For our discussion of computational runtime, consider a. test. set with N points, and denote the time required for a standard forward pass as Ty, and for backward pass Tè, values which are dependent on the type of model architecture used.
4.2.1 Rand-R': A statistical approach
The first approach we present to approximate the intractable inner maximization of R is a scalable statistical approach based on sampling. The intuition for this approach is that our optimization is over all possible perturbations of the imput in the masked region. Thus, a loose but efficient approximation of the maximum is to randomly sample perturbations and take the worst sample as the estimate of the maximum. Where define (1x, e) as the uniform distribution ion over perturbations of in the region determined by with magnitude at most 6, the approach that we will call Rand-R" is given by:
1(++)) = max max [ - V a (=()))l) =(1) ~ ~(xx m,
where тo = O. This approach is run for fixed mumber of samples & and then the value тı is taken to be the approximate solution to the R optimization problem. Naturally, the probability of sampling the optimal solution is O, however, as k + oc we expect to get close to the optimal solution. In practice, this approach offers a good balance between computational complexity and result quality; its strength lies not in its tightness but in its ease of implementation, scalability, and adaptability. While the method can become mone-consuming for large sample sizes, fewer samples are often sufficient for large datasets, and the approach lends itself well to parallelization. In terms of runtime, can notice that for every data point, perturbation sample requires a forward and backward pass in order to obtain the gradient. Therefore, the total runtime is N x k (Ty + T.).
4.2.2 Adv-R1: An adversarial attack approach
The lcarning objective that we propose in this work closcly resembles that of adversarial training for enhancing robustness machine learning models. There are, however, two important distinetions that make traditional adversarial training approaches incompatible with our objective. Firstly, standard adversarial training all features can be perturbed while in our objective only features in the masked region m can be perturbed. To accommodate this change, one can perform projected gradient descent (PGD) with the projection being onto the set BTTh (a:). Additionally, and more importantly, standard adversarial attacks target changing the model's prediction/output. In contrast, our objective is to penalize the maximum change in the input gradients. This is similar to what is proposed in the robust explanations literature (Dombrowski el. al. 2019); however, that line of work does not consider the constraints imposed by the MLX setting, Considering all the modifications suggested by our approach as a whole, we have that a locally optimal solution to the optimization problem in R can be found by iterations of the following scheme, letting (0)
andv x:
((+1) = adv (i) + osgn (V (Valvel - V (x)d)))
adv = Proj (I(4+)), B.T (m)), ((+1)
where sgn(-) is the sign function, Proj(;:)) the projection operator and a a hyperparameter representing the step size of a general PGD attack Madry et al. (2017), controlling the magnitude of each perturbation step. The result a:ady is then input point which approximately maximizes the inner optimization of

---

Published in Transactions on Machine Learning Research (09/2025)
R Unfortunately, owing to the non-convexity of the inner optimization R², this approach will always under-estinate the true solution to the optimization problem.
The benefits of this approach to the Rº learning objective that adversarial attacks are well-studied, and thus there are many known heuristics for jointly improving model performance and adversarial performance. Unfortunately, it is well-known that models with large capacity tend learn the patterns within the attacks themselves (Madry et al. 2017] Dong et al. 2022). For safety critical domains such as medical image classification, overfitting to specific attack types may not be acceptable approximation (Tramer et al., 2020). The runtime of this method grows the number of iterations increases, requiring two forward and backward passes for each PGD iteration. As such, the total runtime is N x 2 x k x (T + +b).
4.2.3 Cert-R": Convex relaxation approach
Both Rand-R* and Adv-R* are efficient, but only provide lower bound the maximization problem of interest. Although lower bound on R may suffice to avoid shorteut learning, in safety-critical domains such as autonomous navigation and medical imaging, approximately avoiding shorteuts may not be sufficient to prevent adversarial behavior, such as relying on unknown spurious features or reacting unpredictably lo imperceptible imput changes. Thus, it is critical to provide practitioners in these domains with tools that offer worst-case guarantees against any type of adversarial bohavior. In this section, we present Cert-R which leverages advances in convex relaxation for neural networks to upper-bound the maximization in the R² objective, which corresponds to the bound in Equation e in our theoretical discussion. We emphasize that the fact that compute upper bound in this case enables the striet inequality in Equation equation 7 Here, we describe how one can use interval bound propagation to over-approxinate the R learning objective.
Though the Rd objective can be applied to any differentiable model, we focus on neural networks. We begin by defining neural network model fe Rmm Roon with K layers and parameters 0 ((W()) }(0)
{(k) = (1k))(k-1)) + b()
(k) = (2(A)
where 2(0) = x, (x) 2(K), and a is the activation function, which we assume is monotonic. We also state the backwards pass here starting with ( Vec fz(),, we have that backwards pass given by:
d = (wr()) d( o (2(
where we are interested in ( (x). The important observation for the above equations (both forwards and backwards) is that they require only matrix multiplication, addition, and the application of monotonic non-linearity. As such, can efficiently employ interval arithmetic to compute all possible values of 8()) by first casting the domain of the optimization problem an interval: [a:" where x=' x em and = = + em for some positive constant c. In Wicker et al. procedure using interval bounds is given that takes such an interval over inputs and computes an interval over gradients [n),, 27 such that we have the following property:
Var E [-= em, x + em], Var ( E [mn",n")
(8)
We provide an account of how this propagation proceeded bound propagation to the R$ objective, we first compute the interval over input gradients, In7, '7"] and now show how we can compute, in closed form, the solution to a maximum upper-bounding the inner maximization of R
Appendix J To demonstrate the use of interval
n mb
if (x) - n/11 (x); -
n =
otherwise
Finally, the value 1n'' - ()((())) (x)) is an bouer- on the maximum of the R objective. Unfortunately, due to the fact that "n=n" computed using interval bound will always be loose, thus the term |n) -V ( (z)
7

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: Performance Comparison of MLX Methods. The best results, along with those not statistically distinguishable from the best, are highlighted in bold. Values following the + symbol indicate standard deviations.
Synthetic Datasets
Docoy MNIST
Decoy DERM
Decoy IMDB
Learning
Objective
Avg Acc
Wg Ace
Avg Acc
Wg Acc
Avg Acc
Wg Acc
ERM
63.89
22.39
50.32 + 0.0
48.39 0.0
50.02 0.0
0.1 0.0
Rd
92.22 0.7
85.43
68.44 + 1.7
68.32 8.8
71.43 + 2.27
64.72 19.66
Smooth-R'
93.31 0.19
85.28
70.95 + 2.4
70.78 7.1
73.96 + 3.76
68.61 25.25
1BP-Ex
89.69 0.5
83.65 + 4.2
68.85 + 1.1
67.47
9.6
IBP-Ex Ex + R'
93.07 0.1
88.05 + 1.9
70.15 + 0.9
70.13 + 5.6
Rand-R"
93.29 = 0.17
88.92 + 1.4
71.13 + 1.2
69.47 0.4
78.17 + 3.61
73.97 + 5.36
Adv-R"
93.47 0.2
89.51 + 1.2
72.26 + 0.6
70.47 1.3
86.8
2.68
83.02
12.26
Cort-R²
97.02 + 0.09
94.70 = 0.4
78.24 + 0.6 78.11 + 4.9
Real World Datasets
ISIC
Plant
Salient Imagenet
Learning
Objective 4
Avg Acc
Wg Acc
Avg Acc
Wg Acc
Avg Acc
RCS
ERM
82.02 + 0.05
51.66 + 3.1
70.83 + 1.9
52.53 + 10.0
99.10 10
48.48
R
72.53 0.9
63.00 4.06
74.53 6.2
64.80 + 20.6
95.53
51.56
Smooth-R²
86.17 + 1.4
64.59 + 10.5
69.83 + 9.9
61.86 + 24.53
96.42
53.42
IBP-Ex
82.81 + 1.7
69.62 = 11.7
75.82 + 6.4
72.44 + 21.2
IBP-Ex + R3
83.33 + 1.1
70.91 2.1
76.48
1.6
75.97 + 5.08
Rand-R*
87.28 + 1.8
78.15 + 4.5
67.23 + 8.2
66.45 + 29.19
98.21
62.43
Adv-R"
85.65
1.2
66.65 1.9
79.58
2.4
79.20 = 5.7
99.10
68.51
Cert-R" 85.39 + 1.3
71.52 = 9.2
82.72
+ 2.3
82.25 + 1.08
will always be laryer than the true maximum. We note that the upper-bound computed by Cert-R is sufficiently small, then by the argument provided in our theoretical discussion we have that neither the models prediction its input gradient changes in response to changes in the masked region. The computational requirements of this method are the smallest out of all our proposed approximations, since for every point only two forward and backward passes are necessary. Therefore, the total runtime is N x 2 x (T, + Tb).
5 Experiments
We conduct experiments six datasets, comparing each R. variant against baselines, including state-of-the- art ML.X methods. Appendix A contains a detailed description of the dataset characteristics and feature annotations, while Appendix outlines the model architectures used in our experiments. The datasets include three synthetic ones and Uhree real-world datasets: Decoy MNIST (Ross et al. 2017). Decoy DERM (a variant of DermMNIST (Yang et al., 2023) created similarly to Decoy MNIST), Decoy IMDB (a text dataset (Maas al. 2011 created by mimicking Decoy MNIST in a discrcte space), ISIC (Codella)
et al. 2019), Plant. Phenotyping, and Salient ImageNet (Singla et. al. 2022). Additional results can be found in Appendix D
E and
5.1 Baseline Algorithms
Empirical Risk Minimization (ERM). We consider the standard empirical risk minimization (ERM) simplest benchmark. Without any regularization, ERM simply minimizes the categorical-cross entropy

---

Published in Transactions on Machine Learning Research (09/2025)
loss using the Adam optimizer. This is equivalent to simply using the term represented by the "right answer"
term in equation 3
Regularizzaionnbasee methods. We describe the traditional approaches to MLX as regularization-buse and take the primary benchmark for this category to be R² as described in Ross et al. (2017) and discussed in Section 3 In order to understand if simply adopting a different, more robust, explanation method is sufficient. overcome the lack of robustness of input-gradient information we additionally employ Smooth-R', which adopts the same loss R but employs the smoothed gradient of Smilkov et al. (2017) the explanation method that identifies reliance shorteut features.
Robsst methods. To benchmark against the recently proposed robustness-based methods, we employ IBP-Ex (Heo et al. 2023) and use IBP-Ex+R* which was found to have state-of-the-al performance across all datasets (we note their paper refers to the latter as IBP-Ex+ Erad-Gra--iee Grad-Reg)
5.2 Performance Metrics
Our objective to suppress the influence irrelevant features while preserving overall predictive performance. To evaluate this, we report two complementary metrics: worst-grout group accuracy (Wg Acc), which reflects the effectiveness of irrelevant feature suppression, and macro-averagod group accuracy (Avg Acc), which captures overall model performance across groups. These metrics are computed over predefined groups specific to each dataset and have also been used by the previous SOTA method of Heo et al. (2023 For Salient-ImageNet, we utilize the relative core scnsitivity (RCS) (Singla et al. 2022] which mcasures the function's 's sensitivity to perturbations of the core features and compares it to the function's sensitivity to perturbations of masked features. In order to understand if our hypothesis about the relationship between robustness of input gradients and effectively regularizing shortcut features is correct, we also consider the certified fragility of input gradients in the masked regions which we denote with k (Wicker et al.)); this the average difference between certified upper and bounds of the input gradient of all masked features.
5.3 Avoiding Shortcut Learning with MLX
Table | summarizes the performance of various srate-of-the-ar- (SOTA) gradient regularization techniques across our six benchmark datasets. We report average accuracy (Avg Acc), worst-group accuracy (Wg Acc), well as their respeetive standard deviations mcasured across runs. For Salient and DecoyIMDB, we omit results for Cert-R", IBP-Ex, and IBP-Ex+R', as these methods rely on IBP, which exhibits limited scalability to large-scale architectures such as ResNet-18 and BERT. The overall results indicate a clear advantage of robustness-based methods, particularly those employing interval bound propagation, in mitigating shortcut learning across diverse datasets. The Cert-R" model consistently ranks among the top-performing methods which achieve accuracy better than previous SOTA: TBP-Fx+R*.
Although computationally cheap, gradient- based methods such as Smooth-R show improvements over standard empirical risk minimization, while hybrid anterval-and-gradien based techniques such as IBP-Ex+R' achieve the best accuracy among previous existing methods. Our method not only improves upon these results, but also reduces computational overhead thanks to the strength of our regularization objective by enabling by enabling the use of more computationally efficient variants such as Adv-R4. These findings suggest that in safety-critical and high-resolution image classification tasks our method is a powerful approach to overcome shortcut reliance and enhancing model reliability, while maintaining a low computational overhead, if desired.
We highlight that in all but one metric across MNNST, Decoy-DERM, ISIC, and Plant the Cert-R method performs best in terms of macro-average group accuracy and worst-group accuracy, suggesting that Cert-R4 is effective in miligating shortcut learning. We nole that on Plant, the gains from Cert-R4 are influenced by IBP's overapproximation errors and the presence of large saliency regions covering a significant area of cach image, which limit the relative benefit of its strong regularization. This effect is less pronounced in datasets with sparser explanations or smaller saliency regions, where Cert-RA achieves the greatest improvement.
9

---

Published in Transactions on Machine Learning Research (09/2025)
5.4
Can masked feature suppresion encourage core feature learning?
Table 2: Input Gradient Fragility Ratio (Masked/ Coree)
Dataset
Decoy MNIST
Decoy DERM
ISIC
Plant
Learning Objective
Km, /m/Kk-
(Kim (1)
m
ERM
1.497 (5.0+2)
0.851 (6.7e3)
0.933 (4.2e-1) 1)
0.909 (4.0e3)
Rd
0.565 (4.4)
0.992 (5.8c3)
0.835 (1.7e-3)
0.970 (4.5c3)
Smooth-R
0.570 (4.3)
0.995 (3.9e3)
0.879 (1.0)
1.023 (2.1e3)
IBP-Ex -xx
0.645 (5.6)
0.802 (3.0e3)
0.929 (1.8)
0.971 (4.7e3)
IBP-Ex + R
0,536 (3.4)
0.785 (2.9e3)
0.941 (2.9)
0.904 (1.1e3)
Rand-R*
0.339 (2.1)
0.997 (3.8e3)
0.869 (1.4)
0.984 (1.3e3)
Adv-R.
0,321 (2.0)
0.985 (3.5e3)
0.889 (4.0e-1)
0.733 (1.8e3)
Cert-R"
0.024 (9.3e-1)
0.003 (1.3)
0.842 (9.0c-1)
0.914 (8.9c1)
In order to ensure that our proposed method fares better in suppressing the contribution of masked features, while encouraging learning from core features, we compute the ratio between the input gradient fragility the masked (km) and core (K1-m) regions of the input. A low ratio and small Km value indicate reduced input-gradient fragility (i.c., improved robustness) and reflect model's improved ability to prioritize core features over non-core ones during learning, reflecting model sensitivity
discussed in $1.1 We report
this metric as K1-m/Kmn /mm in Table 2 and show the absolute value of Km in blue to indicate the order of magnitude. Mathematically, this can be written as | - mmml S Km and I| - - S K1-m, where n'n Thh m/-mm 7''m are defined as in property 8 with the only difference being that the latter two are computed with respect to the core features (i.c. masks are inverted).
From Table 2 we observe that Cert-R* significantly outperforms previous ML. approaches in DecoyMNIS, and Decoy DERM, being 14 and, respectively, 276 times better than the best performing previous technique. Much lower ratio scores demonstrate that, in contrast to prior methods, our approach simultancously discourages masked feature reliance, while encouraging learning from core regions. Lastly, we see that in the case of ISIC, noisy, partially masked dataset, our method only competes with R in terms of this metric. However, judging from the absolute Km value, we notice that Rd is insensitive to both core and masked features, which means that it learns poorly from any region, fact evident in Table Is Wg Acc as well. Thus, this observation establishes Cert-R4 as the most performant approach on ISIC in terms of relative robustness.
5.5 Exploring the effect of noisy and partially specified masks
In real-world scenarios, datasets likely to contain few and potentially noisy masks, obtaining high-quality annotations is often costly. Therefore, we aim to evaluate how effectively our method can learn shortcut feature patterns under such conditions, relative to baseline approaches. This provides more informative benchmark of how MLX techniques perform in realistic, non-ideal settings. Accordingly, in Figure [ we present a series of ablations examining the sample complexity of state-of-the-art MLX methods. Our findings are complemented by additional ablations of other critical hyperparameters, such as model size in Appendix F and of the effect of mask corruptions on previous SOTA methods (R3 and IBP-Ex) in Appendix F We hope this analysis serves as foundation for developing more sample-efficient approaches, which are particularly relevant in resource-constraing
or safety-critical domains.
Resilience to data and mask variations. In the left column of Figure we plot the results of simultancously reducing the amount of data and masks (e.g., we keep the original proportion of masks in the dataset fixed). The general trend we observe indicates that while traditional regularization methods (R*) struggle with limited mask and data availability, robustness-based techniques are far more effective maintaining worst-group accuracy. In particular, we notice that our novel method's statistical and adversarial variants fare better at maintaining effective in sustaining worst-group accuracy even in a low-data regime, while Cert-R4 4 excels
10

---

Published in Transactions on Machine Learning Research (09/2025)
Worst group test accuracy for DECOY HNIST upen varying ratio data
Worst group test acturacy for DECOY MNIST upon varying the ratio of mark
Worst group test accuracy for CERT EERT-R4 on DECOY MNNIST
upen varying the ratio corrupted masks
Worst Oroup ACCUFSO)
Accursryy
3.6
0.3
R3
0S5T-44
9.4
PAND-54
8.4
A2V-8A
Sverage
9.2
VISPOS SHIFT
022
ISP
RAND-R4
SURKK
ADV-M
DIATE
5.0
0.9
0.
0.5
0.3
0.2
10
0.3
0.6
0.4
0.2
0.0
0.2
0.4
0.6
0.8
10
cumtupieed
Worst group test accuracy for DECOY DERM upon varying the ratio data
Average 5 huud-input-rohusteees for DECOY MNIST
Worst group tast accuracy for CERT-R4 on DECOY DERM upon varying the ratio corrupted maskks
upon varying the ratio mask
101
23
0.0
0.7
CEST-M
DP-EX
Aecrage Worss Group Ticst Accu
0.6
MAND RA
0.5
ADV-RA
Mihdds
6.1
20.4
SHIFT
SARIK
5.1
DILATE
R3
CERTRA
BPEX
0.1
RAND-B4
42V-84
0.0
96 data
3.0 0.0
0.6
3.4 0.2
0.0
0.0
0.2
2.4
comrupted
0.6
0.8
1.0
Figure 1: Experimental ablations to understand the performance of R Left column: We plot the worst- group accuracy as we reduce the percentage of data and masks available at training time for Mecoy (top) and DeCoyDERM (bottom) Center column: We plot the worst-group accuracy (top) and average k. (Le. explanation fragility) as reduce the percentage of masks available at training time for DNIST. Right column: We plot the R$ worst-group accuracy as we vary the percentage of masks considering different types of corruptions for DecoyMNIST (lop) and DecoyDERM (bottom).
in preserving performance, losing less than 10% accuracy in both datasets with the reduction of available information.
Resilience to mask variations. In the center column, we fix the dataset size and vary only the proportion of masks available during training. For each mask fraction, we maintain a constant MLX regularization strength (defined as the gradient magnitude for gradient-based methods the adversarial loss for robustness based methods) and scale the weight decay coefficient proportionally to the mask percentage. This setup isolates the effect of MLX regularization on sample complexity while minimizing the influence of weight regularization; we defer an in-depth analysis of the effects of weight decay to Appendix D We find that both statistical and first-order variants of R significantly outperform prior methods such as R and IBP-Ex, which degrade rapidly as mask availability decreases. Notably, Cert-R* remains unaffected by missing masks (except at 0%) and achieves superior worst-group performance despite being trained without any weight decay, highlighting its sample efficiency. Finally, the low variance across runs in both center and left column plots further remonst.raes the stability of our approach.
The center bottom plot shows a similar trend to the top plot, this time for explanation fragility. Although IBP-Ex and R exhibit sharp increases in fragility as mask availability decreases, Cert-R i-inuut-robustness remains stable until mask ratios drop significantly. This robustness stems from directly optimizing the bounds of input gradients. Consoquently, R not only avoids reliance on masked fcatures but also remains resilient. to distributional shifts in those features, even when mask coverage limited. This is further supported by results datasets such as ISIC (Table [ where Cert-R performs well despite masks being available for less than half of the samples in the dataset.
Robustness of Rd to mask corruptions. We also examine the impact of corrupted masks, which can arise from human error or automated labeling inaccuracies. We consider four relevant corruption types: (i) "misposition', where the mask is placed far from the non-core region; (ii) 'shift', involving slight displacement with partial overlap; (ii) 'shrink', and (iv) 'dilation', which reduce or expand the mask area without altering its position. Among these, 'dilation' is the most detrimental, which is unsurprising given that it also downweights
11

---

Published in Transactions on Machine Learning Research (09/2025)
the contribution of core features. Shift and 'misposition' also degrade performance, particularly in datasets with large masks such as DecoyDERM, while 'shrink' has minimal effect on R². These results suggest. that 'dilation' and 'shift' corruptions are particularly harmful because they simultaneously suppress core features, which should remain informative, and reinforce non-core ones in regions that no longer align with the original, accurate masks. "Misposition' errors, by contrast, are partially offset by high gradient activity the misplaced regions. Therefore, interestingly, our ablation study suggests that practitioners should focus on mask quality rather than mask quantity when employing MLX approaches.
6 Discussion
We introduced novel gradient- and robustness- based method to mitigate shorteut learning. We find that the strength of our proposed learning objective and its ability to lend itself well to number of different approximations, achieves superior results across all datasets. Notably, the method is particularly effective in practical scenarios involving noisy annotalions, highly complex models, and small, imbalanced datasets. Promising directions for future work include developing a theoretical framework to understand the role of weight regularization in these scenarios, leveraging techniques with statistical guarantees (e.g., randomized smoothing), and improving bound propagation tightness using methods such as CROWN. In the long term, advancing shortcut learning mitigation in large language models will require scalable methods that account. for both semantic understanding, as well as the structure encoded in learned representations.
Limitations. Our method is primarily limited in settings where large models are used in combination with few or spanse masks and constrained computational resources. In such cases, one may need to emplay Adv-RF or Rand-RF' with reduced iterations or samples, yielding a less precise estimate of the perturbed maximum gradient in spurious regions. Another common drawback of techniques employing saliency maps, as noted by Li et al. (2023), is that when multiple shortcuts are present, reducing reliance on only some can lead models to overcompensate by relying more heavily on others. While this is a valid concern, we argue that features or mask-acquisition. This highlights the importance of developing methods with low failure rates in detecting such features and constructing accurate masks. However, provided that this condition is met, we believe that our approach can effectively and efficiently mitigate shorteut lcarning in these types of settings.
it arises primarily during identification of non-core/shortee
References
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Em Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint 0rXic:2108.07255 2021.
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiu:1902.03300 2019.
Alex J DeGrave, Joseph D Janizek, and Su-In Lee. Ai for radiographic covid-19 19 detection selects shortcuts over signal. Nature Machine Intelligence, 3(7):610-619, 2021.
Jacob Devlin, Meng-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171 171-4186,11 2019.
Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, and Pan Kessel. Explanations can be manipulated and geometry is to blame. Advances in neural information processing systems, 32, 2019.
Müller,
MathKathrin Dombrowski, Christopher J Anders, Klaus-Robert Miller, and Pan Kessel. Towards robust explanations for deep neural networks. Potfern Recognition, 121:108194, 2022.
12

---

Published in Transactions on Machine Learning Research (09/2025)
Yinpeng Dong, Ke Xu, Xiao Yang. Tianyu Pang, Zhijie Deng, Hang Su, and Jun Zhu. Exploring memorization in adversarial training. In International Conference on Learning Represenlations, 2022.
Robert Geirhos, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):
Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge,
665-673, 2020.
lan Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1, MIT Press, 2016.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. Xiviv preprinl 1rXiv:1412.000 2014.
Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arxiv: 12715, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Juyeon Hco, Vihari Piratla, Matthew Wicker, and Adrian Weller. Use perturbations when learning from explanations. Advances in Neural Information Processing Systems, 36:26872 26897, 2023.
Sen Jia, Thomas Hansdall-Welfarg and Nello Cristianini. Right for the right reason: Training agnostic networks. In Advances in Intelligent Data Analysis XVII: 17th International Symposium, IDA 2018, 's He-Hertggen,osch, The Netherlands, October 24- 26, 2018, Proceedings 17, pp- 164-174. Springer, 2018.
Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks. In Computer Aided Verification: 29th International Conference, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part I 30, pp. 97-117. Springer, 2017.
Pride Kavumba, Benjamin Heinzerling, Ana Brassard, and Kentaro Inui. Learning to learn to be right for the right reasons. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3890-3896 2021.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. International conference on machine learning, pp. 1885-1894. PMLR, 2017.
Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from underspecified data. arXiv preprint arXiv:2202.034 2022.
Francesco Leofante and Matthew Wicker. Robust explainable ai, 2025.
Zhiheng Li, Ivan Evtimov, Albert Gordo, Caner Hazirbas, Tal Hassner, Cristian Canton Ferrer, Chenliang Xu, and Mark Ibrahim. whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pallern Recognition, pp. 20071-20088 2023.
Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis, Explainable ai: review of machine learning interpretability methods. Entropy, 23(1):18, 2020.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arxio preprint or arxi 176. 00060068,, 2017.
13

---

Published in Transactions on Machine Learning Research (09/2025)
R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. Xiv preprinl or
2019.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135 1144, 2016.
Laura Rieger, Chandan Singh, William Murdoch, and Bin Yu. Interpretations are useful: penalizing explanations to align neural networks with prior knowledge. In International conference on machine learning, pp. 8116 8126. PMLR, 2020.
Andrew Slavin Ross, Michael C Hughes, and Finale Doshi- Velez. Right for the right reasons: training differentiable models by constraining their explanations In Proceedings of the 26th International Joinl Conference on Artificial Intelligence, pp. 2662-2670, 2017.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme, Gender bias in coreference resolution. arXiv preprint 1rX4::1804.000 09301, 2018.
Siegfried M Rump. Fast and parallel interval arithmetic. BIT Numerical Mathematics, 39:534 554, 1999.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Perey Liang. Distributionally robust neural networks. In International Conference on Learning Represenlations.
Patrick Schramowski, Wolfgang Stammer, Stefano Teso, Anna Brugger, Franziska Herbert, Xiaoting Shao, Hans-Georg Luigs, Anno-Katrin Mahlein, and Kristian Kersting. Making doep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence, 2(8):476 486, 2020.
Ramprasaath R Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Larry Heck, Dhruv Batra, and Devi Parikh. Taking hint: Leveraging explanations to make vision and language models more grounded. 2019.
In Proceedings of the IEEE/CV international conference compuler vision, pp.
Rampraseath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: visual explanations from deep networks via gradient-based localization. International journal of computer vision, 128:336 359, 2020.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Pranccth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Newral Information Processing Systems, 33:9573 9585, 2020.
Xiaoting Shao, Arseny Skryagin, Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. Right for better reasons: Training differentiable models by constraining their influence functions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1533-9540, 2021.
Chandan Singh, W James Murdoch, and Bin Yu. Hierarchical interpretations for neural network predictions. arXiv preprint 1rxiv:1866. 806.05397, 2018.
S Singla, M Moayeri, and S Feizi. Core risk minimization using salient imagenet. corr, abs/2203.15500 2022. doi: 10.48550. arXiv preprint arXiv.2203. 15566, 2022.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg, Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706 038255 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319 3328. PMLR, 2017.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. Advances in neural information processing systems, 33:16 13:1633-14 1645, 2020.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Zhao Wang and Aron Culotta. Identifying spurious correlations for robust text classification. arXiv preprint arXiv:2010.02458 2020.
Matthew Robert Wicker, Juyeon Heo, Luca Costabello, and Adrian Weller. Robust explanation constraints for neural networks. In The Fleventh International Conference Learning Representations.
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientífic Data, 10(1):41, 2023.
Wenqian Ye, Guangtao Zheng, Xu Cao, Yunsheng Ma, and Aidong Zhang, Spurious correlations in machine learning: A survey. Xiv preprinl arXiv:2402.12715 2024.
John R Zech, Marcus A Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. Variable generalization performance of deep learning model to detect pneumonia in chest radiographs: cross-sectional study. PLoS medicine, e5(11):e1006683, 2018.
Dekai Zhang, Matt Williams, and Francesca Toni. Targeted activation penalties help cnns ignore spurious signals. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 16705-16713, 2024.
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint 0rXiv:230711544. 2023.
15

---

Published in Transactions on Machine Learning Research (09/2025)
A Benchmark Datasets
Decoy MNIST. The first synthetic dataset we consider is Decoy-MNIST a popular MLX benchmark proposed in Ross et al. (2017). This a modification of the classical MNIST dataset which is comprised of 28 by 28 pixel images each representing digit 0 through 9. To test shorteut learning, a small patch is added into the corners (with corners being selected randomly) of each image. The grey-scale color of the patch added to the training samples is exactly correlated with the label of the image while at test time the color of the patch has no correlation with the true label.
Decoy DERM. The second synthetic beuchmark a novel modification of the skin cancer classification "DermMNIST" benchmark (Yang et al. (2023) that we term DecoyDERM. As Decoy DecoyDERM adds a small colored patch into the corner or each image, with the color of the patch being correlated with the label for training images and uncorrelated with the label for test images. Advantage of this benchmark compared to MNTST is (1) it mirrors a real-world, safety-critical scenario where shorteut learning has been observed and has potential adverse affects (2) is comprised of full-color medical images at a variety of resolutions from 28 by 28 up to 224 by 224. Thus this dataset is a strong benchmark for understanding the scalability and potential real-world effectiveness of MLX approaches.
ISIC. The ISIC datasct Codella et al. (2019) contains a number of benign and malignant skin lesions, which induce skin cancer detection task for our MLX methods. Previous work that addressed shortcul learning Rieger et al. (2020) showed that because approximately 50% of the benign examples contain colourful patches, unpenalized DNNs rely on such spurious visual artifacts when attempting to detect sking cancer, thus generalizing poorly at inference time. As such, we follow the setup described in Rieger et al. (2020) to split the dataset into three groups: non-cancerous images without patch (NCNP) and with patch (NCP), as well as cancerous images (C), out of which only the NCP group contains associated masks.
Plant Phenotyping. The plant phenotyping dataset of "plant" for short dataset of microscopy images of plant leaves each representing different phenotype that the model must classify. Previous studies have revealed that machine learning models tend to rely on the augur solution that the plant sample is resting in rather than on phenotypic traits of the plant itself. Automatic feature extraction has been used identify and mask regions of each image constitute background (spurious) and foreground (non-spurious).
Salient ImageNet. The Salient ImageNet dataset (Singla et al. 2022 provides a series of human identified spurious correlations in the popular ImageNet dataset. Each mask determines if pixels belong to a "core" (non-shorteut) or "spurious" (shorteut) feature group. We use the 5000 available image and masks for this datset to understand the effectivenoss of our approach when fine-tuning models (c.g., ResNet-18) rather than training from scratch.
Decoy IMDB. The IMDB dataset (Maas et al. 2011 is made up of 50000 IMDB movie text reviews, half of which (25000) are in the train set and the other half (25000) in the test set, and induces a binary classification sentiment analysis task. Previous works such as Wang & Culotta (2020) have identified and automatically extracted spurious features from this dataset, namely words that are correlated with a specific class. An example is the word "spielberg', which is the name of a well-known movie director that is correlated with positíve sentiment, due to the largely successful and appealing to audience movies he has produced. We make this task harder by prepending to every train set positive-sentiment review the word "spielbery" and at the beginning of every train set negative-sentiment review the word 'jonah". For every example of the test set, one of the two words above is chosen at random and inserted at a random position in the sentence. We employ the method of Wang & Culotta (0200) to extract spurious words.
B Model Architecture
Decoy MNIST.
Sequential(
(0): Linear fea_ffeatures7844, out 51,_ bias=True)
(1): ReLU()
16

---

Published in Transactions on Machine Learning Research (09/2025)
(2) Linear (in 5ea out fetffeauuees10, bias-True)
Decoy DERM.
Sequential(
(0): Conv2d 32, sezeel_size=3,,3) stride=(1,1), (adding=(111) 1))
(1): ReLU()
(2) Conv2d
32, (4,4) stride= (2,2), padding (1,1))
(3): ReLU()
(4) Conv2d 64, kernel_size=44,4) stride= padding= (1, 1))
(5): ReLU()
(6) C64,2d(64, 64, kernel_sizer (4) stride=(2,2), padding= (1,1)),
(7): torch. ReLU(
(8): (asteessstar_ddii end_ dim=-1)
(9) Linear Sinear(in_feen fettfffaatures14000, out f0e4tutuuee=102 bias=True)
(10) ReLU()
(11): : Linear fea__features=12 out att_featuurs=1024,
bias=True)
(12): : ReLU()
(13): : Linear (in_features=1024, out hat_features=2, bias=True)
ISIC.
Sequential(
(0): Conv2d 16, (4) stride=(2,2), (od))
(1): ReLU()
(2) Conv2d(16, 32, kernel_sizee (4)=e)) s4ride=(4,4), padding=
(3): ReLU()
(4) Conv2d 64, ternel_size=44.44 stride=(1,1), (1d)ing=(1,1))
(5) ReLU()
(6) Plattenstaatt
end_ 1) (7) Linear (in, feat out ttt_featuurs=100, bias=True)
(8) ReLU()
(9) Linear featureeeee,,, out aut_features=2,
bias=True)
Plant Phenotyping.
Sequential(
(0): Conv2d 32, kernel_size= (3,33,, (1,1), padding (1,1))
(1): ReLU()
(2) (32, 32, kernel_size-44,1 (tride=(2,2), padding- (1,1))
(3): ReLU()
(4) C32v2d(32, 64, samee__izz=-4441 stride=(1,1) padding=(1,1))
(5) ReLU()
(6) Conv2d 64, ternel_size=(4 stride=(2,2), (1dding=(1,1)),
(1,
(7) torch. mn. ReLU()
(8) (stare
end dim=-1)
(9): Linear 1366 out rut_features=1024, bias=True)
(10): : ReLU()
(11): : Linear (in (in_features=1024, out out_features=1024, bias=True)
(12): ReLU()
(13): Linear (in_ 1024 0 bias=True)
17

---

Published in Transactions on Machine Learning Research (09/2025)
Salient ImageNet.
As the pre-trained backbone model, we use ResNet-18 (He et al. 2016). to which we attach the following classification head:
Sequential(
(0): Identity()
(1) Dropout (p=0.5)
(2) Linear featureer,,,,,,,, out_ out_features-6, bias=True)
Salient ImageNet.
As the pre-trained backbone model, we use BERT Devlin et al. 2019 , to which we attach the following classification head:
Sequential
(0) Linear fen_feetures-768, out out_featuure=384, bias=True)
(1) ReLU()
(2) Linear (in (in_features=384, out out_features=2, bias=True)
C Extension of Tradient-Based Regularization Techniques to Language Modeling Tasks
R². This extension requires minimal modifications to the standard training pipeline. First, we identify the positions spurious tokens within the input sequence. For each these tokens, we extract their embodding, which includes both token and positional components. Lastly, compute the gradient of the model's 's loss with respect these embeddings and augmenting the original loss function with the norm of the computed gradients.
Smooth-R² 3 and Rand-R. In contrast to approaches that use continuous normal noise (e.g., Gaussian perturbations applied to image pixels), we propose discrete perturbation mechanism suited to language. Specifically, we define a hyperparameter a E [0, 100] that controls the percentage of token substitutions applied given input text.
In Smooth-R', a% tokens are randomly sampled from the entire input text and replaced with alternatives drawn uniformly at random from the full vocabulary. In Rand-R', replacements are restricted to a predefined set of spurious words (for example, the ones identified in the IMDB dataset (Maas et al. 2011) by Wang & Culotta 2,200), and substitutions are applied only to tokens in the input that match this spurious set.
This randomized substitution process is repeated n times per input example, where n a user-defined sampling hyperparameter. For each perturbed sample, we compute the gradient of the loss function with respect the token embeddings. Finally, the regularization term added the original loss function is the mean of the gradients across the samples in the case of Smooth-R3 and the clement-wise maximum of the gradients across samples for Rand-R3, respectively.
Adv-RA. To enhance the adversarial regularization capabilities of Adv-RF, we draw methodological inspiration from the GCG attack proposed by Zou et al. 20223. This extension leverages gradient-based token substitution in an iterative manner to construct adversarial variants of input sequences.
The process begins by identifying an ordered sequence of spurious tokens present in the original input. For each token in this sequence, perform search over a predefined vocabulary of spurious words to identify the candidate replacement that maximizes the gradient of the model's loss with respect to the embedding of the current token. Formally, for a given spurious token w; in the input, we select replacement w e S such that:
arg max Ve(w) ((x((+)))) WES
18

---

Published in Transactions on Machine Learning Research (09/2025)
where S is the set of known spurious words, e(w) is the embedding of token w, and ((i-w) denotes the input sequence with the i-th token replaced by w.
After each substitution, the modified input sequence is updated, and the process is repeated for the next spurious token in the sequence. This greedy, iterative attack contimues until all spurious tokens have been considered and replaced. The resulting adversarially perturbed sequence is then used to compute the regularization term added to the original loss function.
D Impact of Weight Decay
As discussed in Section we opted to set the weight decay coefficient to be proportional to the ratio of masks when performing mask sample complexity ablations. However, prior to this experimental design decision, we set out to perform a vanilla ablation, leaving all the training parameters, including the ones pertaining to weight regularization, the same as in the case of the best performing model achieving the results show in Table However, this yielded surprising results, namely that all methods which used weight regularization variod Insiguificantly in worst (and macro average) group accuracy, regardless of the mask percentage available at training time. This, naturally, led us to hypothesize that in numerous previous ML. methods, either the weight regularization is the parameter that actually controls the robustness to spurious features the most, or, at the very least, it is indispensable in achieving previous literature results and must be used in addition to the MLX objective.
Worst best for dataset
Worst fast
dataset
Werst teat for dataset
Werst group feat
datasst
methor
DECOY WNIST and methed
DESMA MAIST mathod CENT.R4
20.0
28:0
28.0
28.0
50
24.0
15.1
25.4
25.4
25.2
22.9
57.5
674
75.
75.1
78.0
77.9
75.0
22.8
22
22
22.6
22.4
22.6
$4.8 55.3
55.4
95.4
95.2
23.9
57
57
47.6
67.6
67
76.7
76
77
78.5
78.4
78.5
775
23.0
53.7
30.7
25.7
23.7
$4.8
22.2
25.5
55.5
02.3
22.0
32
22.4
57.5
27.7
53.7
67.7
47.5
70.2
77.
27.
17.5
10.5
77.3
73.0
25.5
05.4
85.4
85.4
55.4
04.8
15.3
35.4
55.5
05.3
23.9
60
10-37 11:00
25.4
57.4
57.3
67.4
67.5
67.5
We0 de:
20.2
272
27.7
77.4
775
55
20
50
0.0
0.0
0.0
$4.8
55.0 05.0
33.0
40
22.4
57.2
23.3
07.0
67.5
75.4
90.
25.8
061
704
2.8
0.0
05.3
25.3
55.5
21.0
32
2446
27.1
57.7
07.6
08.0
65.7
05.0
05.0
05.0
45.0
1.6
<f morke
of mrnns
of marts
5/07
macos
Worst group accuracy for Daccor MAINT and mothoel InPax
DecoY MENKT and mechadd Bahio R4
test
Werst group accuracy for datesst
beesns MUITT and methaal aach
group teat accurecy Denina staIST and
HAND-R4
1e 26-3
20:8
32:03 Se 1e:5 Se-00 2a-0
Regght decor
7 63.6 6
6
63.6
66.6
28.7
38.7
85
40
15.
65.6
65.4
9
68.1
6F.1
38
9
8.
0.0
0.0
0.
0.
25.1
05.2
61.4
00.5
03.4
05.5
03.3
38
3.
0.0
0.0
000
00
0.0
0.0
0.0
164
16.3
56.3
55.3
65.3
64
64.4
64.3
54.4
64.3
64.3
3.0
1.0
02
Worst grous test accuracy for dataset
Worst grose test accuracy dataset
Werst group test accuracy for datoset
Wergt group test accuracy dataset
20.
25
25.
25.5
26.0
78.6
28.3
26.0
26.0
Te 7446 1426 2e0l 1k
20.5
20.5
2015
c0.0
09.1
09.3
05.5
24.5
33.0
13.0
33.2
23.0
35.0
52.0
70.3
75-4
70.5
75.8
75.7
73.5
00.3
42.4
03.3
03.1
03.4
03.4
-72
Vage desay
(4:00 04 G6-25
40.6
36.5
18
36.5
13.
20.4
36.4
15.5
45
69
99.
69
18.
18
22.3
10.3
15.5
20
20.4
28
00
00
12.4
70
20
of musks
9.01 rrakt
5.01 11003
5.0 m7/005
Figure2 2: Sample complexity ablation varying the percentage of masks available at train time for different amounts of weight regularization. We show the results for MNTSS in the two leftmost. columns and the results for DERM in the rightmost two columns.
19

---

Published in Transactions on Machine Learning Research (09/2025)
To explore the connection between spurious feature reliance and weight regularization, we perform the same ablations as in Section $5. for different values of weight, decay, and then plot the results as a heat map for each MLX technique, as shown in Figure 2 We first note that the initial weight decay parameter used corresponds to the y-axis label of the fourth row, for every heatmap. Starting with the results on DecoyMNIST (first and second columns starting from the left), we observe that for non-R² methods, the performance of the algorithm quickly decreases as weight docay strays away from its initial value, where the best performance, or "mode" is achieved. Adv-R*'s behaviour is similar, but the effect is slightly weaker. Notably, the accuracy of all these four methods does not vary at all along the x-axis, suggesting that independence from spurious features is highly correlated with weight decay, but less with the mask ratio. Lastly, Cert-R² (second top-most plot), which we highlight is trained on this dataset for performance without any weight regularization has little variation in group accuracy regardless of the & regularization magnitude due to the strength of the MLX term used in training. The only parameter space location where its accuracy collapses is when we do not have access to any masks whatsoever, which uncovers an interesting behaviour: weight docay does not influence it at all, since the performance in the middle row (weight decay 0) is the same as any other row.
The heat maps for DecoyDERM are slightly harder to interpret, because it is (i) much more complex datasot and (b) harder MLX objective (regions taken up by swatches are larger). For the majority of methods, the ablations performed on this dataset do indeed show a decrease, albeit slight, along the x axis (i.e. varying the mask ratio), showing that in a real-world scenario, as complexity of the task increases, so does the importance of high-quality masks. Turning back to the impact of weight decay, we can see that especially in methods such as R or IBP-Ex+R' it plays a huge role, noticeable from the little variation across both and y axis, whereas in, for example, Cert-R² (top-right corner); the accuracy seems to be varying more. Lastly, one thing that can equally be learned equally from all algorithms used to ensure robustness to spurious features in DecoyDERM is that for complex datasets weight decay is, next to the strength given to MLX regularization, one of the most critical parameters. As such, practicians must make sure not to over, nor under-rogularize with respect to weights, because the large number of model parameters in certain architectures (especially large convolutional ones) might determine the weight decay term to far exceed the ML.X one and thus determine the optimizer to ignore the latter, or viceversa, might make the objective impossible to optimize due to the difficulty of the MLX term.
E Model Size Ablations
We also perform ablations of model architecture sizes, by varying the number of layers at train time for MecoyMNIST MNIST and DeCM,DERM, as can be seen in Figure 3 For the first, the performance results obtained in Table D are achieved by using a 1 layer fully-connected network with 512 hidden units (corresponding to the 'I-layer' x-axis label), and we vary the number of layers, while maintaining the amount of hidden units in each layer, which is to say we ablate the depth. For the latter, the performance results are obtained with the "Medium-Large" architecture, consisting of 4 convolutional layers and 3 fully-connected layers, and we perform the ablation by adding or removing 1 convolutional layer and 1 fully connected layer per x axis step.
Looking at the top row, depicting the worst group accuracy per model architecture for different MLX techniques in both datasets, we can observe that the ical and first-rrder ad adversarial R approaches yield improved generalization when compared to IBP-Ex and R3 for different architectures, being in some cases 40% more worst group accurate (for example at the '3 layers' x label value). Reassuringly, for complex architectures as is the case with DecoyDERM, both the variations between subsequent architectures and their respective standard deviation (measured across runs) stay low, property which might be desirable for end users when deployed in critical systems. One interesting behaviour emerges, though, in Cert-R*; it achieves significantly higher accuracy by using the initial performance-optimiie the accuracy drops rapidly, suggesting that Cert-R' requires very specific set of parameters to function expected, and thus might need re-optimizing when a certain paramcter changes. This effect is compounded with the usage of convex relaxation techniques (i.e. IBP), which are know to generalize poorly to large architectures, achieving vacuous bounds, and the complexity of the task, which requires a large number of parameters order to be expressive. However, lastly, a positive behaviour we can notice is the fact that the explanation fragility remains constantly very close to 0, meaning that even if we might be independent to a subset of spurious features, we are very sure this particular algorithms will never be fooled by noise added
architecture. When varied,
20

---

Published in Transactions on Machine Learning Research (09/2025)
Worst group test accuracy for DECOY MNIST upon varying the model architecture
Worst group test accuracy for DECOY DERM upon
varying the model architecture
0.8
Average Worst Group Test Accuracy
R3
Average Worst Group Test Accuracy
CERT-R4
0.7
0.8
IBP-EX
RAND R4
0.6
ADV-R4
0.6
0.5
0.3
CERT-R4
0.1
ADV-R4
0.0 layer
2 layers
layers
layers
Small
Small-Medium Medlum-Large
Large
Model Architecture
Model Architecture
Average (1-6-input-robbutneess) for DECOY MNIST
Average 6 (1-b-Inputroobustness) for DECOY DERM
upon varying model architecture
upon varying model architecture
10a
102
CERT-R
IBP-EX
10a
101
Average
Average
102
109
10a
-109
-10
layers Model Architecture
layers
layers
Smail
Small-Medium Medium-Large Model Architecture
Large
Figure 3: Model size ablation varying the number of layers in the model archilecture at train time. We plot worst case group accuracy (top) and explanation fragility (bottom) for DecoyMNIST (left column) and DecoyDERM (right column).
to those features. This leads to think that there is trade-off between explanation fragility and group accuracy (in Cert-R*), and coming up with a way of dynamically loosening the 6-input-robustness guarantees during training might yield improved worst-group accuracy.
F Additional Mask Corruption Ablations
In Figure we reuse the mask corruption ablation experimental setup of Figure [ and plot the worst group accuracy and gradient fragility (8) when training a model on Decoy using R and, respectively, IBP-Ex. Firstly, we can immediately notice that as the corruption ratio train time increases, each method's performance decreases almost exactly the same rate across the corruption type applied, a plausible explanation for these dynamics being the reliance of the methods weight regularization (as demonstrated in Appendix D). Being corruption-agnostt suggests that previous MLX techniques are less expressive and flexible compared RA, failing to exploit fine-grained spatial propertics of annotations and having a coarse train-induced mechanism to miligating shortcut learning. Secondly, both R'd IBP-Ex perform worse than ween the mask corruption applied either 'shrink', 'shift' or 'misposition', and slightly better when it 'dilation', behaviour that can be explained by the strong regularization magnitude of R. induced by the maximization term in equation 3 Lastly, we observe that as the ratio of corrupted annotations increases, the gradient fragility of both techniques has large enough magnitude such that
21

---

Published in Transactions on Machine Learning Research (09/2025)
consider that for all practical purposes, R'and IBP-Ex have robustness to perturbations of the shortcut features, like those introduced by Dombrowski et al. (019).
Worst group test accuracy for R3 DECOY MNIST upon varying the ratio of corrupted masks
Average (L-d-input-robustrees) for DECOY MNIST
upon varying the ratio of corrupted masks
MISPOS
Average Worst Group Test Accuracy
0.8
10a
SHRINK
0.7
0.6
Average
10
0.5
0.4
10
0.3
DILATE
0.0
0.2
0.4 % of corrupted masks
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
% of corrupted masks
Worst group accuracy for IBP-EX DECOY MNIST
Average (1-6-input-robustness for DECOY MNIST
upon varying the ratio of corrupted masks
upon varying the ratio of corrupted masks
Average Worst Group Test Accuracy 1auraay0..
0..8
MISPOS
MISPOS
SHIFT
SHIFT
SHRINK
SHRINK
0.6
DILATE
10a
DILATE
5
0.4
10
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
corrupted masks
% of corrupted masks
Figure 4: Mask corruption ablations using the experimental setup in Figure 1 We plot worst case group accuracy (left column) and explanation fragility (right column) for DecoyMNIST using R (top row) and IBP-Ex (bottom row).
G Additional Experimental Details
Table 3: Robust to Perturbation of Spurious Features
Learning Objective
ERM
R3
Smooth- R
IBP-Ex
IBP-Ex Ex + R²
Rand-I -
Adv-R*
Cert-R
Dataset
Avg
Avg 5
Avg 8
Avg
Avg
Avg
Avg 6
Avg
Decoy MNIST
497.44
4.35
4.33
5.61
3.43
2.10
2.02
0.93
Decoy DERM
6742.41
5762.81
3906.13
2979.48
2892.32
3790.20
3504.75
1.297
ISIC
0.423
0.00167
1.029
1.784
2.878
1.392
0.400
0.901
Plant.
4011.40
4519.85
2082.06
4742.33
1105.28
1285.71
1787.99
89.38
22

---

Published in Transactions on Machine Learning Research (09/2025)
Table Robustness to Perturbation Core Features
Learning Objective
ERM
R²
Smooth-R"
IBP-Ex
IBP-Ex R²
Rand-R"
Adv-R*
Cert-R
Dataset +
Avg 8
Avg
Avg
Avg 6
Avg 6
Avg 8
Avg
Avg d 39.4
DecoyMNIS.
332.3
7.7
7.6
8.7
6.4
6.2
6.3
DeCOyDERM
7918.7
5811.9
3925.1
3714.5
3684.6
3800.2
3558.9
453.7
ISIC
0.45
0.002
1.17
1.92
3.06
1.6
0.45
1.07
Plant
4410.9
4657.3
2034.7
4881.83
1223.02
1307.2 2
2439.8
97.8
H
Model Outputs Upper Bound for Adversarially Perturbed Spurious Regions Proof
We firstly restate our claim and then provide proof.
Proposilion 1. Given a function f induced by a model parametrized by 0, trained on datasel D = {(xl1) y(o), m with loss function R* as in equation adversarial perturbations of magnitude at most (i.e, with x' E (x)) and post-hoc computed d-input robustness magnitude at most 8 then the output difference norm between any perturbed input and its standard counterpart is upper bounded by:
(x) - (x')) |= =lel)))
Proof. We begin by writing the first-order Taylor approximation, with second-order remainder term at the spurious (masked) region of input x', i.e. (x'). In that case, the function ( is eraclly equal to:
(x ff (x') V2S
(x)) (a')
(a x'))
R2(5((()
function change
second-order remsinder
By rearranging and applying the norm on both sides, we have:
(x) - (z = |V ( (m (x- x')) + (2(f2(2))))
|
We note that, as mentioned in $4.1 by optimizing R objective, we can bound the gradient magnitude for any point in the e-ball of the masked region: Vx* E BTT [[V),/[V>/ (x*)|l < 8t. Using the property given by our tochnique, as well as the triangle incquality, we have that:
10))) S 8 1 (x 2')l RRa ( (1 (x') (x'))]
Firstly, we remind the reader of Taylor's Theorem with Lagrange Remainder. We write the n-th order Taylor's approximation for function at point x' E [x, + h] and assume that f is n + differentiable in this interval. The theorem tells us that the norm of the remainder term of order + is upper bounded by:
M
(n + 1)! 1All, where M supee(ee00 ( f(x))
(V
In our case, we have by definition that x' E (x)), which implics | -x==x' x'1 S I 3 cl < ||]]], since m E [0, 1]. We also note that ||^]| = m (x-xx')) x')l I| in the above equation. Lastly, as mentioned in $4.1 because the norm of the gradient in the masked e-ball is bounded by 8*, then trivially the norm of the Hessian in the region is also bounded by 5" and corresponds with the notion of M defined above, for first-order Tppooi-appooi with second-order remainder. Lastly, since the matrix norm is submultiplicative, we obtain the following strict upper bound:
[ - (0(')) s (1]lel( + Illl))
23

---

Published in Transactions on Machine Learning Research (09/2025)
Why is R$ better than IBP-Ex?
The following result demonstrates the merit of R$ over IBP-Ex.
Proposition 2. Given a dataset = (0(0)y())) we wish a linear regressor that predicts the output well without using the pre-spocified i irrelevant features (of the total features), which we assume are the last k dimensions of æ without loss of generality. We denote by the weights fitted using IBP-Ex and R$ by wy and we respectively. The following result holds for the norm of irrelevant features.
W O ml S VKT//lell
[WR O mmll ST
where 0, 0,
and
max BED,E'EB'C
[ x' w
We require that the norm of the irrelevant features to be as small as possible. However, the bound on the norm of irrelevant features given by IBP-Ex is much weaker than that of R². Because the upper bound given by IBP-Ex gets weaker with the number of irrelevant features and further exacerbated by the practically small value of On the other hand, R$ fitted wa bounds the norm of irrelevant features well even when their dimension high. Intuitively, the error in suppressing the function deviation of the inner maximization loop blows up with the number of dimensions in IBP-Ex. On the other hand, since R suppresses the norm directly, the support of irrelevant features is kept. in check.
Proof. We will generalize the definition of T to work for both IBP-Ex or R by generalizing the function w x to fzx) The more general definition of r is as follows.
If(x') - f(x)]
max wED,x' (B))))
We first prove the result for wy, which bounds the function value deviation in the e-neighborhood.
Since we are fitting a linear regressor, the coefficients of the 7th irrelevant feature
w; is (f(x [0, (0.-...0, 0, ||||| o,.... 0)) f(=) S </||c||- Therefore, the norm of the irrelevant features
is bounded by /kT//le|l-
The result for wo follows directly from observing that R$ minimizes the gradient norm of irrelevant features, which is wn @ m, and from the generalized definition of r.
J Interval Arithmetic for Bounding Input Gradients
We begin by recalling the form of the forward and backwards pass sated in the main text where we have a neural network model f I R nout with K layers and parameters ((W(), b()) K
as:
2(k) w((4 (x-)) + (k))
(A) (2(*)
where 200) = x, (x) = 2(K), and a is the activation function, which we assume is monotonic. We have the backwards pass starting with 8(2) g (xx), we have that backwards pass is given by:
1(k-1)
(wlt)) g(k) oo (2(k-1))
Where we are interested in ((0) V (
As highlighted in the paper the above forwards and backwards bounds consist solely of matrix multiplication, addition, and the application of non-linearity.
24

---

Published in Transactions on Machine Learning Research (09/2025)
Where we start with imput interval " must define interval versions of the above operations such that they maintain that the output interval of our interval operations provably contains all possible outputs of their non-interval counterparts. We will represent interval matrices with bold symbols ie., A = [A,,Au Ay ] C R We denote interval vectors as a = (az, av] with analogous operations.
Definition J.1 (Interval Matrix Arithmetic). Let A = [Az. Au] and B = BL. Bu] be intervals over matrices. Let @B, & @ represent interval matrix addition, matrix multiplication and element-wise multiplication, such that
A+ B E A B A B E
AΛ A, B B,
AΛ A, B B, VÁ A, B.B.
Ao O BJ
Both the addition (defined element-wise) and element-wise multiplication of these bounds can be accomplished by simply taking all 4 combinations of the interval end points and returning the maximum and minimum. The matrix multiplication is slightly more complex but be bounded using Rump's algorithm Rump 1999).
These operations are standard interval arithmetic techniques and are computed in at most 4x the cost of standard forward and backward pass.
For the non-linearity, we have assumed the function is monotonic which is defined by: x < y = o()) < ( Thus the element-wiss, application of to an interval over vectors [V v"] we simply have that the interval output (a(v"),
Taken together, these suffice to bound the output of forward and backwards pass. For more rigorous treatment we reference readers to (Wicker et al.).
25


Reviews:
[{"review_id": "MUT0VmZ8TB", "reviewer": "Reviewer_ZDng", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper provides a meaningful contribution to the robustness of MLX approaches, offering a framework that is both mathematically grounded and supported by empirical evidence. The authors conduct extensive experiments demonstrating the robustness of the $R^4$ framework in comparison to previous methods. They also provide thoughtful discussions and explanations that further emphasize the significance of their approach. However, the paper could benefit from a more detailed discussion of the computational requirements of the proposed variants, especially in comparison to prior work. Additionally, it would be interesting to explore how different types of mask corruption affect the performance of earlier methods, which would offer a clearer picture of how Cert-$R^4$ compares to the existing literature under varied conditions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces $R^4$, a Machine Learning from Explanations (MLX) framework that promotes robustness in feature attribution by minimizing the model's reliance on irrelevant (non-core) features. Unlike previous methods that primarily focus on robustness to small perturbations in non-core inputs, $R^4$ instead targets robustness of feature importance scores, which helps ensure that explanations remain stable even when the non-core components of an input vary. $R^4$ achieves this by attempting to minimize the impact of the perturbation which would maximally increases attribution to non-core features, which is intractable.  Therefore, the authors propose three approximation strategies that differ in complexity and performance. Rand-$R^4$ approximates the worst case by randomly sampling perturbations and selecting the one with the greatest shift. Adv-$R^4$ uses a first-order gradient-based optimization procedure, similar to adversarial attacks, to find a perturbation that maximizes sensitivity. Cert-$R^4$ instead computes an upper bound on the worst-case attribution using interval bound propagation, allowing for worst-case guarantees. Experimental results across several datasets demonstrate that $R^4$, particularly the Cert-$R^4$ variant, achieves greater robustness in attribution while maintaining strong predictive performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses part above.\n\nIn addition, there are several points where changes are needed:\n\n1. In the left column of Figure 1, it appears that screenshots of plots were used, as indicated by the visible top and left image borders. Cleaning up these borders would improve the visual quality and professionalism of the figure.\n\n2. In the right column of Figure 1, the authors show the performance of their method under corrupted masks. It would strengthen the paper to include how previous methods perform under the same corrupted mask conditions.\n\n3. The limitations section mentions that if multiple shortcuts are present in the data, $R^4$ is believed to handle them effectively and efficiently. However, this claim would be much stronger with supporting empirical evidence. If possible, the authors should include experiments or examples that directly demonstrate this capability.", "broader_impact_concerns": "No concerns on the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UeffujRRax", "reviewer": "Reviewer_Br7p", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "***Strengths:***\n1. This paper provides theoretical proof on the derivation of the training objectives, which makes the paper more solid.\n2. It is a good practice to follow the evaluation protocol in prior study.\n\n\n***Weakness:***\n1. Some claims or arguments in the paper should be supported by appropriate references. For instance, \"The best approach to mitigate ... \" in Introduction should reference some papers. \"Unfortunately, even for ...\" also needs reference in Section 3. Also, the notations in Section 4.4.\n\n2. This paper include a number of equations, which also introduce some notations. However, many notations are not defined which makes it hard to follow. For example, $H(f^\\theta)$ means Hessian matrix? $sng$ is the signum function? What does $\\alpha$ indicate in Section 4.3? If it is a hyperparameter, how is it selected or computed? There are also a few more notations undefined. \n\n3. The proposal of the method is motivated by the observation in Section 4. Given its importance of such a finding, it would be beneficial to introduce some appropriate reference or conduct some experiments to illustrate it.\n\n4. In the experiments, authors only compare the IBP-Ex  $+ R^3$. However, the best performant methods in [a] are IBP-Ex+Grad-Reg and PGD-Ex+Grad-Reg. Please also compare these two methods.\n\n   [a] Use perturbations when learning from explanations.\n\n5. Compared methods are outdated, where the latest method is published in 2023. It may be helpful to include [b].\n\n   [b] Targeted Activation Penalties Help CNNs Ignore Spurious Signals.\n\n6. On Decoy IMDB and Salient Imagenet, why are the performance of some methods missing?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a simplified, theoretically grounded training objective to mitigate shortcut learning—where models rely on irrelevant features—without depending on unreliable salience-based explanations. The method jointly promotes explanation robustness and discourages shortcut use. Experiments across multiple tasks show a reduction in test-time misclassifications over compared methods. The authors also highlight that annotation quality has a greater impact than quantity.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness section to (1) include proper references (2) define all notations in introduced equations (3) expand the comparison methods in the experiments (4) add reference or experiment for the motivation of proposing the method (5) complete the results in table 1.", "broader_impact_concerns": "This paper aims to improve the robustness of machine learning algorithm, which is beneficial to the community. Thus, no concerns on the ethical implication of this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MpogsPlZPD", "reviewer": "Reviewer_NY1T", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThis is a technically solid paper, and in my opinion merits acceptance (given the requested changes below).  In particular, the authors offer a well-motivated approach to a problem of high interest in the community.  Technical material is clearly presented, and experimental results are generally strong.  Code and other necessary information to reproduce and build on the experimental results are include. \n\n\n## Weaknesses\n\nI think the primary weakness of this paper is that the experimental results focus too much on showing the superiority of R4 to other methods.  This leaves a missed opportunity to analyze and discuss where and when R4 does not perform as strongly, which is important information to share with researchers aiming to build on your work.  \n\nFor example, it would be interesting to have the authors discuss why Cert-R4 performs poorly on Plant in Table 2.  The current results discussion includes discussion of the other 3 datasets in Table 2, but does not mention this result.  Also, why is Cert-R4 not applied to Decoy IMDB and Salient Imagenet in Table 1?  Are there some situations where each of the 3 proposed variants are more or less appropriate, or possible cannot be applied?\n\nAlso in the same vein, I think the main paper should include some results and discussion regarding the ablations of model architectures, which are currently in the appendix, and not referenced in any detail in the main paper.  Appendix Figure 3 makes it seem like model architecture plays a significant role in determining the relative success of the various methods, so (at least some of) those results should be presented and discussed in the main paper.  \n\nMy second high-level critique of the paper is that the writing would benefit from further careful editing and polish.  For example:\n- In the introduction, “Learning such shortcuts poses a great challenge with deployment in safety-critical domains” is unclear (i.e., could be read that it’s hard to learn the shortcuts) and somewhat awkwardly phrased (“poses a great challenge with deployment”)\n- In related works, “Mitigating shortcut learning has been approached from multiple fronts, the three major themes are…”\n- In preliminaries, first sentence, the definition of a machine learning model actually seems to be the definition of a classification model.\n- Section 4.1, the following needs a comma: “We begin by describing the theoretical intuition of our learning objective describing its benefits relative to prior works.”\n\nAlso, I found myself confused at the beginning of the paper as to exactly what types of “features” were being considered, before I finally realized that the paper focuses on gradient-based saliency maps.  For example, the term “masked” is introduced before it is specified that we are interested in gradient-based saliency maps, and thus it is unclear what it means at first and I had to re-read those paragraphs to fully understand.  The introduction should be edited for logical flow and clarity.\n\nFinally, I would suggest that the paper could benefit from some sort of illustrative figure 1 orienting the reader towards the problem at hand (e.g., a 3 panel figure showing 1 typical saliency map, 1 illustrating brittleness to small perturbations, and 1 showing that r4 is robust to the same perturbation).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Modern neural network classifiers are highly accurate, especially when testing on data drawn from the same distribution as the training data.  However, these models are known to learn shortcuts in the data, such that the patterns that they learn to associate with each class label are not robust to distribution shifts or minority groups in the data.  To combat this, a variety of approaches to machine learning from explanations (MLX) have been proposed, where ground-truth feature annotations are used to guide the model towards learning more robust and interpretable solutions.  In particular, model explanations are used to depress the amount of attention paid to spurious features, so that the model can be “right for the right reasons”.  One shortcoming of this approach to MLX is that the salience maps produced by inspecting input gradients are known to be brittle, where small perturbations to the input cause large changes to the map.  \n\nThe contribution of this work is to address this challenge, proposing R4, an extension of R3 with a focus on robustifying the gradient-based attributions.  As their initial formulation is intractable, the authors propose 3 methods for computing and optimizing their objective.  Results are offered across a range of tasks and in comparison to a range of baselines, showing R4 to be a strong choice of optimizer for MLX with feature salience.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In the experiments section:\n- Explain why Cert-R4 is not applied to Decoy IMDB and Salient Imagenet in Table 1.\n- Explain why Cert-R4 performs (relatively) poorly on Plant in Table 2.\n- Move some architecture ablation results and discussion to main paper.\n\nWriting:\n- End-to-end proofread and polish\n- Edit introduction for clarity \n\n(Suggested) Illustrative Figure 1", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)