Title: Use perturbations when learning from explanations
Venue: NeurIPS 2023
Accepted: yes
Keywords: Learning from explanation

Robustness

Interpretability

Shortcuts

Explanations
Abstract: Machine learning from explanations (MLX) is an approach to learning that uses human-provided explanations of relevant or irrelevant features for each input to ensure that model predictions are right for the right reasons. Existing MLX approaches rely on local model interpretation methods and require strong model smoothing to align model and human explanations, leading to sub-optimal performance. We recast MLX as a robustness problem, where human explanations specify a lower dimensional manifold from which perturbations can be drawn, and show both theoretically and empirically how this approach alleviates the need for strong model smoothing. We consider various approaches to achieving robustness, leading to improved performance over prior MLX methods. Finally, we show how to combine robustness with an earlier MLX method, yielding state-of-the-art results on both synthetic and real-world benchmarks.

Use Perturbations when Learning from Explanations

Abstract

Machine learning from explanations (MLX) is an approach to learning that uses human-provided explanations of relevant or irrelevant features for each input to ensure that model predictions are right for the right reasons . Existing MLX approaches rely on local model interpretation methods and require strong model smoothing to align model and human explanations, leading to sub-optimal performance. We recast MLX as a robustness problem, where human explanations specify a lower dimensional manifold from which perturbations can be drawn, and show both theoretically and empirically how this approach alleviates the need for strong model smoothing. We consider various approaches to achieving robustness, leading to improved performance over prior MLX methods. Finally, we show how to combine robustness with an earlier MLX method, yielding state-of-the-art results on both synthetic and real-world benchmarks. Our implementation can be found at: https://github.com/vihari/robust_mlx .

1 Introduction

Deep neural networks (DNNs) display impressive capabilities, making them strong candidates for real-wold deployment. However, numerous challenges hinder their adoption in practice. Several major deployment challenges have been linked to the fact that labelled data often under-specifies the task (D'Amour et al., 2020). For example, systems trained on chest x-rays were shown to generalise poorly because they exploited dataset-specific incidental correlations such as hospital tags for diagnosing pneumonia (Zech et al., 2018; DeGrave et al., 2021). This phenomenon of learning unintended feature-label relationships is referred to as shortcut learning (Geirhos et al., 2020) and is a critical challenge to solve for trustworthy deployment of machine learning algorithms. A common remedy to avoid shortcut learning is to train on diverse data (Shah et al., 2022) from multiple domains, demographics, etc, thus minimizing the underspecification problem, but this may be impractical for many applications such as in healthcare.

Enriching supervision through human-provided explanations of relevant and irrelevant regions/features per example is an appealing direction toward reducing under-specification. For instance, a (human-provided) explanation for chest x-ray classification may highlight scanning artifacts such as hospital tag as irrelevant features. Learning from such human-provided explanations (MLX) has been shown to avoid known shortcuts (Schramowski et al., 2020). Ross et al. (2017) pioneered an MLX approach based on regularizing DNNs, which was followed by several others (Schramowski et al., 2020; Rieger et al., 2020; Stammer et al., 2021; Shao et al., 2021). Broadly, existing approaches employ a model interpretation method to obtain per-example feature saliency, and

regularize such that model and human-provided explanations align. Since saliency is unbounded for relevant features, many approaches simply regularize the salience of irrelevant features. In the same spirit, we focus on handling a specification of irrelevant features, which we refer to as an explanation hereafter.

Existing techniques for MLX employ a local interpretation tool and augment the loss with a term regularizing the importance of irrelevant region. Regularization-based approaches suffer from a critical concern stemming from their dependence on a local interpretation method. Regularization of local, i.e. example-specific, explanations may not have the desired effect of reducing shortcuts globally, i.e. over the entire input domain (see Figure 1). As we demonstrate both analytically and empirically that previous MLX proposals, which are all regularization-based, require strong model smoothing in order to be globally effective at reducing shortcut learning.

In this work, we explore MLX using various robust training methods with the objective of training models that are robust to perturbations of irrelevant features. We start by framing the provided human explanations as specifications of a local, lower-dimensional manifold from which perturbations are drawn. We then notice that a model whose prediction is invariant to perturbations drawn from the manifold ought also to be robust to irrelevant features. Our perspective yields considerable advantages. Posing MLX as a robustness task enables us to leverage the considerable body of prior work in robustness. Further, we show in Section 4.1 that robust training can provably upper bound the deviation on model value when irrelevant features are perturbed without needing to impose model smoothing. However, when the space of irrelevant features is high-dimensional, robust-training may not fully suppress irrelevant features as explained in Section 4.2. Accordingly, we explore combining both robustness-based and regularization-based methods, which achieves the best results. We highlight the following contributions:

We theoretically and empirically demonstrate that existing MLX methods require strong model smoothing owing to their dependence on local model interpretation tools.

We study learning from explanations using robust training methods. To the best of our knowledge, we are the first to analytically and empirically evaluate robust training methods for MLX.

We distill our insights into our final proposal of combining robustness and regularization-based methods, which consistently performed well and reduced the error rate over the previous best by 20-90%.

2 Problem Definition and Background

We assume access to a training dataset with N training examples, D T = { ( x ( i ) , y ( i ) ) } N i =1 , with x ( i ) ∈ R d and y ( i ) label. In the MLX setting, a human expert also specifies input mask m ( n ) for an example x ( n ) where non-zero values of the mask identify irrelevant features of the input x ( n ) . An input mask is usually designed to negate a known shortcut feature that a classifier is exploiting. Figure 2 shows some examples of masks for the datasets that we used for evaluation. For example, a mask in the ISIC dataset highlights a patch that was found to confound with non-cancerous images. With the added human specification, the augmented dataset contains triplets of example, label and mask, D T = { ( x ( i ) , y ( i ) , m ( i ) ) } N i =0 . The task therefore is to learn a model f ( x ; θ ) that fits observations well while not exploiting any features that are identified by the mask m .

The method of Ross et al. (2017) which we call Grad-Reg (short for Gradient-Regularization), and also other similar approaches (Shao et al., 2021; Schramowski et al., 2020) employ an local interpretation method (E) to assign importance scores to input features: IS ( x ) , which is then regularized with an R ( θ ) term such that irrelevant features are not regarded as important. Their training loss takes the form shown in Equation 1 for an appropriately defined task-specific loss ℓ .

We use ⊙ to denote element-wise product throughout. CDEP (Rieger et al., 2020) is slightly different. They instead use an explanation method that also takes the mask as an argument to estimate the contribution of features identified by the mask, which they minimize similarly.

About obtaining human specification mask. Getting manually specifying explanation masks can be impractical. However, the procedure can be automated if the nuisance/irrelevant feature occurs systematically or if it is easy to recognize, which may then be obtained automatically using a procedure similar to Liu et al. (2021); Rieger et al. (2020). A recent effort called Salient-Imagenet used neuron activation maps to scale curation of such human-specified masks to Imagenet-scale (Singla & Feizi, 2021; Singla et al., 2022). These efforts may be seen as a proof-of-concept for obtaining richer annotations beyond content labels, and towards better defined tasks.

3 Method

Our methodology is based on the observation that an ideal model must be robust to perturbations to the irrelevant features. Following this observation, we reinterpret the human-provided mask as a specification of a lower-dimensional manifold from which perturbations are drawn and optimize the following objective.

The above formulation uses a weighting α to trade off between the standard task loss and perturbation loss and κ > 0 is a hyperparameter that controls the strength of robustness. We can leverage the many advances in robustness in order to approximately solve the inner maximization. We present them below.

Avg-Ex : We can approximate the inner-max with the empirical average of loss averaged over K samples drawn from the neighbourhood of training inputs. Singla et al. (2022) adopted this straightforward baseline for supervising using human-provided saliency maps on the Imagenet dataset. Similar to κ , we use σ to control the noise in perturbations as shown below.

PGD-Ex : Optimizing for an estimate of worst perturbation through projected gradient descent (PGD) (Madry et al., 2017) is a popular approach from adversarial robustness. We refer to the approach of using PGD to approximate the second term of our loss as PGD-Ex and denote by ϵ ∗ ( x ( n ) , θ, m ( n ) ) the perturbation found by PGD at x ( n ) . Given the non-convexity of this problem, however, no guarantees can be made about the quality of the approximate solution x ∗ .

IBP-Ex : Certified robustness approaches, on the other hand, minimize a certifiable upper-bound of the second term. A class of certifiable approaches known as interval bound propagation methods (IBP) (Mirman et al., 2018; Gowal et al., 2018) propagate input intervals to function value intervals that are guaranteed to contain true function values for any input in the input interval.

We define an input interval for x ( n ) as [ x ( n ) -κ m ( n ) , x ( n ) + κ m ( n ) ] where κ is defined in Eqn. 2. We then use bound propagation techniques to obtain function value intervals for the corresponding input interval: l ( n ) , u ( n ) , which are ranges over class logits. Since we wish to train a model that correctly classifies an example irrespective of the value of the irrelevant features, we wish to maximize the minimum probability assigned to the correct class, which is obtained by combining minimum logit for the correct class with maximum logit for incorrect class: ˜ f ( x ( n ) , y ( n ) , l ( n ) , u ( n ) ; θ ) ≜ l ( n ) ⊙ ¯ y ( n ) + u ( n ) ⊙ ( 1 -¯ y ( n ) ) where ¯ y ( n ) ∈ { 0 , 1 } c denotes the one-hot transformation of the label y ( n ) into a c-length vector for c classes. We refer to this version of the loss as IBP-Ex, summarized

below.

Despite its computational efficiency, IBP is known to suffer from scaling issues when the model is too big. Consequently, it is better to use IBP-Ex only when the model is small (less than four layers of CNN or feed-forward) and if computational efficiency is desired. On the other hand, we do not anticipate any scaling issues when using PGD-Ex.

Combined robustness and regularization : PGD-Ex+Grad-Reg, IBP-Ex+Grad-Reg. We combine robustness and regularization by simply combining their respective loss terms. We show the objective for IBP-Ex+Grad-Reg below, PGD-Ex+Grad-Reg follows similarly.

λ R ( θ ) and α, ˜ f are as defined in Eqn. 5 and Eqn. 3 respectively. In Section 4, 5.2, we demonstrate the complementary strengths of robustness and regularization-based methods.

4 Theoretical Motivation

3

2

1

0

1

2

3

4

20

15

10

5

0

5

10

15

20

(a) Toy data

(b) Grad-Reg with

β

=0

(c) Grad-Reg with

β

=1

(d) IBP-Ex with

β

=0

20

20

20

10

10

10

0

D

0

10

-10

-10

20

20

-20

2

0

2

Z-

1

2

In this section, we motivate the merits and drawbacks of robustness-based over regularization-based methods. Through non-parametric analysis in Theorems 1, 2, we argue that (a) regularization methods are robust to perturbations of irrelevant features (identified by the mask) only when the underlying model is sufficiently smoothed, thereby potentially compromising performance, (b) robust training upper-bounds deviation in function values when irrelevant features are perturbed, which can be further suppressed by using a more effective robust training. Although our analysis is restricted to nonparametric models for the ease of analysis, we empirically verify our claims with parametric neural network optimized using a gradient-based optimizer. We then highlight a limitation of robustness-based methods when the number of irrelevant features is large through Proposition 1.

4.1 Merits of Robustness-based methods

Consider a two-dimensional regression task, i.e. x ( n ) ∈ X and y ∈ R . Assume that the second feature is the shortcut that the model should not use for prediction, and denote by x ( n ) j the j th dimension of n th point. We infer a regression function f from a Gaussian process prior f ∼ GP ( f ; 0 , K ) with a squared exponential kernel where k ( x, ˜ x ) = exp( -∑ i 1 2 ( x i -˜ x i ) 2 θ 2 i ) . As a result, we have two hyperparameters θ 1 , θ 2 , which are length scale parameters for the first and second dimensions respectively. Further, we impose a Gamma prior over the length scale: θ -2 i ∼ G ( α, β ) .

Theorem 1 (Grad-Reg) . We infer a regression function f from a GP prior as described above with the additional supervision of [ ∂f ( x ) /∂x 2 ] | x ( i ) = 0 , ∀ i ∈ [1 , N ] . Then the function value deviations to perturbations on irrelevant feature are lower bounded by a value proportional to the perturbation strength δ as shown below.

Full proof of Theorem 1 is in Appendix A, we provide the proof outline below.

Proof sketch. We exploit the flexibility of GPs to accommodate observations on transformed function values if the transformation is closed under linear operation (Hennig et al., 2022) (see Chapter 4). Since gradient is a linear operation (contributors, 2022), we simply revise the observations y to include the N partial derivative observations for each training point with appropriately defined kernel. We then express the inferred function, which is the posterior f ( x | D T ) in closed form.

Wethen derive a bound on function value deviations to perturbation on the second feature and simplify using simple arithmetic and Bernoulli's inequality to derive the final expression.

We observe from Theorem 1 that if we wish to infer a function that is robust to irrelevant feature perturbations, we need to set α β to a very small value. Since the expectation of gamma distributed inverse-square length parameter is E [ θ -2 ] = α β , which we wish to set very small, we are, in effect, sampling functions with very large length scale parameter i.e. strongly smooth functions. This result brings us to the intuitive takeaway that regularization using Grad-Reg applies globally only when the underlying family of functions is sufficiently smooth. The result may also hold for any other local-interpretation methods that is closed under linear operation. One could also argue that we can simply use different priors for different dimensions, which would resolve the over-smoothing issue. However, we do not have access to parameters specific to each dimension in practice and especially with DNNs, therefore only overall smoothness may be imposed such as with parameter norm regularization in Eqn. 1.

We now look at properties of a function fitted using robustness methods and argue that they bound deviations in function values better. In order to express the bounds, we introduce a numerical quantity called coverage (C) to measure the effectiveness of a robust training method. We first define a notion of inputs covered by a robust training method as ˆ X ≜ { x | x ∈ X , ℓ ( f ( x ; θ ) , y ) < ϕ } ⊂ X for a small positive threshold ϕ on loss. We define coverage as the maximum distance along second coordinate between any point in X and its closest point in ˆ X , i.e. C ≜ max x ∈X min x ∈ ˆ X | x 2 -ˆ x 2 | . We observe that C is small if the robust training is effective. In the extreme case when training minimizes the loss for all points in the input domain, i.e. ˆ X = X , then C=0.

Theorem 2. When we use a robustness algorithm to regularize the network, the fitted function has the following property.

δ max and f max are maximum values of ∆ x 2 and f ( x ) in the input domain ( X ) respectively.

Proof sketch. We begin by estimating the function deviation between an arbitrary point and a training instance that only differ on x 2 , which is bounded by the product of maximum slope ( max x ∂f ( x ) /∂x 2 ) and ∆ x 2 . We then estimate the maximum slope (lipschitz constant) for a function inferred from GP. Finally, function deviation between two arbitrary points is twice the maximum deviation between an arbitrary and a training instance that is estimated in the first step.

Full proof is in Appendix B. The statement shows that deviations in function values are upper bounded by a factor proportional to C , which can be dampened by employing an effective robust training method. We can therefore control the deviations in function values without needing to regress α β (i.e. without over-smoothing).

Further remarks on sources of over-smoothing in regularization-based methods. We empirically observed that the term R ( θ ) (of Eqn. 1), which supervises explanations, also has a smoothing effect on the model when the importance scores (IS) are not well normalized, which is often the case. This is because reducing IS( x ) everywhere will also reduce saliency of irrelevant features.

Empirical verification with a toy dataset. For empirical verification of our results, we fit a 3-layer feed-forward network on a two-dimensional data shown in Figure 1 (a), where color indicates the label. We consider fitting a model that is robust to changes in the second feature shown on y-axis. In Figures 1 (b), (c), we show the Grad-Reg fitted classifier using gradient ( ∂f/∂x 2 for our case) regularization for two different strengths of parameter smoothing (0 and 1 respectively). With weak smoothing, we observe that the fitted classifier is locally vertical (zero gradient along y-axis), but curved overall (Figure 1 (b)), which is fixed with strong smoothing (Figure 1 (c)). On the other hand, IBP-Ex fitted classifier is nearly vertical without any parameter regularization as shown in (d). This example illustrates the need for strong model smoothing when using a regularization-based method.

4.2 Drawbacks of Robustness-based methods

Although robust training is appealing in low dimensions, their merits do not transfer well when the space of irrelevant features is high-dimensional owing to difficulty in solving the inner maximization of Eqn. 2. Sub-par estimation of the maximization term may learn parameters that still depend on the irrelevant features. We demonstrate this below with a simple exercise.

Proposition 1. Consider a regression task with D + 1 -dimensional inputs x where the first D dimensions are irrelevant, and assume they are x d = y, d ∈ [1 , D ] while x D +1 ∼ N ( y, 1 /K ) . The MAP estimate of linear regression parameters f ( x ) = ∑ D +1 d =1 w d x d when fitted using Avg-Ex are as follows: w d = 1 / ( D + K ) , d ∈ [1 , D ] and w D +1 = K/ ( K + D ) .

We present the proof in Appendix C. We observe that as D increases, the weight of the only relevant feature ( x D +1 ) diminishes. On the other hand, the weight of the average feature: 1 D ∑ D d =1 x d , which is D/ ( D + K ) approaches 1 as D increases. This simple exercise demonstrates curse of dimensionality for robustness-based methods. For this reason, we saw major empirical gains when combining robustness methods with a regularization method especially when the number of irrelevant features is large such as in the case of Decoy-MNIST dataset, which is described in the next section.

5 Experiments

We evaluate different methods on four datasets: one synthetic and three real-world. The synthetic dataset is similar to decoy-MNIST of Ross et al. (2017) with induced shortcuts and is presented in Section 5.2. For evaluation on practical tasks, we evaluated on a plant phenotyping (Shao et al., 2021) task in Section 5.3, skin cancer detection (Rieger et al., 2020) task presented in Section 5.4, and object classification task presented in Section 5.5. All the datasets contain a known spurious feature, and were used in the past for evaluation of MLX methods. Figure 2 summarises the three datasets, notice that we additionally require in the training dataset the specification of a mask identifying irrelevant features of the input; the patch for ISIC dataset, background for plant dataset, decoy half for Decoy-MNIST images, and label-specific irrelevant region approved by humans for Salient-Imagenet.

5.1 Setup

5.1.1 Baselines

We denote by ERM the simple minimization of cross-entropy loss (using only the first loss term of Equation 1). We also compare with G-DRO(Sagawa et al., 2019), which also has the objective of avoiding to learn known irrelevant features but is supervised through group label (see Section 6). Although the comparison is unfair toward G-DRO because MLX methods use richer supervision of per-example masks, it serves as a baseline that can be slightly better than ERM in some cases.

Class 0

Class 1

Mask

Multi-class

ISIC

Plant

phenotyping

decoy

MNIST

Salient

Imagenet

T

Regulaization-based methods. Grad-Reg and CDEP, which were discussed in Section 2. We omit comparison with Shao et al. (2021) because their code is not publicly available and is non-trivial to implement the influence-function based regularization.

Robustness-based methods. Avg-Ex, PGD-Ex, IBP-Ex along with combined robustness and regularization methods . IBP-Ex+Grad-Reg, PGD-Ex+Grad-Reg that are described in Section 3.

5.1.2 Metrics

Since two of our datasets are highly skewed (Plant and ISIC), we report (macro-averaged over labels) accuracy denoted 'Avg Acc' and worst accuracy 'Wg Acc' over groups of examples with groups appropriately defined as described below. On Salient-Imagenet, we report using average accuracy and Relative Core Sentivity (RCS) following the standard practice.

Wg Acc. Worst accuracy among groups where groups are appropriately defined. Different labels define the groups for decoy-MNIST and plant dataset, which therefore have ten and two groups respectively. In ISIC dataset, different groups are defined by the cross-product of label and presence or absence of the patch. We denote this metric as 'Wg Acc', which is a standard metric when evaluating on datasets with shortcut features (Sagawa et al., 2019).

RCS proposed in Singla et al. (2022) was used to evaluate models on Salient-Imagenet. The metric measures the sensitivity of a model to noise in spurious and core regions. High RCS implies low dependence on irrelevant regions, and therefore desired. Please see Appendix E.2 for more details.

5.1.3 Training and Implementation details

Choice of the best model. Wepicked the best model using the held-out validation data. We then report the performance on test data averaged over three seeds corresponding to the best hyperparameter.

Network details. We use four-layer CNN followed by three-fully connected layers for binary classification on ISIC and plant dataset following the setting in Zhang et al. (2019), and three-fully connected layers for multi classification on decoy-MNIST dataset. Pretrained Resnet-18 is used as the backbone model for Salient-Imagenet.

More details about network architecture, datasets, data splits, computing specs, and hyperparameters can be found in Appendix E.

5.2 Decoy-MNIST

Decoy-MNIST dataset is inspired from MNIST-CIFAR dataset of Shah et al. (2020) where a very simple label-revealing color based feature (decoy) is juxtaposed with a more complex feature (MNIST image) as shown in Figure 1. We also randomly swap the position of decoy and MNIST parts, which makes ignoring the decoy part more challenging. We then validate and test on images where decoy part is set to correspond with random other label. Note that our setting is different from decoy-mnist of CDEP (Rieger et al., 2020), which is likely why we observed discrepancy in results reported in our paper and CDEP (Rieger et al., 2020). We further elaborate on these differences in Appendix G.

We make the following observations from Decoy-MNIST results presented in Table 1. ERM is only slightly better than a random classifier confirming the simplicity bias observed in the past (Shah et al., 2020). Grad-Reg, PGD-Ex and IBP-Ex perform comparably and better than ERM, but when combined (IBP-Ex+Grad-Reg,PGD-Ex+Grad-Reg) they far exceed their individual performances.

In order to understand the surprising gains when combining regularization and robustness methods, we draw insights from gradient explanations on images from train split for Grad-Reg and IBP-Ex. We looked at s 1 = M [∥ ∥ m ( n ) × ∂f ( x ( n ) ) ∂ x ( n ) ∥ ∥ ] and s 2 =

∥ ∥ M [∥ ∥ ∥ m ( n ) × ∂f ( x ( n ) ) ∂ x ( n ) ∥ ∥ ∥ /∥ ∥ ∥ ( 1 -m ( n ) ) × ∂f ( x ( n ) ) ∂ x ( n ) ∥ ∥ ∥ ] , where M [ · ] is median over all the examples and ∥ · ∥ is ℓ 2 -norm. For an effective algorithm, we expect both s 1 , s 2 to be close to zero. However, the values of s 1 , s 2 is 2.3e-3, 0.26 for the best model fitted using Grad-Reg and 6.7, 0.05 for IBP-Ex. We observe that Grad-Reg has lower s 1 while IBP-Ex has lower s 2 , which shows that Grad-Reg is good at dampening the contribution of decoy part but also dampened contribution of non-decoy likely due to over-smoothing. IBP-Ex improves the contribution of the non-decoy part but did not fully dampen the decoy part likely because high dimensional space of irrelevant features, i.e. half the image is irrelevant and every pixel of irrelevant region is indicative of the label. When combined, IBP-Ex+Grad-Reg has low s 1 , s 2 , which explains the increased performance when they are combined.

5.3 Plant Phenotyping

Plant phenotyping is a real-world task of classifying images of a plant leaf as healthy or unhealthy. About half of leaf images are infected with a Cercospora Leaf Spot (CLS), which are the black spots on leaves as shown in the first image in the second row of Figure 2. Schramowski et al. (2020) discovered that standard models exploited unrelated features from the nutritional solution in the background in which the leaf is placed, thereby performing poorly when evaluated outside of the laboratory setting. Thus, we aim to regulate the model not to focus on the background of the leaf using binary specification masks indicating where the background is located. Due to lack of out-of-distribution test set, we evaluate with in-domain test images but with background pixels replaced by a constant pixel value, which is obtained by averaging over all pixels and images in the training set. We replace with an average pixel value in order to avoid any undesired confounding from shifts in pixel value distribution. In Appendix F.5, we present accuracy results when adding varying magnitude of noise to the background, which agree with observations we made in this section. More detailed analysis of the dataset can be found in Schramowski et al. (2020).

Table 1 contrasts different algorithms on the plant dataset. All the algorithms except CDEP improve over ERM, which is unsurprising given our test data construction; any algorithm that can divert focus from the background pixels can perform well. Wg accuracy of robustness (except AvgEx) and combined methods far exceed any other method by 5-12% over the next best baseline and by 19-26% accuracy point over ERM. Surprisingly, even Avg-Ex has significantly improved the performance over ERM likely because spurious features in the background are spiky or unstable, which vanishes with simple perturbations.

We visualize the interpretations of models obtained using SmoothGrad (Smilkov et al., 2017) trained with five different methods for three sample images from the train split in Figure 3. As expected, ERM has strong dependence on

non-leaf background features. Although Grad-Reg features are all on the leaf, they appear to be localized to a small region on the leaf, which is likely due to over-smoothing effect of its loss. IBP-Ex, IBP-Ex+Grad-Reg on the other hand draws features from a wider region and has more diverse pattern of active pixels.

5.4 ISIC: Skin Cancer Detection

ISIC is a dataset of skin lesion images, which are to be classified cancerous or non-cancerous. Since half the non-cancerous images in the dataset contains a colorful patch as shown in Figure 2, standard

DNN models depend on the presence of a patch for classification while compromising the accuracy on non-cancerous images without a patch (Codella et al., 2019; Tschandl et al., 2018). We follow the standard setup and dataset released by Rieger et al. (2020), which include masks highlighting the patch. Notably, (Rieger et al., 2020) employed a different evaluation metric, F1, and a different architecture - a VGG model pre-trained on ImageNet. This may explain the worse-than-expected performance of CDEP from Rieger et al. (2020). More details are available in Appendix G.

Traditionally, three groups are identified in the dataset: non-cancerous images without patch (NCNP) and with patch (NCP), and cancerous images (C). In Table 2, we report on pergroup accuracies for different algorithms. Detailed results with error bars are shown in Table 5 of Appendix F. The Wg accuracy (of Table 1) may not match with the worst of the average group accuracies in Table 2 because we report average of worst accuracies.

We now make the following observations. ERM performs the worst on the NPNC group confirming that predictions made by a standard model depend on the patch. The accuracy on the PNC group is high overall perhaps because PNC group images that are consistently at a lower scale (see middle column of Figure 2 for an example) are systematically more easier to classify even when the patch is not used for classification. Although human-explanations for this dataset, which only identifies the patch if present, do not full specify all spurious correlations, we still saw gains when learning from them. Grad-Reg and CDEP improved NPNC accuracy at the expense of C's accuracy while still performing relatively poor on Wg accuracy. Avg-Ex performed no better

than ERM whereas PGD-Ex, IBP-Ex, IBP-Ex+Grad-Reg, and PGD-Ex+Grad-Reg significantly improved Wg accuracy over other baselines. The reduced accuracy gap between NPNC and C when using combined methods is indicative of reduced dependence on patch. We provide a detailed comparison of PGD-Ex and IBP-Ex in Appendix F.3

5.5 Salient-Imagenet

Salient-Imagenet (Singla & Feizi, 2021; Singla et al., 2022) is a large scale dataset based on Imagenet with pixel-level annotations of irrelevant (spurious) and relevant (core) regions. Salient-Imagenet is an exemplary effort on how explanation masks can be obtained at scale. The dataset contains 232 classes and 52,521 images (with about 226 examples per class) along with their core and spurious masks. We made a smaller evaluation subset using six classes with around 600 examples. Each of the six classes contain at least one spurious feature identified by the annotator. Please refer to Appendix E.4 for more details. We then tested different methods for their effectiveness in learning a model that ignores the irrelevant region. We use a ResNet-18 model pretrained on ImageNet as the initialization.

In Table 1, we report RCS along with overall accuracy. Appendix F.2 contains further results including accuracy when noise is added to spurious (irrelevant) region. The results again substantiate the relative strength of robustness-methods over regularization-based. Furthermore, we highlight the considerable improvement offered by the novel combination of robustness-based and regularization-based methods.

5.6 Overall results

Among the regularization-based methods, Grad-Reg performed the best while also being simple and intuitive.

Robustness-based methods except Avg-Ex are consistently and effortlessly better or comparable to regularization-based methods on all the benchmarks with an improvement to Wg accuracy by 3-10% on the two real-world datasets. Combined methods are better than their constituents on all the datasets readily without much hyperparameter tuning.

In Appendix, we have additional experiment results; generality to new explanation methods (Appendix F.6) and new attention map-based architecture (Appendix F.7), and sensitivity to hyperparameters of PGD-Ex on Plant and ISIC dataset (Appendix F.4) and other perturbation-based methods on Decoy-MNIST dataset (Appendix F.8).

6 Related Work

Sub-population shift robustness. Sagawa et al. (2019) popularized a problem setting to avoid learning known spurious correlation by exploiting group labels, which are labels that identify the group an example. Unlike MLX, the sub-population shift problem assumes that training data contains groups of examples with varying strengths of spurious correlations. For example in the ISIC dataset, images with and without patch make different groups. Broadly, different solutions attempt to learn a hypothesis that performs equally well on all the groups via importance up-weighting (Shimodaira (2000); Byrd & Lipton (2019)), balancing subgroups(Cui et al. (2019); Izmailov et al. (2022); Kirichenko et al. (2022)) or group-wise robust optimization (Sagawa et al. (2019)). Despite the similarities, MLX and sub-population shift problem setting have some critical differences. Subpopulation shift setting assumes sufficient representation of examples from groups with varying levels of spurious feature correlations. This assumption may not always be practical, e.g. in medical imaging, which is when MLX problem setting may be preferred.

Learning from human-provided explanations. Learning from explanations attempts to ensure that model predictions are 'right for the right reasons' (Ross et al., 2017). Since gradient-based explanations employed by Ross et al. (2017) are known to be unfaithful (Murphy, 2023) (Chapter 33) (Wicker et al., 2022), subsequent works have proposed to replace the explanation method while the overall loss structure remained similar. Shao et al. (2021) proposed to regularize using an influence function, while Rieger et al. (2020) proposed to appropriately regularize respective contributions of relevant or irrelevant features to the classification probability through an explanation method proposed by Singh et al. (2018). In a slight departure from these methods, Selvaraju et al. (2019) used a loss objective that penalises ranking inconsistencies between human and the model's salient regions demonstrated for VQA applications. Stammer et al. (2021) argued for going beyond pixel-level importance scores to concept-level scores for the ease of human intervention. On the other hand, Singla et al. (2022) studied performance when augmenting with simple perturbations of irrelevant features and with the gradient regularization of Ross et al. (2017). This is the only work, to the best of our knowledge, that explored robustness to perturbations for learning from explanations.

Robust training. Adversarial examples were first popularized for neural networks by Szegedy et al. (2013), and have been a significant issue for machine learning models for at least a decade (Biggio & Roli, 2018). Local methods for computing adversarial attacks have been studied (Madry et al., 2017), but it is well known that adaptive attacks are stronger (i.e., more readily fool NNs) than general attack methods such as PGD (Tramer et al., 2020). Certification approaches on the other hand are guaranteed to be worse than any possible attack (Mirman et al., 2018), and training with certification approaches such as IBP have been found to provide state-of-the-art results in terms of provable robustness (Gowal et al., 2018), uncertainty (Wicker et al., 2021), explainability (Wicker et al., 2022), and fairness (Benussi et al., 2022).

7 Conclusions

By casting MLX as a robustness problem and using human explanations to specify the manifold of perturbations, we have shown that it is possible to alleviate the need for strong parameter smoothing of earlier approaches. Borrowing from the well-studied topic of robustness, we evaluated two strong approaches, one from adversarial robustness (PGD-Ex) and one from certified robustness (IBP-Ex). In our evaluation spanning seven methods and four datasets including two real-world datasets we found that PGD-Ex and IBP-Ex performed better than any previous approach, while our final proposal IBP-Ex+Grad-Reg and PGD-Ex+Grad-Reg of combining IBP-Ex and PGD-Ex with a light-weight interpretation based method respectively have consistently performed the best without compromising compute efficiency by much.

Limitations. Detecting and specifying irrelevant regions per-example by humans is a laborious and non-trivial task. Hence, it is interesting to see the effects of learning from incomplete explanations, which we leave for future work. Although robustness-based methods are very effective as we demonstrated, they can be computationally expensive (PGD-Ex) or scale poorly to large networks (IBP-Ex).

Broader Impact. Our work is a step toward reliable machine learning intended for improving ML for all. To the best of our knowledge, we do not foresee any negative societal impacts.

References

Benussi, E., Patane, A., Wicker, M., Laurenti, L., and Kwiatkowska, M. Individual fairness guarantees for neural networks. arXiv preprint arXiv:2205.05763 , 2022.

Biggio, B. and Roli, F. Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition , 84:317-331, 2018.

Byrd, J. and Lipton, Z. What is the effect of importance weighting in deep learning? In International conference on machine learning , pp. 872-881. PMLR, 2019.

Codella, N., Rotemberg, V., Tschandl, P., Celebi, M. E., Dusza, S., Gutman, D., Helba, B., Kalloo, A., Liopyris, K., Marchetti, M., et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.03368 , 2019.

contributors, W. Linearity of differentiation. Wikipedia , 10 2022. URL https://en.wikipedia. org/wiki/Linearity_of_differentiation .

Cui, Y., Jia, M., Lin, T.-Y., Song, Y., and Belongie, S. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 9268-9277, 2019.

DeGrave, A. J., Janizek, J. D., and Lee, S.-I. Ai for radiographic covid-19 detection selects shortcuts over signal. Nature Machine Intelligence , 3(7):610-619, 2021.

D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research , 2020.

Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. Shortcut learning in deep neural networks. Nature Machine Intelligence , 2(11):665-673, 2020.

Gowal, S., Dvijotham, K., Stanforth, R., Bunel, R., Qin, C., Uesato, J., Arandjelovic, R., Mann, T., and Kohli, P. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arXiv:1810.12715 , 2018.

Hennig, P., Osborne, M. A., and Kersting, H. P. Probabilistic Numerics: Computation as Machine Learning . Cambridge University Press, 2022. doi: 10.1017/9781316681411.

Izmailov, P., Kirichenko, P., Gruver, N., and Wilson, A. G. On feature learning in the presence of spurious correlations. Advances in Neural Information Processing Systems , 35:38516-38532, 2022.

Kirichenko, P., Izmailov, P., and Wilson, A. G. Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.02937 , 2022.

Liu, E. Z., Haghgoo, B., Chen, A. S., Raghunathan, A., Koh, P. W., Sagawa, S., Liang, P., and Finn, C. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning , pp. 6781-6792. PMLR, 2021.

Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 2017.

Miao, K., Gokul, A., Singh, R., Petryk, S., Gonzalez, J., Keutzer, K., and Darrell, T. Prior knowledgeguided attention in self-supervised vision transformers. arXiv preprint arXiv:2209.03745 , 2022.

Mirman, M., Gehr, T., and Vechev, M. Differentiable abstract interpretation for provably robust neural networks. In International Conference on Machine Learning , pp. 3578-3586. PMLR, 2018.

Murphy, K. P. Probabilistic Machine Learning: Advanced Topics . MIT Press, 2023. URL http: //probml.github.io/book2 .

Rieger, L., Singh, C., Murdoch, W., and Yu, B. Interpretations are useful: penalizing explanations to align neural networks with prior knowledge. In International conference on machine learning , pp. 8116-8126. PMLR, 2020.

Ross, A. S., Hughes, M. C., and Doshi-Velez, F. Right for the right reasons: Training differentiable models by constraining their explanations. arXiv preprint arXiv:1703.03717 , 2017.

Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731 , 2019.

Schramowski, P., Stammer, W., Teso, S., Brugger, A., Herbert, F., Shao, X., Luigs, H.-G., Mahlein, A.-K., and Kersting, K. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence , 2(8):476-486, 2020.

Selvaraju, R. R., Lee, S., Shen, Y., Jin, H., Ghosh, S., Heck, L., Batra, D., and Parikh, D. Taking a hint: Leveraging explanations to make vision and language models more grounded. In Proceedings of the IEEE/CVF international conference on computer vision , pp. 2591-2600, 2019.

Shah, H., Tamuly, K., Raghunathan, A., Jain, P., and Netrapalli, P. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems , 33:9573-9585, 2020.

Shah, R., Varma, V., Kumar, R., Phuong, M., Krakovna, V., Uesato, J., and Kenton, Z. Goal misgeneralization: Why correct specifications aren't enough for correct goals. arXiv preprint arXiv:2210.01790 , 2022.

Shao, X., Skryagin, A., Stammer, W., Schramowski, P., and Kersting, K. Right for better reasons: Training differentiable models by constraining their influence functions. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pp. 9533-9540, 2021.

Shimodaira, H. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference , 90(2):227-244, 2000.

Singh, C., Murdoch, W. J., and Yu, B. Hierarchical interpretations for neural network predictions. arXiv preprint arXiv:1806.05337 , 2018.

Singla, S. and Feizi, S. Salient imagenet: How to discover spurious features in deep learning? arXiv preprint arXiv:2110.04301 , 2021.

Singla, S., Moayeri, M., and Feizi, S. Core risk minimization using salient imagenet. arXiv preprint arXiv:2203.15566 , 2022.

Smilkov, D., Thorat, N., Kim, B., Viégas, F., and Wattenberg, M. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825 , 2017.

Stammer, W., Schramowski, P., and Kersting, K. Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 3619-3629, 2021.

Sundararajan, M., Taly, A., and Yan, Q. Axiomatic attribution for deep networks. In International conference on machine learning , pp. 3319-3328. PMLR, 2017.

Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 , 2013.

Tramer, F., Carlini, N., Brendel, W., and Madry, A. On adaptive attacks to adversarial example defenses. Advances in Neural Information Processing Systems , 33:1633-1645, 2020.

Tschandl, P., Rosendahl, C., and Kittler, H. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data , 5(1):1-9, 2018.

Wicker, M., Laurenti, L., Patane, A., Chen, Z., Zhang, Z., and Kwiatkowska, M. Bayesian inference with certifiable adversarial robustness. In International Conference on Artificial Intelligence and Statistics , pp. 2431-2439. PMLR, 2021.

Wicker, M., Heo, J., Costabello, L., and Weller, A. Robust explanation constraints for neural networks. arXiv preprint arXiv:2212.08507 , 2022.

Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., and Oermann, E. K. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine , 15(11):e1002683, 2018.

Zhang, H., Chen, H., Xiao, C., Gowal, S., Stanforth, R., Li, B., Boning, D., and Hsieh, C.-J. Towards stable and efficient training of verifiably robust neural networks. arXiv preprint arXiv:1906.06316 , 2019.

Supporting material for "Use Perturbations when Learning from Explanations"

A Proof of Theorem 1

We restate the result of Theorem 1 for clarity.

We infer a regression function f from a GP prior as described above with the additional supervision of [ ∂f ( x ) /∂x 2 ] | x ( i ) = 0 , ∀ i ∈ [1 , N ] . Then the function value deviations to perturbations on irrelevant feature are lower bounded by a value proportional to the perturbation strength δ as shown below.

Proof outline. We first show that the posterior mean of the function estimates marginalised over hyperparameters with Gamma prior has the following closed form where d ( x, y ) = ( x -y ) 2 / 2 and ˜ y denotes original observations y augmented with observations on gradients, which is described in more detail further below.

We then derive the following lower bound on the function value deviations and finally use simple inequalities to arrive at the final result.

Proof. We first derive the augmented set of observations ( ˆ y ) and ˆ K explained in the main section.

These results follow directly from the results on covariance between observations of f and its partial derivative below (Hennig et al., 2022).

The posterior value of the function at an arbitrary point x would then be of the form p ( f ( x ) | D ) ∼ N ( f ( x ); m x , k x ) where m x and k x are have the following closed form for Gaussian prior and

Gaussian likelihood in our case.

Since m x , k x are functions of the parameters θ 1 , θ 2 , we obtain the closed form for posterior mean by imposing a Gamma prior over the two parameters. For brevity, we denote by d ( x, ˜ x ) = ( x -˜ x ) 2 / 2 and ˜ y ( i ) is the i ( th ) component of ˆ K -1 XX ˆ y .

Overall, we have the following result.

We now derive the sensitivity to perturbations on the second dimension for ∆ x = [0 , δ ] T .

Using Bernoulli inequality, (1 + x ) r ≥ 1 + rx if r ≤ 0 , we derive the following inequalities.

Similarly,

Using inequalities 9, 10 in Equation 7, we have the following.

Using the inequality (1 + x ) r ≥ 1 + rx if r ≤ 0 , we have

B Proof of Theorem 2

We restate the result of Theorem 2 for clarity.

When we use an adversarial robustness algorithm to regularize the network, the fitted function has the following property.

δ max and f max are maximum value of ∆ x 2 and f ( x ) in the input domain ( X ) respectively. ˆ X denotes the subset of inputs covered by the robustness method. C therefore captures the maximum gap in coverage of the robustness method.

Proof. We begin by estimating the Lipschitz constant of a GP with squared exponential kernel.

We denote with δ max the maximum deviation of any input from the training points, i.e. we define δ max as max x ∈X min n ∈ [ N ] | x 2 -x ( n ) 2 | . Also, we denote by f max the maximum function value in the input domain, i.e. f max ≜ max x ∈X f ( x ) . We can then bound the partial derivative wrt second dimension as follows.

For any arbitrary point x , the maximum function deviation is upper bounded by the product of maximum slope and maximum distance from the closest point covered by the adversarial distance method.

Therefore,

Marginalising θ -2 with the Gamma prior leads to the final form below.

C Proof of Proposition 1

We restate the result here for clarity.

Consider a regression task with D + 1 -dimensional inputs x where the first D dimensions are irrelevant, and assume they are x d = y, d ∈ [1 , D ] while x D +1 ∼ N ( y, 1 /K ) . The MAP estimate of linear regression parameters f ( x ) = ∑ D +1 d =1 w d x d when fitted using Avg-Ex are as follows: w d = 1 / ( D + K ) , d ∈ [1 , D ] and w D +1 = K/ ( K + D ) .

Proof. Without loss of generality, we assume α, σ 2 parameters of Avg-Ex are set to 1. In effect, our objective is to fit parameters that predict well for inputs sampled using standard normal perturbations, i.e. x ( n ) + m ϵ, ∀ n ∈ [1 , N ] , ϵ ∼ N (0 , 1) , m = [1 , 1 , . . . , 1 , 0] T ∈ { 0 , 1 } D +1 . The original problem therefore is equivalent to fitting on transformed input ˆ x such that ˆ x ( n ) i ∼ N ( y, σ 2 i ) where σ 2 i = 1 for all i ≤ D and is 1 /K when i = D +1 .

Likelihood of observations for the equivalent problem is obtained as follows.

Substituting, the value of σ i defined as above, we have P=D+K and the MLE estimate for the linear regression parameters are as shown in the statement. The MAP estimate also remains the same since we do not impose any informative prior on the regression weights.

D Parametric Model Analysis

In this section we show that a similar result to what is shown for non-parametric models also holds for parametric models. We will analyse the results for a two-layer neural networks with ReLU activations. We consider a more general case of D dimensional input where the first d dimensions identify the spurious features. We wish to fit a function f : R D → R such that f ( x ) is robust to perturbations to the spurious features. We have the following bound when training a model using gradient regularization of Ross et al. (2017).

Proposition 2. We assume that the model is parameterised as a two-layer network with ReLU activations such that f ( x ) = ∑ j β j ϕ ( ∑ i w ji x i + b j ) where ⃗ β ∈ R F , ⃗ w ∈ R F × D , ⃗ b ∈ R F are the parameters, and ϕ ( z ) = max( z, 0) is the ReLU activation. For any function such that gradients

wrt to the first d features is exactly zero, i.e. ∂f ∂x i | x ( n ) i = 0 ∀ i ∈ [1 , d ] , n ∈ [1 , N ] , we have the following bound on the function value deviations for input perturbations from a training instance x : ˜ x -x = ∆ x = [∆ x T 1: d , 0 T d +1: D ] T .

For a two-layer network trained to regularize gradients wrt first d dimensions on training data, the function value deviation from an arbitrary point ˜ x from a training point x such that ˜ x -x = ∆ x = [∆ x T 1: d , 0 T d +1: D ] T is bounded as follows.

Proof. Recall that the function is parameterised using parameters ⃗ w, ⃗ b, ⃗ β such that f ( x ) = ∑ j β j ϕ ( ∑ i w ji x i + b j ) where ⃗ β ∈ R F , ⃗ w ∈ R F × D , ⃗ b ∈ R F are the parameters, and ϕ ( z ) = max( z, 0) is the ReLU activation.

Since we train such that ∂f ( x ) ∂x i = 0 , i ∈ [1 , d ] , we have that ∂f ( x ) x i = ∑ j β j ˆ ϕ ( ∑ i w ij x i + b i ) w ij where ˆ ϕ ( a ) = max( a | a | , 0) .

We now bound the variation in the function value for changes in the input when moving from x → ˜ x where x is an instance from the training data. We define four groups of neurons based on the sign of ∑ i w ji x i + b j and ∑ i w ji ˜ x i + b j . g 1 is both positive, g 2 is negative and positive, g 3 is positive and negative, g 4 is both negative. By defining groups, we can omit the ReLU activations as below.

Since we have that ∑ j ∈ g 1 ∪ g 3 β j w ij = 0 , ∀ i ∈ [1 , d ] , we have

retaining only the terms that depend on ∆ x = ˜ x -x , the expression is further simplified as a term that grows with ∆ x and a constant term that depends on the value of x

E Further Experiment Details

E.1 Hyperparameters.

We picked the learning rate, optimizer, weight decay, and initialization for best performance with ERM baseline on validation data, which are not further tuned for other baselines unless stated otherwise. We picked the best λ for Grad-Reg and CDEP from [1, 10, 100, 1000]. Additionally, we also tuned β (weight decay) for Grad-Reg from [1e-4, 1e-2, 1, 10]. For Avg-Ex, perturbations were drawn from 0 mean and σ 2 variance Gaussian noise, where σ was chosen from [0.03, 0.3, 1, 1.5, 2]. In PGD-Ex, the worst perturbation was optimized from ℓ ∞ norm ϵ -ball through seven PGD iterations, where the best ϵ is picked from the range 0.03-5. We did not see much gains when increasing PGD iterations beyond 7, Appendix F contains some results when the number of iterations is varied. In IBP-Ex, we follow the standard procedure of Gowal et al. (2018) to linearly dampen the value of α from 1 to 0.5 and linearly increase the value of ϵ from 0 to ϵ max , where ϵ max is picked from 0.01 to 2. We usually just picked the maximum possible value for ϵ max that converges. For IBP-Ex+Grad-Reg, we have the additional hyperparameter λ (Eqn. 4), which we found to be relatively stable and we set it to 1 for all experiments.

E.2 Metrics

Relative Core Sensitivity (RCS) (Singla et al., 2022) . The metric measures the relative dependence of the model on core features and is normalised such that the best value is 100. Higher value of RCS imply that the model is exploiting core features more than the spurious features.

where z ∼ N ( 0 , I ) and σ = 0 . 25 for all experiments. The interpretation of acc ( C ) is the accuracy when noise is added outside of core region, and acc ( S ) is the accuracy when noise is added outside of spurious region.

E.3 Data splits

We randomly split available labelled data in to training, validation, and test sets in the ratio of (0.75, 0.1, 0.15) for ISIC, (0.65, 0.1, 0.25) for Plant (similar to Schramowski et al. (2020)) and (0.6, 0.15, 0.25) for Salient-Imagenet. We use the standard train-test splits on MNIST.

E.4 Datasets

ISIC dataset The ISIC dataset consists of 2,282 cancerous (C) and 19,372 non-cancerous (NC) skin cancer images of 299 by 299 size, each with a ground-truth diagnostic label. We follow the standard setup and dataset released by Rieger et al. (2020), which included masks with patch segmentations. In half of the NC images, there is a spurious correlation in which colorful patches are only attached next to the lesion. This group is referred to as patch non-cancerous (PNC) and the other half is referred to as not-patched non-cancerous (NPNC) Codella et al. (2019). Since trained models tend to learn easy-to-learn and useful features, they tend to take a shortcut by learning spurious features instead of understanding the desired diagnostic phenomena. Therefore, our goal is to make the model invariant to such colorful patches by providing a human specification mask indicating where they are.

decoy-MNIST dataset The MNIST dataset consists of 70,000 images of handwriting digit from 0 to 9. Each class has about 7,000 images of 28 by 28 size. We use three-fully connected layers for multi classification with 512 hidden dimension and 3 channels.

Salient-Imagenet. The six classes we considered are Rhinoceros Beetle, Dowitcher, Alaskan Tundra Wolf, Dragonfly, Gorilla, Snoek Fish .

E.5 Computational cost

Run time and memory usage Table 3 presents the computation costs, including run time and memory usage, for each method using GTX 1080 Ti. It is worth noting that IBP-Ex has significantly less run time and memory usage compared to PGD-Ex, with a 10-fold reduction in run time and a 2.5-fold reduction in memory usage. Considering that PGD-Ex and IBP-Ex have similar performance in terms of worst group accuracy, as shown in Table 5, IBP-Ex+Grad-Reg appears to be comparably effective and efficient for model modification. Additionally, the combined method IBP-Ex+Grad-Reg, which presents the best performance in terms of averaged and worst group accuracy compared to PGD-Ex, also has a 3-fold reduction in run time and a 2-fold reduction in memory usage compared to PGD-Ex.

E.6 Network Architecture

Model architecture on the decoy-MNIST dataset

F Additional Results

F.1 Standard deviations

We repeated all our experiments on Decoy-MNIST, Plant and ISIC dataset three times and report the mean and standard deviation in Table 4. Similarly, we report in Table 5, the standard deviations for the corresponding Table 2 of the main paper.

F.2 Additional results on Salient-Imagenet

In Table 6, we show average accuracy and accuracy when noise (drawn from standard normal) is added to spurious or irrelevant regions (N-Acc column). We observe that for PGD-Ex + Grad-Reg, the accuracy did not diminish by much when noise is added to the spurious region.

F.3 Comparison of PGD-Ex and IBP-Ex

In Table 5, it is difficult to compare the worst group accuracy of IBP-Ex (64.2) and PGD-Ex (64.4) due to the comparably high standard deviation of PGD-Ex (4.3). Therefore, we additionally compare

the accuracy drop when colorful patches are removed from images in the PNC group in Table 7. We replace the colorful patch of the image with its mean value, making it looks like a background skin color. Note that we evaluate the robustness to concept-level perturbations rather than pixellevel perturbations, as our focus is on avoiding spurious concept features rather than robustness to adversarial attacks. Interestingly, the accuracy drops about 17% and 37% in IBP-Ex and PGD-Ex, respectively, showing that IBP-Ex is more robust to concept perturbations. This can be explained by the effectiveness of robustness methods in covering the epsilon ball with the center of each input point defined in a low-dimensional manifold annotated in the human specification mask. IBP guarantees robustness on any possible pixel combination within the epsilon ball while PGD only considers the worst case in the epsilon ball. When the inner maximization to find the PGD attack is non-convex, an inappropriate local worst case is found instead of the global one. Thus, IBP-Ex shows better robustness when spurious concepts are removed, which involves large perturbations on irrelevant parts within the defined epsilon ball. The combined method IBP-Ex+Grad-Reg, where Grad-Reg compensates for the practical limitations of the training procedure of IBP-Ex, shows about 1% higher worst group accuracy than IBP-Ex alone.

F.4 Results of PGD-Ex with different epsilon and iteration number.

We experimented with different values of epsilon and iteration numbers on the ISIC and Plant phenotyping datasets. The epsilon values tested were 0.03, 0.3, 1, 3, and 5, and the iteration numbers were 7 and 25. In Figure 4, the results on the ISIC dataset showed that using an iteration of 7 with different epsilon values resulted in stable results, but using an iteration of 25 resulted in unstable worst group accuracy. However, in the Plant phenotyping dataset, we found that both average and worst group accuracy were similar regardless of the epsilon and iteration values used.

PGD-Ex on the ISIC dataset

avg_iter7

wg_iter7

avg_iter25

wg_iter25

1. 00

0.75

0.50

0.25

0.00

0.03

0.3

1

3

5

Epsilon

(a) PGD-Ex on the ISIC data

PGD-Ex on the Plant phenotyping dataset

avg_iter7

wg_iter7

avg_iter25

wg_iter25

1. 00

0.75

0.50

0.25

0.00

0.03

0.3

1

3

5

Epsilon

(b) PGD-Ex on the Plant phenotying data

F.5 Out-of-distribution scenarios on the Plant data

In the main paper, we follow the dataset construction from Schramowski et al. (2020) to replace background with the average pixel value, which is obtained from train split. Here, in Table 8 we evaluated on a test set obtained by adding varying magnitude of noise to the background to test methods under out-of-distribution scenarios. We observe that robustness and regularization methods when combined led to a model that is far more robust to noise in the background, aligning with our original results on the plant dataset.

F.6 Generality to new explanation methods: Integrated-gradient and CDEP

In Table 9, we introduced evaluation using Integrated-gradient (Sundararajan et al., 2017) based regularization and also added evaluation with PGD-Ex+CDEP that we did not originally include on Decoy-MNIST dataset.

F.7 Generality to new model architecture: Attention map-based (ViT)

Using a Visual transformer architecture (of depth 3 and width 128), we evaluated regularization using local explanations obtained using an attention map - regularization based on attention maps was used to supervise prior knowledge in Miao et al. (2022) and is called SPAN. We obtained saliency explanations on inputs using the procedure proposed in Miao et al. (2022), shown in Table 10

F.8 Sensitivity to hyperparameters on Decoy-MNIST dataset

In Table 11, we show sensitivity to hyperparameters on Decoy-MNIST dataset. In summary, results are broadly stable with the choice of hyperparameters, and we did not extensively search for the best hyperparameters.

G Discussion on poor CDEP performance

Regarding ISIC dataset discrepancy: In Table 5, CDEP demonstrates better performance in worst group accuracy compared to ERM on the ISIC dataset. However, it fails to surpass RRR, which contradicts results from previous research in Rieger et al. (2020) where CDEP was found to perform better than RRR. This discrepancy may be attributed to the fact that Rieger et al. (2020) used different metrics (F1 and AUC) and employed a pretrained VGG model to estimate the contribution of mask features, whereas in our study we used worst group accuracy and employed a four-layer CNN followed by three fully connected layers without any pretraining. We do not use a pre-trained model for CDEP in order to make a fair comparison to other methods. As a result, CDEP also fails to improve worst group accuracy over ERM on the Plant Phenotyping and Decoy-MNIST datasets. We further illustrate the interpretations of CDEP on the Plant Phenotyping dataset using Smooth Gradient in Figure 5. In comparison to the interpretations of other methods shown in Figure 3 in the main paper, CDEP appears to focus primarily on the spurious agar part instead of the main leaf part.

Regarding DecoyMNIST dataset discrepancy: Note that our Decoy-MNIST setting is inspired from decoy-mnist of CDEP (Rieger et al., 2020), but not the same. All the methods were found to be equally good on the original decoy-mnist dataset (Rieger et al., 2020), which is why we had to alter the dataset to be more challenging. A key difference is that the volume of spurious/simple features in our version of decoy-mnist dataset is much higher, making it harder to remove dependence of a model on decoy/spurious features. This explains why there is the performance gap on this dataset reported in our paper and CDEP (Rieger et al., 2020).

CDEP

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel approach to Machine Learning from Explanations (MLX), the setting in which data points for learning are paired with human-annotated “explanations”, usually in the form of image masks marking regions that are known to be irrelevant with respect to the downstream prediction task. Previous methods for MLX made use of regularization-based techniques, forcing the learnt function to be locally independent of the irrelevant features for the training data points. The authors show that these methods require strong model smoothing to be effective, to the extent that downstream task performance is heavily penalized. They then propose “robustness-based” methods, which force the model to be robust to perturbation across the irrelevant features, and show theoretically and practically that these techniques work even without heavy regularization smoothing. The authors then show, using three benchmarks, that their technique works in practice, but best results are obtained by combining robustness and regularization-based methods.\n\nSTRENGTHS:\nThe paper improves on methods from an existing problem setting, applies robustness techniques where they had not been tried before, and convincingly shows both with theory and experiments that such techniques work.\n\nThe theorems, despite their simplified setting, intuitively demonstrate why the approach works better than the alternatives.\n\nThe experimental results convincingly show that the robustness-based techniques work better than regularization-based techniques.\n\nWEAKNESSES:\nWhile the contribution itself is nicely self-contained and perfectly valid when taken at face value (that in the MLX setting, training GPs or simple CNNs, robustness-based methods are better than regularization-based methods), one cannot help but wonder whether the entire setup is relevant or interesting, considering current state of the art architectures and how they attend to their input. Attention-based architectures like ViTs, due to intermediate-layer attention maps, lend themselves to very natural investigations commensurable to the kind of pixel-wise explanations used in this work.\n\nOne can imagine that, as these transformer models also easily incorporate masking, the whole MLX problem could be reframed to be about either manipulating attention maps, or dynamically learning attention masks. And example is [Yang et al. (2023)](https://openaccess.thecvf.com/content/CVPR2023/html/Yang_Improving_Visual_Grounding_by_Encouraging_Consistent_Gradient-Based_Explanations_CVPR_2023_paper.html). Addressing these concerns either in the introduction or related work could help contextualize the paper’s contributions.\n\nQUESTIONS:\nIt would be interesting, to place the present work in perspective, to at least spend some time in related work to investigate how “explanations” are currently treated in the literature on transformer-based image models like ViTs. Is there any existing literature showing techniques for MLX (maybe not labeled as such) that are dependent on a particular architecture choice? If so, what makes the present work still interesting?\n\nEdit: Increased the score after author rebuttal addressing the points above - see comment below.\n\nLIMITATIONS:\nThe authors briefly mention limitations related to the possibility, in future work, to learn from incomplete explanation data. It would be interesting to add a segment related to the points raised in Weaknesses and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper reinterprets Machine learning from explanations (MLX) as a robustness problem, where human explanations define a lower-dimensional manifold for perturbations. The paper points out that the previous regularization-based MLX approaches require strong model smoothing in order to be globally effective at reducing shortcut learning. The authors propose a novel approach that combines robust training methods with an earlier MLX technique, achieving state-of-the-art results on both synthetic and real-world benchmarks. The theoretical and empirical analyses explaining how the combination of robustness and regularization can reduce the need for strong model smoothing are provided.\n\nSTRENGTHS:\n1. The paper is well-structured and clearly organized, making it easy to follow;\n\n2. The authors offer theoretical analyses to explain how the combination of robustness and regularization can minimize the need for strong model smoothing, which adds to the rigor and solidity of the paper.\n\nWEAKNESSES:\n1. The contribution of the proposed method is restricted to the combination of two established robust training techniques with an existing MLX approach;\n\n2. Although the authors thoroughly review various regularization-based MLX approaches in the introduction, the paper only showcases the effectiveness of combining robust training methods with a single MLX approach, Grad-Reg. It remains unclear whether the proposed approach can be applied to other MLX methods, and further investigation is necessary to determine its generalizability.\n\nIn addition, Table 1 does not include the results of the more recent MLX approaches that are well discussed in the introduction section.\n\nQUESTIONS:\n1. Is it possible to apply the proposed methods to other MLX approaches? To better understand the generalizability of the approach, it would be helpful to see the performance of robust training methods combined with various MLX methods.\n\n2. The authors mention in lines 277-279 that they evaluate in-domain test images with background pixels replaced by a constant pixel value. Have you tested the methods' performance under other out-of-distribution (OOD) scenarios, such as replacing background pixels with the background pixels from randomly selected images?\n\n3. A sensitivity analysis on the hyper-parameters should be provided.\n\nLIMITATIONS:\nSee weaknesses.\n\nAt this moment, the main limitations of the paper revolve around the limited novelty and generalizability of the proposed method. I tend to reject the paper primarily for this reason. However, I may reconsider my evaluation if the authors can provide solid evidence demonstrating the generalizability of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-23"}

comments: SUMMARY:
This paper proposes a new approach to machine learning from explanations (MLX). This new approach is still based on human-provided explanations of (ir)relevant features for each input in training, but recasts the MLX problem itself essentially into a robustness problem. The authors achieve SOTA performance when combining their method with previous ones on several benchmarks.

STRENGTHS:
The authors show that the need for strong parameter smoothing of earlier approaches can be overcome, and they achieve SOTA performance on several benchmarks. Their method is intuitive and easy to understand.

WEAKNESSES:
Obtaining human-specified masks is at best a lot of effort and in many cases simply not available, which limits the scope of problems for which their method can be applied.

QUESTIONS:
Very minor: What is the bolding criteria for Table 1? Usually, the best model is in bold, but here there are 1, 2, or 3 models bolded depending on the dataset, and it's not clear what determined the number of bolded models for each.

LIMITATIONS:
It would be good to expand the "Limitations" section and provide more detail on, e.g., when the scaling breaks to large NNs, i.e. provide some guidance here about when one should expect the robustness methods to no longer be feasible to use.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is in the domain of MLX (machine learning from explanations). In this approach, human annotated data for each input example is available denoting features that are _relevant_ and which are _irrelevant_. It is desired that the model doesn't learn from irrelevant features.\n\nIn this paper, the authors utilize robustness for this domain. To elaborate, they expect the model to be robust to perturbations along the features which are considered irrelevant. According to the authors, this is the first use of this technique in the domain of MLX. They also combine with existing regularization-based approaches.\n\nA theoretical framework is provided and approach evaluated on three datasets -- Decoy-MNIST, Plant, ISIC -- which are designed to capture the extent to which the models are learning from the irrelevant features. Combination of the robustness-based approaches with regularization-based approaches is shown to outperform prior approaches.\n\nSTRENGTHS:\nThe use of robustness for MLX is well-motivated, and as per the paper, novel.\n\nThe empirical results appear to back the utility of combining robustness with existing techniques.\n\nThe paper is well-written.\n\nWEAKNESSES:\nThe prior works (such as Sagawa et al., 2019; Piratla et al., 2021) have several more evaluation datasets, which the approach is not evaluated on. The generality and robustness (pardon the pun) of the approach is not completely clear.\n\nQUESTIONS:\nQuestions\n- Please address the choice of limiting the evaluation to the 3 datasets used.\n\nMinor suggestion (no response needed)\n- L63: State the domain of m^(n)\n- Consider placing figure closer to their first reference. For example, Figure 2 is referenced on Pg 2 but appears on Pg 6.\n- L70: \"while not exploiting\" -- please provide a the definition for this at this stage (addendum: if there is one at all)\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-27"}