SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Use Perturbations when Learning from Explanations
Juyeon Heo*  
University of Cambridge  
jh2324@cam.ac.uk
Vihari Piratla*  
University of Cambridge  
vp421@cam.ac.uk
Matthew Wicker Alan Turing Institute
Adrian Weller  
Alan Turing Institute and University of Cambridge
Machine learning from explanations (MLX) is an approach to learning that uses human-provided explanations of relevant or irrelevant features for each input to ensure that model predictions are right for the right reasons. Existing MLX approaches rely on local model interpretation methods and require strong model smoothing to align model and human explanations, leading to sub-optimal performance. We recast MLX as a robustness problem, where human explanations specify a lower dimensional manifold from which perturbations can be drawn, and show both theoretically and empirically how this approach alleviates the need for strong model smoothing. We consider various approaches to achieving robustness, leading to improved performance over prior MLX methods. Finally, we show how to combine robustness with an earlier MLX method, yielding state-of-the-art results on both synthetic and real-world benchmarks. Our implementation can be found at: https://github.com/vihari/robust_mlx.
Deep neural networks (DNNs) display impressive capabilities, making them strong candidates for real-wold deployment. However, numerous challenges hinder their adoption in practice. Several major deployment challenges have been linked to the fact that labelled data often under-specifies the task (D'Amour et al., 2020). For example, systems trained on chest x-rays were shown to generalise poorly because they exploited dataset-specific incidental correlations such as hospital tags for diagnosing pneumonia (Zech et al., 2018; DeGrave et al., 2021). This phenomenon of learning unintended feature-label relationships is referred to as shortcut learning (Geirhos et al., 2020) and is a critical challenge to solve for trustworthy deployment of machine learning algorithms. A common remedy to avoid shortcut learning is to train on diverse data (Shah et al., 2022) from multiple domains, demographics, etc, thus minimizing the underspecification problem, but this may be impractical for many applications such as in healthcare.
Enriching supervision through human-provided explanations of relevant and irrelevant regions/features per example is an appealing direction toward reducing under-specification. For instance, a (human-provided) explanation for chest x-ray classification may highlight scanning artifacts such as hospital tag as irrelevant features. Learning from such human-provided explanations (MLX) has been shown to avoid known shortcuts (Schramowski et al., 2020). Ross et al. (2017) pioneered an MLX approach based on regularizing DNNs, which was followed by several others (Schramowski et al., 2020; Rieger et al., 2020; Stammer et al., 2021; Shao et al., 2021). Broadly, existing approaches employ a model interpretation method to obtain per-example feature saliency, and
regularize such that model and human-provided explanations align. Since saliency is unbounded for relevant features, many approaches simply regularize the salience of irrelevant features. In the same spirit, we focus on handling a specification of irrelevant features, which we refer to as an explanation hereafter.
Existing techniques for MLX employ a local interpretation tool and augment the loss with a term regularizing the importance of irrelevant region. Regularization-based approaches suffer from a critical concern stemming from their dependence on a local interpretation method. Regularization of local, i.e. example-specific, explanations may not have the desired effect of reducing shortcuts globally, i.e. over the entire input domain (see Figure 1). As we demonstrate both analytically and empirically that previous MLX proposals, which are all regularization-based, require strong model smoothing in order to be globally effective at reducing shortcut learning.
In this work, we explore MLX using various robust training methods with the objective of training models that are robust to perturbations of irrelevant features. We start by framing the provided human explanations as specifications of a local, lower-dimensional manifold from which perturbations are drawn. We then notice that a model whose prediction is invariant to perturbations drawn from the manifold ought also to be robust to irrelevant features. Our perspective yields considerable advantages. Posing MLX as a robustness task enables us to leverage the considerable body of prior work in robustness. Further, we show in Section 4.1 that robust training can provably upper bound the deviation on model value when irrelevant features are perturbed without needing to impose model smoothing. However, when the space of irrelevant features is high-dimensional, robust-training may not fully suppress irrelevant features as explained in Section 4.2. Accordingly, we explore combining both robustness-based and regularization-based methods, which achieves the best results. We highlight the following contributions:
- We theoretically and empirically demonstrate that existing MLX methods require strong model smoothing owing to their dependence on local model interpretation tools.
- We study learning from explanations using robust training methods. To the best of our knowledge, we are the first to analytically and empirically evaluate robust training methods for MLX.
- We distill our insights into our final proposal of combining robustness and regularization-based methods, which consistently performed well and reduced the error rate over the previous best by  $20 - 90\%$ .
2 Problem Definition and Background
We assume access to a training dataset with  $N$  training examples,  $\mathcal{D}_T = \{(\mathbf{x}^{(i)},y^{(i)})\}_{i = 1}^N$ , with  $\mathbf{x}^{(i)}\in \mathbb{R}^d$  and  $y^{(i)}$  label. In the MLX setting, a human expert also specifies input mask  $\mathbf{m}^{(n)}$  for an example  $\mathbf{x}^{(n)}$  where non-zero values of the mask identify irrelevant features of the input  $\mathbf{x}^{(n)}$ . An input mask is usually designed to negate a known shortcut feature that a classifier is exploiting. Figure 2 shows some examples of masks for the datasets that we used for evaluation. For example, a mask in the ISIC dataset highlights a patch that was found to confound with non-cancerous images. With the added human specification, the augmented dataset contains triplets of example, label and mask,  $\mathcal{D}_T = \{(\mathbf{x}^{(i)},y^{(i)},\mathbf{m}^{(i)})\}_{i = 0}^N$ . The task therefore is to learn a model  $f(\mathbf{x};\theta)$  that fits observations well while not exploiting any features that are identified by the mask  $\mathbf{m}$ .
The method of Ross et al. (2017) which we call Grad-Reg (short for Gradient-Regularization), and also other similar approaches (Shao et al., 2021; Schramowski et al., 2020) employ an local interpretation method (E) to assign importance scores to input features:  $IS(\mathbf{x})$ , which is then regularized with an  $\mathcal{R}(\theta)$  term such that irrelevant features are not regarded as important. Their training loss takes the form shown in Equation 1 for an appropriately defined task-specific loss  $\ell$ .
$$
\begin{array}{l} I S (\mathbf {x}) \triangleq E (\mathbf {x}, f (\mathbf {x}; \theta)). \\ \mathcal {R} (\theta) \triangleq \sum_ {n = 1} ^ {N} \| I S (\mathbf {x} ^ {(n)}) \odot \mathbf {m} ^ {(n)} \| ^ {2}. \\ \theta^ {*} = \arg \min  _ {\theta} \left\{\sum_ {n} \ell \left(f \left(\mathbf {x} ^ {(n)}; \theta\right), y ^ {(n)}\right) + \lambda \mathcal {R} (\theta) + \frac {1}{2} \beta \| \theta \| ^ {2} \right\}. \tag {1} \\ \end{array}
$$
We use  $\odot$  to denote element-wise product throughout. CDEP (Rieger et al., 2020) is slightly different. They instead use an explanation method that also takes the mask as an argument to estimate the contribution of features identified by the mask, which they minimize similarly.
About obtaining human specification mask. Getting manually specifying explanation masks can be impractical. However, the procedure can be automated if the nuisance/irrelevant feature occurs systematically or if it is easy to recognize, which may then be obtained automatically using a procedure similar to Liu et al. (2021); Rieger et al. (2020). A recent effort called Salient-Imagenet used neuron activation maps to scale curation of such human-specified masks to Imagenet-scale (Singla & Feizi, 2021; Singla et al., 2022). These efforts may be seen as a proof-of-concept for obtaining richer annotations beyond content labels, and towards better defined tasks.
Our methodology is based on the observation that an ideal model must be robust to perturbations to the irrelevant features. Following this observation, we reinterpret the human-provided mask as a specification of a lower-dimensional manifold from which perturbations are drawn and optimize the following objective.
$$
\theta^ {*} = \arg \min  _ {\theta} \sum_ {n} \left\{\ell \left(f \left(\mathbf {x} ^ {(n)}; \theta\right), y ^ {(n)}\right) + \alpha \max  _ {\epsilon : \| \epsilon \| _ {\infty} \leq \kappa} \ell \left(f \left(\mathbf {x} ^ {(n)} + (\epsilon \odot \mathbf {m} ^ {(n)}); \theta\right), y ^ {(n)}\right) \right\} \tag {2}
$$
The above formulation uses a weighting  $\alpha$  to trade off between the standard task loss and perturbation loss and  $\kappa > 0$  is a hyperparameter that controls the strength of robustness. We can leverage the many advances in robustness in order to approximately solve the inner maximization. We present them below.
Avg-Ex: We can approximate the inner-max with the empirical average of loss averaged over  $K$  samples drawn from the neighbourhood of training inputs. Singla et al. (2022) adopted this straightforward baseline for supervising using human-provided saliency maps on the Imagenet dataset. Similar to  $\kappa$ , we use  $\sigma$  to control the noise in perturbations as shown below.
$$
\theta^ {*} = \arg \min  _ {\theta} \sum_ {n} \left\{\ell \left(f \left(\mathbf {x} ^ {(n)}; \theta\right), y ^ {(n)}\right) + \frac {\alpha}{K} \sum_ {\boldsymbol {\epsilon} _ {j} \sim \mathcal {N} (0, \sigma^ {2} I)} ^ {K} \ell \left(f \left(\mathbf {x} ^ {(n)} + \left(\boldsymbol {\epsilon} _ {j} \odot \mathbf {m} ^ {(n)}\right); \theta\right), y ^ {(n)}\right) \right\}
$$
PGD-Ex: Optimizing for an estimate of worst perturbation through projected gradient descent (PGD) (Madry et al., 2017) is a popular approach from adversarial robustness. We refer to the approach of using PGD to approximate the second term of our loss as PGD-Ex and denote by  $\epsilon^{*}(\mathbf{x}^{(n)},\theta ,\mathbf{m}^{(n)})$  the perturbation found by PGD at  $\mathbf{x}^{(n)}$ . Given the non-convexity of this problem, however, no guarantees can be made about the quality of the approximate solution  $\mathbf{x}^*$ .
$$
\theta^ {*} = \arg \min _ {\theta} \sum_ {n} \left\{\ell \left(f (\mathbf {x} ^ {(n)}; \theta), y ^ {(n)}\right) + \alpha \ell \left(f (\mathbf {x} ^ {(n)} + (\boldsymbol {\epsilon} ^ {*} (\mathbf {x} ^ {(n)}, \theta , \mathbf {m} ^ {(n)})); \theta), y ^ {(n)}\right) \right\}
$$
IBP-Ex: Certified robustness approaches, on the other hand, minimize a certifiable upper-bound of the second term. A class of certifiable approaches known as interval bound propagation methods (IBP) (Mirman et al., 2018; Gowal et al., 2018) propagate input intervals to function value intervals that are guaranteed to contain true function values for any input in the input interval.
We define an input interval for  $\mathbf{x}^{(n)}$  as  $[\mathbf{x}^{(n)} - \kappa \mathbf{m}^{(n)},\mathbf{x}^{(n)} + \kappa \mathbf{m}^{(n)}]$  where  $\kappa$  is defined in Eqn. 2. We then use bound propagation techniques to obtain function value intervals for the corresponding input interval:  $\mathbf{l}^{(n)},\mathbf{u}^{(n)}$ , which are ranges over class logits. Since we wish to train a model that correctly classifies an example irrespective of the value of the irrelevant features, we wish to maximize the minimum probability assigned to the correct class, which is obtained by combining minimum logit for the correct class with maximum logit for incorrect class:  $\tilde{f} (\mathbf{x}^{(n)},y^{(n)},\mathbf{l}^{(n)},\mathbf{u}^{(n)};\theta)\triangleq$ $\mathbf{l}^{(n)}\odot \bar{\mathbf{y}}^{(n)} + \mathbf{u}^{(n)}\odot (\mathbf{1} - \bar{\mathbf{y}}^{(n)})$  where  $\bar{\mathbf{y}}^{(n)}\in \{0,1\} ^c$  denotes the one-hot transformation of the label  $y^{(n)}$  into a c-length vector for  $c$  classes. We refer to this version of the loss as IBP-Ex, summarized
$$
\mathbf {l} ^ {(n)}, \mathbf {u} ^ {(n)} = I B P (f (\bullet ; \theta), [ \mathbf {x} ^ {(n)} - \kappa \times \mathbf {m} ^ {(\mathbf {n})}, \mathbf {x} ^ {(n)} + \kappa \times \mathbf {m} ^ {(\mathbf {n})} ])
$$
$$
\tilde {f} \left(\mathbf {x} ^ {(n)}, y ^ {(n)}, \mathbf {l} ^ {(n)}, \mathbf {u} ^ {(n)}; \theta\right) \triangleq \mathbf {l} ^ {(n)} \odot \bar {\mathbf {y}} ^ {(n)} + \mathbf {u} ^ {(n)} \odot (\mathbf {1} - \bar {\mathbf {y}} ^ {(n)})
$$
$$
\theta^ {*} = \arg \min  _ {\theta} \sum_ {n} \ell \left(f \left(\mathbf {x} ^ {(n)}; \theta\right), y ^ {(n)}\right) + \alpha \ell \left(\tilde {f} \left(\mathbf {x} ^ {(n)}, y ^ {(n)}, \mathbf {l}, \mathbf {u}; \theta\right), y ^ {(n)}\right) \tag {3}
$$
Despite its computational efficiency, IBP is known to suffer from scaling issues when the model is too big. Consequently, it is better to use IBP-Ex only when the model is small (less than four layers of CNN or feed-forward) and if computational efficiency is desired. On the other hand, we do not anticipate any scaling issues when using PGD-Ex.
Combined robustness and regularization: PGD-Ex+Grad-Reg, IBP-Ex+Grad-Reg. We combine robustness and regularization by simply combining their respective loss terms. We show the objective for IBP-Ex+Grad-Reg below, PGD-Ex+Grad-Reg follows similarly.
$$
\theta^ {*} = \arg \min  _ {\theta} \sum_ {n} \ell \left(f \left(\mathbf {x} ^ {(n)}; \theta\right), y ^ {(n)}\right) + \alpha \ell \left(\tilde {f} \left(\mathbf {x} ^ {(n)}, y ^ {(n)}, \mathbf {l}, \mathbf {u}; \theta\right), y ^ {(n)}\right) + \lambda \mathcal {R} (\theta). \tag {4}
$$
$\lambda \mathcal{R}(\theta)$  and  $\alpha, \tilde{f}$  are as defined in Eqn. 5 and Eqn. 3 respectively. In Section 4, 5.2, we demonstrate the complementary strengths of robustness and regularization-based methods.
4 Theoretical Motivation
(a) Toy data
Figure 1: Illustration of the uneasy relationship between Grad-Reg and smoothing strength. (b) The decision boundary is nearly vertical (zero gradient wrt to nuisance y-axis value) for all training points and yet varies as a function of y value when fitted using Grad-Reg with  $\beta = 0$ . (c) Grad-Reg requires strong model smoothing ( $\beta = 1$ ) in order to translate local insensitivity to global robustness to y-coordinate. (d) IBP-Ex, on the other hand, fits vertical pair of lines without any model smoothing.
(b) Grad-Reg with  $\beta = 0$
(c) Grad-Reg with  $\beta = 1$
(d) IBP-Ex with  $\beta = 0$
In this section, we motivate the merits and drawbacks of robustness-based over regularization-based methods. Through non-parametric analysis in Theorems 1, 2, we argue that (a) regularization methods are robust to perturbations of irrelevant features (identified by the mask) only when the underlying model is sufficiently smoothed, thereby potentially compromising performance, (b) robust training upper-bounds deviation in function values when irrelevant features are perturbed, which can be further suppressed by using a more effective robust training. Although our analysis is restricted to nonparametric models for the ease of analysis, we empirically verify our claims with parametric neural network optimized using a gradient-based optimizer. We then highlight a limitation of robustness-based methods when the number of irrelevant features is large through Proposition 1.
4.1 Merits of Robustness-based methods
Consider a two-dimensional regression task, i.e.  $\mathbf{x}^{(n)}\in \mathcal{X}$  and  $y\in \mathbb{R}$ . Assume that the second feature is the shortcut that the model should not use for prediction, and denote by  $\mathbf{x}_j^{(n)}$  the  $j^{th}$  dimension of  $n^{th}$  point. We infer a regression function  $f$  from a Gaussian process prior  $f\sim GP(f;0,K)$  with a squared exponential kernel where  $k(x,\tilde{x}) = \exp (-\sum_{i}\frac{1}{2}\frac{(x_i - \tilde{x}_i)^2}{\theta_i^2})$ . As a result, we have two hyperparameters  $\theta_{1},\theta_{2}$ , which are length scale parameters for the first and second dimensions respectively. Further, we impose a Gamma prior over the length scale:  $\theta_i^{-2}\sim \mathcal{G}(\alpha ,\beta)$ .
Theorem 1 (Grad-Reg). We infer a regression function  $f$  from a  $GP$  prior as described above with the additional supervision of  $[\partial f(\mathbf{x}) / \partial x_2]|_{\mathbf{x}^{(i)}} = 0, \quad \forall i \in [1, N]$ . Then the function value deviations to perturbations on irrelevant feature are lower bounded by a value proportional to the perturbation strength  $\delta$  as shown below.
$$
f (\mathbf {x} + [ 0, \delta ] ^ {T}) - f (\mathbf {x}) \geq \frac {2 \delta \alpha}{\beta} \Theta \left(x _ {1} ^ {2} x _ {2} ^ {6} + \delta x _ {1} ^ {2} x _ {2} ^ {5}\right) \tag {5}
$$
Full proof of Theorem 1 is in Appendix A, we provide the proof outline below.
Proof sketch. We exploit the flexibility of GPs to accommodate observations on transformed function values if the transformation is closed under linear operation (Hennig et al., 2022) (see Chapter 4). Since gradient is a linear operation (contributors, 2022), we simply revise the observations  $y$  to include the  $N$  partial derivative observations for each training point with appropriately defined kernel. We then express the inferred function, which is the posterior  $f(\mathbf{x} \mid \mathcal{D}_T)$  in closed form.
We then derive a bound on function value deviations to perturbation on the second feature and simplify using simple arithmetic and Bernoulli's inequality to derive the final expression.
We observe from Theorem 1 that if we wish to infer a function that is robust to irrelevant feature perturbations, we need to set  $\frac{\alpha}{\beta}$  to a very small value. Since the expectation of gamma distributed inverse-square length parameter is  $\mathbb{E}[\theta^{-2}] = \frac{\alpha}{\beta}$ , which we wish to set very small, we are, in effect, sampling functions with very large length scale parameter i.e. strongly smooth functions. This result brings us to the intuitive takeaway that regularization using Grad-Reg applies globally only when the underlying family of functions is sufficiently smooth. The result may also hold for any other local-interpretation methods that is closed under linear operation. One could also argue that we can simply use different priors for different dimensions, which would resolve the over-smoothing issue. However, we do not have access to parameters specific to each dimension in practice and especially with DNNs, therefore only overall smoothness may be imposed such as with parameter norm regularization in Eqn. 1.
We now look at properties of a function fitted using robustness methods and argue that they bound deviations in function values better. In order to express the bounds, we introduce a numerical quantity called coverage (C) to measure the effectiveness of a robust training method. We first define a notion of inputs covered by a robust training method as  $\hat{\mathcal{X}}\triangleq \{\mathbf{x}\mid \mathbf{x}\in \mathcal{X},\ell (f(\mathbf{x};\theta),y) < \phi \} \subset \mathcal{X}$  for a small positive threshold  $\phi$  on loss. We define coverage as the maximum distance along second coordinate between any point in  $\mathcal{X}$  and its closest point in  $\hat{\mathcal{X}}$ , i.e.  $C\triangleq \max_{\mathbf{x}\in \mathcal{X}}\min_{\mathbf{x}\in \hat{\mathcal{X}}}|\mathbf{x}_2 - \hat{\mathbf{x}}_2|$ . We observe that C is small if the robust training is effective. In the extreme case when training minimizes the loss for all points in the input domain, i.e.  $\hat{\mathcal{X}} = \mathcal{X}$ , then  $\mathrm{C} = 0$ .
Theorem 2. When we use a robustness algorithm to regularize the network, the fitted function has the following property.
$$
\left| f (\mathbf {x} + [ 0, \delta ] ^ {T}) - f (\mathbf {x}) \right| \leq 2 C \frac {\alpha}{\beta} \delta_ {\max } f _ {\max }. \tag {6}
$$
$\delta_{max}$  and  $f_{max}$  are maximum values of  $\Delta x_2$  and  $f(\mathbf{x})$  in the input domain  $(\mathcal{X})$  respectively.
Proof sketch. We begin by estimating the function deviation between an arbitrary point and a training instance that only differ on  $x_{2}$ , which is bounded by the product of maximum slope  $(\max_{\mathbf{x}} \partial f(\mathbf{x}) / \partial x_{2})$  and  $\Delta x_{2}$ . We then estimate the maximum slope (lipschitz constant) for a function inferred from GP. Finally, function deviation between two arbitrary points is twice the maximum deviation between an arbitrary and a training instance that is estimated in the first step.
Full proof is in Appendix B. The statement shows that deviations in function values are upper bounded by a factor proportional to  $C$ , which can be dampened by employing an effective robust training method. We can therefore control the deviations in function values without needing to regress  $\frac{\alpha}{\beta}$  (i.e. without over-smoothing).
Further remarks on sources of over-smoothing in regularization-based methods. We empirically observed that the term  $\mathcal{R}(\theta)$  (of Eqn. 1), which supervises explanations, also has a smoothing effect on the model when the importance scores (IS) are not well normalized, which is often the case. This is because reducing  $\mathrm{IS}(\mathbf{x})$  everywhere will also reduce saliency of irrelevant features.
Empirical verification with a toy dataset. For empirical verification of our results, we fit a 3-layer feed-forward network on a two-dimensional data shown in Figure 1 (a), where color indicates the label. We consider fitting a model that is robust to changes in the second feature shown on y-axis. In Figures 1 (b), (c), we show the Grad-Reg fitted classifier using gradient  $(\partial f / \partial x_{2}$  for our case) regularization for two different strengths of parameter smoothing (0 and 1 respectively). With weak smoothing, we observe that the fitted classifier is locally vertical (zero gradient along y-axis), but curved overall (Figure 1 (b)), which is fixed with strong smoothing (Figure 1 (c)). On the other hand, IBP-Ex fitted classifier is nearly vertical without any parameter regularization as shown in (d). This example illustrates the need for strong model smoothing when using a regularization-based method.
4.2 Drawbacks of Robustness-based methods
Although robust training is appealing in low dimensions, their merits do not transfer well when the space of irrelevant features is high-dimensional owing to difficulty in solving the inner maximization of Eqn. 2. Sub-par estimation of the maximization term may learn parameters that still depend on the irrelevant features. We demonstrate this below with a simple exercise.
Proposition 1. Consider a regression task with  $D + 1$ -dimensional inputs  $\mathbf{x}$  where the first  $D$  dimensions are irrelevant, and assume they are  $x_{d} = y$ ,  $d \in [1, D]$  while  $x_{D+1} \sim \mathcal{N}(y, 1 / K)$ . The MAP estimate of linear regression parameters  $f(\mathbf{x}) = \sum_{d=1}^{D+1} w_{d} x_{d}$  when fitted using Avg-Ex are as follows:  $w_{d} = 1 / (D + K)$ ,  $d \in [1, D]$  and  $w_{D+1} = K / (K + D)$ .