SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We present the proof in Appendix C. We observe that as  $D$  increases, the weight of the only relevant feature  $(x_{D + 1})$  diminishes. On the other hand, the weight of the average feature:  $\frac{1}{D}\sum_{d = 1}^{D}x_{d}$ , which is  $D / (D + K)$  approaches 1 as  $D$  increases. This simple exercise demonstrates a curse of dimensionality for robustness-based methods. For this reason, we saw major empirical gains when combining robustness methods with a regularization method especially when the number of irrelevant features is large such as in the case of Decoy-MNIST dataset, which is described in the next section.
We evaluate different methods on four datasets: one synthetic and three real-world. The synthetic dataset is similar to decoy-MNIST of Ross et al. (2017) with induced shortcuts and is presented in Section 5.2. For evaluation on practical tasks, we evaluated on a plant phenotyping (Shao et al., 2021) task in Section 5.3, skin cancer detection (Rieger et al., 2020) task presented in Section 5.4, and object classification task presented in Section 5.5. All the datasets contain a known spurious feature, and were used in the past for evaluation of MLX methods. Figure 2 summarises the three datasets, notice that we additionally require in the training dataset the specification of a mask identifying irrelevant features of the input; the patch for ISIC dataset, background for plant dataset, decoy half for Decoy-MNIST images, and label-specific irrelevant region approved by humans for Salient-Imagenet.
Figure 2: Sample images and masks for different datasets. The shown mask corresponds to the image from the second column.
5.1.1 Baselines
We denote by ERM the simple minimization of cross-entropy loss (using only the first loss term of Equation 1). We also compare with G-DRO(Sagawa et al., 2019), which also has the objective of avoiding to learn known irrelevant features but is supervised through group label (see Section 6). Although the comparison is unfair toward G-DRO because MLX methods use richer supervision of per-example masks, it serves as a baseline that can be slightly better than ERM in some cases.
Regulaization-based methods. Grad-Reg and CDEP, which were discussed in Section 2. We omit comparison with Shao et al. (2021) because their code is not publicly available and is non-trivial to implement the influence-function based regularization.
Robustness-based methods. Avg-Ex, PGD-Ex, IBP-Ex along with combined robustness and regularization methods. IBP-Ex+Grad-Reg, PGD-Ex+Grad-Reg that are described in Section 3.
Since two of our datasets are highly skewed (Plant and ISIC), we report (macro-averaged over labels) accuracy denoted "Avg Acc" and worst accuracy "Wg Acc" over groups of examples with groups appropriately defined as described below. On Salient-Imagenet, we report using average accuracy and Relative Core Sensitivity (RCS) following the standard practice.
Wg Acc. Worst accuracy among groups where groups are appropriately defined. Different labels define the groups for decoy-MNIST and plant dataset, which therefore have ten and two groups respectively. In ISIC dataset, different groups are defined by the cross-product of label and presence or absence of the patch. We denote this metric as "Wg Acc", which is a standard metric when evaluating on datasets with shortcut features (Sagawa et al., 2019).
RCS proposed in Singla et al. (2022) was used to evaluate models on Salient-Imagenet. The metric measures the sensitivity of a model to noise in spurious and core regions. High RCS implies low dependence on irrelevant regions, and therefore desired. Please see Appendix E.2 for more details.
5.1.3 Training and Implementation details
Choice of the best model. We picked the best model using the held-out validation data. We then report the performance on test data averaged over three seeds corresponding to the best hyperparameter.
Network details. We use four-layer CNN followed by three-fully connected layers for binary classification on ISIC and plant dataset following the setting in Zhang et al. (2019), and three-fully connected layers for multi classification on decoy-MNIST dataset. Pretrained Resnet-18 is used as the backbone model for Salient-Imagenet.
More details about network architecture, datasets, data splits, computing specs, and hyperparameters can be found in Appendix E.
Dataset →	Decoy-MNIST	Plant	ISIC	Salient-Imagenet
Method↓	Avg Acc	Wg Acc	Avg Acc	Wg Acc	Avg Acc	Wg Acc	Avg Acc	RCS
ERM	15.1	10.5	71.3	54.8	77.3	55.9	96.4	47.9
G-DRO	64.1	28.1	74.2	58.0	66.6	58.5	-	-
Grad-Reg	72.5	46.2	72.4	68.2	76.4	60.2	88.3	52.5
CDEP	14.5	10.0	67.9	54.2	73.4	60.9	-	-
Avg-Ex	29.5	19.5	76.3	64.5	77.1	55.2	-	-
PGD-Ex	67.6	51.4	79.8	78.5	78.7	64.4	93.8	58.7
IBP-Ex	68.1	47.6	76.6	73.8	75.1	64.2	-	-
P+G	96.9	95.8	79.4	76.7	79.6	67.5	94.6	65.0
I+G	96.9	95.0	81.7	80.1	78.4	65.2	-	-
Table 1: Macro-averaged (Avg) accuracy and worst group (Wg) accuracy on (a) decoy-MNIST, (b) plant dataset, (c) ISIC dataset along with relative core sensitivity (RCS) metric on (d) Salient-Imagenet. Statistically significant numbers are marked in bold. Please see Table 4 in Appendix for standard deviations. I+G is short for IBP-Ex+Grad-Reg and P+G for PGD-Ex+Grad-Reg.
5.2 Decoy-MNIST
Decoy-MNIST dataset is inspired from MNIST-CIFAR dataset of Shah et al. (2020) where a very simple label-revealing color based feature (decoy) is juxtaposed with a more complex feature (MNIST image) as shown in Figure 1. We also randomly swap the position of decoy and MNIST parts, which makes ignoring the decoy part more challenging. We then validate and test on images where decoy part is set to correspond with random other label. Note that our setting is different from decoy-mnist of CDEP (Rieger et al., 2020), which is likely why we observed discrepancy in results reported in our paper and CDEP (Rieger et al., 2020). We further elaborate on these differences in Appendix G.
We make the following observations from Decoy-MNIST results presented in Table 1. ERM is only slightly better than a random classifier confirming the simplicity bias observed in the past (Shah et al., 2020). Grad-Reg, PGD-Ex and IBP-Ex perform comparably and better than ERM, but when combined (IBP-Ex+Grad-Reg,PGD-Ex+Grad-Reg) they far exceed their individual performances.
In order to understand the surprising gains when combining regularization and robustness methods, we draw insights from gradient explanations on images from train split for Grad-Reg and IBP-Ex. We looked at  $s_1 = \mathcal{M}\left[\left\| \mathbf{m}^{(n)} \times \frac{\partial f(\mathbf{x}^{(n)})}{\partial \mathbf{x}^{(n)}} \right\|\right]$  and  $s_2 = \mathcal{M}\left[\left\| \mathbf{m}^{(n)} \times \frac{\partial f(\mathbf{x}^{(n)})}{\partial \mathbf{x}^{(n)}} \right\| / \left\| (\mathbf{1} - \mathbf{m}^{(n)}) \times \frac{\partial f(\mathbf{x}^{(n)})}{\partial \mathbf{x}^{(n)}} \right\|\right]$ , where  $\mathcal{M}[\bullet]$  is median over all the examples and  $\| \cdot \|$  is  $\ell_2$ -norm. For an effective algorithm, we expect both  $s_1, s_2$  to be close to zero. However, the values of  $s_1, s_2$  is 2.3e-3, 0.26 for the best model fitted using Grad-Reg and 6.7, 0.05 for IBP-Ex. We observe that Grad-Reg has lower  $s_1$  while IBP-Ex has lower  $s_2$ , which shows that Grad-Reg is good at dampening the contribution of decoy part but also dampened contribution of non-decoy likely due to over-smoothing. IBP-Ex improves the contribution of the non-decoy part but did not fully dampen the decoy part likely because high dimensional space of irrelevant features, i.e. half the image is irrelevant and every pixel of irrelevant region is indicative of the label. When combined, IBP-Ex+Grad-Reg has low  $s_1, s_2$ , which explains the increased performance when they are combined.
5.3 Plant Phenotyping
Plant phenotyping is a real-world task of classifying images of a plant leaf as healthy or unhealthy. About half of leaf images are infected with a Cercospora Leaf Spot (CLS), which are the black spots on leaves as shown in the first image in the second row of Figure 2. Schramowski et al. (2020) discovered that standard models exploited unrelated features from the nutritional solution in the background in which the leaf is placed, thereby performing poorly when evaluated outside of the laboratory setting. Thus, we aim to regulate the model not to focus on the background of the leaf using binary specification masks indicating where the background is located. Due to lack of out-of-distribution test set, we evaluate with in-domain test images but with background pixels replaced by a constant pixel value, which is obtained by averaging over all pixels and images in the training set. We replace with an average pixel value in order to avoid any undesired confounding from shifts in pixel value distribution. In Appendix F.5, we present accuracy results when adding varying magnitude of noise to the background, which agree with observations we made in this section. More detailed analysis of the dataset can be found in Schramowski et al. (2020).
Table 1 contrasts different algorithms on the plant dataset. All the algorithms except CDEP improve over ERM, which is unsurprising given our test data construction; any algorithm that can divert focus from the background pixels can perform well. Wg accuracy of robustness (except Avg-Ex) and combined methods far exceed any other method by  $5 - 12\%$  over the next best baseline and by  $19 - 26\%$  accuracy point over ERM. Surprisingly, even Avg-Ex has significantly improved the performance over ERM likely because spurious features in the background are spiky or unstable, which vanishes with simple perturbations.
We visualize the interpretations of models obtained using SmoothGrad (Smilkov et al., 2017) trained with five different methods for three sample images from the train split in Figure 3. As expected, ERM has strong dependence on
Figure 3: Visual heatmap of salient features for different algorithms on three sample train images of Plant data. Importance score computed with SmoothGrad (Smilkov et al., 2017).
non-leaf background features. Although Grad-Reg features are all on the leaf, they appear to be localized to a small region on the leaf, which is likely due to over-smoothing effect of its loss. IBP-Ex, IBP-Ex+Grad-Reg on the other hand draws features from a wider region and has more diverse pattern of active pixels.
5.4 ISIC: Skin Cancer Detection
ISIC is a dataset of skin lesion images, which are to be classified cancerous or non-cancerous. Since half the non-cancerous images in the dataset contains a colorful patch as shown in Figure 2, standard
DNN models depend on the presence of a patch for classification while compromising the accuracy on non-cancerous images without a patch (Codella et al., 2019; Tschandl et al., 2018). We follow the standard setup and dataset released by Rieger et al. (2020), which include masks highlighting the patch. Notably, (Rieger et al., 2020) employed a different evaluation metric, F1, and a different architecture — a VGG model pre-trained on ImageNet. This may explain the worse-than-expected performance of CDEP from Rieger et al. (2020). More details are available in Appendix G.
Traditionally, three groups are identified in the dataset: non-cancerous images without patch (NCNP) and with patch (NCP), and cancerous images (C). In Table 2, we report on per-group accuracies for different algorithms. Detailed results with error bars are shown in Table 5 of Appendix F. The Wg accuracy (of Table 1) may not match with the worst of the average group accuracies in Table 2 because we report average of worst accuracies.
We now make the following observations. ERM performs the worst on the NPNC group confirming that predictions made by a standard model depend on the patch. The accuracy on the PNC group is high overall perhaps because PNC group images that are consistently at a lower scale (see middle column of Figure 2 for an example) are systematically more easier to classify even when the patch is not used for classification. Although human-explanations for this dataset, which only identifies the patch if present, do not full specify all spurious correlations, we still saw gains when learning from them. Grad-Reg and CDEP improved NPNC accuracy at the expense of C's accuracy while still performing relatively poor on Wg accuracy. Avg-Ex performed no better
Method	NPNC	PNC	C
ERM	55.9	96.5	79.6
Grad-Reg	67.1	99.0	63.2
CDEP	72.1	98.9	62.2
Avg-Ex	62.3	97.8	71.0
PGD-Ex	65.4	99.0	71.7
IBP-Ex	68.4	98.5	67.7
I+G	66.6	99.6	68.9
P+G	69.6	98.8	70.4
Table 2: Per-group accuracies on ISIC.
than ERM whereas PGD-Ex, IBP-Ex, IBP-Ex+Grad-Reg, and PGD-Ex+Grad-Reg significantly improved Wg accuracy over other baselines. The reduced accuracy gap between NPNC and C when using combined methods is indicative of reduced dependence on patch. We provide a detailed comparison of PGD-Ex and IBP-Ex in Appendix F.3
5.5 Salient-Imagenet
Salient-Imagenet (Singla & Feizi, 2021; Singla et al., 2022) is a large scale dataset based onImagenet with pixel-level annotations of irrelevant (spurious) and relevant (core) regions. Salient-Imagenet is an exemplary effort on how explanation masks can be obtained at scale. The dataset contains 232 classes and 52,521 images (with about 226 examples per class) along with their core and spurious masks. We made a smaller evaluation subset using six classes with around 600 examples. Each of the six classes contain at least one spurious feature identified by the annotator. Please refer to Appendix E.4 for more details. We then tested different methods for their effectiveness in learning a model that ignores the irrelevant region. We use a ResNet-18 model pretrained on ImageNet as the initialization.
In Table 1, we report RCS along with overall accuracy. Appendix F.2 contains further results including accuracy when noise is added to spurious (irrelevant) region. The results again substantiate the relative strength of robustness-methods over regularization-based. Furthermore, we highlight the considerable improvement offered by the novel combination of robustness-based and regularization-based methods.
5.6 Overall results
Among the regularization-based methods, Grad-Reg performed the best while also being simple and intuitive.
Robustness-based methods except Avg-Ex are consistently and effortlessly better or comparable to regularization-based methods on all the benchmarks with an improvement to Wg accuracy by  $3 - 10\%$  on the two real-world datasets. Combined methods are better than their constituents on all the datasets readily without much hyperparameter tuning.
In Appendix, we have additional experiment results; generality to new explanation methods (Appendix F.6) and new attention map-based architecture (Appendix F.7), and sensitivity to hyperparameters of PGD-Ex on Plant and ISIC dataset (Appendix F.4) and other perturbation-based methods on Decoy-MNIST dataset (Appendix F.8).
Sub-population shift robustness. Sagawa et al. (2019) popularized a problem setting to avoid learning known spurious correlation by exploiting group labels, which are labels that identify the group an example. Unlike MLX, the sub-population shift problem assumes that training data contains groups of examples with varying strengths of spurious correlations. For example in the ISIC dataset, images with and without patch make different groups. Broadly, different solutions attempt to learn a hypothesis that performs equally well on all the groups via importance up-weighting (Shimodaira (2000); Byrd & Lipton (2019)), balancing subgroups(Cui et al. (2019); Izmailov et al. (2022); Kirichenko et al. (2022)) or group-wise robust optimization (Sagawa et al. (2019)). Despite the similarities, MLX and sub-population shift problem setting have some critical differences. Subpopulation shift setting assumes sufficient representation of examples from groups with varying levels of spurious feature correlations. This assumption may not always be practical, e.g. in medical imaging, which is when MLX problem setting may be preferred.
Learning from human-provided explanations. Learning from explanations attempts to ensure that model predictions are "right for the right reasons" (Ross et al., 2017). Since gradient-based explanations employed by Ross et al. (2017) are known to be unfaithful (Murphy, 2023) (Chapter 33) (Wicker et al., 2022), subsequent works have proposed to replace the explanation method while the overall loss structure remained similar. Shao et al. (2021) proposed to regularize using an influence function, while Rieger et al. (2020) proposed to appropriately regularize respective contributions of relevant or irrelevant features to the classification probability through an explanation method proposed by Singh et al. (2018). In a slight departure from these methods, Selvaraju et al. (2019) used a loss objective that penalises ranking inconsistencies between human and the model's salient regions demonstrated for VQA applications. Stammer et al. (2021) argued for going beyond pixel-level importance scores to concept-level scores for the ease of human intervention. On the other hand, Singla et al. (2022) studied performance when augmenting with simple perturbations of irrelevant features and with the gradient regularization of Ross et al. (2017). This is the only work, to the best of our knowledge, that explored robustness to perturbations for learning from explanations.
Robust training. Adversarial examples were first popularized for neural networks by Szegedy et al. (2013), and have been a significant issue for machine learning models for at least a decade (Biggio & Roli, 2018). Local methods for computing adversarial attacks have been studied (Madry et al., 2017), but it is well known that adaptive attacks are stronger (i.e., more readily fool NNs) than general attack methods such as PGD (Tramer et al., 2020). Certification approaches on the other hand are guaranteed to be worse than any possible attack (Mirman et al., 2018), and training with certification approaches such as IBP have been found to provide state-of-the-art results in terms of provable robustness (Gowal et al., 2018), uncertainty (Wicker et al., 2021), explainability (Wicker et al., 2022), and fairness (Benussi et al., 2022).
By casting MLX as a robustness problem and using human explanations to specify the manifold of perturbations, we have shown that it is possible to alleviate the need for strong parameter smoothing of earlier approaches. Borrowing from the well-studied topic of robustness, we evaluated two strong approaches, one from adversarial robustness (PGD-Ex) and one from certified robustness (IBP-Ex). In our evaluation spanning seven methods and four datasets including two real-world datasets we found that PGD-Ex and IBP-Ex performed better than any previous approach, while our final proposal IBP-Ex+Grad-Reg and PGD-Ex+Grad-Reg of combining IBP-Ex and PGD-Ex with a light-weight interpretation based method respectively have consistently performed the best without compromising compute efficiency by much.
Limitations. Detecting and specifying irrelevant regions per-example by humans is a laborious and non-trivial task. Hence, it is interesting to see the effects of learning from incomplete explanations, which we leave for future work. Although robustness-based methods are very effective as we demonstrated, they can be computationally expensive (PGD-Ex) or scale poorly to large networks (IBP-Ex).
Broader Impact. Our work is a step toward reliable machine learning intended for improving ML for all. To the best of our knowledge, we do not foresee any negative societal impacts.
MW acknowledges support from Accenture. MW and AW acknowledge support from EPSRC grant EP/V056883/1. AW acknowledges support from a Turing AI Fellowship under grant EP/V025279/1, and the Leverhulme Trust via CFI. We thank Pingfan Song, Yongchao Huang, Usman Anwar and multiple anonymous reviewers for engaging in thoughtful discussions, which helped greatly improve the quality of our paper.
Benussi, E., Patane, A., Wicker, M., Laurenti, L., and Kwiatkowska, M. Individual fairness guarantees for neural networks. arXiv preprint arXiv:2205.05763, 2022.
Biggio, B. and Roli, F. Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition, 84:317-331, 2018.
Byrd, J. and Lipton, Z. What is the effect of importance weighting in deep learning? In International conference on machine learning, pp. 872-881. PMLR, 2019.
Codella, N., Rotemberg, V., Tschandl, P., Celebi, M. E., Dusza, S., Gutman, D., Helba, B., Kalloo, A., Liopyris, K., Marchetti, M., et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.03368, 2019.
contributors, W. Linearity of differentiation. Wikipedia, 10 2022. URL https://en.wikipedia.org/wiki/Linearity_of_differentiation.
Cui, Y., Jia, M., Lin, T.-Y., Song, Y., and Belongie, S. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9268-9277, 2019.
DeGrave, A. J., Janizek, J. D., and Lee, S.-I. Ai for radiographic Covid-19 detection selects shortcuts over signal. Nature Machine Intelligence, 3(7):610-619, 2021.
D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research, 2020.
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-673, 2020.
Gowal, S., Dvijotham, K., Stanforth, R., Bunel, R., Qin, C., Uesato, J., Arandjelovic, R., Mann, T., and Kohli, P. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arXiv:1810.12715, 2018.
Hennig, P., Osborne, M. A., and Kersting, H. P. Probabilistic Numerics: Computation as Machine Learning. Cambridge University Press, 2022. doi: 10.1017/9781316681411.
Izmailov, P., Kirichenko, P., Gruver, N., and Wilson, A. G. On feature learning in the presence of spurious correlations. Advances in Neural Information Processing Systems, 35:38516-38532, 2022.
Kirichenko, P., Izmailov, P., and Wilson, A. G. Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.02937, 2022.
Liu, E. Z., Haghgoo, B., Chen, A. S., Raghunathan, A., Koh, P. W., Sagawa, S., Liang, P., and Finn, C. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
Miao, K., Gokul, A., Singh, R., Petryk, S., Gonzalez, J., Keutzer, K., and Darrell, T. Prior knowledge-guided attention in self-supervised vision transformers. arXiv preprint arXiv:2209.03745, 2022.
Tschandl, P., Rosendahl, C., and Kittler, H. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
Wicker, M., Laurenti, L., Patane, A., Chen, Z., Zhang, Z., and Kwiatkowska, M. Bayesian inference with certifiable adversarial robustness. In International Conference on Artificial Intelligence and Statistics, pp. 2431-2439. PMLR, 2021.
Wicker, M., Heo, J., Costabello, L., and Weller, A. Robust explanation constraints for neural networks. arXiv preprint arXiv:2212.08507, 2022.
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., and Oermann, E. K. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine, 15(11):e1002683, 2018.
Zhang, H., Chen, H., Xiao, C., Gowal, S., Stanforth, R., Li, B., Boning, D., and Hsieh, C.-J. Towards stable and efficient training of verifiably robust neural networks. arXiv preprint arXiv:1906.06316, 2019.
Supporting material for "Use Perturbations when Learning from Explanations"
We restate the result of Theorem 1 for clarity.