Machine Learning from Explanations
Authors: Jiashu Tao
Reza Shokri
Venue: Submitted to ICLR 2023 (2023)
Keywords: model explanations
trustworthy machine learning
explainable ai
interpretable machine learning

Abstract: Machine learning needs a huge amount of (labeled) data, as otherwise it might not learn the right model for different sub-populations, or even worse, they might pick up spurious correlations in the training data leading to brittle prediction mechanisms.  Also, for small training datasets, there is a huge variability in the learned models on randomly sampled training datasets, which makes the whole process less reliable.  But, collection of large amount of useful representative data, and training on large datasets, are very costly.  In this paper, we present a technique to train reliable classification models on small datasets, assuming we have access to some simple explanations (e.g., subset of influential input features) on labeled data.  We also propose a novel two stage training pipeline that optimizes the model's output and fine-tunes its attention in an interleaving manner, to help the model to agree with the provided explanation while learning from the data. We show that our training pipeline enables faster convergence to better models, especially when there is a severe class imbalance in the population or spurious features in the training data.

Under review as a conference paper at ICLR 2023
MACHINE LEARNING FROM EXPLANATIONS
Anonymous authors
Paper under double-blind review
ABSTRACT
Machine learning needs a huge amount of (labeled) data, as otherwise it might not learn the right model for different sub-populations, even worse, they might pick up spurious correlations in the training data leading to brittle models. Also, for small training datasets, there is a huge variability in the learned models on randomly sampled training datasets, which makes the whole process less reliable. But, collection of large amount of useful representative data, and training on large datasets, are very costly. In this paper, we present a technique to train reliable classification models on small datasets, assuming we have access to some simple explanations (e.g., subset of important input features) on labeled data. We also propose a two stage training pipeline that optimizes the model's 's output and fine- tunes its attention in interleaving manner, to help the model to discover the reason behind the provided explanation while learning from the data. We show that our training pipeline enables faster convergence to better models, especially when there is a severe class imbalance or spurious features in the training data.
INTRODUCTION
Machine learning has an excellent performance in many challenging tasks (Dosovitskiy et al. 2020
Liu et al.
2021 Zhang et al. 2020), reaching or even outperforming humans (Silver et al. 2016
et al.
2022). in controlled experiments. However, their real-life performance is often drastically
worse, especially when there are natural spurious correlations in training data (Arjovsky et al. 2019
Ribeiro et al. 2016), and in so many cases where the training data sel is small, heterogeneous, or unbalanced, All this makes it hard for the existing learning algorithms to learn the right set of robust rules that can generalize well from training data. This is serious issue in many critical applications where machine learning promises to assist humans. For example, the training data used in training human-level medical Al models often contain spurious features, and models overly rely on spurious features to classify medical images, which results in untrustworthy diagnoses in practice (Rieger et al., 2020). To make it worse, models tend to extract more spurious reasons from the smaller sets, making models more useless for minority groups (Sagawa et al. 2020). If the model does not learn the true reason, it usually finds reasons favor the majority in the training set because it is empirical risk minimizer. Even if the model up- up-weights the minority group to prevent them from being left out, which is common practice in long-tailed classification, the model cannot classify minorities well on unseen data because the reasons extracted cannot be generalized. This discrepancy in group performance is also the base of the algorithmic (un)fairness problem in machine learning.
Getting more data seems to help to solve some obvious problems, with a significant cost (for obtaining a large amount of high-quality, accurately labeled data, and for training models on large datasets). However, it does not seem to necessarily help to learn stable and generalizable models. We observe that running the same training algorithms on the same datasets can yield models of similar accuracies that have learned drastically different decision functions and have low prediction agreements among themselves (Ross et al. 2017 D Amour et al., 2020 Watson et al. 2022). This comes as no surprise, given the digh-dimensionaliiy of input space and parameter space, for (classification) tasks with so few labels. To fully eliminate the ambiguity in parameter space, we need datasets of huge sizes, which prohibitively increases the cost of using machine learning for most real-world problems. Collecting more data is also not always possible. For example, for (rare) disease diagnoses, there are not many new patients every year to expand the datasets. In manufacturing industries, defect detection systems for new products also do not have much data available. This shows there is a need to develop better learning algorithms in small data regimes.

---

Under review as a conference paper at ICLR 2023
In this paper, we show that guiding machine learning with simple explanations can significantly improve its performance, reduce its sample complexity, and increase its stability. We assume that for some training data points, in addition to their label, we also have an expert explanation for the assigned labels. The explanation can have different degrees of complexity. However, in its simplest form, we assume that small subset of input features is highlighted contain the high-level reason for the assigned label. We propose an effective algorithm for machine learning with explanations, where we guide the model to identify the correct latent features most consistent with the explanations while optimizing the model's 's overall accuracy based on labels. Our approach significantly outperforms the baselines in convergence rate, accuracy, robustness to spurious correlations, and stability (with influences generalizability), especially for heterogeneous data. Under these settings, we show that baselines not only struggle to learn complex tasks, but also fail at properly learning the right reasons for classifying data for extremely simple tasks (e.g., detecting geometric shapes). Thus, we argue that it is necessary to incorporate explanations in learning algorithms if we aim at deploying trustworthy ML models that do not latch onto spurious counter-intuitive signals.
Some prior works have explored the idea of using prior knowledge to improve machine learning Ross et al. 2017 Rieger et al., 2020; Schramowwsii al. 2020 Shao et al., (021)) Although their objective is to be "right for the right reasons", these methods are actually penalizing models when they learn the wrong reasons. As we show in our analysis, this does not necessarily result in learning the right reasons, thus it has a limited advantage to learning only from labels. This is also fundamentally different from our proposed approach. Conceptually, providing the explanation for right reasons is much easier than enumerating all possible ways that the models might make mistakes, and doing so during the data collection phase. If there are known spurious correlations, a more straightforward and more effective solution (compared to penalizing the model on learning the spurious correlations) would be to remove them and train models on clean data (Friedrich et al. 2022)
2 LEARNING FROM EXPLANATIONS
2.1 PROBLEM STATEMENT
Given a labeled dataset D {(x, y):}. we also have access to explanations e(x) x, which are subset of the input features, of the label for each input point x. The explanations are informative enough that they can sufficiently explain the labels. We want to train a model to produce outputs similar to the given labels and base their decisions using reasons close to the given explanations.
2.2
WHY DO PREVIOUS METHODS NOT WORK WELL?
In this work, we focus on the image domain, where a wide range of model explanation methods have been studied. There are some prior works (Ross et al. 2017 Rieeer et al. [220 Schramowski ct al. 2020 Shao et al., 202T) on a similar problem to ours. Explanations in their settings are bounding boxes of either the main object or the spurious features, which differ from our definition of "informative and sufficient subsets of input features".. Technically, they all adopt a loss- based approach, adding an explanation misalignment loss to the label loss:
Cjoint
Clabel + ACeppl-
The additional loss is computed by taking the difference between the feature attributions attr(x) computed by certain model explanation methods and the given explanation e(x):
Geepl = ||attr(x) e(x)]l-
(2)
While it is tempting to reuse the existing algorithms for our problem, they do not lead to higher test accuracy than vanilla training in practice. The first reason is that using model explanation methods to generate models' attributions is questionable. [Adebyy et al. (2018) have shown that many popular saliency map based explanations do not even pass the sanity check. Using them as proxy of models' attention is thus unreliable. The second reason is with training. Optimization of the joint loss is often done via gradient descent. However, gradients of the two loss terms may point to different directions, creating a race condition that pulls and pushes the model into a bad local optimum. Imagine the two gradients counteract each other. The weights are then updated with negligible aggregated gradients.
2

---

Under review as a conference paper at ICLR 2023
This leads to models converging more slowly not getting updated in the worst case. Rieger et al. (2020) and Friedrich et al. (2022) have empirically verified that these methods often do not outperform the vanilla models traindd with labels only, and sometimes they strictly worse than vanilla models.
2.3 TUNING MODELS' ATTENTION ON LATENT FEATURES TO LEARN FROM THE EXPLANATION
To design a working algorithm that teaches the reason to models, the first step is to teach models to identify the predictive features highlighted by explanations. However, as input pixels are less semantically meaningful for image data, we need guide the model recognize important features in the latent space. Since explanations are sufficient, the latent representations of the given explanations should also be distinctive enough for the classifier. Hence, our core idea is to make the latent features extracted by our models more similar to the those of the explanations. We minimize the distributional difference between normalized latent features Xfeat and Xeau by optimizing the feature misalignment loss, which is the KL divergence between two normalized feature maps using softmax:
Lient (xeett: X
K LXLLxes
X
(3)
where N is the size of the latent features. There are two important design choices in this loss function. The first one is using normalized feature maps. Without normalization, latent features can be very large or very small, potentially resulting in exploding or vanishing gradients. Secondly, we use KL divergence as the loss criterion. The main reason that we want the distributions to more similar, which can imply that the model's focus is primarily the reasons region. Other loss functions may work empirically, but do not provide similar distributional intuition.
Since also need to make models accurate, we minimize the label loss, which is the cross entropy loss between the prediction y' and true label y:
(CEly/ y) = -y log(s') - (1 - y) log( -/).
To avoid creating the same race condition in prior work, we propose to optimize them sequentially. Our training algorithm alternates between minimizing the label loss and the feature misalignment loss. Algorithm [ describes our two-stage optimization.
Algorithm 1 Two-stage optimization
Require:
Input data x, model c f consists of feature extractor f. mapping layer m, and fully connected layers c, target y, explanation e(x), learning rates 7)1 and 172 for cross entropy loss and feature map loss
fCE f- -ylog((xx)) - (1 v) log(1 hxx))
2: On On
Ve, LCB
3: x' + x 2 e(x)
4: Xtrat t softmax(/
5: Xicat f- softmax(m) (f))))))
6: Licat + KLxx Xteat)
7: 8m + 6m 112 V Vo., Lieat
Fig [illustrates our two-stage training process. Given convolutional neural network (CNN) h. we first decompose it into two parts: (i) the feature extractor f. which contains all the convolutional layers, and (ii) the classifier c, which contains all the fully connected (FC) layers. We insert a mapping layer m in between f and c. The mapping layer is linear layer that maps features extracted by f back to the same feature space. It will be trained to filter out irrelevant information and amplify signals from the reasons region. This addition is introduced because we do not want to update the weights of the feature extractor in the second stage. Otherwise, would cause similar problems in prior work where two gradients counteract each other.
In the first stage, we do a normal forward pass and then backpropagate the label loss to update the entire model. In the second stage, we use the explanation e(x) as binary mask over the input image x to obtain the masked input x'. We can then pass the masked input to the feature extractor to obtain its feature map X softmax( fxx)). We then compute the feature map Xteat softmax (m (xx)) and minimize its difference with the reasons' feature map x/cat using Equation We subsequently update the weights of the mapping layer m by backpropagating the feature misalignment loss.
3

---

Under review as a conference paper at ICLR 2023
Convolutional Layers f
Mapping Layer m
Fully-comnected Layers
X
Output
softmax
softmax
x
X feat
Xfeat
Figure I: demonstration how our proposed training pipeline uses the explanation. Solid line: First stage of the optimization, forward pass, and backpropagation of the cross entropy loss in the same way as the conventional machine learning training pipeline. this example, an image x passes through the convolutional layers, which are the feature extractor, the self mapping layer, and the fully connected (FC) layers to reach the output. The cross entropy loss (or any other loss functions) computed, and the gradients are backpropagated to the weights. Dotted line: Second stage of the optimization, the new addition to the existing training pipeline that tunes the feature maps. In this diagram, the masked input x' based on the explanation is fed the convolutional layers, and feature map of Xteat will be obtained (after applying softmax). We want the feature maps Xieat fed to the FC layers to contain as much information as X/em Hence we introduce self mapping layer between the convolutional layers and the FC layers to learn mapping function that filters out irrelevant information and extracts predictive features from the feature map. We compute the loss between the two feature maps and back- pooppppop the loss the mapping layer.
3 EMPIRICAL ANALYSIS
We focus on binary classification with three datasets in our empirical analysis. Two of the datasets are synthetic geometric datasets that are easy for humans but challenging for machines, and one is real dataset that is difficult even for normal humans:
Triangle Orientation Dataset: If triangle is pointing upwards, it is assigned to be Class 1. Otherwise, Class O. The explanation rectangular area highlighting the spatial arrangement of the vertex and the bottom. (Details in Appendix A
Fox Cat Dataset: Foxes (Class 1) have triangular heads and cats (Class 0) have round heads. The explanation a square highlighting either the vertex or the are. (Details in Appendix A.2)
We select Indigo Buntings and Blue Grosbeaks from the CUB-200-201 dataset (Wah et al. 2011) to form our binary Bird dataset. The two species are visually iden- tical except for the sizes of their beaks. The explanation then small square highlighting the beaks. (Details in Appendix A.3
Bird Dataset:
We start our empirical analysis from simple geometric datasets where human can apply very simple rule to perfectly classify all images. However, such simple datasets are surprisingly challenging for machine learning models if they are trained with labels only. Not only do models fail to learn the simple rule from more than enough data, but they also often pick up wrong rules that do not generalize well. We have presented the intuition of our algorithm in the previous section, to empirically verify that models trained with reasons using our training pipeline are better. After observing how conventional ML training algorithms fail on the easy geometric synthetic dataset, naturally extend our analysis to the harder Bird dataset where domain knowledge or expert advice is needed explain the decision labels.
want
4

---

Under review as a conference paper at ICLR 2023
10 Samples
50 Samples
100 Samples Triangle
Classes are balanced
Classes are balanced
0.9
0.9
0.9
TestAccuraey
0.8
0.8
0.8
0.7
0.7
0.7
0.6
0.6
0.6
0.5
0.5
0.5
0
50
100
0
50
100
0
50
100
Epoch
Epoch
Epoch
10 Samples
50 Samples Class ratio 9
100 Samples
Class ratio 9
Class ratio 9
0.8
0.8
0.8
TestAccuracy
0.7
0.7
0.7
0.6
0.6
0.6
0.5
0.5
0.5
100
100
50
100
Epoch
Epoch
Epoch
No Expl
Grad Reg + Focal Loss
Ours
Ours(Sparse)
Figure 2: Test performance on the Triangle Orientation dataset. Top row: sizes with balanced classes. Bottom row: Training with small sample sizes when positive data only make up 10% of the training set.
Training with small sample
We want explore the following problems in empirical analysis:
1. Do models benefit from learning with explanations? (Sec 3.1
2. Do models learn the reasons suggested by the provided explanations? (Sec 3.2
3. Does training become more consistent we present explanations to models? (See 3.3
4. Can models still learn the suggested reason when there are multiple spurious reasons for the same decision? (Sec3.. 3.4)
3.1 Do MODELS BENEFIT FROM LEARNING WITH EXPLANATIONS?
We train 30 models with early stopping for all datasets and sample sizes under each training setup. For balanced datasets, we use the vanilla training algorithm that trains with labels only. We also use the gradient regularization (Grad Reg) method proposed by Ross et al. (2017) as a baseline. For imbalanced datasets, we add two more settings. Firstly, we replace the cross entropy loss with focal loss (Lin et al.] (017). popular loss function for imbalanced datasets, to train models with labels only. Then we consider sparse explanation setting where we only have explanations on the minority class. We train models with the normal cross entropy loss and tune their feature maps points with explanations. The implementation details and hyper- pappe--ersssters choices can be found in Appendix B
We observe from the top rows in Fig and and Fig 4 that training pipeline learning process and make final models generalize better on unseen data across all datasets when classes balanced in the training Even when there is severe class imbalance, where the
accelerate the
5

---

Under review as a conference paper at ICLR 2023
50 Samples
100 Samples
500 Samples
Classes are balanced
Classes are balanced
Classes are balanced
0.9
0.9
0.9
TestAccuracy
0.8
0.8
0.8
0.7
0.7
0.7
0.6
0.6
0.6
0.5
0.5
0.5
0
50
100
0
50
100
0
50
100
Epoch
Epoch
Epoch
50 Samples Class ratio
100 Samples
500 Samples Class ratio
Class ratio
0.7
0.7
0.8
TestAccuracy
0.65
0.65
0.7
0.6
0.6
0.6
0.55
0.55
0.5
0.5
0.5
50
100
50
100
0
50
100
Epoch
Epoch
Epoch
No Expl
Grad Reg
Focal Loss
Ours
Ours(Sparse)
Training with small sample sizes with balanced classes. Bottom row: Training with small sample sizes when positive data only make up 10% the training set.
Figure 3: Test performance on the Fox vs Cat dataset. Top row:
class ratio in the training set is 9 : 1, the bottom rows in Fig 2 and show our models can reach higher test accuracy balanced test set. We report the standard deviations over 30 trials in Table
S in the appendix, and we observe our models are more consistent with standard deviations one magnitude smaller.
30 Samples
60 Samples
0.8
0.8
TestAccuracy
0.7
0.7
0.6
0.6
0.5
0.5
0
100
200
0
100
200
Epoch
Epoch
No Expl
Grad Reg
Ours
Figure 4: Test performance the Bird dataset.

---

Under review as a conference paper at ICLR 2023
To save annotation costs, sometimes it makes to only query explanations data points from the under-represented groups. We simulate this realistic scenario by only training with explanations when the data are from the minority class (Class 1). The cyan lines in Fig 2 and demonstrate that our training strategy still outperforms others, even with sparse explanations. This is very sample
significant given the small sample sizes in experiments. For example, when size of 50, the additional information only explains five images. Nevertheless, our algorithm can significantly improve the generalization performance and convergence speed. The reason for our method's s usefulness in imbalanced settings without using popular re- re-weighting or sampling- based solutions shows that our models extract very useful features for minority groups under the guidance of explanations, useful that the classification layers can easily distinguish the two classes in the latent space.
In addition, Fig 3 and reveal that the existing algorithm from Ross et al. (2017) rarely outperforms the baseline, as the orange lines are often below the blue lines. This is aligned with our claim that existing methods do not work well in practice.
3.2 Do MODELS LEARN THE REASONS SUGGESTED BY THE PROVIDED EXPLANATIONS?
While we have empirically observed that presenting explanations to models helps (Section 3.1 we know if the superiority stems from the fact that models learn the suggested reasons better. To analyze the alignment between the suggested reasons and the reasons models learned, previous work (Ross al., 2017 Rieger al. 2020 Shao et 2021] Schramowski al.) 2020) relies mainly on model explanation methods to visualize saliency maps and inspecting the input features with high attribution values. However, many researchers the community have found out, many popular model explanation methods are bad proxies of models' behavior (Adebayo et al.) [2018) (hhhh et al.) 202I). Hence, to make our analysis more convincing, we construct additional test datasets that share the same classification rules the synthetic datasets:
Pentagon Orientation Dataset:
We construct this dataset by taking vertical slice from the middle of each triangle in the test set of the Triangle Orientation dataset. The vertical slice will be pentagon. If the triangle points upwards, the pentagon should also point upwards.
Triangle vs Circle Dataset: We construct this dataset by removing the rectangles in the test set of the Fox vs Cat dataset.
Table 1: Average test accuracy of models on test datasets sharing the same rules. For each row in the table, report the test dataset use, the training dataset, the size of the training dataset, the class ratio the training dataset, and the test accuracy of different models obtained under different training schemes.
Test set
Train set
Size
Class ratio
No expl
Grad
Ours
Ours(sparse)
1
0.587
0.616
0.772
50
1 1:9
0,561
0.549
0.618
0.580
1:1
0.623
0.606
0.813
100
1:9
0.573
0.574
0.657
0.642
Triangle vs Circle
Fox vs Cat
1:1
0.764
0.795
0.890
500
1:9
0,731
0.707
0,811
0.790
0.846
0.822
0.916
1000
1:9
0.834
0.824
0.825
0.824
1:1
0.561
0.565
0.710
10
1:9
0.537
0.555
0.721
0.555
1:1
0.713
0.596
0.833
Pentagon
Triangle
50
1:9
0.573
0.570
0.799
0.618
1:1
0.673
0.605
0.855
100
1:9
0.597
0,550
0.712
0.621

---

Under review as a conference paper at ICLR 2023
If models have learned the suggested reasons from the training datasets, they should be able apply the same reasons to the additional datasets and reach high accuracy. From the results listed in Table [ observe that our models almost always perform the best these datasets, suggesting have learned the most from the suggested reasons. Moreover, our approach outperforms the other training methods
models
in the sparse explanation setting. Hence, safely claim that our models generalize better and faster because they grasp the suggested reasons better.
3.3 DOES TRAINING BECOME MORE CONSISTENT IF WE PRESENT EXPLANATIONS TO MODELS?
Without guidance from the reasons, models have too much freedom in choosing their decision functions. This often results in huge variances and discrepancies in models trained even with the same data. Since we have shown our models learn the given reasons better, it is important to see they are more consistent with each other. We measure the pairwise agreement of final models by computing the percentage of identical predictions on test data. We then report the results in Table Our models always very consistent, implying they most probably the same reasons for predictions. Models trained with labels only become more consistent when dataset size increases, supporting our hypothesis that these models can slowly infer the true reasons the dataset is large enough.
Table 2: Average pairwise prediction agreement of 10 models when they trained the same dataset but with different strategies. Our models are always consistent, while models trained with labels only need large datasets become consistent.
Training dataset
Size
No expl
Grad Reg
Ours
50
0.560
0,600
0.927
Fox Cat
100
0.589
0.470
0.924
500
0.782
0.707
0.944
Triangle
50
0.711
0.669
0.949
Bird
60
0.584
0.753
0.870
3.4 CAN MODELS STILL LEARN THE SUGGESTED REASON WHEN THERE ARE MULTIPLE OR SPURIOUS REASONS FOR THE SAME DECISION?
To further demonstrate the usefulness of our method, we inject spurious features into training data. Appendix A.4 details how we introduce the spurious features. If models can learn the right reasons to make decisions, training on spurious features should minimally affect the test performance on clean test sets. On the other hand, if models learn the spurious reasons in the training set, they will fail to generalize on the clean test set. Fig 5 shows our models are consistently the best-performing ones across all datasets and sample sizes on the clean test set, suggesting our models have learned the most out of the true reasons even when there are strong spurious signals. More results are provided in Fig g and Table the appendix.
4 RELATED WORK
Previous work on making machine learning models learn desired reasons is mostly done via regular- ization with a given explanation. Most papers are from natural language processing (NLP), where the attention vector is natural source of feature attribution. Pruthi et al. (2022) trained student model to simulate the teacher model better using attention scores, Bao et al. (2018) utilized human attention on textual data to guide machine attention. Liu & Avei (2019) treated feature attribution scores as priors to force the language models to produce similar feature attributions. Besides attention, CChefer et al. (2022) regularized relevancy scores to shift transformers' focus to the foreground. However, some
8

---

Under review as a conference paper at ICLR 2023
TestAccuracy
0.8
0.7
0.5
Fox 50
Fox 100
Fox 500
Fox 1000
Triangle 10
Triangle 50
Triangle 100
Bird
No Expl
I Grad Reg
I Ours
Figure 5: Test accuracy the clean test when models trained on spurious training set. Our models consistently achieve the highest accuracy, implying that they have learned more from the suggested reasons.
methods may only be useful in NLP due to the distinct characteristic of attention- language models. Input pixels images are also less semantically meaningful than words in language data. To our best knowledge, Ross al. (2017) firstly explored the possibility of regularizing model explanation in training. However, their main point is that their algorithm can train diverse models focusing on different features with similar test accuracy. Du al. (2019) focused the alignment between model explanations and given attribution scores by optimizing the difference and enforcing sparsity. but the resulting test accuracy did not improve. Rieger al. (2020) considered another form of explanation and focused on avoiding learning spurious correlations. Similarly, (Shao et al. (202T) considered second-order derivatives as explanations and [Schramowski et al. (2020) used GradCAMs. However, these methods use the knowledge of "wrong reasons" explanations, which fundamentally different from our approach.
5 DISCUSSION
We have shown that we can improve the convergence rate and generalization performance of ma- chine learning models by inserting intermediate linear layer and performing additional back- propagation in each training iteration if we can query an expert for explanations. Our approach is efficient and effective, especially in small data regime where models trained with conventional pipelines struggle to extract useful information and learn correct patterns. We also consider the data,
usability of our approach with imbalanced data. While standard training methods require particularly from the minority class, to learn the under-represented class, our method substantially improves the classification performance for the minority class when explanations make it more realistic and reduce the annotation cost, we consider the situation where only data from the minority class has explanations. We demonstrate that models trained with our algorithm can still learn faster and better, even if the additional information is only the explanation of one or five data points.
available. To
CONCLUSION
In this paper, we study very important and useful machine learning setting: learning from expla- nations. We explain why previous papers fail to solve this problem and why previous loss- based methods do not bring substantial improvement. We propose novel two-stage optimization pipeline that teaches the model to extract features according to the given explanation. We demonstrate that our algorithm significantly outperforms the current training pipeline and previous loss-based methods in both synthetic and real datasets. We also show that our models can learn the suggested reasons better and apply the reasons to applicable alternative datasets. We think our method can naturally extend to multi-class classification, and leave future work. We hope that work can inspire more researchers to adopt this training pipeline such that more real world datasets with explanations be curated.

---

Under review as a conference paper at ICLR 2023
7 ETHICS STATEMENT
Our work does not have direct ethical concerns. Instead, it can benefit the under-represented groups.
REPRODUCIBILITY STATEMENT
We provide the source code our of algorithms in the submission. It comes with the default parameter values and random seeds that we use to run experiments. We also provide some details about implementation Appendix
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint
2019.
Yujia Bao, Shiyu Chang, Mo Yu, and Regina Barzilay. Deriving machine attention from human rationales. In EMNLP, 2018.
Hila Chefer, Idan Schwartz, and Lior Wolf. Optimizing relevance maps of vision transformers improves robustness. arXiv preprint arXiv:2206.01161, 2022.
Alexander A'mour Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew Hoffman, al. Underspecification presents challenges for credibility in modern machine learning. arXiv preprint arXhv:2011.03395. 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1999, 2020.
Mengnan Du, Ninghao Liu, Fan Yang, and Xia Hu. Learning credible deep neural networks with rationale regularization. In 20/9 IEEE International Conference on Data Mining (ICDM), pp. 150-159. IEEE, 2019.
Felix Friedrich, Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. A typology to explore and guide explanatory interactive machine learning. arXiv preprint arXiv:2203.0366 2022.
Yujia Li, David Choi, Junyoung Chung. Nate Kushman, Julian Schrittwesser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang. Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode, Feb 2022.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Frederick Liu and Besim Avei. Incorporating priors with feature attribution on text classification. In ACL, 2019.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. arXiv preprint arXiv:21 (0t::2/11.0900 1.09883, 2021.
10

---

Under review as a conference paper at ICLR 2023
Danish Pruthi, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary Chase Lipton, Graham Neubig, and William W. Cohen. Evaluating explanations: How much do explanations from the teacher aid students? Transactions of the Association for Computational Linguistics, 10: 359-375, 2022.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1 1135-1144, 2016.
Laura Rieger, Chandan Singh, William Murdoch, and Bin Yu. Interpretations are useful: penalizing explanations to align neural networks with prior knowledge. International conference on machine learning, pp. 8116-8126. PMLR, 2020.
Andrew Slavin Ross, Michael C Hughes, and Finale Doshi-Velez. Right for the right reasons: training differentiable models by constraining their explanations. In Proceedings of the 26th International Joint Conference Artificial Intelligence, pp. 2662 2270, 2017.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine Learning, pp. 8346-8356. PMLR, 2020.
Patrick Schramowski. Wolfgang Stammer, Stefano Teso, Anna Brugger, Xiaoting Shao, Hans-Gre Heons Luigs, Anne- Mahlein, and Kristian Kersting. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence, 476-486, 2020.
Harshay Shah, Prateek Jain, and Praneeth Netrapalli. Do input gradients highlight discriminative features? Advances in Neural Information Processing Systems, 14:2046-2059 2021.
Xiaoting Shao, Arseny Skryagin, Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. Right for better reasons: Training differentiable models by constraining their influence functions. In AAAI, 2021.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Joannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report ENS-TR-2011-001 California Institute of Technology, 2011.
Matthew Watson, Bashar Awwad Shiekh Hasan, and Noura Al Moubayed. Agree disagree: When deep learning models with identical architectures produce distinct explanations. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 875-884, 2022.
Yu Zhang, James Qin, Daniel Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoe V Le, and Yonghui Wu. Pushing the limits of semi-supervised learning for automatic speech recognition. arXiv preprint arXiv:2010.10500 2020.
A DATASETS
We use two synthetic geometric datasets and one real datasets that involves expert domain knowledge our paper.
A.1 TRIANGLE ORIENTATION
The first synthetic dataset is binary classification dataset on the orientation of triangle. A triangle is classified into class if is pointing upwards, otherwise The explanation is the spatial arrangement of the vertex and the bottom line: if the vertex is above the bottom line, it is pointing upwards. The explanation is then represented as a mask highlighting the vertex and a portion of the bottom line. Each image is of grayscale with size of 64.

---

Under review as a conference paper at ICLR 2023
A.2
FOX CAT
The second synthetic dataset a minimalistic fox vs cat dataset. A fox has triangular head and rectangular body, while cat has circular head and rectangular body. The explanation should then focus on their heads. However, an explanation does not have to highlight the entire head region. only needs highlight distinctive feature of the heads that sufficiently distinguishes fox from cat. Hence, the explanation for fox is mask highlighting the vertex of the triangular head, and the explanation for a mask highlighting the are of the circular head (See Fig 6). Each image is of grayscale with size of 64.
(a) Fox
(b) Fox with a mask high- lighting the vertex of its triangular head
(c) Cat
(d) Cat with mask high- lighting the are of its round head
Figure 6: Fox (Left) vs Cat (Right).
A.3 BIRD
We also extend our experiments to real datasets. We use subset of the CUB-200-2011 (We will refer the Bird dataset the of the paper) dataset (Wah al. 201T). fine ien-ggriieed classification dataset with photos of 200 species of birds. From the ground truth distribution of attributes for each individual class, we identify two species that are most similar to each other: the Indigo Bunting and the Blue Grosbeak (See 7a and and 7ch. If we refer to their most likely attributes from the ground truth distribution, they are indistinguishable as they share the identical set of attributes. However, according bird watcher website | Blue Grosbeaks have bigger and heavier beaks. Hence, the explanations for our Indigo Bunting vs Blue Grosbeak classification are masks highlighting the beaks if they visible (See Fig 7b arb 7db. We use the ground truth bounding box of birds to crop all images, before resizing them to the same size for training.
(a) An Indigo Bunting
(b) An Indigo Bunting with an explanation mask on its beak
(c) Blue Grosbeak
A Blue Grosheak with an explanation mask on its beak
Figure 7: Indigo Buntings (Left) vs Blue Grosbeaks (Right). The difference is that Blue Grosbeaks have larger breaks. The explanations hence highlight their beaks.
A.4 INJECTING SPURIOUS FEATURES
For the Fox Cat dataset, add spurious features fixed locations. For foxes, add square to the top left corner, for cats, we add a square to the top right corner For the Triangle Orientation
https:: htt::///wwwww...... sdakotabi rds cwm/d ffids
bunttakkkbbuinniii htm
12

---

Under review as a conference paper at ICLR 2023
dataset, we add square to random location the leftmost column for positive data. We add the square to a random location on the rightmost column for negative data. For the CUB bird dataset, we add green square to random place to the negative class, and add red square to random place to the positive class. We make sure that these spurious features do not block the reasons masks. However, they can be arbitrarily close to the reasons.
(a) Spurious cat
(b) Spurious down
triangle
pointing
(c) Spurious Indigo Bunting
(d) Spurious fox
(e) Spurious triangle pointing up
(l) Spurious Blue Grosbeak
Figure 8: Top row: images with spurious features injected from the negative class. Bottom row: images with spurious features injected from the positive class. Left: square is added to the top right corner for cat images and the top left corner for fox images. Mid: a square is added to the top left corner for triangles pointing down and the bottom left corner for triangles pointing up. Right: green square is added to the lower right corner for indigo buntings, and a red square is added to the lower left corner for blue grosbeaks.
B IMPLEMENTATION
B. SYNTHETIC DATASETS
For the two synthetic datasets, we CNN with convolutional layers and 1 FC layer. We Adam with learning rate of 0.01 when we train with no explanations. When we add the gradient misalignment loss the joint loss function, we optimize the joint loss with Adam with the same learning rate mentioned above. We set the balancing coefficient À of the gradient misalignment loss be 0.001 in equation [] which keeps the two loss terms similar magnitude to ensure maximum performance. This choice of hyper-parameter is mentioned by Ross et al. (2017) and also observed by us in typer-parameter tuning. When using our method, we empirically evaluated the model with three learning rates for both 7) and 7/2: 10.1, 0.01, 0.001 J. We then pick the best combination. Note that we did not tune these better with more tuning. In summary, both optimizers are Adam with learning rates of 0.001 when the class ratio is balanced with no spurious correlations. When there is a class imbalance or spurious correlation in data, we increase the learning rate 7/2 of the feature loss optimizer to 0.01. The
to great depth, so it is very likely our model can perform
13

---

Under review as a conference paper at ICLR 2023
empirical evidence that 7 needs to be increased in these two scenarios is intuitive, because models need to update their focus more to generalize well to class balanced test set.
B.2 BIRD DATASET
For the bird dataset, we use simple CNN with 2 convolutional layers and FC layer. This is simpler than the model used in synthetic datasets because the size of training set is smaller for this dataset. To reduce the chance of overfitting, we make the model simpler. We use SGD as the optimizer for this dataset. For the baseline method without explanations, the learning rate is 0.01. For GradReg, the balancing coefficient is computed make the two loss terms of similar magnitudes to ensure best performance. Similar to the combination of 7Ji and 7/2 from empirical results. Across all settings, 7/1 is always set to be 0.001. When the class ratio is balanced with spurious correlations, set 7)2 to be 0.001. On the other hand, when there is a class imbalance or spurious correlation data, we increase the learning rate 722 of the feature loss optimizer to 0.01.
selection procedure explained above, pick the best
C ABLATION STUDIES
There are two novel designs in our algorithm:
1. The mapping layer
2. Two-stage optimization
We conduct ablation studies to see any new designs bring significant improvement. Table 3 shows that the combination of the two new designs yields the best final performance and fastest convergence rate, especially on the Bird dataset. With the mapping layer, the feature difference loss is only backpropagated to the mapping layer, otherwise it will be backpropagated to all convolutional layers. As argued before, updating the convolutional layers twice with different gradients may be less effective, and the gradients might counteract each other. On the other hand, doing joint training is conceptually inferior to our two-stage training. The only difference would be that the feature difference loss in the joint optimization scheme will be computed before the feature extractor is updated by the label loss. This is less ideal because the gradient of feature difference loss might be outdated compared the gradient in the two-stage training scheme, so the mapping layer will be less tuned. This inefficiency results in the slower convergence rate displayed in Table 3
D COST ANALYSIS
In this section, we analyze the benefit of training with explanations from the cost perspective. First of all, Table shows that our method leads to lower sample complexity as the dataset size needed to reach a certain performance threshold is much smaller than that if using labels only. The table also shows that our method can achieve at least a 3x reduction sample sizes, sometimes as much as 10x if the sample size is very small. It takes an Amazon Turk worker 1.2 cents to label an image, and 3.6 cents to provide bounding boxes. The explanation used in our method similar to bounding boxes where an annotator is only required to highlight an important region in the image. Hence the cost to obtain explanations for each image would be approximately 3.6 cents, three times the cost of labels. However, considering how much reduction in dataset sizes our methods can achieve, believe it is more cost-efficient to query for both labels and explanations than simply collecting more data to train models if model creators start from set of unlabeled data.
E ADDITIONAL RESULTS
We present some additional empirical results in this section.
Numbers taken from http://taaeesse...o.oooco//o///a
14

---

Under review as a conference paper at ICLR 2023
Table 3: Model performance with standard deviation of two-stage training with mapping layer, two-stage training without mapping layer, and joint training with mapping layer. The numbers are computed 20 epochs, 40 epochs and convergence synthetic datasets, and 20 epochs, 80 epochs and convergence on Bird datasets because models take longer time to converge.
Dataset
Size
Ours
No Mapping Layer
Joint Opt
0.704 + 0.051
0.567 0.089
0.632 = 0.099
50
0.751 + 0.029
0.686 + 0.076
0.749 + 0.051
0.750 + 0.029
0.721 0.062
0.749 0.051
0.699 + 0.095
0.599 0.101
0.683 = 0.105
100
0.799 +: 0.019
0.691 + 0.094
0.783 = 0.071
Fox vs Cat
0.799 + 0.019
0.733 0.078
0.788 = 0.072
0.801 + 0.080
0.780 + 0.081
0.605 + 0.155
500
0.854 + 0.030
0.801 + 0.067
0.760 + 0.159
0.874 + 0.012
0.803 + 0.108
0.796 = 0.135
0.738 + 0.158
0.698 + 0.155
0.664 = 0.160
1000
0.860 + 0.093
0.797 0.167
0.756 0.167
0.903 + 0.017
0.909 + 0.016
0.834 = 0.034
0.682 + 0.110
0.681 + 0.121
0.557 = 0.089
10
0.732 + 0.082
0.703 0.124
0.627 + 0.112
0.732 + 0.082
0.703 0.124
0.627 + 0.112
Triangle
0.766 + 0.100
0.552 0.077
0.753 = 0.111
50
0.836 + 0.026
0.774 + 0.088
0.827 = 0.060
0.839 + 0.028
0.796 0.052
0.848 0.041
0.730 0.115
0.577 0.106
0.731 + 0.113
100
0.843 + 0.043
0.778 + 0.089
0.843 + 0.051
0.874 + 0.018
0.801 0.076
0.871 = 0.044
0.562 + 0.088
0.495 + 0.053
0.500 - 0.007
30
0.672 + 0.101
0.530 + 0.091
0.515 = 0.024
0.777 + 0.044
0.522 + 0.038
0.545 = 0.057
Bird
0.734 + 0.058
0.570 + 0.087
0.515 + 0.039
60
0.797 + 0.085
0.692 0.069
0.613 0.089
0.860 + 0.079
0.783 + 0.022
0.772 = 0.039
15

---

Under review as a conference paper at ICLR 2023
Table 4: Test performance of models trained with explanations and method under different training set sizes.
Dataset
Size
No expl
Ours
50
0.594
0.750
100
0.621
0.799
300
0.707
0.848
Fox Cat
500
0.754
0.874
1000
0.837
0,903
1500
0.881
0.920
10
0.595
0.732
50
0.726
0.839
Triangle
100
0,687
0.874
200
0.827
0.900
500
0.863
0.913
0.8
TestAccuracy
0.6
0.5
Fox 50
Fox 100
Fox 500
Fox 1000
Triangle 10
Triangle 50
Triangle 100
I Expl
DrGrd Reg
I
Figure 9: Test accuracy on test sets sharing the same true reasons when models are trained on spurious training set. Our models always do better, suggesting they have learned more of the true reasons.
16

---

Under review as a conference paper at ICLR 2023
Table 5: Black numbers: Final models performance on clean test sets when they are trained under different settings. Red numbers: Final model performance on additional datasets sharing the same reasons with the training datasets when models are trained with balanced datasets. Blue numbers: Final model performance balanced testsets when the class ratio in the training set is 1 9. Numbers in bold are best performances in each setting.
Dataset
Size
No expl
Grad Reg
Ours
0.594 0.119
0.610 + 0.110
0.750 0.029
50
0.587 = 0.120
0.616 + 0.122
0.772 + 0.037
0.552 = 0.086
0.548 0.084
0.607 + 0.065
0.621 = 0.119
0.601 0.123
0.799 + 0.019
100
0.623 = 0.136
0.606 + 0.133
0.813 + 0.033 0.635 0.069
Fox vs Cat
0.559 = 0.094
0.568 0.110
0.754 = 0. 150
0.785 0.127
0.874
500
0.764 = 0.163
0.795 + 0.134
0.725 = 0.142
0.698 0.145
1.806
0.837 = 0.116
0.815 + 0.070
0.903 0.017
1000
0.846 = 0.122
0.822 + 0.083
4.916
0.822 = 0.081
0.819 + 0.034
0
0.595 = 0.118
0.576 + 0.124
0.732 + 0.082
10
0.562 = 0.092
0.565 0.114
0.710 + 0.104
0.550 0.093
0.570 + 0.120
0.735 + 0.101
Triangle
0.726 = 0.154
0.649 + 0.134
0.839 + 0.028
50
0.713 = 0.158
0.596 + 0.122
0.833 + 0.040
0.595 0.137
0.594 + 0.124
0.803 + 0.070
0.687 = 0.175
0.643 0.148
0.874
100
0.673 = 0.176
0.605 + 0.137
0.855 + 0.034
0.612 = 0.142
0.567 0.129
0.752 4
30
0.627 = 0.128
0.673 0.121
0.777 + 0.044
Bird
60
0.684 + 0.186
0.749 0.177
0.860 + 0.079
17

---

Under review as a conference paper at ICLR 2023
Table Black numbers: Models performance on clean test sets when they are trained under different settings on datasets with spurious features. Red numbers: Model performance on additional datasets sharing the same reasons with the training datasets. Results show that models trained with our proposed setup perform the best across all three datasets and different dataset sizes, which means our models learn the most from the given reasons in the presence of strong spurious correlations.
Spurious dataset
Size
No expl
Grad Reg
Ours
0.620
0.573
0.729
50
0.642
0.563
0.760
0.570
0.601
0.727
100
0.601
0.613
0.751
Fox vs Cat
0.611
0.561
0.698
500
0.612
0.571
0.708
0.601
0.605
0.728
1000
0.558
0.618
0.693
0.697
0.677
0.726
10
0.698
0.691
0.711
0.694
0.600
0.796
Triangle
50
0.671
0.574
0.776
0.606
0.575
0.784
100
0.603
0.555
0.780
Bird
60
0.698
0.733
0.878
18


Reviews:
[{"review_id": "qoBrtxbDlLS", "reviewer": "Reviewer_pQVm", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to train more accurate image classification models by leveraging human explanations in the form of input masks to increase test accuracy. While the pitched idea is promising, this work is fundamentally flawed in a number of ways and needs to be completely rethought.", "summary_of_the_review": "The experimental results have several major flaws (as denoted above), and for this reason I believe there are significant structural changes that need to be made in order to make this a valuable contribution.", "strength_and_weaknesses": "Strengths:\n- The proposed setting of leveraging human explanations in order to increase performance is interesting.\n\nWeaknesses:\n- The main issue with this work is that the evaluation setup is not realistic at all. For an experimental paper like this, verifying its applicability on real-world datasets is important. Yet, 2 datasets are synthetically generated and only 1 is of real birds. This birds dataset, too, is very simple, in that the feature is very easily identifiable (the beak), and it is not clear if this method scales to more realistic distributions where the features are not as simple.\n- Another huge issue is that experiments are only conducted at the extremely small-sample regime, up to 500 samples on the synthetic datasets of shapes and up to 60 examples on the bird dataset. No one is deploying machine learning trained on 60 samples. If the method was to train on all labeled data, and only incorporate some additional explanations, then that would be much more reasonable. But that is not what is happening here.\n\nAdvice:\n- The idea of leveraging a few human annotations to increase performance is interesting, but the rest of the paper needs to be completely reworked. Here's what a great version of this paper would look like:\n- Consider a suite of real-world datasets, such as those in the WILDS benchmark. Do not include any synthetic data experiments (they add no value) and report performance on the specific metric for each dataset. Another benefit of this is that experiments are run on non-binary tasks as well.\n- Train on *all* available labeled data. The WILDS dataset contains training data splits. You should compare two main methods primarily: 1) the baseline of training on the labeled data, and 2) the new method of training on the labeled data, plus incorporating input mask explanation annotations for a few (say, 60) examples.\n- Use modern backbone baselines (say, Resnet50 or DenseNet121) for the feature extraction layer - 3 conv layers is definitely too small for anything non-synthetic.\n- I have to say that even given this version of the idea, I am skeptical this would work (lots of such robustness/domain invariance interventions have been proposed and have failed). But this is just my opinion, my advice, and the rest of this review is independent of this viewpoint.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "iChQr07oYe-", "reviewer": "Reviewer_ZDog", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors address the important topic explanations value for improving deep learning performance.\nSpecifically, the authors examine the problem of visual binary classification, where explanations in the form of attention/masking (carefully selected sunset of input pixels) are made available to the model while training.\n\nThe explanations are used by minimizing the kl divergence between the normalized explanation features and the normalized full image features passed through a linear filter/mapping layer constructed prior to the classification head of simple convolution architectures.\n\nThe relative performance of this scheme is evaluated on two synthetic and one natural (birds derived) data sets, under three scenarios:\n\n1. The authors demonstrate that within the range of evaluation — up to few hundred samples  in the synthetic datasets and several tens in the birds dataset — there is a generalization benefit throughout the learning curve when training with the explanations. The evaluation is conducted both in balanced and unbalanced settings with explanations given to the minority class.\n\n2. The authors examine the generalization on a modified dataset than trained on, but which shares the explanation classification rule — as a basis of evaluating if the models learned the ‘suggested rules provided by the explanation’.\n\n3. The authors examine the effect of adding disturbances to the image which do not overlap with the explanation.\n", "summary_of_the_review": "Utilizing human insight to improve model performance and/or cost (direct or indirect by way of data curation) is a very important topic, with strong ties to explainability and fairness as well.\n\nThis paper showcases simple visual cases where performance can be improved at the small scale.\n\nHowever it does not show how these results can carry over either from theoretical ground or empirical demonstration into the practical setting of real tasks and scale, nor does it address the core question of ‘how better can one perform with explanations?’ (from a cost / data / performance scaling perspectives)\n\nSignificant work is required for addressing these issues, and I will be happy to upgrade my review if they are adequately resolved.\n", "strength_and_weaknesses": "$\\bf{Strengths:}$ \n\nThe notion of using explanations in order to improve training is a very important one. \nThe authors offer a filtration layer and optimization method which allows assessing agreement between feature representations (which are not known in advance) by using characteristics which are assumed to be known as important for the labeling — hence explanations.\n\n$\\bf{Weaknesses:}$\n\nThis paper has several structural and generalizability weakness:\n\nThe datasets are extremely simple/limited. In itself that is not a problem (and can actually be a virtue!) if they were to provide theoretical insight (which is not demonstrated in the paper)  *or* empirical insight that is evaluated and carries over also in practical settings of interest.\n\nHowever both are limited. And it is not demonstrated how these results can be generalized.\nCan the authors show how they improve performance by providing explanations for (e.g. improving mis-classifed samples) cifar10/100 (at the least) and imagenet ?\n\nExperimental rigor — the baseline no-explanation deteriorates with increasing number of samples (figure 2). What does that mean about the confidence in proper optimization of the experiments. Why is this happening? Is it overfitting? What happens with more samples where we know that performance of vanilla no-explanation should scale and improve.\n\nConceptual applicability — evaluated only in the small data regime. Is performance quickly surpassed in practice by no-explanation training with sufficient data? Then the practical utility is unclear. \nTable 1 is showing that this may be the case?\n\nFurther, providing explanations in practice may be very costly. How much is an explanation worth — e.g. in terms of equivalent amount of unexplained data resulting in the same performance? How would this scale ?\n\nIt would be very exciting to show that few explanations can be used to significantly improve performance of large models. But this paper offers no insight into this practical domain of improving actual performance or showing why, in principle, it can be improved in large scale settings.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "4oo30yNSF7u", "reviewer": "Reviewer_YcFy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "ML models are known to latch to incidental correlations in the training data, which emphasises the need for controlling what they learn. This paper proposes a learning algorithm that extracts features for classification that can only be revealed from a specified region of pixels (explanation). For example, the explanation may identify the beak of a bird if we wish to learn a bird classifier that discriminated only based on beak shape/colour etc.  ", "summary_of_the_review": "I agree that we should focus on a handful of “right reasons” instead of ignoring innumerable “wrong reasons”. The paper does a good job of proposing a method and showing that it is better than GradReg, but I have some conceptual and experiment related concerns that should be addressed.\n", "strength_and_weaknesses": "Strengths.\n1. Well motivated problem. \n2. Presentation is mostly clear. \n3. Experimental evaluation is mostly through and convincing. I particularly like the imbalanced settings and evaluating if we can get away by providing explanations only for the minority class. \n\n\nWeakness/questions.\n1. There are mistakes in the method section that hindered understanding. Training is a two-step procedure, in the first step the parameters of h: (f, c) are updated and in the second step the parameters of mapper (m) are updated. If the outputs only depend on f, c (and not m), how does updating of m parameters make any difference? Cold be an error in the presentation here.  \n2. How are lambda1, lambda2 or any other parameters tuned? Especially for the imbalanced settings?  \n3. (minor) Line 2 and 8 of Alg 1 should have subtract rather than add for gradient updates. The mapping layer in Figure 1 is not consistent with writing. Does the mapping layer affect outputs at all? \n4. To justify the need for a mapping layer, ablation experiments without the mapping layer should have been reported.\n5. Authors report their method’s superiority on imbalanced datasets, but the algorithm is not designed to handle such settings, it is merely an accident if it so. In any case, some justification for why their method can handle imbalanced settings is expected. \n6. Std deviation should be reported. Grad-Reg on Fox vs Cat with size 100 in Table 2 has inconsistent performance, which made me wonder if there is large variation in numbers. \n7. Conceptually, I do not see why original and masked (through explanation) inputs should have the same representation. For example, when the explanation is the beak of a bird, then in order to extract the features of a beak we need to first identify that it’s a beak, which then requires spatial orientation with respect to other features of a bird. However, in the masked image, we may not have sufficient information to identify a beak as a beak and therefore may draw only poor feature representation. For this reason, I think it is repressive to constrain the masked and unmasked image representations as being alike. \n8. Presence of self-sufficient explanations localized to only small regions seems like a strong assumption to me. To prove otherwise, authors should work with real-world datasets that can be explained this way. The toy experiments and bird classification experiments are not very convincing. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "0VJEKVuBvI", "reviewer": "Reviewer_C28x", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an approach to augment a ML model's training process when ground truth explanations (in this case, subsets of relevant features) are available. The approach is based on encouraging a model's learned representation to be similar with the full input and with the subset of relevant features.", "summary_of_the_review": "The paper develops a method for training models when ground truth explanations are available. My recommendation is marginally below acceptance, but I could be swayed if the authors include the requested ablation experiments and compare with a stronger set of baselines. ", "strength_and_weaknesses": "--- Strengths ---\n\n- Improving a model's training when ground truth explanations are available is a good idea. In cases where we don't have many training examples but we know which parts of the input is important, it would be ideal if we could improve the sample complexity for learning accurate models.\n- The proposed approach seems straightforward to implement. It simply involves augmenting the standard cross-entropy loss (for encouraging accurate predictions) with a secondary loss that captures the internal representation's similar with all inputs and with the subset of relevant inputs.\n\n--- Weaknesses ---\n\nHigh-level issues\n- The method only works when we have access to ground truth set of important features - an uncommon situation in practice.\n- The evaluation focuses primarily on toy datasets.\n- The method has several strange and seemingly unjustified design choices (see below).\n\nThere are a couple choices in the method's design that don't make sense to me, it would be helpful if the authors could explain these:\n- Why is $\\mathcal{L}_{\\text{feat}}$ a KL divergence penalty based on normalized representations, rather than something simpler like a squared error loss? How would we normalize if the activation functions don't produce non-negative results, e.g., if we used ELU or GELU activations? A squared error loss seems more universally applicable, did the authors test this?\n- The argument that the classifier and mapping layer can't be jointly optimized doesn't make much sense - people train networks with competing objectives all the time and the networks don't fail to train. Did the authors conduct ablation experiments to test whether performing joint rather than sequential training leads to meaningfully different results?\n- Figure 1 seems to indicate that the classifier $h$ uses the mapping layer $m$, but this is somewhat unclear in Algorithm 1 - can the authors clarify? And it's strange that we compare the representation from $f$ with the representation from $m$, can the authors explain why we need $m$? Aside from a theoretical justification, did the authors conduct ablation experiments to test whether removing $m$ altogether leads to meaningfully different results?\n- I follow how the method works, and I agree that it constitutes \"learning from explanations,\" but I don't see what it has to do with \"being right for the right reasons.\" The model is basically encouraged to have the same internal representation with or without the unimportant features. But the reasons for the model's prediction are not known (there's no explanation of the model's prediction) and are not explicitly encouraged to resemble the correct reasons. Wouldn't it make more sense to describe the approach slightly differently, perhaps as encouraging the model to automatically identify and disregard irrelevant input information? What are the authors' thoughts on this? \n\nAbout the experimental comparisons:\n- Among the baselines, only the GradReg one is in the same category as the proposed method (focal loss has nothing to do with leveraging knowledge about important features). Why did the authors not include the methods from Rieger et al (2020), Schramowski et al (2020) or Shao et al (2021)? There are two others that also could have been included mentioned below (Erion et al, 2021 and Chefer et al, 2022).\n- The results from different methods presumably converge when given large enough training datasets. Could the authors provide results showing how large the training sets must be, either in the main text or supplement? E.g., there could be a plot showing the peak accuracy (or accuracy after 100 epochs) given different training dataset sizes.\n\nAbout prior work:\n- The claim that \"most popular model explanations do not even pass the sanity check\" in Adebayo et al (2018) seems like a bit of an exaggeration. The methods considered in that paper are a subset of gradient-based methods, but what about other popular methods like RISE, LIME, or SHAP? It's safe to say that there are many explanation methods that don't work as well as desired, but this claim should be dialed back.\n- The authors present a hypothetical argument to explain the issue of training with a joint loss function accounting for prediction and explanation accuracy: \"However, gradients of the two loss terms may point to different directions, creating a race condition that pulls and pushes the model into a bad local optimum. Imagine the two gradients counteract each other. The weights are then updated with negligible aggregated gradients.\" Many ML/DL methods involve training with multiple objectives and work just fine, why would it be especially problematic here? It seems correct that previous methods in this area don't work that well, but is there any evidence that this is the reason why? And even if it were, couldn't it be mitigated by pre-training with the prediction accuracy loss only and then turning on the explanation loss? This relates to one of my requests above for an ablation experiment.\n- Referring to prior works that penalize explanations, the authors write: \"Explanations in their settings are bounding boxes of either the main object or the spurious features, which differ from our definition of “informative and sufficient subsets of input features”.\" Actually, bounding boxes around the main object sound very similar to “informative and sufficient subsets of input features.” What's the significant difference? I suppose a segmentation mask would be more precise than a bounding box, but that's not a huge difference. Also, note that the authors actually use a bounding box around the beak in their one non-toy dataset.\n- Referring to the same set of prior works, the authors write: \"Although their objective is to be “right for the right reasons”, these methods are actually penalizing models when they learn the wrong reasons.\" What's the significant difference? Would you also argue that cross-entropy loss doesn't encourage classifiers to make the right predictions, they just penalize them for making the wrong predictions? This is not a well thought out criticism by the authors, but I suppose my bigger point is: I agree that there are differences between this work and prior work, but this is not helpful in clarifying what that difference is. The difference is that prior methods (at least those I'm aware of) penalize the explanation of the model being trained, whereas this method penalizes the internal representation in a manner that doesn't require generating any form of explanation. \n- Two related works belonging to the Ross et al. 2017 category are \"Improving performance of deep learning models with axiomatic attribution priors and expected gradients\" by Erion et al (2021) and \"Optimizing relevance maps of vision transformers improves robustness\" by Chefer et al (2022).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject