Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper proposes a method to train more accurate image classification models by leveraging human explanations in the form of input masks to increase test accuracy. While the pitched idea is promising, this work is fundamentally flawed in a number of ways and needs to be completely rethought.
strength_and_weaknesses: Strengths:
- The proposed setting of leveraging human explanations in order to increase performance is interesting.

Weaknesses:
- The main issue with this work is that the evaluation setup is not realistic at all. For an experimental paper like this, verifying its applicability on real-world datasets is important. Yet, 2 datasets are synthetically generated and only 1 is of real birds. This birds dataset, too, is very simple, in that the feature is very easily identifiable (the beak), and it is not clear if this method scales to more realistic distributions where the features are not as simple.
- Another huge issue is that experiments are only conducted at the extremely small-sample regime, up to 500 samples on the synthetic datasets of shapes and up to 60 examples on the bird dataset. No one is deploying machine learning trained on 60 samples. If the method was to train on all labeled data, and only incorporate some additional explanations, then that would be much more reasonable. But that is not what is happening here.

Advice:
- The idea of leveraging a few human annotations to increase performance is interesting, but the rest of the paper needs to be completely reworked. Here's what a great version of this paper would look like:
- Consider a suite of real-world datasets, such as those in the WILDS benchmark. Do not include any synthetic data experiments (they add no value) and report performance on the specific metric for each dataset. Another benefit of this is that experiments are run on non-binary tasks as well.
- Train on *all* available labeled data. The WILDS dataset contains training data splits. You should compare two main methods primarily: 1) the baseline of training on the labeled data, and 2) the new method of training on the labeled data, plus incorporating input mask explanation annotations for a few (say, 60) examples.
- Use modern backbone baselines (say, Resnet50 or DenseNet121) for the feature extraction layer - 3 conv layers is definitely too small for anything non-synthetic.
- I have to say that even given this version of the idea, I am skeptical this would work (lots of such robustness/domain invariance interventions have been proposed and have failed). But this is just my opinion, my advice, and the rest of this review is independent of this viewpoint.
clarity,_quality,_novelty_and_reproducibility: The paper makes a number of unsubstantiated claims with exaggerated language throughout. To give an example:
- "we argue that it is necessary to incorporate explanations in learning algorithms, if we aim at using machine learning in real-world scenarios"
- "we need datasets of astronomical sizes"

These claims are littered all over and need cites or need to be removed.

Code seems to be provided (but no README on how to interact with it), so there is partial reproducibility.

The paper was often unclear, specifically:
- specify that the explanations e(x) are binary masks much before the bottom of page 3.
- in the network (Figure 1), which layers are frozen and which layers are trained?
- equation 3 - are you computing the KL divergence between two *probability distributions* over features induced by the feature maps? In it's current form, equation 3 is dividing 2 tensors and take its log, which mathematically does not make any sense.
summary_of_the_review: The experimental results have several major flaws (as denoted above), and for this reason I believe there are significant structural changes that need to be made in order to make this a valuable contribution.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: ML models are known to latch to incidental correlations in the training data, which emphasises the need for controlling what they learn. This paper proposes a learning algorithm that extracts features for classification that can only be revealed from a specified region of pixels (explanation). For example, the explanation may identify the beak of a bird if we wish to learn a bird classifier that discriminated only based on beak shape/colour etc.  
strength_and_weaknesses: Strengths.
1. Well motivated problem. 
2. Presentation is mostly clear. 
3. Experimental evaluation is mostly through and convincing. I particularly like the imbalanced settings and evaluating if we can get away by providing explanations only for the minority class. 


Weakness/questions.
1. There are mistakes in the method section that hindered understanding. Training is a two-step procedure, in the first step the parameters of h: (f, c) are updated and in the second step the parameters of mapper (m) are updated. If the outputs only depend on f, c (and not m), how does updating of m parameters make any difference? Cold be an error in the presentation here.  
2. How are lambda1, lambda2 or any other parameters tuned? Especially for the imbalanced settings?  
3. (minor) Line 2 and 8 of Alg 1 should have subtract rather than add for gradient updates. The mapping layer in Figure 1 is not consistent with writing. Does the mapping layer affect outputs at all? 
4. To justify the need for a mapping layer, ablation experiments without the mapping layer should have been reported.
5. Authors report their method’s superiority on imbalanced datasets, but the algorithm is not designed to handle such settings, it is merely an accident if it so. In any case, some justification for why their method can handle imbalanced settings is expected. 
6. Std deviation should be reported. Grad-Reg on Fox vs Cat with size 100 in Table 2 has inconsistent performance, which made me wonder if there is large variation in numbers. 
7. Conceptually, I do not see why original and masked (through explanation) inputs should have the same representation. For example, when the explanation is the beak of a bird, then in order to extract the features of a beak we need to first identify that it’s a beak, which then requires spatial orientation with respect to other features of a bird. However, in the masked image, we may not have sufficient information to identify a beak as a beak and therefore may draw only poor feature representation. For this reason, I think it is repressive to constrain the masked and unmasked image representations as being alike. 
8. Presence of self-sufficient explanations localized to only small regions seems like a strong assumption to me. To prove otherwise, authors should work with real-world datasets that can be explained this way. The toy experiments and bird classification experiments are not very convincing. 

clarity,_quality,_novelty_and_reproducibility: Clarity: lacking in parts
Reproducibility: code not released

summary_of_the_review: I agree that we should focus on a handful of “right reasons” instead of ignoring innumerable “wrong reasons”. The paper does a good job of proposing a method and showing that it is better than GradReg, but I have some conceptual and experiment related concerns that should be addressed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an approach to augment a ML model's training process when ground truth explanations (in this case, subsets of relevant features) are available. The approach is based on encouraging a model's learned representation to be similar with the full input and with the subset of relevant features.
strength_and_weaknesses: --- Strengths ---

- Improving a model's training when ground truth explanations are available is a good idea. In cases where we don't have many training examples but we know which parts of the input is important, it would be ideal if we could improve the sample complexity for learning accurate models.
- The proposed approach seems straightforward to implement. It simply involves augmenting the standard cross-entropy loss (for encouraging accurate predictions) with a secondary loss that captures the internal representation's similar with all inputs and with the subset of relevant inputs.

--- Weaknesses ---

High-level issues
- The method only works when we have access to ground truth set of important features - an uncommon situation in practice.
- The evaluation focuses primarily on toy datasets.
- The method has several strange and seemingly unjustified design choices (see below).

There are a couple choices in the method's design that don't make sense to me, it would be helpful if the authors could explain these:
- Why is $\mathcal{L}_{\text{feat}}$ a KL divergence penalty based on normalized representations, rather than something simpler like a squared error loss? How would we normalize if the activation functions don't produce non-negative results, e.g., if we used ELU or GELU activations? A squared error loss seems more universally applicable, did the authors test this?
- The argument that the classifier and mapping layer can't be jointly optimized doesn't make much sense - people train networks with competing objectives all the time and the networks don't fail to train. Did the authors conduct ablation experiments to test whether performing joint rather than sequential training leads to meaningfully different results?
- Figure 1 seems to indicate that the classifier $h$ uses the mapping layer $m$, but this is somewhat unclear in Algorithm 1 - can the authors clarify? And it's strange that we compare the representation from $f$ with the representation from $m$, can the authors explain why we need $m$? Aside from a theoretical justification, did the authors conduct ablation experiments to test whether removing $m$ altogether leads to meaningfully different results?
- I follow how the method works, and I agree that it constitutes "learning from explanations," but I don't see what it has to do with "being right for the right reasons." The model is basically encouraged to have the same internal representation with or without the unimportant features. But the reasons for the model's prediction are not known (there's no explanation of the model's prediction) and are not explicitly encouraged to resemble the correct reasons. Wouldn't it make more sense to describe the approach slightly differently, perhaps as encouraging the model to automatically identify and disregard irrelevant input information? What are the authors' thoughts on this? 

About the experimental comparisons:
- Among the baselines, only the GradReg one is in the same category as the proposed method (focal loss has nothing to do with leveraging knowledge about important features). Why did the authors not include the methods from Rieger et al (2020), Schramowski et al (2020) or Shao et al (2021)? There are two others that also could have been included mentioned below (Erion et al, 2021 and Chefer et al, 2022).
- The results from different methods presumably converge when given large enough training datasets. Could the authors provide results showing how large the training sets must be, either in the main text or supplement? E.g., there could be a plot showing the peak accuracy (or accuracy after 100 epochs) given different training dataset sizes.

About prior work:
- The claim that "most popular model explanations do not even pass the sanity check" in Adebayo et al (2018) seems like a bit of an exaggeration. The methods considered in that paper are a subset of gradient-based methods, but what about other popular methods like RISE, LIME, or SHAP? It's safe to say that there are many explanation methods that don't work as well as desired, but this claim should be dialed back.
- The authors present a hypothetical argument to explain the issue of training with a joint loss function accounting for prediction and explanation accuracy: "However, gradients of the two loss terms may point to different directions, creating a race condition that pulls and pushes the model into a bad local optimum. Imagine the two gradients counteract each other. The weights are then updated with negligible aggregated gradients." Many ML/DL methods involve training with multiple objectives and work just fine, why would it be especially problematic here? It seems correct that previous methods in this area don't work that well, but is there any evidence that this is the reason why? And even if it were, couldn't it be mitigated by pre-training with the prediction accuracy loss only and then turning on the explanation loss? This relates to one of my requests above for an ablation experiment.
- Referring to prior works that penalize explanations, the authors write: "Explanations in their settings are bounding boxes of either the main object or the spurious features, which differ from our definition of “informative and sufficient subsets of input features”." Actually, bounding boxes around the main object sound very similar to “informative and sufficient subsets of input features.” What's the significant difference? I suppose a segmentation mask would be more precise than a bounding box, but that's not a huge difference. Also, note that the authors actually use a bounding box around the beak in their one non-toy dataset.
- Referring to the same set of prior works, the authors write: "Although their objective is to be “right for the right reasons”, these methods are actually penalizing models when they learn the wrong reasons." What's the significant difference? Would you also argue that cross-entropy loss doesn't encourage classifiers to make the right predictions, they just penalize them for making the wrong predictions? This is not a well thought out criticism by the authors, but I suppose my bigger point is: I agree that there are differences between this work and prior work, but this is not helpful in clarifying what that difference is. The difference is that prior methods (at least those I'm aware of) penalize the explanation of the model being trained, whereas this method penalizes the internal representation in a manner that doesn't require generating any form of explanation. 
- Two related works belonging to the Ross et al. 2017 category are "Improving performance of deep learning models with axiomatic attribution priors and expected gradients" by Erion et al (2021) and "Optimizing relevance maps of vision transformers improves robustness" by Chefer et al (2022).
clarity,_quality,_novelty_and_reproducibility: The writing is clear and the method is novel, at least to my knowledge. My main concerns are with the method design, significance (due to the assumption of known ground truth important features), and evaluation with respect to strong baselines.
summary_of_the_review: The paper develops a method for training models when ground truth explanations are available. My recommendation is marginally below acceptance, but I could be swayed if the authors include the requested ablation experiments and compare with a stronger set of baselines.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors address the important topic explanations value for improving deep learning performance.
Specifically, the authors examine the problem of visual binary classification, where explanations in the form of attention/masking (carefully selected sunset of input pixels) are made available to the model while training.

The explanations are used by minimizing the kl divergence between the normalized explanation features and the normalized full image features passed through a linear filter/mapping layer constructed prior to the classification head of simple convolution architectures.

The relative performance of this scheme is evaluated on two synthetic and one natural (birds derived) data sets, under three scenarios:

1. The authors demonstrate that within the range of evaluation — up to few hundred samples  in the synthetic datasets and several tens in the birds dataset — there is a generalization benefit throughout the learning curve when training with the explanations. The evaluation is conducted both in balanced and unbalanced settings with explanations given to the minority class.

2. The authors examine the generalization on a modified dataset than trained on, but which shares the explanation classification rule — as a basis of evaluating if the models learned the ‘suggested rules provided by the explanation’.

3. The authors examine the effect of adding disturbances to the image which do not overlap with the explanation.

strength_and_weaknesses: $\bf{Strengths:}$ 

The notion of using explanations in order to improve training is a very important one. 
The authors offer a filtration layer and optimization method which allows assessing agreement between feature representations (which are not known in advance) by using characteristics which are assumed to be known as important for the labeling — hence explanations.

$\bf{Weaknesses:}$

This paper has several structural and generalizability weakness:

The datasets are extremely simple/limited. In itself that is not a problem (and can actually be a virtue!) if they were to provide theoretical insight (which is not demonstrated in the paper)  *or* empirical insight that is evaluated and carries over also in practical settings of interest.

However both are limited. And it is not demonstrated how these results can be generalized.
Can the authors show how they improve performance by providing explanations for (e.g. improving mis-classifed samples) cifar10/100 (at the least) and imagenet ?

Experimental rigor — the baseline no-explanation deteriorates with increasing number of samples (figure 2). What does that mean about the confidence in proper optimization of the experiments. Why is this happening? Is it overfitting? What happens with more samples where we know that performance of vanilla no-explanation should scale and improve.

Conceptual applicability — evaluated only in the small data regime. Is performance quickly surpassed in practice by no-explanation training with sufficient data? Then the practical utility is unclear. 
Table 1 is showing that this may be the case?

Further, providing explanations in practice may be very costly. How much is an explanation worth — e.g. in terms of equivalent amount of unexplained data resulting in the same performance? How would this scale ?

It would be very exciting to show that few explanations can be used to significantly improve performance of large models. But this paper offers no insight into this practical domain of improving actual performance or showing why, in principle, it can be improved in large scale settings.

clarity,_quality,_novelty_and_reproducibility: The paper is clear, although leaves more than desired for the reader to devine in terms of the experimental results. Multiple referrals to appendices, and some unclarities (this reader found himself referring to the code, e.g. to clarify what is meant by Appendix B “W set the balancing coefficient $\lambda$...", but did not find readily clear trace of it).

For quality, as mentioned above, the experimental rigor need to be improved.

If improved such that a explanations can be demonstrably used for the improvement in the practical setting, or within the realm of toy datasets and models, applicable theory can be showen, then there would be room for a novel significant contribution. Currently the novelty is limited.

Reproducibility given the code is adequately addressed by the code, and I commend the authors for providing it (could use some cleanup --- there is a lot of clutter such as mse loss that does not seem to be used etc.).

Finally, the paper would do well to engage more deeply with prior work.
The paper does not sufficiently engage with known methods of explainability and feature extraction. In particular, the notion of attention is central and pervasive both in natural language processing and in vision (e.g. vision transformers). Attention is structurally available both in the input (pixels) layer and in feature layers (i.e. for deeper than 1 layer transformers). 

summary_of_the_review: Utilizing human insight to improve model performance and/or cost (direct or indirect by way of data curation) is a very important topic, with strong ties to explainability and fairness as well.

This paper showcases simple visual cases where performance can be improved at the small scale.

However it does not show how these results can carry over either from theoretical ground or empirical demonstration into the practical setting of real tasks and scale, nor does it address the core question of ‘how better can one perform with explanations?’ (from a cost / data / performance scaling perspectives)

Significant work is required for addressing these issues, and I will be happy to upgrade my review if they are adequately resolved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new learning paradigm to train reliable classification models on small datasets, assuming access to some simple explanations (e.g., subset of important input features) on labeled data. Experiments on several synthetic and real datasets showed promise of the proposed method. While all the reviewers consider the proposed setting of leveraging human explanations to increase model performance interesting, they are not convinced by the applicability of the proposed method in real world settings. Aiding learning with explanation is probably more helpful in small data regime. The authors however focused the experiments on image classification tasks which often has abundant training examples.  Application domains such as medical data where labeled data is hard to come by, might be able to better showcase the value of the proposed paradigm and method. 

justification_for_why_not_higher_score: Reviewers are not convinced on the practical utility of the proposed paradigm and method.
Decision: Reject