Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a data augmentation framework called BRACE that utilizes concept-based explanations from existing interpretability methods (such as Comprehensible Convolutional Neural Network (CCNN) and GradCAM) to identify informative examples to be added to the training set. Specifically, the proposed approach defines a utility function that focuses on finding images that are 1) under-represented in a class (i.e., images containing visual features of the class but are predicted as other classes with high confidence by the classifier), and 2) contain visual concepts that have led to the misclassification. Experiments on CUB, CUB-Families and Tiny Imagenet show better performance for the proposed approach as compared to previous works and baselines.
main_review: Strengths:
- The paper presents a very nice demonstration of how explainability can be used to improve image classification systems.
- The paper is very well written and easy to follow.
- The ablation studies clearly highlight the importance of both of the main components of the proposed framework.


Weaknesses:
- There seems to be some mismatch between the motivation and the proposed approach. In Line 87, the authors claim that the queried set of images is typically noisy but then their proposed solution focuses on under-representation and visual features that lead to misclassification. Shouldn’t the solution just be to throw out the noisy data? They seem to be orthogonal to each other. Please clarify.
- If there are two highly similar images in the queried set of images such that they both result in high utility scores, the proposed method would choose to add both to the training set, correct? This doesn’t seem ideal. Is there a way to enforce some diversity in the selected set of images to maximize the information gain?
- I appreciate the experiment showing better generalization with BRACE as compared to the baselines. But, how does BRACE compare against previous works in this experiment?
- The derivation of the set of concepts that caused the classifier to misclassify images of class c to c_bar as well as their overlap with the visual features in an image is specific to an interpretability method. I think it is okay to consider this issue as out of scope for this work but it would be good for the paper to discuss some initial ideas about how they can be generalized to a family of interpretability methods.
- Where are the feature vectors extracted from that are used in equation 2?
- Lines 64-66: How does GradCAM explain model decisions in terms of human-friendly concepts?
- The citation for the RCNN model in line 155 is incorrect.


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
I thank the authors for their response. It answers all my concerns.

After reading other reviews and the authors’ responses to them, I think all the major concerns from the other reviewers have also been answered. 

In my opinion, the paper proposes a significant contribution that would be useful to the Explainable AI (XAI) community and, therefore, I would like to see the paper getting accepted (hence, keeping my acceptance rating). 

I would encourage the authors to incorporate the additional experiments and the qualitative results they provided in the rebuttal period into the revised version (camera-ready, if accepted) of their paper.
limitations_and_societal_impact: The paper discusses their work’s societal impact satisfactorily. However, I think the paper can expand on the limitations a bit more, e.g., it might be difficult to derive the set of concepts that lead to misclassification for some types of interpretability methods, etc.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an explanation-based data augmentation method for image datasets. Technically, this method defines a utility function to choose the data samples collected from the Internet. The utility function is defined to select the samples that could better reflect the under-represented region in the data distribution. The authors compare the proposed method with some existing data augmentation techniques on three datasets and showcase the effectiveness of the proposed method to helping achieve higher finetuning accuracies.  
main_review: 1. The paper mentioned two types of existing data augment methods: the first type of methods conduct certain transformations to input samples to explore the local regions around the given inputs. It is relatively straightforward to understand that these methods cannot explore the under-represented region because of the local exploration nature. The second type of methods collect new data from other sources and add those data into the training set. The paper first argues that these methods will introduce noise or OOD samples, which is again quite obvious. However, the authors then argue in the L59 these methods can neither explore the under-represented region. I am kind of doubting this argument. I could understand that the authors want to argue that these methods do not explicitly include a mechanism to explore the under-represented region. But this does not mean that they cannot achieve this goal. In fact, collecting diverse samples from other sources may also serve the purpose of exploring under-represented regions. I would suggest the authors provide some empirical supports on this argument other than the finetuning accuracy. For example, the authors could show the utility scores of samples selected by these methods to show whether they could uncover the under-represented region. 

2. My second comment is that the first and second terms in the utility function may have a similar effort and select a similar set of samples. My hypothesis is that most misclassified samples are wrongly labeled because they contain the specific features that cause the model to make mistakes. To this end, It seems that the second term (explanation term) only helps to filter out the highly confusing samples that are caused by randomness. However, since the first terms select the samples that are most deviated from the class center of their true class. It is less likely that their misclassifications are caused by randomness. I appreciate that the authors provide an ablation study to showcase the effectiveness of each term. I would suggest the authors also report the difference in the sets of samples selected by each term and showcase some examples. IMHO, this will help better understand the different effects of the two terms and why the second term works better than the first one. 

3. Regarding the proposed utility function, I found the following questions to be addressed to make it clearer. 
(1) What is the GAP layer, and what is $o$_{i} in line 126? (2) For the post-hoc explanation models, why not directly using Grad-CAM to generate explanation? What is the insight of choosing R-CNN? Why not selecting other explanation methods? (3) In Eqn. (6), what is the insight of adding the first term (i.e., max_{w \in W}).

4. Regarding the evaluation, I have the following concerns: (1) Major concern: the performance improvement in most setups is marginal. I also checked the results with stds shown in the supplement and found out that in some cases, there are overlaps between the accuracies obtained by baselines and the proposed method. To better demonstrate the superiority of the proposed method, I would suggest the authors conduct some statistical testings (e.g., paired t-test) on the results; (2) I found some inconsistencies among the experiments. More specifically, I would suggest the authors comment what's the reason for only applying two baselines in the self-explainable model experiment. (3) As I mentioned earlier, besides the finetuning accuracy, I would also suggest reporting the results of other metrics to showcase the effectiveness of the proposed method (e.g., the utility scores of samples selected by each method and the sample overlapping rate in the ablation study, the error collection rate (the number of originally errors that are corrected after finetuning) and the number of newly introduced errors after finetuning). (4) I would suggest adding a hyper-parameter sensitivity test on \epsilon and the explanation method used in the black-box case. 

5. I would suggest the authors run a grammar check on the current version. I found some typos and grammar errors (e.g., Line 141 convepts -> concepts).

==== After rebuttal. ====
The rebuttal responses addressed most of my comments (especially the potential overlap between the $\beta$ and $\Delta$ and the statistical tests). With that, I am happy to raise my score to above the acceptance threshold. I would further appreciate that if the authors could include the new experiments and other parts of the responses in the revised paper (Or the camera-ready version if get accepted).


limitations_and_societal_impact: This paper discusses one possible limitation in the conclusion section. I would appreciate it if the authors could include a more comprehensive discussion of the limitations and future works, for example, the scalability of the proposed method, the generalizability of the proposed method in other application domains, etc. Regarding the social impact, the authors could discuss the potential threats of the proposed method if attackers use it to generate poison data.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes to use interpretation to select effective instances for augmenting training data. A utility function is developed, where images with high scores mean that: 1) the images are helpful for better learning class-c data, 2) the images could significantly correct the drawbacks of the model M. The paper discusses two interpretation scenarios, interpretable models (GAP+linear model) and post-hoc interpretation. The experiment includes a comparison to two types of baseline methods, data augmentation methods and data selection methods. Ablation studies are also conducted to show the effectiveness of the two components in the utility function. Different parts in the current version are somehow fragmented. 
main_review: Originality: The paper proposes an interesting application of interpretation to improve models. The current approach mainly combines existing interpretation methods, but the overall design is reasonable and well-motivated.

Quality: In general, the proposed approach is technically sound. The core contribution is an interpretation-driven utility function with two terms. The first term encourages selecting instances for better learning class c which was not well learned in the model. The second term encourages selecting instances containing crucial concepts. In this way, interpretation is fully utilized. The effectiveness of the two terms is also supported in the ablation study. 
Meanwhile, there are several factors that could be further clarified or improved. First, why do we need NLP-based content for interpretation? Second, what is the running time of the approach? Third, the proposed method is more like an active learning method than data augmentation, but the baseline methods are not balanced to reflect this (i.e., typical methods for active learning or data selection are not included as baseline methods).

Clarity: The paper is, in general, well written and well organized. Figure 1 is also helpful for understanding the motivation. However, Section 4.2 could be extended with more details, especially for the last paragraph.

Significance: The proposed work should be interesting to XAI researchers. A potential limitation is that the interpretable model used in Section 4.1 is not a commonly used model in practice, which may limit the applicability of the proposed method. However, post-hoc methods, which are more flexible, are not fully discussed in this work.
limitations_and_societal_impact: Please refer to the main review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a utility function to rank new training samples with respect to their potential contributions of improving the model performance. The proposed utility contains two parts, one is for the under-represented region, another is based on explanation algorithms for finding the concepts that cause the misclassification. Experiments proved the effectiveness of the proposed method.
main_review: Overall I find the idea of this paper interesting: using explanations that indicate the reasons of misclassification to improve the model performance is interesting and practically useful.

However, there are two main issues: 

1. The sample selection from this paper is on web datasets, while authors only compared with two trivial methods, Random and Confidence. Though I am not familiar with the sample selection, I believe there exists some other more advanced baselines and I suggest authors compare with advanced baselines. Because, fundamentally, the proposed approach is for sample selection from an open dataset, instead of data augmentation, according to my understanding. Related key words may be learning from web images (found in the cited paper [29]), but others words may be also pertinent. I hope authors could address this issue with clear explanations and/or experiments.

2. The presentation is not good. (1) The samples in Figure 1 are mis-labeled by the dataset or mis-classified by the model? I didn’t understand the presentation of Figure 1. (2) Better highlight the definition of under-represented sample/region, or mention it at beginning (Line 89). (3) The designed scores should be better explained, with intuitions and remarks. For example, which samples will obtain high utility scores, and in which way these samples will help the model to generalize better. Without some intuitive introductions, the formulas are difficult to understand. (4) The designed formulas are strange and there are no explanations. For example, why exponential is needed in eq.(2)? Why log is needed in eq. (6)? Will some other alternatives be also useful? e.g., self-entropy as utility? 

and several detailed questions:

1. How many samples have been selected for each dataset? I have seen the reported numbers for two methods. I believe the number of samples used by this paper is much smaller.
2. I briefly looked at the cited paper [29], which is a weakly supervised method. Authors used the collected samples from [29] for trainings on CUB. How the labels of these new samples are obtained? Or did I miss something?
3. Are the compared methods using samples from the same image repositories?



[29] C. Zhang, Y. Yao, H. Liu, G.-S. Xie, X. Shu, T. Zhou, Z. Zhang, F. Shen, and Z. Tang, “Web supervised network with softly update-drop training for fine-grained visual classification,” in AAAI, 2020.
limitations_and_societal_impact: .
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a form of data augmentation that uses explanations to select external images from poorly performing classes.  Specifically, it looks for images that:  
- are "under-represented" meaning they are similar (according to the model) to images of the target class but are predicted as being part of a different class
-  have the same concepts that are often present when the model mistakes images of the target class as being part of this different class
main_review: 
# Originality 

The methodology is novel and the idea of using explanations to guide data augmentation is an interesting one.  The connection between this work and prior work is clear.  


# Quality

Most of the claims are well-supported, but there are a few things that need clarification.  

*L128:  "We apply a softmax function to obtain the percentage contributions and select the set of concepts whose contribution is greater than some threshold."*

How is this threshold chosen?   


*L150: "We consider the image patches whose pixels have importance scores greater than some threshold as the concepts that have contributed to the model decision."*

How is this threshold chosen? 

  
*L175: "We report our results on the validation set."*

Based on the "Implementation Details" in the Supplement, it seems like the hyper-parameters for this method were tuned while those from the baselines were not.  How were the hyper-parameters tuned?  If the validation set was used, this places the results on this dataset into doubt.  This is especially concerning because CUB(-Family) is a small dataset, where we would expect augmentation to be more helpful than on larger datasets, and TinyImageNet is the only larger dataset tested.  


*Table 2.*

Why are the results for the other "external source" data augmentation methods not shown?


*5.2 Comparison of Sample Selection Methods*

Why are these results based on ResNet101/DenseNet161 while the previous sections' results are based on ResNet34?


# Clarity

Most of the paper is clearly written.  Detailed comments follow

*2.  Related Work*
-  It would be nice if this section was clearly divided into "Augmentation" and "Explanation" based sections
-  It would be nice if relationship between [6] and [19] was clarified 
-  It is a little unusual to describe methods like GradCAM as concept-based because they just point to parts of the image when methods like TCAV/ACE/CCNN are directly based on concepts.   Relatedly, it is surprising that only GradCAM and CCNN are used in Section 4.  

*3.  Proposed Framework*
-  "A high β value indicates that x is similar to some image in class c but the model M has classified it as class c ̄. " This doesn't seem quite right because Equation 2 is based on similarity with the mean of the class representation rather than the max of the similarity images of the class.  
-  Generally, a simple illustration would help explain the intuition behind the beta term (which otherwise requires some effort to realize why it is a "under-represented region") 

*4.1 Fully Interpretable Classifier with Linguistic Explanation*
-  This section relies heavily on the reader being familiar with CCNN

*4.2 Post-hoc Explanation*
-  It would be nice if the intuition behind Equation 6 was explained the same way it was for Equation 4

*L170: "We create under-represented regions by removing some species in the test split of [28] from each family in the train dataset."*

It should be made clear that the purpose of this is to do something like zero-shot learning.  Otherwise, this experiment looks like one that is clearly designed for this method to work.  


*Table 5*

This caption should be explicit about the fact that these are classes where the training data was deliberately removed.  

# Significance

If the concerns over hyper-parameter selection for TinyImageNet and why the Part/Meta-based augmentation strategies were excluded from Table 2 are addressed, this does seem like a significant contribution.  

# Update based on discussion

The reviewer's main concerns have been addressed and the rating has been revised upwards by a point.  
limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper has novelty as it proposes a form of data augmentation that uses explanations to select external images from poorly performing classes. All five reviewers gave a 6 or 7 rating. The authors provided detail responses to reviewers' comments. The reviewers and AC reached the agreement to accept the paper. 
Decision: Accept (Poster)