Venue: ICLR 2023 Conference

Reviews:

final_score: rating: null
review_content: I'm making a brief top-level post because I think this is a key point to clarify. 

My understanding is as follows:
1. The MaxBoxAccV2 **metric** is class-agnostic.
2. The WSOL **methods** considered in Choe et al. are not class agnostic. In particular, they are trained for multi-class classification. 

Authors, please let me know if you disagree with either of these statements - this is a fundamental point and we need to arrive at consensus. 
review_title: The WSOL metric in Choe et al. in class-agnostic, the WSOL methods are not. 

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper measures the WSOL capability of existing heatmap-based XAI method applied on ResNet50 and improves them.
strength_and_weaknesses: This paper finds that XAI methods perform WSOL with very sub-standard MaxBoxAcc scores.

The experiment is then repeated for the same model trained with Neural Backed Decision Tree (NBDT) and finds that vanilla CAM yields significantly better WSOL performance after NBDT training.
clarity,_quality,_novelty_and_reproducibility: The proposed method in this paper is incremental.
summary_of_the_review: The overall contributions of this paper are not significant.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: In this paper, the authors discussed the work of (Choe et al., 2020) for the evaluation of WSOL methods. and the work of (Wan et al., 2021)  for the neural-backed decision tree (NBDT) method. The authors describe both works, and also evaluate some heatmap-based explainable methods such as GradCAM, saliency, Deeplift, and Guided Backpropagation, and compare them to the CAM method. They also compare with and without NBDT training. Evaluation is done on the Imagenet dataset.
strength_and_weaknesses: Strength: 
+ The paper concerns WSOL tasks and explainability, which are very important and relevant.
+ The empirical results show some benefits of the method, and some ablation studies are provided.

Weakness:
+ There is a lack of contribution and novelty.

clarity,_quality,_novelty_and_reproducibility: + The paper is clearly written and organized. 
+ While WSOL methods are important for machine learning models, in the context of interpretability and explainability, this paper fails to provide a novelty or contribution to the field.
summary_of_the_review: + This paper reads more like a tutorial rather than a research paper. It lacks any innovative contribution. In addition, it is not clear why authors refer to WSOL methods (Choe et al 2020) as class-agnostic - they are class-aware methods.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper considers the problem of weakly supervised object localization (WSOL), in which the goal is to accurately predict bounding boxes using only image-level labels at training time. The goal of this paper is to assess the performance of different explainable AI (XAI) methods (GradCAM, Saliency, DeepLIFT, GBP) for WSOL. The paper also asses the utility of neural backed decision tree (NBDT) training for the backbone and studies the utility of using different weighted sums of activation maps from different layers to produce localization heatmaps. All experiments are performed on ImageNet. 
strength_and_weaknesses: # Strengths
* WSOL is an interesting problem that may lead to label-efficient methods for predicting bounding boxes.
* Benchmarking new methods on WSOL is a worthwhile service to the community. 
* I agree that MaxBoxAcc is a somewhat odd metric, and I appreciate critical thinking about its pros and cons. The paper's related discussion of thresholds is interesting. 
* The paper considers a number of different XAI methods, and the idea to train using NBDT is interesting. 
* It is interesting to see the qualitative results in Figure 1 showing that XAI methods can produce heatmaps that are unsuitable for WSOL. 
* The visualizations of "cascading bounding boxes" in Figure 3 are interesting. 
* The per-layer experiments in Figure 4 are interesting. 

# Weaknesses

* I suspect there may be some fundamental misunderstandings about WSOL and in particular [Choe]. The paper makes reference to "WSOL training" and says that [choe] performs "a class agnostic training to optimize" for WSOL performance. I have read [choe] and this claim does not seem right to me. The typical approach in WSOL (and in [choe]) is to (i) train a CNN end-to-end on a multi-class classification task, with various bells and whistles (which is the main differentiator between WSOL methods) and (ii) generate boxes by using CAM, thresholding, and then picking a connected component to draw a box around. Nothing about the training process is "class agnostic" - the different WSOL methods in [choe] differ in their method for training the backbone, but all of them train multi-class classifiers. This seeming confusion is repeated on Page 6 ("However, their performance on MaxBoxAcc have not been reported.") where it is used to insinuate that [choe] did not report MaxBoxAcc results for methods "without WSOL training" because performance was low. I find this all very strange, and I am eager to hear a clarification. 
* Related to the previous point, the paper says that in [choe] "models are fine-tuned specifically to improve WSOL performance" - this is not true to the best of my knowledge. 
* The numbers reported in Table 1 are extremely low compared to [choe]. The paper mentions that "Vanilla CAM is equivalent to GradCAM applied to layer 4" - if so, then that result in Table 1 disagrees with the exact same result in [choe] by a large margin. Unless an excellent explanation is provided, implementation issues seem likely. 
* The paper uses only one dataset, which is not standard in the WSOL literature. Most papers use CUB and ImageNet. [choe] introduces a third. [cole] introduces a fourth.
* The paper says that it "attempts to simultaneously maintain the model's predictive power as we perform WSOL" unlike the methods in [choe] which improve localization "at the expense of predictive performance". However, there is no comparison provided between the predictive performance of the proposed methods and the methods in [choe]. On Page 6, there is a claim that "final class label prediction accuracy results have not been reported" by [choe] after "WSOL training" because it "might have degraded significantly" - the numbers in question can be found at [choe-results], which is linked on the GitHub page for [choe]. In fact, the results from [choe] for ResNet-50 on ImageNet are similar to those in Table 2. 
* Related to the previous point, I would caution against publicly suggesting that other authors deliberately omitted unflattering results without providing any evidence. One instance is described in the previous bullet point, and another instance is where the paper concludes: "Finally, we should mention that the technical and subtler details in the appendix are worth perusing. We have used the codes from two main papers (NBDT and Choe’s) mostly as they are presented in their respective github repositories. There might be some concerning details that readers would like to pay attention to regardless of the fact that they have been published in top conferences." This seems to ominously imply that there are significant flaws in those papers. When we turn to the appendix, there seems to be just a repeated allegations against [choe] based on what I perceive to be a deep misunderstanding of that paper, which I have addressed above. Identifying flaws in published work is absolutely crucial, but it should be done professionally, and with great care and attention to detail. 

[choe]:

@inproceedings{choe2020evaluating,
  title={Evaluating weakly supervised object localization methods right},
  author={Choe, Junsuk and Oh, Seong Joon and Lee, Seungho and Chun, Sanghyuk and Akata, Zeynep and Shim, Hyunjung},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={3133--3142},
  year={2020}
}

[choe-results]:

https://docs.google.com/spreadsheets/d/1O4gu69FOOooPoTTtAEmFdfjs2K0EtFneYWQFk8rNqzw/edit#gid=0

[cole]:

@article{cole2022label,
  title={On Label Granularity and Object Localization},
  author={Cole, Elijah and Wilber, Kimberly and Van Horn, Grant and Yang, Xuan and Fornoni, Marco and Perona, Pietro and Belongie, Serge and Howard, Andrew and Mac Aodha, Oisin},
  booktitle={European Conference on Computer Vision},
  year={2022}
}
clarity,_quality,_novelty_and_reproducibility: # Clarity and Quality
* I found the paper to be a rather confusing experience in its current form. The overall structure seems a bit disorganized. Perhaps more signposting would be helpful: "here's what we're going to do, here's why, here's what we found" etc. 
* It would be better to use consistent notation in the presentation of CAM and MaxBoxAcc.
* There is a lot of "jargon dropping" without context or explanation - what should the reader take away from mentions of "mode collapse" or "embedded decision rules for soft inferences"? Please make sure to explain these terms and only include them if they contribute to the reader's understanding. 
* [Page 2: "The problems with these simple metrics are well described in... First, WSOL with only image-level labels can be an ill-posed problem."] As presented, a reader might think that this is a motivation for the introduction of the MaxBoxAcc metric. Per Choe et al., this is not the case. My understanding is that ill-posedness of WSOL is brought up in Choe et al. to make it clear that bounding boxes are necessary in the validation set. 

# Novelty
* The paper benchmarks a number of XAI methods on WSOL with and without NBDT, which is novel to the best of my knowledge. 

# Reproducibility
* The section "CAM-inspired sum of weighted feature maps" is confusing, and does not seem to be described in sufficient detail for reproducibility. 
* Are any hyperparameters tuned for this work? If so, how? If not, why is this not problematic for the experimental results in e.g. Table 1? 


summary_of_the_review: This paper addresses and interesting problem and undertakes a worthy benchmarking project. However, this work seems to be built on significant misunderstandings of prior work, and makes serious negative claims about prior work that are insufficiently supported. Of course it is possible that I am the one with a profound misunderstanding, in which case I hope to be corrected.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This manuscript aims to improve the localization performance of XAI models. The authors claim that incorporating NBDT into CAM can result in improved BoxAcc metric.
strength_and_weaknesses: In summary, this manuscript is of very bad quality. 

Introduction:
- The research question is insufficiently motivated. What makes it worthwhile to improve the localization performance of XAI methods?
- What's the difference in definition between XAI methods and WSOD methods?
- "The NBDT original paper (Wan et al., 2021) trains larger models from scratch. However, we successfully fine-tuned pre-trained ResNet50 without causing the collapse of predictive performance." I don't see why this is a contribution.
- "Firstly, WSOL with only image-level labels can be an ill-posed problem." I don't see why.

Method & Experiment:
- Why choose MaxBoxAcc over mAP (detection) and Dice (segmentation)?
- Table 1, please report absolute performance.




clarity,_quality,_novelty_and_reproducibility: The manuscript is poorly presented. The arrangement of materials is chaotic. Language is poor and it's difficult to understand most of the materials.

This manuscript lacks novelty. The proposed method applies an existing training scheme to a well-known WSOD method. 


summary_of_the_review: This manuscript resembles more a poorly-written school report than a research paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: All reviewers agree that while this work tackles an important problem, there are some significant concerns and flaws in the current version of the work, including the characterization of WSOL in prior related methods. The contribution of this paper is also considered not significant enough to be publishable in ICLR at this time. This paper is therefore recommended to be rejected. The reviewers and authors have engaged in an active discussion, and we hope the authors will find this useful and can incorporate the feedback into revising and resubmitting the work elsewhere.
justification_for_why_not_higher_score: There are significant technical flaws in the work in its current form, as described in the meta-review, and the majority of reviewers also recommend strong reject.
Decision: Reject