Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: -Summary: 
This paper investigates the previous masking-based saliency map methods. By detailly formulate the masking-based saliency methods and sufficient experimentation, the paper gives a simple formulation and practical training strategies.

-Strength: 
The paper is well organized and the presentation is easy to follow.
The experiments are comprehensive.

-Weakness:
Although the experiments are enough, the paper simply adopts the previous masking-based methods, evaluation metrics, architectures, and sanity check analysis. The theoretical analysis is insufficient.  More theoretical experiments instead of performance analysis need to be added.


review_title: The paper has sufficient experiments on strategies of masking-based saliency methods , but lacks theoretical analysis.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes an approach to improve masked based prediction explanation. They train an auxiliary model which predicts a mask that must satisfy two terms 1) maximize classification accuracy when applied to the image and 2) maximize entropy over softmax when the inverse mask is applied to the image (they also experiment with minimizing the classification accuracy instead here).

The paper also positions their work to be classifier agnostic in the text which is not clear to me. I think this aspect is a negative because it is not in the list of contributions and seems like a distraction of a remnant of an old direction of the paper:

> "Whereas a FIX approach seeks a saliency map that explains what regions are most salient to a given classifier, a CA approach tries to identify all possible salient regions for any hypothetical classifier (hence, classifier-agnostic). In other words, a CA approach may be inadequate for interpreting a specific classifier and is better suited for identifying salient regions for a class of image classification models." 

I'll focus now on studying how this paper addresses the 4 contributions they claim in the intro:

> (1) We find that incorporating both masked-in classification maximization and masked-out entropy maximization objectives leads to the best saliency maps, and continually training the classifier improves the quality of generated maps. 

There are no standard deviations reported from multiple runs so nothing can be statistically claimed. Also, looking at table 1 this is not clear. It seems MinClass works the best compared to almost all other methods in terms of PxAP. In terms of other metrics it seems like no best can be determined.

> (2) We find that the masking model requires only the top layers of the classifier to effectively generate saliency maps. 

This is reported in the table. It is not clear if this is a strong contribution as it would just be specific to this method and without a standard deviation we cannot conclude anything.

> (3) Our final model outperforms other masking-based methods on WSOL and PxAP metrics. 

The reported difference between the methods is 48.6 vs 48.4.

> (4) We find that a small number of examples—as few as ten per class—is sufficient to train a masker to within the ballpark of our best performing model.
This is reported but the paper doesn't have a section detailing the experiments or showing how this number is derived.


The paper could be improved by refining the contributions and detailing what evidence should be observed to support these claims and then providing a significant amount of evidence. Right now the paper is not focused in general and does not focus on supporting the claims made in the introduction and therefore is not ready for publication.	
review_title: Unsupported claims

final_score: rating: 7: Good paper, accept
review_content: Summary: 

By the first look, this work itself does not introduce any new architecture or novel algorithm. It takes what is considered as the popular choices in generating classifier saliency masks, and conducts quite extensive sets of experiments to dissect the components by their importance. The writing is pretty clear in narrative and the experimental findings are surprising and significant. 

Good things to mention:

1. The importance of using CA seems quite evident from Table 1.
2. The importance of using multiple resolutions is also evident.
3. The findings of few shot learning ability for masker is quite interesting, class diversity seems to play a very import role.

Some questions:
1. Given the fact that all experiments are done with ImageNet, which is widely considered as a solved task with almost super-human classifiers, do you think the superb performance of those classifiers could inflate your findings? Do you think the results in Table 1 and Figure 5 would still hold consistently when the classifier is not as performant?
2. I am not sure I understand the point of the figure on the right of Figure 4. What is the take-way message from it?
3. Compare (e,f,i,j) in Table 1, it seems I does not make a difference as long as O is used. What is the reason there?
review_title: Thorough experimentation and important insight on the matter of deriving saliency from trained classifiers

final_score: rating: 6: Marginally above acceptance threshold
review_content: /*************************Post-Rebuttal**********************/
The authors address many of my concerns well, and I agree with their rebuttal.

The modified manuscript also looks good, too.

I raise my rating.

/*************************Pre-Rebuttal**********************/

Pros.:
1. The proposed method is simple and reasonable but better than the current state-of-the-art methods.
2. The paper is clearly presented and well-organized.
4. The authors conduct many ablation studies to validate the different variants of the proposed methods. 
5. The few-shot localization built in the proposed method is also interesting, and worth being explored.


Cons.:
1. Limited Novelty

The proposed method is similar to [Ref. 1, 2], but there is no discussion in the related work. After reading this paper and [Ref. 1, 2], I don't see any significant difference in [Ref. 1, 2], the authors focus on top-down saliency detection, and use the masked image scheme to train the network. They proposed a CNN framework that contains two CNN modules, an image-level classifier and a pixel-level map generator. They also use the saliency from the map generator as the classifier's input and then optimize the classifier scores to derive the saliency map and train the generator. In this paper, the loss is L2 loss and binary-class setting, but I think the main idea is almost the same. Therefore, I think the novelty is very limited. I would like to see that the authors point the significant difference between the proposed method and [Ref. 1, 2] and give a detailed discussion.

[Ref. 1] Hsu et al., Weakly Supervised Saliency Detection with A Category-Driven Map Generator, BMVC'17

[Ref. 2] Hsu et al., Weakly Supervised Salient Object Detection by Learning A Classifier-Driven Map Generator, TIP'19

2. No significant improvement

Compared to Zolna et al. (2020), the improvement of the proposed method is not significant, 48.6 v.s. 48.4 in OM,  36.1 v.s. 35.8 in LE, 61.4 v.s. 62.0 in F1. I think this improvement is minor. 

3. About few-shot experiments

The few-shot experiments are interesting, but why aren't other methods conducted for these experiments? The authors claim that the few samples are sufficient to train the proposed method, but other methods may also require a small number of training samples to achieve similar performance. Besides, I am not clear why the proposed method could work well under the few-shot setting because there is no specific module or design for the few-shot setting.   Could the authors explain it in detail?

4. Other issue:

What is the difference between the proposed method and other weakly supervised or unsupervised object saliency detection, such as [Ref 3, 4, 5, 6, 7]? In these works, their goal is also to train a network to predict a class-agnostic saliency map and highlight the most salient object in the images under the unsupervised or weakly supervised setting. Therefore, I would like to know the major difference between the tasks. If they are similar to the solved problem, the discussion and comparison should be conducted.

[Ref. 3] Zhang et al., Zhang. Supervision by fusion: Towards unsupervised learning of deep salient object detector, ICCV'17

[Ref. 4] Li et al., Weakly supervised salient object detection using image labels, AAAI'19

[Ref. 5] Wang et al., Learning to detect salient objects with image-level supervision, CVPR'17

[Ref. 6] Zhang et al., Deep unsupervised saliency detection: A multiple noisy labeling perspective, CVPR'18


review_title: The proposed method is simpler and better than the previous methods, but my concerns are the limited novelty and no significant improvement

Meta-review: While the reviewers found parts of the paper interesting, the main concern about this paper was lack of novelty and marginal improvements obtained by the proposed methods.
Decision: Reject