Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: I Summary
This paper has two major contributions:
- A method to infer robust saliency maps using distribution preserving decoys (generated perturbated images that resemble the intermediate representation of the original image in the neural network)
- A decoy-enhanced saliency score which compensates for gradient saturation and takes into account joint activation patterns
The authors show the performance of their methods on three different saliency methods, quantitatively and qualitatively, but also when it is submitted to adversarial perturbations.

II Comments
1. Content
The paper is very clear and easy to read, I would like to point out how well structured it is. 
The method yields very interesting results, the whole work is very complete experimentally (saliency methods used, adversarial perturbations, models etc) and the process coherent. I would love to see it applied to different datasets: Imagenet focuses more on the foreground, whereas "Places" focuses more on background objects and could be interesting to use. 
The authors referred to Sanity Checks for Saliency Maps (Adebayo et al) without using it for their results, it would be nice to add it to the experiments.


2. Typos
- intro, paragraph 2, l2: use -> uses
- paragraph 5, l9: indendently ->  independently
- 4.2, paragraph 4, last line: target word twice
- 4.3, paragraph 2, l5: fisrt -> first
- l9: not able -> may not be able
- 4.5, l5: salinency -> saliency


review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Given a model and an image, the proposed method generates perturbations of the image, such that the model output in intermediate layers of the network does not change. Their method first generates such perturbed images, and then uses those to generate a saliency map in order to interpret the classification.  The idea is overall interesting, but its relationship with many other methods is not adequately examined. Significant portion of recent literature is ignored, especially studies on counterfactuals and decision boundaries in the context of image classification.  For numerical results, they compare their results with raw scores generated by three other methods, but the results are quite thin and disorganized with respect to adversarial robustness and interpretability.

Your proposed computational method possibly needs to be changed, too.   Details are explained below:  

Authors have cited the paper on "Sanity Checks for Saliency Maps", but have not used it to verify their own results, not with respect to saliency map, and not with respect to computational cost.  Several recent methods on saliency maps have not been considered, for example: 
1. Certifiably robust interpretation in deep learning 
2. Interpreting Neural Networks Using Flip Points 
3. Explaining Image Classifiers by Counterfactual Generation
 4. Counterfactual Visual Explanations 

It's not clear how your proposed saliency scores relate to methods that consider counterfactuals and the decision boundaries. For any image, there is a counterfactual image closest to it. One can also compute the closest image on the decision boundary of the model. Such image can reveal which pixels should be changed in order to change or to keep the classification. The relationship between the current method and those methods needs to be established.
 Assume that for an image, you compute the closest image to it on the decision boundary and let’s call the distance between them r (distance measured in l1 or l2 norm). Then, consider a ball centered at that image with radius r. All images inside that ball would have the same label as the original image and you can obtain all of them only by solving one optimization problem for finding closest point on the decision boundary. Would this be a less expensive computation compared to solving your proposed decoy optimization many times?

There are many hyper parameters in your optimization problem, and it seems that for each image, hyper-parameters should be tuned separately. However, if you seek the closest image on the decision boundary, your results would be independent of hyper-parameters. 

As you have mentioned, your optimization problem seems quite hard to solve, but no details are given on how long it takes to solve it for a single image. You have mentioned sometimes you cannot solve it if c is not chosen well. This information needs to be reported. Is it solvable at all with a large mask? How many trials did it take to obtain your saliency maps?

Your optimization is nonlinear, non-convex, non-differentiable, …. . How did you deal with non-convexity? Do you arrive at the same solution each time you solve it? These are vital information to about an optimization method.

Proposition 1 is quite obvious. Its discussion seems obvious, too, given how you have defined the Z. In your proof, you have considered a single layer network, i.e. a simple linear transformation. It might be better to explain your proof for Proposition 1 as an observation and use it as the motivation on how you have defined Z.
 Overall, you are taking a data-driven approach to interpret the classification of an image. The additional computational cost of your proposed method needs to be compared to other methods that are not data-driven. For example, how does your computational cost compare with the references above?

On page 12, one equation is referenced with ??.

At the beginning of section 3.4, you are using a standard optimization technique to augment a constraint as a penalty in the objective function. You can explain it as such, instead of saying we solve an alternative formulation. See Numerical Optimization by Nocedal and Wright.

The necessity to transform the variable via tanh, instead of performing projection is not explained. Does it make the process faster?

Strategy for choosing a good initial value for c (in the penalty term) and how to increase it is not explained well. How many iterations does it take to find the solution?

Any classification model is defined by its decision boundaries, so interpreting the classification without considering the decision boundaries seems inconclusive.

In summary, it would be best to compare your results with other methods, some of which are mentioned above. It would also be necessary to justify the additional data-driven cost that you are prescribing and to explain why your goals cannot be achieved otherwise with less expensive computation.



review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The authors tackle the important problem of generating saliency maps. First, a decoy image is defined (in short, it is a perturbed version of the image that leads to very similar activations) and then, the method that leverages decoy images is proposed. The method can be understood as a improvement that can be applied to enhance an existing saliency extractor. For a given image, the method generates a bunch of decoys images, and then a given saliency extractor is applied to not only the original image but all generated decoy images. The resulting saliency maps are aggregated to output the final saliency map. I found this idea technically sound.

However, there are two keys reasons why this paper should be rejected.
(1) The idea of using decoy images is interesting but computationally expensive. In practice (as showed in the paper) using blurred images leads to comparable results.
(2) The paper misses one very important experiment which is a quantitative comparison with the existing works on Imagenet ILSVRC’14 localization task. This is the standard experiment and is used in a few works cited (Fong & Vedaldi, 2017; Dabkowski & Gal, 2017).

Even though the idea of using decoy images is technically sound, I find the algorithm for generating these decoy images very complex and computationally expensive. First, under Eq.6 one can find "Our strategy is to set c to a small value initially and run the optimization. If it fails, then we double c and repeat until success." And then below Eq.8 there is "After each iteration, if the second term in Equation 8 is zero, indicating that τ is too large, then we reduce τ by a factor of 0.95 and repeat". Hence, the algorithm is nested and the paper does not provide any details how long the procedure is in practice. On top of that, since a population of decoy images is needed (12 in the experiments), this expensive procedure has to be run a few times. When decoy images are replaced with blurry images the results are almost the same and hence, unfortunately, using decoy images is not justified.

In general, the experimental results are decent and prove the possible benefits of using the population of images (decoys or blurred ones). However, I think that Imagenet ILSVRC’14 localization task is the standard experiment that provides the quantitative view and should be performed. The paper evaluates on Imagenet data set already, and hence adding this experiment should be straightforward.

The paper does a good job motivating the problem and covering related work. The paper states that the key limitation of existing saliency maps is that they "evaluate pixel-wise importance in an isolated fashion by design, implicitly assuming that other pixels are fixed" and "the presence of gradient saturation". While it is valid for gradient-based methods, it is not for perturbation-based methods. The latter are just briefly mentioned and then it is stated that "for any perturbation-based method, a key challenge is ensuring that the perturbations are effective yet preserving the training distribution". However, I do not find this reason to be strong enough to exclude these work from consideration, because there are works that train saliency extractor and the classifier simultaneously and then the argument mentioned does not hold (Fan et al., 2017; Zolna et al., 2018).

There is one more thing that I would like to ask about. In Eq.3, Z_j is defined as max(E˜_j ) − min(E˜_j ). Hence, if a given pixel is very important for all decoy images, all elements of E˜_j will be high and then Z_j will be very small. As a result, a saliency value assigned to this pixel will be low which seems to be counter-intuitive. Can you please elaborate on that?


(Fong & Vedaldi, 2017): Ruth C Fong, and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation
(Dabkowski & Gal, 2017): Piotr Dabkowski, and Yarin Gal. Real time image saliency for black box classifiers
(Fan et al 2017): Lijie Fan, Shengjia Zhao, and Stefano Ermon. Adversarial localization network
(Zolna et al 2018): Konrad Zolna, Krzysztof J. Geras, and Kyunghyun Cho. Classifier-agnostic saliency map extraction
review_title: Official Blind Review #1

Meta-review: This submission proposes a method to explain deep vision models using saliency maps that are robust to certain input perturbations.

Strengths:
-The paper is clear and well-written.
-The approach is interesting.

Weaknesses:
-The motivation and formulation of the approach (e.g. coherence vs explanation and the use of decoys) was not convincing.
-The validation needs additional experiments and comparisons to recent works.

These weaknesses were not sufficiently addressed in the discussion phase. AC agrees with the majority recommendation to reject.
Decision: Reject