Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: [score raised from weak reject to weak accept after rebuttal - on a more fine-grained scale, I would rate this paper now an accept (7), but not a strong accept (8), however since this year's scale is quite coarse, I'll stick with a score of 6]

Summary
The paper proposes a new perturbation-based measure for computing input-saliency maps for deep RL agents. As reported in a large body of literature before, such saliency maps are supposed to help “explain” why a deep RL system picked a certain action. The measure proposed in the paper aims at combining two aspects: specificity and relevance, which should ensure that the saliency map highlights inputs that are relevant for a particular action to be explained (specificity), and this particular action only (relevance). The paper shows illustrative examples of the proposed approach and two previously proposed alternatives on Chess, Go and three Atari games. Additionally the paper evaluates the method and the two previous alternatives on two interesting chess-tasks: chess-puzzles where human players were shown to be able to solve puzzles faster and with higher accuracy when given the proposed saliency map in addition, and evaluation against a curated dataset of human-expert saliency maps for 100 chess puzzles. 

Contributions
i) Novel, perturbation-based saliency measure composed of two parts. Main idea of specificity is (similar to Iyer et al. 2018) to use State-Action value function (Q-function) with a specific action, instead of State-Value function only. Main idea of relevance is to “normalize” by taking into account change in Q-value for all other actions as well. Both parts are combined in a heuristic fashion.

ii) More objective/reproducible assessment of how saliency maps produced by different methods overlap with human judgement of saliency of pieces in chess. To this end: two experiments with human chess players/experts (puzzle solving, and expert-designed saliency maps).

Quality, Clarity, Novelty, Impact
The paper is well written and most sections are easily understandable, though for some parts it might help if the reader is quite familiar with Chess/Go. The proposed saliency measure seems to address some shortcomings of previously proposed measures - my main criticism is that the construction of the measure seems rather ad-hoc and heuristic. It would have been great to formally define specificity and relevance (e.g. in an information-theoretic framework as Sparse Relevant Information) and then derive a suitable measure that is shown to satisfy/approximate the formal definitions. At least, there is one ablation study that justifies parts of the heuristic construction to some degree. 
The proposed measure seems to do reasonably well on board-game domains, in particular chess. However it might also be the case that the measure does particularly well for generating saliency maps for Stockfish (which is the agent that happens to be evaluated in the chess domain), which might be quite different from previously reported methods that have been designed for deep neural network RL agents. The illustrative examples on Atari and Go do not allow for a statistically significant judgement of the quality of the proposed method.

On a conceptual level, a bigger issue (of many saliency methods in general, but the criticism also applies to the paper) is that the “explanations” drawn from saliency maps are rarely evaluated rigorously. The paper makes a nice attempt by trying to establish some “ground-truth” saliency in chess using humans to increase the degree of objectivity, which I greatly appreciate. However, it remains unclear whether explanations that happen to coincide with human notions of saliency really are of higher quality for assessing how an artificial system makes its decisions. The main goal of explainability/interpretability methods must be to come up with testable hypotheses that tell us something about how the artificial system makes its decisions in novel/unseen situations. The goal is not to explain a decision mechanistically after the fact (which is trivial, given a deterministic, differentiable feed-forward system), but to come up with non-trivial explanations that extrapolate/generalize. Specificity and Relevance are probably important ingredients of such explanations, but I think it’s important to formalize them properly first. Currently I am in favor of suggesting a major revision of the work, but I am happy to reconsider my decision based on the rebuttal and other reviewers’ assessment. Having said that, I do want to reiterate that I think it is great that the authors included some ablation studies and measures of “usefulness” of the saliency method.

Improvements / Major comments
i) Formally define specificity and relevance (e.g. as sparsity and compression?). Ideally derive a saliency measure based on these formal definitions.

ii) Show how saliency maps (of the same situation) change when producing explanations for different actions. I assume that currently the illustrative examples only show the action with the highest Q-value, what changes when using e.g. a less likely action?

iii) Show that the saliency map stays roughly constant for seemingly irrelevant perturbations. In particular, using the chess-dataset with expert annotations, apply various kinds of perturbations to non-salient pieces (e.g. removing them, swapping them for other pieces and potentially moving them in irrelevant ways) and see whether the AUC stays roughly constant.

iv) Apply non-relevant perturbations to the salient pieces. I.e. take the same puzzle/scene and move it across the board, does the saliency-map move in (roughly) the same way.

v) The saliency method might be particularly suited for Stockfish (whose action-value estimates might be strongly influenced by human saliency and chess theory). See whether the method still produces good results for other chess agents (ideally trained without human heuristics or data). If this is hard to do for chess, think about a different application where this is easier.

vi) Add a section that discusses current shortcomings and caveats with the method, and saliency maps for explainability in general.

vii) (Experimental details) For each domain, please explain the perturbations that you used (removing pieces in Chess/Go, blurring pixels in Go, anything else e.g. replacing pieces?). In all experiments, was it always the action with the highest Q-value that was being explained?



Minor comments

a) Table 1: (add error-bars) What is the variance across players? Are the results for the proposed method statistically significant?

b) Chess saliency dataset. Are the expert saliency ratings binary? Why not have multiple degrees of saliency?

c) Would the Greydanus et al. 2018 approach deliver similar results when using a threshold to cut off low-saliency inputs?

d) Why is the saliency map in 3.4 binary (pieces are either salient or not)? How was the binarization threshold chosen? What would the non-binary saliency maps look like?

e) Please provide all experimental details (in the appendix) that are necessary to reproduce the experiments. Referring to a code-repository is not a replacement for describing the methods in detail.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper proposes an algorithm for explaining the move of the agents trained by reinforcement learning (RL) by generating a saliency map.
The authors proposed two desired properties for the saliency map, specificity and relevance.
The authors then pointed out that prior studies failed to capture one of the two properties.
To combine the two components into a single saliency map, the authors proposed using the harmonic mean.

The experimental results demonstrated that the proposed saliency map successfully focused only on important parts while the other method tend to highlight some irrelevant parts also.
The authors also did a great job for evaluating the goodness of the saliency maps, by preparing a human annotated chess puzzle dataset.

I think the paper is well-written, and the basic idea look reasonable and promising.
The experimental evaluations are well designed and the results look convincing.
Saliency map for RL is not yet mature, and I expect to see further improvements (especially, more theoretically principled ones) follow this study.


### Updated after author response ###
The response from the authors seem to be reasonable to me. I therefore keep my score.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: Interpreting the policies of RL agents is an important consideration if we would like to actually deploy them and understand their behaviours. Prior works have applied saliency methods to DRL agents, but the authors note that there are two properties - specificity and relevance - that these methods do not take into account, and therefore result in misleading saliency maps. The authors define (and provide examples) of these properties, propose simple ways to calculate these (and like prior methods, relying on perturbations and therefore applicable to almost black box agents), and combine them neatly using the harmonic mean to provide a new way to calculate saliency maps for agents with discrete action spaces. While the improvements on Atari are hard to quantify, the results on chess and go are more interpretable and hence more convincing. The study using saliency maps to aid human chess players is rather neat, and again adds evidence towards the usefulness of this technique. Finally, the chess saliency dataset is an exciting contribution that can actually be used to quantify saliency methods for DRL agents. The proposed method is relatively simple, but is well-motivated and demonstratively (quantitatively, which is great work for general interpretability methods) better than prior methods, and in addition the authors introduce a new dataset + quantitative measure for saliency methods, so I would give this paper an accept.

Although the authors motivate their choice of perturbation-based saliency methods as opposed to gradient-based methods, they should expand their review of the latter. As the technique the authors introduced is very general, it would be useful to know how it compares to the current state of research in terms of identifying properties that attribution methods should meet - a good example of this is integrated gradients (Sundararajan et al., 2017), which similarly identify "sensitivity" and "implementation invariance" as "axioms" that their method satisfies.
review_title: Official Blind Review #1

Meta-review: A new method of calculating saliency maps for deep networks trained through RL (for example to play games) is presented. The method is aimed at explaining why moves were taken by showing which salient features influenced the move, and seems to work well based on experiments with Chess, Go, and several Atari games. 

Reviewer 2 had a number of questions related to the performance of the method under various conditions, and these were answered satisfactorily by the reviewers.

This is a solid paper with good reasoning and results, though perhaps not super novel, as the basic idea of explaining policies with saliency is not new. It should be accepted for poster presentation.

Decision: Accept (Poster)