Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper has a double aim. First, it is a survey on saliency maps used in explaining deep reinforcement learning models. Second, it is a proposal of a method that should overcome limitations of the current approaches described in the survey.
This double aim makes the paper hard to understand as the survey is not complete and the model is not well explained.
The main limitations the novel model aims to solve seems to be the production of "falsifiable" hypothesis in the explaination with saliency maps. However, experiments are really hard to follow and it is not clear why this is the case.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Abstract:
The author suggests that saliency maps should be viewed as exploratory tools rather than explanation. The explore this idea in the context of a game.

Here is my main issue:
Although I believe there is a value in studies like this. I am not sure ICLR is the right venue for it. The paper is well written but it reads like a long opinion/blog-post. There is no overarching theory or generalizable observation not to mention a solution. 
Yes, I agree that the method of interpreting the black box has a lot of issues and the counterfactual approach/causal approach is probably the right way to go but this is hardly news to the community. 

In short: what the generalizable contribution of the paper?

I am open to change my mind if the discussion is convincing. 
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: [score raised from weak accept to accept due to rebuttal/improvements]

Summary
The paper investigates the practice of using pixel-level saliency maps in deep RL to “explain” agent behavior in terms of semantics of the scene. The main problem, according to the paper, is that pixel-level saliency maps often correlate with semantic objects, however turning these correlations into explanations would require counterfactual analysis / interventions, which are almost never performed in practice. The paper highlights this issue with an extensive literature survey, and proposes a simple method to formulate “explanations” found via saliency maps into falsifiable hypotheses. Three experiments show how to apply the methodology in practice - in all three cases pixel-level correlations cannot be easily mapped to semantic-level explanations that hold (counterfactual) validation.

Contributions
The paper nicely summarizes the main contributions, namely: (i) a literature survey on pixel-saliency methods in deep RL and their use to “explain” agent behavior, (ii) a detailed description of the problem with the latter and a proposal to mitigate the main issues, and (iii) three experimental case-studies to illustrate the problem further and show how the proposed method can help.

Quality, Clarity, Novelty, Impact
The paper addresses a highly important issue in the field of interpretable deep learning. The main message is that a lack of scientific rigor, namely stating falsifiable hypotheses and validation of claimed hypotheses, can easily lead to misinterpretation of deep RL systems. This is a somewhat disenchanting message, but I personally think it is important to ensure that this message is heard in the field of interpretable ML in particular, and in the wider deep learning community in general. It is tempting to give simple answers to complex problems, and while I think saliency maps will play a large role in interpreting deep network decisions, I am also convinced that we need causal explanations, which salience maps (currently) cannot provide on a semantic level. The paper is well written and clear, the literature survey is quite extensive and valuable. The experimental results are nice, however they currently crucially lack quantitative statements that back up the qualitative results (see improvements below). While the latter must be included for publication, I am fairly confident that this can be rectified during the rebuttal phase and therefore (tentatively) vote for acceptance.

Improvements
a) Mandatory for publication! Back the results-plots up by numbers! In particular: visually estimating densities / correlations from scatter plots is often impossible and misleading - while the plots are nice to have, the claims regarding Figure 5, 8, 9 (b) and (c) must be backed up by reporting actual correlations / statistical tests. For instance, it is impossible to judge visually whether there’s any trend in 5 (c). Please report correlations for 5, 8, 9 (b) and perform suitable statistical tests for measuring increase/decrease in correlation for 5, 8, 9 (c).
Similarly, please report an appropriate metric to quantitatively judge the difference between the curves in 4, 6, 7 (c). It’s fine to include tables reporting the quantitative results in the appendix, they don’t necessarily have to be in the main paper.

b) Experimental details. Please report the details required to reproduce the experiments. In particular, what was the precise architecture for A2C and the hyper-parameter settings (particularly since the reference that is cited is not a paper, but a GitHub repo). For the figures, please report how saliency was measured exactly (was there a bounding-box around saliency/enemies? What was its size? Were intensities somehow normalized, were distances to enemies measured between centers of bounding-boxes, …?)

c) (Optional). It would be nice to see an example where the method is used but the original hypothesis is not rejected (i.e. there’s now stronger evidence for the original hypothesis due to the counterfactual analysis). I understand that this is beyond the scope of the rebuttal, and feel free to completely ignore this.


Major Comments
I) Please state whether the paper was written with feed-forward deep RL agents only in mind, or whether the paper is intended to also include recurrent deep RL agents (it would also be helpful to know whether the experiments used a feed-forward, or a recurrent version of A2C). While I think that many aspects carry over from feedforward architectures to recurrent ones, I personally think that some issues with counterfactual analysis could become more intricate with recurrent agents. For instance, on page 6, the described invariance in the first paragraph under ‘Counterfactual Evaluation of Claims’ is fine for feed-forward agents, but could be debatable with recurrent agents.  If you agree, please make this distinction clear in the paper (where appropriate) or state that the paper only applies to feedforward agents. If you disagree please indicate this during the rebuttal discussion. 

II) Page 6, just above Sec. 5: “Since the learned policies should be semantically invariant under manipulations of the RL environment...”. I agree that they should ideally be invariant, for the semantic interventions to make sense, but please comment on whether this is a trivial assumption, how this assumption could (in principle) be verified and the potential consequences of this assumption being violated. I personally think that there’s a fair chance that the semantic space carved up by the agent (that potentially overfits a task/family of tasks) might be quite different from the semantic space given by the latent factors of the environment. This mismatch and its potential interference with the method should be discussed as a current shortcoming.

Minor Comments
I)  A potential subtlety (which I don’t expect you to resolve/discuss in the paper) is that feed-forward agents in an MDP environment can behave like recurrent agents by offloading memory into the environment. E.g. a breakout agent could “memorize” that it is in “tunnel-digging mode” by moving the paddle by a few pixels - this could then potentially shift it’s saliency away from the actual tunnel to the corresponding pixels around the paddle. Such cases might be very hard to interpret via saliency maps or interventional analysis, but I acknowledge that this is perhaps a more exotic case, given the current state of interpretable deep RL. Just a thought for future work perhaps...
review_title: Official Blind Review #2

Meta-review: This was a contentious paper, with quite a large variance in the ratings, and ultimately a lack of consensus. After reading the paper myself, I found it to be a valuable synthesis of common usage of saliency maps and a critique of their improper interpretation. Further, the demonstration of more rigorous methods of evaluating agents based on salience maps using case studies is quite illustrative and compelling. I think we as a field can agree that we’d like to gain better understanding our deep RL models. This is not possible if we don’t have a good understanding of the analysis tools we’re using.

R2 rightly pointed out a need for quantitative justification for their results, in the form of statistical tests, which the authors were able to provide, leading the reviewer to revise their score to the highest value of 8. I thank them for instigating the discussion.

R1 continues to feel that the lack of a methodological contribution (in the form of improving learning within an agent) is a weakness. However, I don’t believe that all papers at deep learning conferences have to have the goal of empirically “learning better” on some benchmark task or dataset, and that there’s room at ICLR for more analysis papers. Indeed, it’d be nice to see more papers like this.
 
For this reason, I’m inclined to recommend accept for this paper. However this paper does have weaknesses, in that the framework proposed could be made more rigorous and formal. Currently it seems rather adhoc and on a task-by-task basis (ie we need to have access to game states or define them ourselves for the task). It’s also disappointing that it doesn’t work for recurrent agents, which limits its applicability for analyzing current SOTA deep RL agents. I wonder if authors can comment on possible extensions that would allow for this.

Decision: Accept (Poster)