Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents an explainable method for deep reinforcement learning agents. A gaussian process is used to capture the correlations and the effects between episodes, and is then used to predict the final reward. Further, the authors introduce a method (EDGE) that can provide high-level (strategy-level) explanations of the agent behaviour. Evaluation is done in 5 RL game domains, with results showing higher fidelity against the compared baseline methods. 
main_review: The proposed method is novel, in that it tries to identify the importance of a time step. This is in contrast to most other Rl explainability methods that are focused on feature/state based explanations. This explainable method enables the understanding of the agents’ policy, and can be used to identify policy errors and weaknesses. The work is positioned well in the explainable RL literature, through a well summarized related section. The technical definitions and equations appear to be sound.
 
The main weekend I see in this paper is in the limited selection of evaluation domains. It is doubtful whether the strength of the proposed model can be demonstrated in these environments (especially in the pong env), as the time step frames can look much similar to each other. This limitation is exacerbated due to the lack of a user-study to evaluate how useful the time-step importance explanations are against other explainable models. In Section 5, authors give their own interpretations of the explanation frames, but this is subjective and can vary from user to user. I encourage authors to add a discussion to address this issue. Other evaluations that focus on policy errors are reported and explained well.  

Quality: 
The technical definitions and equations are sound. As mentioned above evaluation suffers from some limitations. I believe these issues can be addressed with a careful commentary on the nature of the interpretability of the proposed method from a user perspective (or better yet a user study).
 
Clarity:  
 
The Paper is well written and is easy to follow. Methods are defined well enough for the reader to easily follow and understand.
 
Significance: 
 
This paper has a clear contribution that is in contrast to many other explainable RL methods. The use of importance of the time-step is a novel direction. Although there are somewhat similar methods that use policy summarization [1], this method explicitly focuses on the importance of time-steps. The evaluation hinders this paper from highlighting the importance of the contributions. In its current form, it is not clear how these explanations are better than the other types of RL explainability methods (saliency, policy-based etc). The paper can significantly be strengthened by providing evaluations in other domains (can be other Open AI gym envs) that can have more strategic agents. I do believe this approach has promise and would be of great importance to the larger explainable RL community.
 
 
 
[1] Amir, D., & Amir, O. (2018, July). Highlights: Summarizing agent behavior to people. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems (pp. 1168-1176).

limitations_and_societal_impact: The main technical limitation lies in the evaluation as explained above. The authors have discussed the societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a method that uses the correlation between different episodes as well as the correlation between states within an episode to provide explanations on deep RL agents. The explanation module mainly consists of an RNN module and deep gaussian processes. Later in the paper, the authors show that the provided explanations could be used to attack deep RL agents, make them more robust, and repair them.

main_review: The idea presented in the paper is interesting, and the approach seems promising. However, there are a few points that need to be addressed in order to increase the quality of the paper. They are mentioned below.

In methods where an attention module is embedded in the deep RL agent's architecture, provided explanations could be easily misleading because the soundness of such approaches is not apparent. The reason for the unclear inferred explanation is that the attention module parameters are affected by the input observations and memory of the agent, which are ignored when an interpretation is given. This misleading explanation makes it hard to find if the attention patterns embed strategic information or just information highlighted by the attention that is key to decisions.

Regarding lines 56-57: The authors claim that their proposed method is the first to explain deep RL policies from a sequence of its past actions and states. However, that is not an accurate claim. [1] and [2] both try to interpret deep RL policies with respect to the actions the agent has taken and the values it has in its memory, resulting in interesting insights into the workings of deep RL agents. 

Regarding lines 79-80: How do authors compare their method to [2]? Because in [2], the time steps in which critical decisions are made are also identified and investigated.

Regarding lines 269-271: If the policy has a memory in any form of RNN, this is simply invalid because manipulating the trajectory of $(t_i, ..., t_{i+l})$ would affect the policy's memory.  So simply feeding state at $i+l+1$ would not be sound.

Regarding line 275: How does $d_{max}$ is calculated?

Regarding line 277: How about normalizing the first term, $log(p_l)$, of the equation by a coefficient in $[0,1]$? The reason is that it could have the most effects on the fidelity score, thus undermining the second term, $log(p_d)$.

Regarding Section 5: Many other things could be extracted from these sorts of explanations (lines 298-299 and 301-302). For example, one could conclude that the agent is just focusing on hitting the ball, thus not losing, rather than focusing on hitting it in a special way. We need to be very careful about our conclusions.

Regarding lines 297-298: It is mentioned that "EDGE highlights the time steps when the agent hits the ball as the key steps leading to a win." Is this claim consistent across different policies?

Regarding line 307: It is mentioned that the examples extracted from three games are provided in the Supp material. However, to show the soundness of the approach, a diverse set of games should be investigated.

Regarding Table 1: Please provide variance as well as the mean.

Regarding lines 312-313: How about strategically timed attacks? They do the same without the need for any explanations [3]. A comparison between the two methods could improve the claim in the paper.

Regarding Patching Policy Errors: It is mentioned that a table could be formed from the state-action pairs to be used as a remediation policy. How is it possible to form a table from such rich and continuous observations and hoping to see them again? The observation space should become finite in a way to make it make sense.

----------------------------------------------------
Grammar, typos, etc.:

Line 5: "augments an Gaussian process" -> "augments a Gaussian process"

Line 16: "success in automatically policy learning" -> "success in automatic policy learning"

Line 28: "existing methods can not shed" -> "existing methods cannot shed"

Line 204: "likelihood is computational expensive" -> "likelihood is computationally expensive"

Line 344: "patched policy introduce too many" -> "patched policy introduces too many"

Line 382: "demonstrates the our method’s" -> "demonstrates our method’s"

----------------------------------------------------
References:

[1] Koul, Anurag, Sam Greydanus, and Alan Fern. "Learning finite state representations of recurrent policy networks." arXiv preprint arXiv:1811.12530 (2018).

[2] Danesh, Mohamad H., et al. "Re-understanding Finite-State Representations of Recurrent Policy Networks." International Conference on Machine Learning. PMLR, 2021.

[3] Lin, Yen-Chen, et al. "Tactics of adversarial attack on deep reinforcement learning agents." arXiv preprint arXiv:1703.06748 (2017).
limitations_and_societal_impact: The authors have adequately addressed the limitations of their proposed method.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new pipeline for understanding agent behavior + actions, and identifying which timesteps contributed to the final reward. The key component of this method is the pipeline, which consists of the following:

* An RNN which takes in state-action pairs, and "compresses" the information via its RNN-state at every timestep.
* All RNN-states over all time steps are taken and inserted into a zero-mean Gaussian Process (GP), which consists of two independent GPs, one for correlations between time-steps, and another for correlations between episodes.
* A feature vector $\mathbf{f}$ is then sampled from this GP, and used to classify the reward. 

Because a GP is involved, variational inference is needed for training.

The paper then experimentally verifies this result on Atari Pong + Adversarial Mujoco tasks by showing competitive performance on fidelity (i.e. metric based on replacing an intelligent agent with random actions) and visualizing which parts of an episode contribute most to the fidelity score. Finally, the method is also applied to successfully perform adversarial attacks which also beat previous baselines.
main_review: Pros:
* This method is applied to scenarios where access to the agent policy is restricted (or might not even be a neural network), and in offline settings, which makes it broad and useful in many settings, compared to saliency (which is specific to neural networks).
* The experiments seem convincing and strong compared to saliency-based methods, and visualizations confirm the validity of the method.


Cons:
* The entire pipeline is pretty large and complex, and involves many different components. It took quite a while for me to understand the overall picture of what was happening, mainly from staring at Figure 1 and matching with the text. There's also a lot of fine-grained details in Sections 3.2 + 3.3, some of which should be moved into the appendix - is it possible to explain the method at a high level using brief math instead? I think much of the method can be simplified for the average reader to digest.
* The paper requires a somewhat restrictive assumption, which is that in every episode, there are specific timesteps that contributed to the overall reward. I feel like this is only applicable to a limited set of RL environments, such as Pong and Adversarial Mujoco, since they only produce a +1/-1 reward at the very end of the episode. Would this method work for environments that always produce rewards at every timestep? 
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The authors propose a new approach to explaining deep reinforcement learning policies. The approach applies to complex or black box policies, and learns Gaussian Process (GP) model with custom function to learn to predict and assess the importance of state-action pairs on game outcomes. The authors illustrate feasibility of their approach in Atari and MuJoCo domains and demonstrate several application areas.

Initial reviews assessed the contribution as strong, especially because of the method's wide applicability and strong experimental evidence. Several concerns were raised as well, including clarity (e.g., the complexity of the method and need for more precise explanations) and concerns about potentially restrictive assumptions. Suggestions for improvements included additional empirical evaluations, including a proposed user study to test the effectiveness of the approach. The discussion between reviewers and authors has been highly productive, with many concerns addressed. The authors went above and beyond by running the suggested user study during the rebuttal period. 

As a result of the productive discussion, all reviewers indicated that their concerns have been largely addressed and reached the consensus to recommend acceptance. The AC agrees with this recommendation. Given that substantial new insights were developed during the discussion, the authors are strongly encouraged to carefully incorporate all suggestions as they prepare the camera ready version.
Decision: Accept (Poster)