Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a method that offers explanation of action preference in a deep RL agent based on given features by the human. In other words, the model explains why action A is preferred to action B based on some given features. This is done through the embedded self-prediction (ESP) model. The authors also proposed a method for evaluation of importance of features in the learned policy. While the paper benefits from extended experimental result and interesting theoretical analysis in the tabular RL, I think its readability could be improved. For example, I believe generalized value functions should be explained more extensively as (I think) it is less known to the community compared to concepts like MDP and DQN. Also, an analysis  (or at least some discussion) on the effect of the number of features on learning QFs would be helpful. Another question I have, is about the dependence of features. What would happen to the evaluation if some given features are dependent?
Some minor points: 
- It would be helpful if the authors add a figure of their networks similar to what is common in the field, specifying input and output of the networks. 
- The phrase of "greedy policy" was a little confusing for me, especially because "greedy algorithm" is usually one-step look ahead search.  Is it just argmax Q(s, a) (as suggested in page 3)? 
- I think the defense on manually-designed features could be transferred to conclusion since the importance of interpretability has been already mentioned in the first paragraph. 
   
review_title: Interesting paper at the intersection of RL and interpretability

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ### Summary

The paper attempts to improve the interpretability of RL agents' action selection process by (a) proposing embedded self-prediction (ESP), a model that embeds generalised value functions (GVFs) in the action-value function of the agent with a "combining" function to and (b) ESP-DQN, an extension of DQN that augments experience replay tuples with a GVF feature vector and that decomposes the model into separate combining and GVF parameters. These enable to define action-values with respect to predefined feature maps, thus providing more "resolution" into the behaviour of the policy.

### Good stuff

1. The idea of decomposing the policy into GVFs as a way to force explanations wrt. some features is *brilliant*, and it is well executed when combined with the contrastive explanation system.
2. Sections 2-4 provide both a clear introduction to GVFs as well as a detailed and sound description of the overall framework.
3. The related work section is fairly tight, but actually covers a good amount of necessary and relevant related work, which makes it easy to scan through the literature.
4. The experimental section does employ mostly fairly similar environments, but it is clear in the hypotheses being tested, and it is fairly satisfactory considering what it is attempting to evaluate.

### Uncategorised notes

- This is more of a meta-comment, but I enjoyed reading the paragraph about manually-designed features in Section 1. I understand why the authors felt the need to write it, and it is a sad state of affairs that it is now often a requirement to argue what to many people is just reality.
- I wonder if it'd be worth it to test the method on Atari, through possibly the use of MinAtar: https://github.com/kenjyoung/MinAtar -- It feels like there's a gap in difficulty (of learning and analysis) between the ToW setting and the rest of the environments, and something aking to a middle ground would probably be a good setting to add. Complex gridworlds such as BabyAI and the NetHack Learning Environment are probably also good options.

### Final comments

Overall, I'm extremely happy to strongly recommend this paper towards acceptance. It is well written, it introduces a method that attempts to move forward towards solving an important problem in Reinforcement Learning, and there's a significant amount of details in the paper that would make it fairly straightforward to reproduce.

### Typos
#### Sec 1

- RL agents explain its... -> their
- directly "embed"... -> embeds
- train those... -> trains

#### Sec 3
- Combination function update -> Combining? The manuscript is at times a little inconsistent.

review_title: Good submission

final_score: rating: 7: Good paper, accept
review_content: Edit: 

I have read the authors' response as well as the other reviews. Based on the additional results and added feature selection details, I now agree that ESP is generally applicable. 

Summary: 

The authors present ESP, an RL system that can then explain action choices in terms of future feature values. Generalized value functions (GVFs) are used to learn an estimate of total future discounted feature values. These estimated total feature values are then used to estimate each actions Q-value. Since Q-values are based on GVF outputs, these intermediate values can be used as an explanation. The authors select a subset of these values to form a Minimal Sufficient Explanation (MSX). The proposed system is evaluated using three domains. The authors show that performance is comparable to non-GVF agents.

Reasons for score: 

Though the authors present a novel explanation format, the applicability of the method is uncertain. The results appear to rely on specific GVF feature choices. Non-general methods are still of interest, but the limited information on feature construction prevents a fair comparison to other approaches. Additionally, the explanations are not evaluated quantitatively. 

Pros:

-The use of GVFs for explanations in terms of future feature values is a novel line of work. MSXs are a natural way to then produce more concise explanations, and the authors extend MSX to their non-linear use case in a well-justified way. 

-The analysis of the Tug of War explanations was thorough. It clearly showed how ESP explanations would be used to investigate agent behavior. 

Cons:

-ESP is built upon the GVF features, but the choice of GVF features is suspect. Each environment uses a different style for its features. Lunar Lander and CartPole both have continuous features, yet the authors use "deltas" for Lunar Lander and region discretization for Cart Pole. Tug of War uses a bunch of features, including information about feature values when a game ends, and non-linearities are applied to the outputs of the GVF features. Note that these non-linearities are used *only* for Tug of War, and different non-linearities are used for different features (Table 2 in Appendix D). Effectively, each environment appears to use carefully engineered features. Given that DQN-Full performs very poorly for specific settings (i.e., in some cases, the agent cannot learn without the GVF features), the choice of features seems to be important. The authors should indicate the process used for selecting them and how these features should be chosen for other environments. ESP may not be robust to GVF feature choice, but this is insufficiently addressed in the paper.

-In Section 6.3, the authors present potential conclusions that can be drawn from an ESP agent. These conclusions can be evaluated to determine whether valid conclusions can be drawn from the explanations. Such an evaluation would allow the hypotheses of the authors to be tested.

-A substantial reorganization of the paper would improve clarity. Various definitions and descriptions are provided a few sections after the terms/methods are first used. Terms are unnecessarily over-loaded (such as "sound"). 

Questions During Rebuttal Period:

-Please address and clarify the "Cons" above.

-In particular:

a) How were the GVF features chosen? Why does each environment use different features? 

b) What happens when Lunar Lander is given "CartPole-style" (region discretization) features? What happens when CartPole is given "Lunar-Lander-style" (change in features) features? 

Minor comments:

-This work would benefit from another editing pass for tense/plurality matching between subject and verb.

-The proofs in Appendix B would benefit from a re-write; currently, they are hard to parse. 

review_title: ESP appears to rely on careful GVF feature engineering; further information/evaluation needed

Meta-review: This paper tackles the important problem of endowing deep RL agents with added interpretability. Action values are decomposed as the combination of GVFs learned on externally-specified features, offering action explanations in terms of discounted future returns in the space of interpretable quantities. Reviewers praised the approach, as well as the level of detail for reproducibility purposes. R3 had concerns about the generality of the method but follow-up experiments have allayed these concerns. Given the reviewer response and the central importance of the problem considered to the field, I can wholeheartedly recommend acceptance.
Decision: Accept (Oral)