Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper applies an interpretability method based on discovering prototypical inputs to the problem of interpreting reinforcement learning (RL) policies. The authors create a model that uses a time contrastive loss to encodes states that lead to similar actions and that are similar in time as close to each other in embedding space. They connect this encoding to a behavior cloning model that first applies an isometric transformation to the encoding, then extracts a set of prototypes. The similarity between an input image and the prototypes is computed, and these similarity scores are fed to a fully connected layer that outputs scores for each action. Thus, the model can not only predict which action the agent is likely to take from a given state, but show the most relevant prototype state to explain why it takes that action. Experimental results reveal that the model is highly accurate (although not as accurate as a ResNet + BC baseline), and provides intuitive prototypes that help interpret the behavior of the RL agent.  
strengths_and_weaknesses: **Significance:** The results of this paper are likely to be significant to the RL community. As an RL researcher, I can confirm that I would find this method useful for interpreting the behavior of a trained agent and debugging why it is failing. Overall the idea for the paper is crisp and compelling. 

**Novelty:** The novelty of the paper may be somewhat limited. The authors state that the training procedure is "essentially identical to that of the related ProtoPNet [8]" (line 261). I recommend the authors clarify this (including line 240), since ProtoPNet is about image recognition and therefore it seems unlikely that it uses the same type of time contrastive loss. Nevertheless, the time contrastive loss used here is also similar to (but not identical to) that of Time Contrastive Networks [10]. Essentially the novelty seems to be in the combination and application of these existing techniques to the domain of RL. 

**Quality:** The experimental evaluation is relatively thorough, uses fairly complex environments, and compares to reasonable baselines. However, the performance of GAIfO is so poor, that it suggests the authors might need to debug their implementation of GAIfO, or choose a more competitive baseline. The authors should also provide some estimate of variance or error bars for their results in Table 1 and Table 3 (as this is part of the reproducibility checklist).

The "diagnosing bad behaviors" experiment is a compelling illustration of the usefulness of this method. 

The sensitivity to flip points experiment was creative and interesting, but there is not much we can conclude from the results unless they include a comparison to some other baseline (such as ResNet-BC). Is ProtoX more sensitive to flip points than other techniques because of its focus on prototypes? If not, this is not a very interesting result. 

The overview of related work is well written and complete, and clearly outlines the distinction between this paper and prior work. 

**Clarity:** Overall the paper is well written and clear. The examples provided in Figure 1 do a great job of framing the contributions of the paper and making them obvious. 

There are a couple of clarity issues however. One is the discussion of the time contrastive (TC) loss as only focusing on "visual similarity". This happens in the intro (lines 60-68), and again in the method (Section 3.1, line 194). Because the TC loss forces states that have similar actions and occur at similar times to be similar, it is not only focused on visual similarity. In fact it incorporates functional and temporal similarity. So referring to it as visual similarity is confusing and misleading. This leads to some awkwardness in the intro, for example where it concludes "Consequently, visually similar states are closer together in the latent space"... the previous sentences have not described why *visually* similar states would be close together at all. 

A second clarity issue relates to how the encoder is trained. Section 3.1 and Eq 1 provides an explanation for training the time contrastive encoder. But Section 3.3 appears to use very similar losses, and the differences between how they are applied is not clear. Is the procedure to first pre-train the encoder using the TC losses in Section 3.1, and then continue training the whole network using contrastive losses, but with respect to prototypes, in Section 3.3? Is the encoder frozen after pre-training?

It would be helpful to explain $a$ and $y_i$ in Eq 5. 
questions: Please explain the differences between this work and ProtoPNet [8] and clarify that explanation in the text.

Is the process of merging prototypes manual? This seems time consuming. Is there a way to automate it? 

Please add a reasonable baseline to the sensitivity to flip points experiment. 

Clarify how time contrastive losses are used in pre-training vs when optimizing Eq. 7. 

The caption in Figure 8 refers to the policy running Mario as "he", and makes a statement about what "he thinks", which is anthropomorphizing the RL policy and overclaiming. Please fix. 

Since GAIfO performed so poorly, improving its implementation or adding a more competitive baseline would improve the paper. 


limitations: The authors acknowledge the limitations of their method with respect to manually merging prototypes, and the fact that interpreting prototypes requires knowledge of the task. 

The societal implications of the work are not discussed. Better interpretability for RL actually could have a number of beneficial societal implications, since it would help with deploying RL to safety-critical human environments (e.g. for robots). This would be good to discuss.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces a method for explaining black-box RL agents. This is accomplished by learning "prototype" states and identifying which prototype the current state corresponds to.
Overall a good and potentially impactful paper and I'm recommending acceptance but conditional on some clarification.
strengths_and_weaknesses: TL;DR I think the method introduced in this paper is powerful and desperately needed as a tool for the RL community. I just don't have an intuition on how the prototypes are discovered, even after multiple reads. I'm happy to raise my score if the authors address some of my concerns below.

### Strengths

- S.1) Very cool idea and very useful to RL folks.
- S.2) General paper structure good and illustrations helpful.
- S.3) Flip point and fidelity evaluation are good ideas. 
- S.4) Easy-to-reproduce Jupyter notebook included. 

### Weaknesses

- W.1) I don't fully understand how prototypes are automatically generated, end-to-end. I've seen the different losses in sec. 3.3, I've looked at the pseudocode in the appendix and I had a very quick glance at the Jupyter notebook but I still don't have a good intuition on how they come to be. Maybe other reviewers got behind this more easily. If you could make another attempt at giving me a high-level intuition on the mechanism behind the prototype learning in the rebuttal phase, I'd appreciate that.
- W.2) Analogous to this, there are a few expressions in the paper that are too unclear for me, like lines 61 ("If two states..."), 263 ("During this step..."), 272 ("We then also create...")
- W.3) I think using prototypes for explainability calls for some more qualitative analysis. I'd like to see some more examples with annotations on what they indicate, like Fig. 7. The purpose of the paper is to provide a "policy explainer" and I don't have a feeling that a policy was explained, other than in the "bad behavior" example.
- W.4) Using 30 most similar frames to a given frame and combining them transparently is conceptually a good idea but as a reader, I can't really make out a lot there. I'd recommend either using fewer frames or making a grid of frames or maybe even 5 images of 6 overlayed frames or something similar. 
- W.5) 2 minor writing things: (a) I think that figures should stand on their own in a paper and a figure + its caption should contain all the information needed to parse it. In your paper, a lot of times, the caption of the figure is actually in the main text of the paper, not in the caption. For example, the captions for Fig. 6 and 7 are in lines 350+ and 359+ respectively. (b) I like it when at the end of the introduction, there's a list of contributions, either in text or as an actual list but in a very clear and condensed form and highlighted as "contributions".
- W.6) 2 minor experimental things: (a) Why did you choose GAIfO? Why is this a valid baseline? I don't see any similarity or comparability. It's not made to explain anything, it's trained differently, and its networks are different. The only similarity is that it's an imitation-learning method (which is not the focus of your method). (b) In the bad behavior example: you have a policy that always outputs "RIGHT", right? Then if I understand correctly, your ProtoX can't learn nuances between states because all states have the same action. Therefore the conclusion of "it predicts that the danger (goomba) is equal to flat ground" is by definition and a bit pointless, no?
questions: I've included most of my burning questions in the section above. Here's some minor stuff:

- Q.1) In the checklist, you mentioned "the code and data are proprietary" - what data?
- Q.2) You also mentioned all experiments were done with a single seed - did you do a preliminary test to verify that indeed your method isn't sensitive to random seeds?
- Q.3) How sensitive is your method to the loss coefficients?
limitations: The authors adequately addressed limitations. I'd also add that maybe this method could in the future be combined with visual saliency methods to highlight what part of a prototype is responsible for the agent's decisions.
review_title: null

final_score: rating: 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
review_content: summary: The authors seek to explain black-box deep reinforcement learning agents using prototyping. Prototyping groups together agent behaviors and game-states into prototypical states. They achieve this using ProtoX, which is a self-supervised neural network that uses prototypical states in its forward pass. Prototypes are trained so that they represent groups of similar game states according to visual similarity and action similarity. They're used within ProtoX, which is trained to accurately predict the action taken by the agent given the observation. They demonstrate the method on Pong, Seaquest and two levels from Super Mario Bros. The first show that ProtoX has good fidelity compared with alternative methods (fidelity measure by whether the methods groups states with the same action together). Then they show that ProtoX is sensitive to flip points, which are states that are temporally adjacent but where the agent chooses a different action. Then they demonstrate ProtoX on seaquest and show that prototypes match the scenarios that activate them and show that prototypes of scenarios where the agent exhibits bad behaviours share a similar action to the true action used in the scenario. 
strengths_and_weaknesses: The authors do a good job of presenting and explaining their method very clearly. I wasn't previously familiar with prototyping methods and have learned about them thanks to the authors. I am not aware of previous deep RL interpretation methods that take a similar approach; the approach is therefore likely original. 

I will judge the paper's quality and significance according to the stated goals of the paper, which are nicely emphasized in the introduction: 

(line 23) 'Popular solutions to interpreting a black-box agent include attention-based methods [3] and saliency maps [4, 5, 6]. These explanations, however, only inform users of where the agent “looks at”, but do not provide a more specific explanation for why an agent takes this action in a given state'
(line 30) 'This paper aims to explain a black-box agent that works with unstructured data.'
In summary, the authors state in the introduction that the goal of the paper is an explanation of _why_ an agent takes an action in a given state. 

At this point it's worth reflecting on what an explanation of an action (in a given state) actually is, since this is the whole point of ML interpretability work and of this paper. An explanation of an action in a given state should give us an understanding of why the state causes to the action; it should illuminate something about the black-box process that results in the agent taking a particular action in a particular state. Unfortunately, to this end, ProtoX does not succeed. ProtoX only tells us which states are similar to states where an agent took a given action. While prototyping is a valid though (arguably weakly informative) black-box interpretation method, this is only true if the prototypes are on the feature-level, i.e. they should tell us what parts of the state they are prototypes of. ProtoX provides protoypes that aren't on the feature-level; it provides us only with a global picture of the state. It reveals nothing about the black-box process that led from the state to the action. The authors justifiably criticise attention-based methods and saliency maps that tell us little about why an input leads to action; but even those methods tell us markedly more about the internals of the black-box than ProtoX, since those methods at least help us to identify features that the black-box decision making process is using.  

Validation of the explanations provided by our interpretability methods is an important step, one that the authors perform in section 4.4 (Diagnosing bad behaviours). But I'm not convinced that this demonstration explains why the agent makes a mistake here (and my reason is in addition to the above criticism that ProtoX doesn't help us to explain the black-box decision making process): ProtoX is trained to group states together that are both visually similar AND that have similar actions. Rather than explain bad behaviour, figure 8 therefore shows only that both the prototype and the real scenario both involve the agent moving right and have similar visual features (because that is what the prototypes were selected to do), but doesn't explain anything about why the bad behaviour happened.  

ProtoX is an interesting network, and the ability to group together similar states (in both the visual and action modality) is a praise-worthy feat. But as an approach to help explain _why_ an agent takes an action in a given state (which was the stated goal of the paper) it does not succeed. 

Minor criticisms:

The authors write (line 280): "We evaluate ProtoX on four classic control problems from OpenAI Gym and compare the results with two black-box and one interpretable baseline". The environments the authors are referring to are Pong, Seaquest and two levels from Super Mario Bros. I'd encourage the authors not to call these "classic control problems from OpenAI Gym" because gym.envs.classic_control is a different set of environments than Atari games. Their conflation will be confusing to many readers. 

(Line 133, footnote). The authors can avoid the abuse of notation by instead describing their RL setup as an MDP: From Wikipedia for POMDP: "an MDP can be reformulated as a POMDP by setting the observation set to be equal to the set of states and defining the observation conditional probabilities to deterministically select the observation that corresponds to the true state."

questions: I could be convinced to change my assessment if the authors could show how ProtoX can explain something about the agent's internals, which is necessary to explain why an agent takes an action in a given state. This could perhaps be achieved by augmenting ProtoX to identify feature-level prototypes. 
limitations: The authors have a good discussion of the method's limitations. They are non-obvious and I commend the authors for highlighting them.
review_title: null

Meta-review: metareview: This paper proposes ProtoX, a method to identify input prototypes of reinforcement learning agents that are representationally similar to tetst-time inputs. This allows one to surface "relevant training examples" matching the action predictive behavior seen at test time. This is proposed as an "interpretability method to explain agent decisions". Two reviewers have advocated strongly for accepting the paper, while one reviewer has advocated for a strong reject, primarily on the grounds that the paper makes unsubstantiated claims that "ProtoX explains why an agent took a particular action". 

After some deliberation with the reviewers, I will recommend accepting the paper. Here is my rationale: 

1. there are several tools one can use to better understand how training data and inputs within an example affect predictions. one can take a "formal, axiomatic approach to attribution" (e.g. integrated gradients) or an "informal, non-axiomatic approach to attribution" (e.g. smoothgrad) when it comes to showing "how inputs/training data explain predictions" . Both types of methods have their uses in the toolkit of every deep learning researcher. This paper proposes an *informal* tool to link training data with test time inputs. ProtoX is dataset-level attribution (via computation of prototypes), which is exciting because most explainability techniques focus on example-level attribution. Even though this paper does not prove implementation invariance of ProtoX method, it  as a useful debugging tool for RL practitioners. 
2. defining "what does it mean for a neural network to be 'interpretable?" and "why did this neural net make a certain decision" is a broad question within the ML community (bordering on irreducible philosophy), and it's too high of a bar to expect any paper to have a definitive, one-size fits all solution to explainability. I suspect reviewer m9zL found the claims of the paper to imply that they were proposing a more axiomatic attribution method, whereas it is really intended as a tool to diagnose any RL agent. 
3. Given that RL agents are very hard to get right, I could see this method (or perhaps an improved version of it with less moving pieces) as a useful tool.  

If I were to point out a weakness of the paper, it is that it bears a lot of similarity to non-parametric imitation learning algorithms, e.g.
VINN paper (Surprising Effectiveness of Representation Learning for Visual Imitation Learning, by Pari et al. 2021 https://arxiv.org/pdf/2112.01511.pdf) where k-NN on training set essentially also surfaces attributable examples from a training set wrt test time images, by design. I have no connection to the VINN paper, but would appreciate it if the authors cited this paper or mentioned prior literature on non-parametric methods in learned embedding spaces as an existing tool already used by the RL community. 

confidence: Less certain
Decision: Accept