Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper focuses on how to construct a wrapper model that is applicable to any arbitrary trained agents which leads to interpretability improvement in deep reinforcement learning. Their technique is called Prototype Wrapper Network (PW-Net) with the key concept of using human knowledge for prototypes and mapping between prototypes and actions.

The main contributions and findings of the paper:
- Using prototype-based learning in deep reinforcement learning
- Pre-hoc interpretability, forcing the agent to learn the human-designed prototypes makes the model inherently explainable
- Performance of the agent doesn’t degrade by using the proposed wrapper
- User trial to measure user’s predictions (failures and successes) of different models
strength_and_weaknesses: Strengths:
- First prototype-based deep reinforcement learning technique
- Can be added to any pre-trained agent
- Experiments fit the task by comparing the proposed method to the black-box agent
- Evaluated the technique in a user study

Weaknesses:
- Experiments section lacks some explanations of the evaluation metrics and results (see questions Q2-Q4)
- Needs hand-crafted prototypes: my concern is the ease-of-use of the method, as it needs extra work from humans to define the prototypes (considering all actions with multiple prototypes) and also the weight matrix. End-to-end learning is usually preferred because it doesn’t need extra hand-crafted elements.

Questions and suggestions for the authors:
Q1: Figure 1.: indices of h, z and p. Even though it is highlighted in the text that the model uses separate mapping, the image of the network shows the same indices (i,j). 

Q2: What is the reason for the different evaluation metrics (MSE vs Accuracy) of discrete and continuous action spaces? 

Q3: There are no results of the VIPER in Car Racing and Atari Pong. Why?

Q4: As your technique doesn’t give any advantage to the trained agent, how could the PW-Net achieve slightly better results in the experiments section?

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and has a good structure. Regarding novelty, it is the first deep reinforcement learning method based on prototypes. The authors of the paper provide both the code and the user surveys to be straightforward to reproduce the results.
summary_of_the_review: Overall, I vote for accepting. I like this paper because the proposed technique can provide inherent interpretability for deep reinforcement learning agents. My main concern is the extra work and time of creating human-defined prototypes and weight matrix that is needed for the wrapper network.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new method, Prototype-Wrapper-Network (PW-Net) to offer explainability of pre-trained RL policies by training a new set of embedding functions on top of the existing feature representation for a learned policy. These embedding functions are each associated with a given prototype/exemplar for salient actions in a given domain. The fine-tuning process learns to map from the learned policy's representations into similarity functions for each prototype, and the final output action is chosen as a pre-defined weighted combination of similarity scores for each prototype. By having direct access to similarity scores for different actions in the domain, and by pre-defining the mapping from prototypes-to-actions, humans are given the opportunity to better understand actions taken in a domain. The PW-Net is evaluated to show that it does not reduce overall performance for pre-trained policies, and a user-study shows that humans can use the given prototypes to better understand how the policies work (and to trust the method more).
strength_and_weaknesses: Strengths:
* The PW-Net is explained clearly and is easy to follow.
* Experimental results verify that black-box policy performance is preserved with the PW-Net wrapper and output mappings.
* Human evaluation shows that the prototypes and PW-Net explanations improve user trust enhance user's abilities to predict policy behavior.

Weaknesses:
* Much of the method must be hand-designed by humans. This is discussed in the paper as a positive, because prior work that simply learns prototypes does not always learn human-usable prototypes, but hand-designing mappings and prototypes is a non-trivial exercise, particularly for something like a 7-DoF robot arm.
* The PW-Net assumes that learned representations are fixed, so any amount of fine-tuning for the policy would require additional fine-tuning for all learned prototypes and embedding functions.
* The prompts and instructions in the user study vary significantly from the control to the proposed method. It is not clear whether human users are rating the control as less trustworthy because the PW-Net prototypes are significantly more useful or explainable, or because participants are being explicitly told that the control has seen "MILLIONS of situations you are NOT being shown." Apart from that very bold text prompt at the top of every question, the two surveys appear to be identical. It therefore seems possible that the user study results are not the result of more interpretable prototypes or a useful mapping function, but just because users are being reminded that one method is less trustworthy than another. Intuitively, the method should improve trust and accuracy, as is reported here, though recent work on explainability methods with humans finds that this is not always the case [1, 2, 3, 4].

[1] Poursabzi-Sangdeh, Forough, et al. "Manipulating and measuring model interpretability." Proceedings of the 2021 CHI conference on human factors in computing systems. 2021.

[2] Silva, Andrew, Mariah Schrum, Erin Hedlund-Botti, Nakul Gopalan, and Matthew Gombolay. "Explainable Artificial Intelligence: Evaluating the Objective and Subjective Impacts of xAI on Human-Agent Interaction." International Journal of Human–Computer Interaction (2022): 1-15.

[3] Peter Hase and Mohit Bansal. 2020. Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5540–5552, Online. Association for Computational Linguistics.

[4] Andrew Anderson, Jonathan Dodge, Amrita Sadarangani, Zoe Juozapaitis, Evan Newman, Jed Irvine, Souti Chattopadhyay, Matthew Olson, Alan Fern, and Margaret Burnett. 2020. Mental Models of Mere Mortals with Explanations of Reinforcement Learning. ACM Trans. Interact. Intell. Syst. 10, 2, Article 15 (June 2020), 37 pages. https://doi.org/10.1145/3366485

clarity,_quality,_novelty_and_reproducibility: The work is clear, and relation to prior work is relatively clearly stated and explained. The user study was not overly clear from a reading of the paper, and required inspection of the source materials to get a better understanding of what participants saw and what their response options were.

While prior work in the field has used exemplars or prototypes for explainability, this work is novel in its use of a hand-defined mapping function from human-specified prototypes to action output, and the method is a valuable contribution to the community.

The work appears to be reproducible and code is included, though I have not tried to run the code to confirm results. The user study materials are included and, with significant effort to rewrite every question, the study could be repeated to confirm those results as well.
summary_of_the_review: The paper contributes an explainability method based on human labels and action prototypes, offering useful explanations for output actions that are verifiable through known prototypes and similarity scores. While the method is sound and the performance results are convincing (i.e., no performance drops with the PW-Net), the user study design is questionable and it is not clear that the trust and accuracy ratings from participants can be fully attributed to the PW-Net rather than simply to the instructions of the study. 

I advocate for acceptance on the basis of the method, but I would like to see a clearer description of the user-study in the main paper and a discussion of possible confounds for the results (such as the instructions telling participants that one method saw millions of other samples). If the study cannot be appropriately defended, I would even advocate removal of the study, on the basis that it is likely not studying explainability, but is instead studying the effects of a text prompt. With clearer explanation of the study or removal of the study, I will increase my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new post-hoc explainable method, called PW-Net, to provide insights into the workings of deep RL policies. The main advantage of the proposed method is to use human-friendly prototypes to make the decision-making process of an RL agent clear. Also, a user study is included to support the main claims of the paper.
strength_and_weaknesses: Following are suggestions, questions, and concerns that I had while reading the paper.

1- It is mentioned in the paper that “This paper builds upon recent work building prototype-based neural networks for interpretable supervised learning.” However, from [1] and [2] we know that borrowing such methods from other learning approaches could often lead to misunderstanding and faulty insights, since they’re not inherently designed for RL. How does this work address such issues?

2- Related work section needs a better discussion. There’re a lot of works missing that could improve the narrative of the paper, and better describe its goals and motives. A few examples of such papers are: [1], [2], [3], [4]

3- Since the proposed method is model agnostic, it’d be good to have results of other black-box pre-trained models in the paper for each particular environment. For example, in car racing, other than PPO, another policy could be studied. Also, providing results on more environment could help to verify the generalizability and robustness of PW-Net.

4- Doesn’t PW-Net change the behavior of the original black-box pre-trained policy? If yes, how can its interpretations be trusted to reflect the original policy? If no, why there’s a performance difference according to the Table 1?

[1] M. H. Danesh, A. Koul, A. Fern, S. Khorram. Re-understanding Finite-State Representations of Recurrent Policy Networks

[2] A. Koul, A. Fern, S. Greydanus. Learning finite state representations of recurrent policy networks.

[3] A. Mott, D. Zoran, M. Chrzanowski, D. Wierstra, D. J. Rezende. Towards interpretable reinforcement learning using attention augmented agents

[4] A. Atrey, K. Clary, D. Jensen. Exploratory not explanatory: Counterfactual analysis of saliency maps for deep rl
clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow, and clear, however, the narrative could be improved. Also, all the source code required to reproduce the results are provided in the supp material which is nice.

summary_of_the_review: See above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method to explain a pretrained DRL policy. At a high level, it adds a wrap network on top of the original policy network and learns this wrap network by minimizing the difference between its output and the policy network output. Then, it generates explanations using this wrap network. The paper tried to prove the wrapped network could mimic the policy network outputs and empirically show it. Finally, the paper designs a user study to demonstrate the utility of the obtained explanation.  
strength_and_weaknesses: Strengths:
+ This paper proposes a new explanation method that encodes human-specified prototypes to improve its human explainability.   
+ This paper well summarizes the existing literature. 

Weaknesses:
- My first major concern about this work is the obtained explanation. After reading the whole paper multiple times, it is still relatively hard to understand what exactly the explanation is. The point I can get is the paper tries to explain the individual actions of an agent. But, It is unclear to me the paper explains the actions through what. For example, most existing works explain individual actions by **highlighting the regions in the input state that are most important to the output action.** In this work, I do not find such a definition. The proposed technique trains the network by predicting the original policy action. After the training, it has prototypical states $S_p$, the weight $W'$, and the encoder $h$. How to generate explanations from these ingredients is not specified. I would highly suggest the authors (1) Give a clear definition of the explanation in this work; (2) Specify how to generate the explanation; (3) Explain the functionality of each component in generating the explanation. (Why we need the prototypes and weight).   

- Regarding the technique, I also have the following questions: (1) How to specify the weight $W'$. Should it be dynamically changed in different states and trajectories? (2) What is the influence of wight and prototypical states on the approximation accuracy (MSE) and explanation quality? 

- IMHO, the theoretical proof is relatively trivial. The final conclusion is if the similarity is proper, the predicted action is accurate. Since the model is actually learning the proper similarity, this is equivalent to saying if the model $h$ is well trained, the output is accurate. This is obviously true. 

- The evaluation is also not that comprehensive. The quantitative evaluation mainly focuses on measuring the difference between the original policy network and the predicted one. This is only the first step. Two networks give the same output may be based on different evidence. The paper misses a quantitative evaluation of the explanation of fidelity or faithfulness. That is, whether the given explanation truly reflects the decision process of the neural policy. The authors could refer to existing evaluation methods for how to design and conduct such an evaluation (e.g., [A Benchmark for Interpretability Methods in Deep Neural Networks]). 

- The user study is not that clearly stated. First, how to define in-distribution and OOD? Second, I would suggest providing an example to demonstrate what is presented to the participants and what are the survey questions. 
clarity,_quality,_novelty_and_reproducibility: I have the following concern about the paper's clarity.
1. The definition of explanation is not clear. What type of explanation does this paper generate? I assume it is an explanation that explains each individual action. 
2. The paper does not specify what exactly the explanation is. It is highlighted features in the input state or weighted prototypes, or something else. 
3, The description of the training process is relatively clear. However, the explanation generation process is not specified. 

IMHO, the idea of this paper is novel in that it proposes a method to integrate human knowledge into the explanation process. However, since the unclear points mentioned above, it is relatively hard to access the benefits of this new method design. As such, I would rank the overall quality as fair. 

Regarding reproducibility, I believe the evaluation in Sections 4 & 5 is reproducible. Still, I have a hard time accessing the reproducibility of the user study due to the unclarity of explanation generation.

summary_of_the_review: The paper proposes an interesting and new explanation method for DRL policies. It seems to be more human-understandable. However, it is unclear what explanation is generated from the proposed model and how to generate it. This imposes challenges for readers to understand the benefits of the proposed method. Besides, the theoretical analysis is relatively trivial and non-rigorous. Therefore, I would rank this paper below the acceptance bar.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes Prototype-Wrapper-Network (PW-Net) to post-hoc explain pre-trained RL policies by learning to map the learned policy's representation into similarity functions to human designed prototypes, and the output action by the learned policy is explained as a pre-defined weighted combination of similarities to the prototypes. Reviewers are unanimously in favor of accepting the work for its novelty, well execution and the inherent interpretability. Reviewers also raised concerns on domain expertise is required to manually define the prototypes and the weight matrix, but were reasonably convinced through the rebuttal.  


justification_for_why_not_higher_score: Prototype-based explanation has been studied for supervised learning counterparts. 
justification_for_why_not_lower_score: Nonetheless, the paper made several well-justified adaptation to enable first use case of prototype-based explanation for deep reinforcement learning.
Decision: Accept: notable-top-25%