Towards Interpretable Deep Reinforcement Learning with Human-Friendly Prototypes
Authors: Eoin M. Kenny
Mycal Tucker
Julie Shah
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Interpretable Machine Learning
Deep Reinforcement Learning
Prototypes
User Study

Abstract: Despite recent success of deep learning models in research settings, their application in sensitive domains remains limited because of their opaque decision-making processes. Taking to this challenge, people have proposed various eXplainable AI (XAI) techniques designed to calibrate trust and understandability of black-box models, with the vast majority of work focused on supervised learning. Here, we focus on making an "interpretable-by-design" deep reinforcement learning agent which is forced to use human-friendly prototypes in its decisions, thus making its reasoning process clear. Our proposed method, dubbed Prototype-Wrapper Network (PW-Net), wraps around any neural agent backbone, and results indicate that it does not worsen performance relative to black-box models. Most importantly, we found in a user study that PW-Nets supported better trust calibration and task performance relative to standard interpretability approaches and black-boxes.


Published as a conference paper at ICLR 2023
TOWARDS INTERPRETABLE DEEP REINFORCEMENT LEARNING WITH HUMAN-FRIENDLY PROTOTYPES
Eoin M. Kenny, Mycal Tucker & Julie A. Shah
Computer Science & Artificial Intelligence Laboratory Massachusetts Institute of Technology (ekenny, mycal, Julis ie a shah) @mi ) tomit.edu edu
ABSTRACT
Despite recent success of deep learning models in research settings, their applica- tion in sensitive domains remains limited because of their opaque decision-makinn processes. Taking to this challenge, people have proposed various eXplainable Al (XAI) techniques designed calibrate trust and understandability of black-box models, with the vast majority of work focused on supervised learning. Here, we focus on making an hinteppeeableeb--deiisp
deep reinforcement learning agent which is forced to use human human-friendly prototypes in its decisions, thus making its reasoning process clear. Our proposed method, dubbed Prototype- Prototype-rrapper Wrapper Net- work (PW-Net), wraps around any neural agent backbone, and results indicate that it does not worsen performance relative to black-box models. Most importantly, we found in user study that PW-Nets supported better trust calibration and task performance relative to standard interpretability approaches and black-boxes.
INTRODUCTION
Deep reinforcement learning (RL) models have achieved state-of-the-art results in Go (Silver et al., 2016), Chess (Silver al.] 2017). Atari (Mnih et 2015), self-driving cars (Kiran et al.) (021). l). and robotic control (Kober al. 2013). However, the usage of these agents in truly sensitive do- mains is limited due to the opaque nature of such systems. Extracting a deep model's rationale in human interpretable format remains challenging problem, but doing so would be highly useful to troubleshoot an agent's 's actions and by extension its possible failure states (Hayes & Shah (017). One popular approach do is with post-hoc explanation methods, which give after- fact tionales for model predictions mostly through some form of saliency map (Bach et al.) (0155 or exemplar (Kenny & Keane 202I). However, whilst popular, these approaches may be incomplete or unsuitable for explanation (Slack et al., 2020 Zhou et al., 2022). and recent work has instead started to focus on pre-hoc interpretability (Rudin 2019). The core idea behind this latter paradigm is to design inherently explainable models, so that you can clearly see and understand their decision- making process such a way that you can calibrate user trust and predict the system's 's capabilities.
In this paper, we present (to the best of our knowledge) the first general, inherently interpretable, well performing, deep reinforcement learning (RL) algorithm that uses an intuitive exemplar-based approach for decision making. Specifically, we train a "wrapper" model called Prototype- Wrapper Network (PW-Net) that can be added to any pre-trained agent, which allows them to be interpretable- by-design, offering the same intuitive reasoning process popular pre-hoc methods (Li et al. 2018). Crucially however, when using PW-Nets, the main advantages of post-hoc methods remain in-tact, in that the black-box model's 's performance is not lost, and it doesn't 't need to be retrained from seratch, which we show across multiple domains notoriously difficult for XAI.
2 RELATED WORK
This paper builds upon recent work building prototype-base based neural networks for interpretable su- pervised learning. Such networks are interpretable by design because they utilize these prototypes in their forward pass by classifying test instances based upon their proximity to these prototypes, thus allowing users to intuitively understand predictions. Perhaps the first notable example of this was by

---

Published as a conference paper at ICLR 2023
Li al. (2018) who learned prototypes in latent space which classified test instances using their L2 distance to each prototype. Work in this area was followed up by Chen et al. (2019) who used image
"parts" rather than the whole instance. This helped spawn many followup works in NLP (Ming
et al. 2019), fairness (Tucker & Shah, 2022). and computer vision tasks (Davoudi & Komeili 2021 Donnelly al., 2022). These prior works establish the value of prototype-basse neural nets, but they focus on traditional classification tasks; we are similarly inspired by such methods but seek to build interpretable agents in RL settings.
Whilst we are not the first to build interpretable RL models (Vorrs 2022 Milani et al. 2022),
almost all prior research uses interpretable proxy models (such as trees) to imitate agents in symbolic domains. However, these techniques do not apply to richer domains with high-ding sigh--imensional inputs (such as raw pixels) that we focus on in this work. To date, most work in these deep RL settings has focused on post-hoc approximations involving attention weights (Zambaldi et al., 2018 Mott et al.) 2019), or trees (Liu al.) 2018), but these methods do not allow transparency the agent's actions or intent (Rudin et al. 2022). Another interesting approach distills recurrent neural network (RNN) policies into fimite-state- machines machhinss (Danesh et al., 2021 Koul et al. 2018), but the approach does not always reveal easily analyzable results, and is restricted RNNs. Perhaps the most relevant approach is that by AAnasammy & Sycara (2019) who learn exemplars to explain Atari games, but their method only works in discrete action spaces, and suffers performance gap relative to black- box counterparts. In contrast to these approaches, PW-Nets are designed to be human-interpretable, generalize to any neural-based agent with any action space, do not lose model performance relative to black-box agents, and do not rely on post-hoc approximations prone to error.
User studies in deep RL trying predict an agent's actions have shown mixed results (Anderson et al. (2020), with the focus now moving towards arguable more useful tasks such as identilying defective models (Olson et al., 202T). the prototype literature, studies have generally focused on how "similar" test instances look to prototypes used (Das et al.) [2200] Rymarczyk et al.) (021), but this does not evaluate if the explanation is useful downstream applications, or appropriately calibrates trust in users (Sanneman & Shah (2022). In contrast to these studies, we ask users to simulate model behaviour and predict failure (and success) cases for the agent, a useful application that should illustrate if trust is appropriately calibrated in the agent's 's abilities.
METHOD
Section 3. first details our assumptions of Markov environment and trained neural agent, before describes the proposed Prototype Wrapper Network (PW-Net), which creates a "wrap- per" around the agent make new end-to-end interpretable policy that reasons with human- defined prototypes. Finally, Section 3.3 gives certain performance guarantees for the system which shows it always closely approximate the performance of black-box agent.
Section 3.2
3.1 MARKOV FRAMEWORK
Our technique assumes access to neural agent pretrained in Markov Decision Process (MDP). MDP is defined by the (S, A, 7) tuple (Sutton Barto 2018). S is the set of states; S is the probabilistic transition
A E RM is the sets of M dimmennsional actions; T: : S:SSXA between states due to actions. We note that actions in RL, unlike outputs of simple classifiers, may be multi-dimeenional (e.g., a must control both steering and acceleration at the same time). We therefore dub each dimension in RM a separate "action". Lastly, y and R define the discount factor and reward function, respectively. In standard RL training methods, the goal is find the policy, : 8 EES S - A, that maximizes the expected discounted reward. Our approach only requires to a pre-trained black box policy, Th6: any existing method from prior may be used to generate such policy (Sutton et al., 2000 Williams] 1992). Assuming a neural net instantiation of this policy with a final linear layer, we may decompose Thi into an encoder fenes alongside the last layer with weights W and bias b as follows: Tss(s) = W (ene(()) (s) + b.
3.2 PROTOTYPE-WRANT
NETWORK
Our primary contribution is a prototype wrapper neural net model, PW-Net, that converts black-box -ox neural models into prototype-based agents for RL by foreing them to use human-unden uuman-understandable
2

---

Published as a conference paper at ICLR 2023
PiJ
Simdantt
Huttor
Score
Dafinse
Turn Left
Weight
Mondt
PiJ
W
State:
Agent
Right
Possible
Steeding
+1)
Senc
Braking
[A maziiishh))
Acceforste 0.25
[A mazzae))
PLI
Weight Values
Siilanty Function
Ascelerate
+]
01
Tonly)
Humon Oefired
ROLUY]
Transformsiions
Pretotype
The framework is instantiated here in the Car Racing do- main from OpenAl's gym. A state is encoded using fenc. before undergoing multiple separate transformations and having its similarity to pre-defined human prototypes measured. Each output action have one or more prototypes associated with representing some understandable con- cept(s), all of which contribute to the output. For example, the action "Steering" sums separate prototypes for turning left and right get a final value, whilst the action "Accelerate" uses only one. The i.j indices are generic placeholders and would be 21,1. 21.2... in the example here for "Turn Left" and "Turn Right", respectively. For simplicity and clarity, we ignore the Tanh(-) function when calculating the "Steering" output in the figure.
Figure 1: Prototype-Wi Prototype-rrapper Wrapper Network:
concepts, thus creating new policy. The overall network diagram depicted in Figure ated for car racing domain in which simulated car must steer, brake, and accelerate.
| instanti-
In the wrapper model, state, s, mapped to latent representation, 2, via the original encoder network:
(s). The network defines, for each action E M]. a set of prototypes that, when combined, dictate action ali]. In Figure I for example, the steering action associated with prototypes for a left and right turn. For each prototype, for each action, separate "projection network", hrj maps from 2 to a specific representation, 2,, for that action and prototype. Within that space for action i and prototype j. we use the similarity function from Chen et al. (2019) to score the distance between ži, and prototype Pt.ji lastly, those similarity scores are combined via a linear weight matrix, W" produce actions. This process is defined more formally in Equation |
Senc(s)
zej = hz)
Vi [1, M1, [1,
(zi. pi,;)²
sim(Zi.j Pi.)) = log
(Zi.} Pi,) +e
(1)
N:
df W% im im P(g)
We created "ouman-friendly" prototypes by specifying set of prototypical states (Spi.)) that, when encoded, were used as the latent prototypes Pi, = ha., (fene (spi.))) in the network. For example, in the self-driving car experiments here, one prototypical state was an example of car turning left and another state was an example of the car turning right; when encoded, these latent prototypes represent different ways turning. Prior literature prototype-based neural networks indicates how one could train our wrapper model to learn its own prototypes (Chen et al.) 2019). However, our human-defined prototypes are motivated by recent literature showing prototype networks benefit from putting humans in the learning loop (Bontempelli et al. 2023), coinciding with work show- ing learning interrraterpee disentangled representations is both empirically and thcoretically impossible without useful inductive biases or human supervision (Locatello al. 2019).
We also added additional human supervision by manually defining W' to control the relationship between prototypes and actions. Intuitively, W corresponds to how prototype Pt, should effect
action d; In our self-driving car example, dn was the turning rate: by setting W -1; W1,2
3

---

Published as a conference paper at ICLR 2023
Algorithm 1:
Training Prototype- Network
Input: f: pre-trained agent with encoder Sene and policy mbe to "wrap".
Output: A PW-Net.
Initialize wrapper, where W" is manually specified weight-matrix.
Sample n state-action pairs from environment for Dataset D + {(s, (s)(s)) P(S))
Choose Human-Internn cottrnn-tterprctable Prototypical States S, S.
foreach batch (s, a) in D do
< fene(s)
foreach in range do
a'li] 1 0
foreach j in range(((Sp))) do
Pis - (fanc
Zi h (2)
a' + a [ Wi sim (Pig, 21.))
Minimize Loss L(a'la; fenci o; W") with gradient descent, updating only 6.
13 Cache all Pi.j (fnc (Spa.)) for testing time inference.
14 return Trained PW-Net
+1, specified that the car's turning rate should equal the difference between the similarity to prototypical left and right turn. In principle, one could combine many similarity scores for the same action, but from performance perspective, never found the need to more than two (although this is likely situation- and rask-dependent). W' guarantees that each prototype is associated with an intuitive action(s), if was learned then each prototype could correspond to multiple actions which may be undesirable. For instance, keeping with the car example, we don' 't want the prototype image for "Accelerate" contribute the output action "Braking" (see Fig. D
Given manually-specified prototypical states and W", the only trainable parts of PW-Net were the projection networks, hy.j, with parameters Φ These multiple transformations were necessary for optimal performance (as opposed to putting all prototypes in single latent space). In experiments, we trained hi.; using gradient-based optimization to minimize a supervision loss between the PW- Net and the black-box policy Tea in specific states, whilst also constraining it to use the prototypes. The training data for this process was generated by running the agent its native domain for n iterations and collating the state action pairs into dataset D {(s, Tbe p(
PW-Net's explanation is classic case-based reasoning one, where explanations are the form "I am going do action because this situation is similar to a previous prototypical sitmation when I also did action x" (Leake 1996; Keane & Kenny] 2019). As our model inherently interpretable, the explanation does not need to be generated, it is naturally produced when deciding each action to take. The explanation can be "local", explaining individual actions of an agent (see Fig. T or "global", explaining the car's understood concepts, which for our study (see Fig. 3)
3.3
PERFORMANCE GUARANTEES
Here, we show that PW-Nets can learn a comparable policy to the black-box model, indicating that performance unlikely to degrade when using our more interpretable architecture. Given that both the black-box model and PW-Net share common encoder, fenc, we therefore seek to prove that the output of PW-Net from z, which we dub fy (z), can equal the output of the black-box model from 2, (2), for all z. For simplicity, we consider only the first action, a1 with:
No W im(21,,
d1
3=1
(2)1(2) P11
(hzz(2) - 2212)2 (h12(2) - (12))2 +
= Wii log
+ W/2 log
(h11(2) - 2 12++2
df +1
d +
= -W log
+ W log
di +c
d2 +
4

---

Published as a conference paper at ICLR 2023
Equation 2 simply restates how action d1 is calculated based on prototypes. For simplicity, only consider two prototypes per action (thus, two entries for W(.)) The range of the similarity function is (0, log(1 /c)]; thus, the range of possible values of af is set by Wij. Assuming W > O; Wi2 < 0, df E [W log(1/c), W log(1/c)]- We note that the similarity function decreases monotonically with distance, it is invertible for legal values. Therefore, even if p.j were fixed, a sufficienty-pprrmeteriee parrameterized hasj could learn to output any similarity score simply by placing zi at the correct distance from Pt.j- Hence, as long as the black box policy outputs always fall within finite range, PW-Net can closely match the output. In experiments, we found one linear layer was not enough to accomplish this, two was sufficient (which we used), and more showed no benefit.
Two important assumptions are worth highlighting. First, we assumed that for each action there were at least two entries in W' and that they had opposite signs. The similarity function always generates positive outputs; thus, to match a black box policy with both positive and negative values, is important to use both positive and negative weights in W". In practice, we found this necessary in the self-driving car experiment. Second, we assumed independent hay for all i. Prior work in prototype-baxee based neural nets often locate all prototypes within a single latent space. However, given a fixed W and without separate fu.j. every state-action pair generated by Tыs creates set of (potentially incompatible) constraints for the relative locations of prototypes. In practice, we found that learning separate hi.j was necessary for performance, which show in the next two sections.
The earlier discussion found that PW-Nets may mimic a black-box policy for almost all 2 by assum- ing that hajj may be learned to output any desired similarity score between žt,j and Pt.j- However, there are a small number of notable exceptions: precisely the corresponding to prototype states. By definition, the distance between between such encodings and the prototype will be zero, which specifies the PW-Net output for such states. Recall however, we are not trying to perfectly mimic the agent, instead we seek to use it as a "guide" whilst learning a new policy that reasons with the prototypes. In doing so no notable performance degradation was noticed, in fact, we often observed performance improvements by forcing the network to utilize the numan-defined prototypes.
4 CONTINUOUS ACTION SPACES
Here test if PW-Nets can perform par with any black-box agent by testing them in deep RL domain with multiple continuous output actions and a state space of only raw pixels. In addition, another domain with symbolic inputs is tested to further display generality. PW-Nets are compared against several baselines and the best methods are determined by the average reward. As a sanity check, we also report how closely the methods imitate the black-box predictions at test time.
4.1 EXPERIMENT SETUP
The deep RL domain tested here is the Car Racing problem from OpenAl's gym environment, the agent receives rewards for driving around the track as fast possible and avoiding swerving off the road. We use a pre-trained model (partly to avoid any bias) from Jain (022) which solved the environment using Proximal Policy Optimization (PPO). The agent has three continuous output actions (see Fig. D. Similarly, for symbolic problem, the Bipedal Walker environment from Open AI's gym Box2d was chosen, where the agent receives rewards for simply walking straight and not falling over. For this latter domain a pre-trained model from Barhate (2022b) was used which solved the problem with the Twin Delayed DDPG (TD3) algorithm. The agent has four continuous output actions (see Appendix A for prototypes used).
We compared first to k-Means in the space & with the same number of clusters as the number of prototypes used in PW-Net, before mapping each centroid onto the nearest training example to get the prototypes. Then, in contrast to the other methods, we allowed k-Means learn the weight parameters in the last layer to help it. As another baseline, we followed the same protocol as PW-Net, but only a single transformation is used which houses all prototypes together in the same latent space which we call PW-Net* In addition, this baseline learns the prototypes as trainable parameters, and once training is complete, they are mapped onto the nearest training example. This can be best thought of as ablation test of PW-Net. however it bears resemblance Chen et al. (2019) and their ProtoPNet model, so we also implemented the clustering and separation losses suggested by the authors (adapted for continuous domains) to help the baseline further (see Appendix B). Lastly,
5

---

Published as a conference paper at ICLR 2023
Table I: Continuous Action Spaces: Various baselines compared to PW-Net. Overall, only PW-
Net is comparable to the performance of the original black-box agent in both domains.
Car Racing
Bipedal Walker
Method
Reward
MSE
Reward
MSE
PW-Net
220.72 + 0.34
1.37 + 0.16
304.77 + 6.72
0.65 + 0.38
VIPER
N/A
N/A
-89.71 + 7.51
101.15 + 8.16
PW-Net
-9.48 + -02.50
16.93 + 3,01
190.41 + 59.51
69.77 + 15.16
k- k-Means
-2.09 -00.94
29.04 + 0.01
-107.72 + 0.13
136.90 + 0.72
Black-Box 218.41 + 03.00
N/A
310.71 + 1.47
N/A
we also compare to VIPER by Bastani et al. (2018), but we adapt their algorithm here by using uniform sampling (which is implemented in their code base as an alternative) instead of the weighted sampling they discuss (which needs q-values), which allows us to use experiments.
Two metrics were used, average reward and mean-squared enror (MSE). Reward was calculated by training each method times and taking the average reward accumulated across 30 simulations, these 5 means then produced a final mean and standard error. The second metric took the MSE of the black-box model's actions compared to the interpretable model's actions each iteration, to how closely methods approximate the oracle. As PW-Net is optimizing a function which (1) matches the original agent, and (2) uses the human-defined prototypes, will not 100% match the policy which reasons
agent's s predictions, and indeed don't want it to, want it to learn with the prototypes and W'. MSE is included to make sure part (1) of the optimization is working.
4.2 RESULTS
Table shows the results in continuous action domains. In the deep RL domain Car Racing, only PW-Net (M=220.72, SE=0.34) was capable of achieving a similar average reward the black- box (M-218.41, SE=3.). Moreover, PW-Net achieved the closest predictions to the black-box (i.e., the MSE was smallest). In Bipedal Walker, again PW-Net (M=304.77, SE=6.72) was capable of achieving near the same average reward as the black-box (M=310.71, SE=1.47), although PW-Net* achieve notable reward too (M=190.41, SE=59.51). k-Means performed the worst overall, with the lowest reward, with VIPER fairing only slightly better. Note that VIPER only works in symbolic domains, so we could not test it on Car Racing.
5 DISCRETE ACTION SPACES
Here again PW-Nets are tested, but in discrete domains to help prove generality. As before, deep RL domain using raw- raw-pixels as state representations is tested, alongside a symbolic domain. The same baselines and metrics are used, although with some minor modifications.
5.1 EXPERIMENT SETUP
The deep RL domain tested the deterministic version of Atari Pong from OpenAl's gym envi- ronment, the agent receives rewards for scoring against the opponent, and losses rewards for being scored against. The agent uses a state space of raw pixels, has six discrete actions corresponding to moving up or down, and we use pre-trained model from [Sentürk (2022) which solved the en- vironment using the DQN algorithm (and 5% stochastic). For a symbolic problem, the Lunar Lander environment the shortest time possible. The agent has four actions corresponding to doing nothing, or firing its engine three different sides. A pre-trained model from Barhate (2022a) was used which solved the problem with the TD3 algorithm (see Appendix A for prototypes).
used, where the agent gets reward by landing the spaceship successfully in
As baselines, again compared to k-Means, but this time by dividing the training data in the space z using each separate action, and clustering each with & = 1, before mapping each centroid onto the nearest training example. Then, in contrast the last experiment, didn't 't allow k-Means to

---

Published as a conference paper at ICLR 2023
Table 2: Discrete Action Spaces: Various baselines are compared. However, again, only PW-Net
consistently performs well.
Atari Pong
Lunar Lander
Method
Reward
Accuracy
Reward
Accuracy
PW-Net
10.72 = 0.26
88.93 + 0.00
216.94 = 16.92
97.63 + 0.00
VIPER
N/A
N/A
408.81
60.98
59.26 1.01
PW-Net*
8,85 = 1.69
84.84 + 0.76
124.54 = 120.53
88.67
+ 0.01
k- X-means
-21.00 = 0.00
11.79 + 4.15
419.46 119.08
10.10
5.87
Black-Box
11.94 = 0.16
N/A
212.94
2.63
N/A
learn the weight parameters in the last layer since each can easily be associated with an appropriate action. PW-Net* was again used here, but this time using the exact same clustering and separation losses suggested by Chen et al. (2019), since they developed the losses for discrete classification problems they will work here. Lastly, we again compare to VIPER by Bastani al. (2018) using the uniform sampling variation of their algorithm.
Two metrics were used, reward and accuracy. Reward calculated the same way in previous experiment. The second metric took the accuracy of the black-box model's actions compared to the interpretable model's actions at time. This has the same functionality as MSE in the previous section, but now adapted for discrete domains.
5.2 RESULTS
Table 2 shows the results of using PW-Nets in discrete action spaces. In Pong PW-Net achieved the closest reward (M=10.72, SE=0.26) to the black-box's 's reward (M=11.94, 1.94, SE=0.16). Interesting divergences happen where PW-Net* achieves lower Accuracy (M=84.84, SE=0.76) compared to PW-Net (M-88.93, SE=0.00), indicating that PW-Net was better at learning from the black-box. In Lunar Lander, again only PW-Net variations were able to reach comparable reward to the black box. Specifically, PW-Net achieved reward (M=216.94, SE=16.92) which was comparable to the black-box (M=212.94, SE=2.63), with PW-Net* doing notably worse (M=124.54, SE=120.53). Larger divergence happens when comparing accuracy, PW-Net (M=97.63,, SE=.0) achieves signif- icantly higher results than PW-Net* (M=88.67, SE=0.001) indicating that the former is again better able to mimic the original black-box function. Although marginally better than k-Means, VIPER performs badly overall (M=-408.8) SEE66009))) perhaps suggesting that decision tree proxy models will struggle in their current form to generalize beyond relatively simple domains such as Cartpole where they are usually tested successfully (Bastani et al., 2018 Silva et al. 2020). Lastly, note the Pong model was 5% stochastic, so reaching high performance with PW-Net was impossible, as was trained on 5% random data, and tested with an additional 5% stochasticity on top of this.
So, considering the two experiments thus far, is clear that PW-Net generalizes the best overall both in terms of reward and MSE/Accuracy, particularly in deep RI. settings which is our focus here. A natural question to ask is how sensitive the performance of PW-Net is to the prototypes chosen, so we performed test on all four domains where the prototypes were randomly chosen from the training data 30 times with 30 simulations ran each time. Results showed no notable difference to the block-box in Car Racing (Reward: M=218.57, SE=0.09, MSE), Atari Pong (Reward: M=10.71, SE=0.29), Lunar Lander (Reward; M=213.78, 78, SE=11.19), or Bipedal Walker (Reward: M=309.44 SE=1.79). So, overall, PW-Nets are very robust to the prototypes chosen. However, shall now see, user task performance crucially depends upon choosing human-friendly prototypes.
USER STUDY
Here we test if PW-Nets can improve a user's ability to predict in distribution (ID) successes and -ft-of-distribution (OOD) failures in a simple self-driving domain, an application seen as highly sought after (Rudin et al. 2022). We setup the materials such that all ID situations were driven successfully by the car, and all OOD situations
ones in which it crashed (see Fig. 5) The
7

---

Published as a conference paper at ICLR 2023
explanation shown to users is global explanation of the car, which for the experiment group using PW-Net is shown in Figure 3
Two different controls were compared our "Human-Friendly Prototype" experiment group which used PW-Net's global explanation. The first is a "Non-Human Friendly" group which uses the same prototypes as network, but rearranges them randomly. So, the weight matrix and actions remain the same, but different prototype images are associated with each. The purpose of this comparison to show that although the performance of the network itself does not hugely hinge on associating human-friendly prototypes with intuitive actions, the performance of humans does. Note this is an actual PW-Net we trained that performed just as well as the other models here (even though the prototypes were purposefully "badly chosen"). This group was easily controlled because they saw the same amount of information in each question as the experiment group, whilst being presented in similar way (see Fig. 6 and 77. Secondly, we compare our network to a full black-box, where people are simply told a "no explanation explanation" in text form with "The car has learned to drive very well by practicing for millions upon millions of simulations". This group was controlled by replacing the prototype images with this text (see Fig. 6 The rationale here is that these images needed to be replaced with comparable information in order to control the cognitive load between groups. After seeing the current situation of the car, participants are shown the explanation, and then asked to predict if the car will drive safely or not scale of 1-5, where 1 corresponds to "Strongly Disagree" and to "Strongly Agree". Note numeric scales were used (not a traditional Likert one) so the data could be confidently analyzed using t-tests (Carifio & Perla 2008).
So, the two hypotheses tested are: HI: Users require human-inferpreetable prototypes to perform well in model simulation tasks. H2: When people receive "no explanation", they will have inappro- priately high trust.
Participants. Comparing to similar studies (Kenny et al. 202T). we anticipated a medium effect size, but pilot studies showed large effect. Hence, anticipating medium-large effectt, we chose an alpha of 0.05, a power of 0.8, and an effect size of 0.65, which predicted through power analysis that a sample size of 30 was required in each group to avoid p-hacking. Hence, 90 people in total were recruited from prolific.co, 30 per group, all paid $12 per hour to complete the study. All participants were restricted to being native English speakers, using desktop computers, in either the U.S. or the U.K. This study passed ethics review of the institution.
Materials. 20 materials were randomly shown to users, 10 in distribution (ID), and 10 out-of- distribution (OOD), the latter was generated from the open source modification to the Car Racing environment provided by Woodcock (2016), of which showed red obstacles the car should avoid, and the other 5 of which showed unusual roads not seen normally in the environment (such as t- junctions). All ID situations were gathered form the original domain, driven successfully by the car, and labelled as "driving safe". The OOD situations were real failure cases of the car, where all trained models either (1) drove off the road crashing, or (2) hit the obstacles. If the car did either of these things, it was labelled as "not driving safely". All these questions can be seen in Figure 5 to make the ID and OOD materials clear. After the questions, users where shown where the car did/didn't drive safe, and asked to complete the NASA TLX perceived workload questionnaire
6.1 RESULTS.
Each participant's scores were averaged into two numbers for both their ID and OOD materials,
which formed the distributions of interest. H1: Fig. 2 A) shows a large difference in predicting ID success. Specifically, those who received FHHuman-Friendll Prototypes" (M=4.48, SD=0.43) com- pared to participants in the "Non-Human Friendly Prototypes" control group (M=2.19, SD=0.78) demonstrated better ability to predict when the system would drive safe, f(58)=13.8, p < .0001, showing human-friendly prototypes had a large effect on people's task performance. This lends evi- dence that prototypes need to be carefully chosen in order to maximize the effectiveness of PW-Net global explanations. H2: Fig. (BB) shows a significant difference in predicting OOD failures be- tween participants who received "Human-Friende" Prototypes" (M=2.24, SD=0.68) and participants in the "Black-Bor -Boxck control group (M=3.48, SD=0.75). The prior demonstrated significantly better
Users were also asked for their confidence in predictions, and an additional system usability questionnaire was asked. However, no results were significant, so they are reported.
8

---

Published as a conference paper at ICLR 2023
5.0
5.0
Non-tammn freend troicopes
4.5
No Effect
0C0
4.5
No Effect
3.5
Bleckhoo
3.5
(A)
3.0
(B)
3.0
2.0-
2.0
1.5
1.5
2.5
3.0
1.5
3.0
Friendly Protonypes
Friendly Frotonyses
Figure 2: User Study Results: Ratings for how confident users were the car would drive safely (5 rating) v. not drive safely (1 rating) for in distribution (ID) out-of-distribution (OOD) data. Each point on the plots represents the average of a single question the survey. Compared to the experimental group, Con-HHumnn-Friendle control group were significantly worse at predicting when the car would drive safely, and the Black-Box group predicting when would crash. The plots can be interpreted as datapoints close to the dashed line showing difference between groups, whilst points far from it showing significant differences.
ability to predict when the system would not drive safe, t(58) -6.53, p .0001, showing that prototypes give a global understanding of the system which allows people to predict OOD-based system failures better than no explanation. This is interesting because it shows that people who no explanation are overly confident the black-box's abilities, leading to inappropriately high trust, which PW-Nets actively avoid by appropriately lowering their trust here.
Aside from testing our original hypotheses, additional post-hoc analysis considered the groups from pure accuracy perspective, where scores of 4/5 were taken as the user predicting the car would drive safely, and scores of 1/2 as the user predicting the car would not drive safely, score of was always marked as incorrect, since the user uncertain either way. The human-friendly group was better at predicting if the car would drive safe or not (M=0.75, SD=0.14), than the non-human friendly group (M=0.37, SD=0.15), 15), and the black-box group (M=0.57, SD=0. 10), with both results achieving statistical significance p .05. This ability to accurately predict a system's capabilities is considered an important proxy metric for trust calibration (Hoffman et al. 2018 Sanneman & Shah 2022). Hence, it is reasonable to posit that PW-Nets may have role to play in calibrating appropriate user trust of Al systems, something which is seen as core application for XAI, but has proven quite difficult demonstrate in practice (Silva et al. 2022 Miller 2022).
Lastly, the NASA TLX questionnaire showed there was significant difference in people's 's perceived workload between participants who received "Human-Friendly Prototypes" (M 5.80, SD 1.74) and participants in the "Non-Human Friendly" control group (M = 7.25, SD = 2.63). The control recorded higher workload, 1(58) -2.45, p .02, showing that badly chosen prototypes make the task of the using the system significantly more taxing on people.
7 CONCLUSION AND FUTURE WORK
The world has grown increasingly excited about possible applications for deep RL. However, with- out transparency in the agent's 's actions and intent, is not feasible to use the systems in truly sensitive accessible way force any deep RL agent to use
domains (Rudin) 2019). Here, we showeased human understandable concepts in an interpretable way with example-based reasoning (Kenny & Keane, 202T). Results demonstrated that the technique does not lose model performance and gen- eralizes across discrete and continuous action spaces. In addition, user study showed real appli- cability of the technique to assist human task performance in predicting model failure (and success) states, hence calibrating appropriate user trust. In future work, it would be interesting to experiment with learning the prototypes and/or weight matrix, particularly in domains people cannot easily de- learning, fine-tur fine-tuning (or
fine concepts. In addition, applying PW-Nets supervised/self training from scratch) the pre- trained encoder, incorporating feature saliency, exploring prototypical trajectories, and adding additional interpretability constraints to the loss could all be explored.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This research was supported in part by Army Research Lab grant #1 #1300552-43128. The authors would like to thank the anonymous ICLR reviewers who greatly helped to improve the presentation of this work. In addition, we are grateful our lab members who supported this work with some early advice and helpful critiques.
REPRODUCIBILITY STATEMENT
The code to reproduce the results is available at:
ICLR23
For the user study, we also included four group surveys as PDFs in the repo, so people may recreate the study results if they wish. If researchers are interested in reproducing the survey results more easily, please contact the corresponding author and we will share the Qualtrics surveys.
ETHICS STATEMENT
One of the potential ethical considerations of using prototypes for explanations is that the examples used must perserve user privacy. Considering this, if PW-Nets were used in downstream applica- tions, some consideration is going need to be given to this. However, when compared more generic nearest neighbor algorithmss, PW-Nets have a big advantage here in that there is much smaller group of exemplars which may be possibly presented to users, so the efforts on algorithm designers to preserve privacy will be much easier.
REFERENCES
Andrew Anderson, Jonathan Dodge, Amrita Sadarangani, Zoe Juozapaitis, Evan Newman, Jed Irvine, Souti Chattopadhyay, Matthew Olson, Alan Fern, and Margaret Burnett. Mental models of mere mortals with explanations of reinforcement learning. ACM Transactions on Interactive Intelligent Systems (TiiS), 10(2):1-37, 2020.
Raghuram Mandyam Annasamy and Katia Sycara. Towards better interpretability in deep q- networks. Proceedings the AAAI conference on artificial intelligence, pp. 4561-4561 2019.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015.
Nikhil Barhate.
Nikilllahhace///etoorrrriicccace Oetor-Critic-Pyye htwp://wwhiioooiiiiiiiiss/////et 2022b. URL
Policy gradient actor-critic py-
torch: Lunar lander v2, 2022a.
URL
Nikhil Barhate.
Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. Advances in neural information processing systems, 31, 2018.
Andrea Bontempelli, Stefano Teso Katya Tentori, Fausto Giunchiglia, and Andrea Passerini.
Concept-level debugging of part-prototype networks. In International Conference on Learning Representations, 2023. URL http://///peerreeeeeeeeeeeeeetts www.
wtwpriiwwww.eeeeeeeeeeeeeeeeeeee net wwtp://teeeeeetteeeeeeee..
James Carifio and Rocco Perla. Resolving the 50-year debate around using and misusing likert scales, 2008
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: Deep learning for interpretable image recognition. Advances in Neural Information Processing Systems, 32:8930-894 8941, 2019.
Mohamad H Danesh, Anurag Koul, Alan Fern, and Saeed Khorram. Re-understanding finite-state representations recurrent policy networks. In International Conference on Machine Learning, pp. 2388-2397. PMLR, 2021.
10

---

Published as a conference paper at ICLR 2023
Subhajit Das, Panpan Xu, Zeng Dai, Alex Endert, and Liu Ren. Interpreting deep neural networks through prototype factorization. 2020 International Conference on Data Mining Workshops (ICDMW), pp. 448 457.. IEEE, 2020.
Seyed Omid Davoudi and Majid Komeili. Toward faithful case-based reasoning through learning prototypes in a nearest neighbor-friendly space. In International Conference on Leurning Repre- sentations; 2021.
Jon Donnelly, Alina Jade Barnett, and Chaofan Chen. Deformable protopnet: An interpretable image classifier using deformable prototypes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10265-10275, 2022.
Sam Greydanus. Greydanus/baby-a3c:: A high-performancce atari a3c agent in 180 lines of pytorch, 2022. htw.Laaese/ggttab....//e/reeeree www.:/www...ooooo....oaaa
Bradley Hayes and Julie A Shah. Improving robot controller transparency through autonomous policy explanation. In Proceedings of the 2017 ACM/IEEE international conference on human- robot interaction, pp. 303-312, 2017.
Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. Metries for explainable ai: Challenges and prospects. arXiv preprint arXiv:18/2.04608 2018.
Jinay Jain. Jinayjain/deep-racing:: Self-driving racecar using reinforcement learning (proxi- mal policy optimization) in pytorch, 2022. URL httt::///ggthaa..co//inneee
Seep-racine
Mark T Keane and Eoin M Kenny. How case-based reasoning explains neural networks: theoreti- cal analysis of xai using post-hoc iellanation-b--xamppe from a survey of ann-cbr twin-systems. In International Conference on Case- Based Reasoning, pp. 155-177 1711. Springer, 2019.
Eoin M Kenny and Mark Keane. Explaining deep learning using examples: Optimal feature weighting methods for twin systems using post-hoc, eaplanation-by-examaee in xai. Knowledge- Based Systems, 233: 107530, 2021.
Eoin M Kenny, Courtney Ford, Molly Quinn, and Mark T Keane. Explaining black-box classifiers using post-hoc pyllanations-b--exappi The effect of explanations and error-rates in xai user studies, Artificial Intelligence, pp. 103459, 2021.
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yoga- mani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 2021.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey, The International Journal of Robotics Research, 32(11):1238-1274 2013.
Anurag Koul, Sam Greydanus, and Alan Fern. Learning finite state representations of recurrent 2018.
policy networks. arXiv preprint arXiv:
David B Leake. Case-Based Reasoning: Experiences, lessons and future directions. MIT press, 1996.
Oscar Li. Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: neural network that explains predictions. Thirty-Seconl AAAI Con- ference Artificial Intelligence, 2018
Guiliang Liu, Oliver Schulte, Wang Zhu, and Qingcan Li. Toward interpretable deep reinforcement learning with linear model u-trees. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 414-429 Springer, 2018.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. international conference on machine learning, pp. 41 4114-412 PMLR, 2019.
11

---

Published as a conference paper at ICLR 2023
Stephanie Milani, Nicholay Topin, Manuela Veloso, and Fei Fang. A survey of explainable rein- forcement learning. arXiv preprint arXiv:2202.08334, 2022.
Tim Miller. Are measuring trust correctly in explainability, interpretability, and transparency research? arXiv preprint arXiv:220900665,, 2022.
Yao Ming, Panpan Xu, Huamin Qu, and Liu Ren. Interpretable and steerable sequence learning via prototypes. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 903-913, 2019. 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature, 518(7540):529-53) 2015.
Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. Towards interpretable reinforcement learning using attention augmented agents. Advances in Neural Information Processing Systems, 32, 2019.
Matthew L Olson, Roli Khanna, Lawrence Neal, Fuxin Li, and Weng-Keon Keen Wong. Counterfac- tual state explanations for reinforcement learning agents via generative deep learning. Artificial Intelligence. 195:103455, 2021.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215, 2019.
Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. In- terpretable machine learning: Fundamental principles and 10 grand challenges. Statistics Surveys, 16:1-85, 2022.
Dawid Rymarczyk, Lukasz Struski, Jacek Tabor, and Bartosz Zieliński. Protopshare: Prototypical parts sharing for similarity discovery in interpretable image classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1420-1430, 2021.
Lindsay Sanneman and Julie A Shah. The situation awareness framework for explainable ai (safe- ai) and human factors considerations for xai systems. International Journal of Human- Computer Interaction, pp. 1-17, 2022.
Andrew Silva, Matthew Gombolay, Taylor Killian, Ivan Jimenez, and Sung-Hyun Son. Optimiza- tion methods for interpretable differentiable decision trees applied reinforcement learning. In International conference on artificial intelligence and statistics, pp. 1855-1865. PMLR, 2020.
Andrew Silva, Mariah Schrum, Erin Hedlund Botti. Nakul Gopalan, and Matthew Gombolay. Ex- plainable artificial intelligence: Evaluating the objective and subjective impacts of xai on human- agent interaction. International Journal of Human Computr Interaction, pp. 1-15, 2022.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieserr, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv: 2017.
Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp. 180-186, 2020.
R. S. Sutton, D. Mcallester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems 12, volume 12, pp. 1057-1063. MIT Press, 2000.
12

---

Published as a conference paper at ICLR 2023
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL http://iincmmmeeaaaaaaaaaaetoot/ html
2nd.
Mycal Tucker and Julie Shah. Prototype based classification from hierarchy to fairness. In International Conference Machine Learning, pp. 2188--21900. PMLR, 2022.
George A Vouros. Explainable deep reinforcement learning: State of the art and challenges. ACM Computing Surveys (CSUR), 2022.
R. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning. 1:229-256, 1992.
Mike Woodcock. Notanymike/gym: An improvement of carracing-v0 from openai gym in order to make the environment complex enough for hierarchical reinforcement learning, Jan 2016. URL www..ot.......eecc www./ootnyy.ike///o/e.a.aao
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap. Edward Lockhart, al. Deep reinforcement learning with relational inductive biases. In International conference on learning representations, 2018.
Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, and Julie Shah. Do feature attribution methods correctly attribute features? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 9623 9023-9633,.00 2022.
Behçet Şentürk. http://gghhub.omo//ccssssspppppp 2022. URL https://githh Com/ http://wwwbccateee//////--------
A
PROTOTYPES CHOSEN
In this appendix we explain how the prototypes were chosen for each domain.
A.1 CAR RACING
In this domain there are three action outputs, steering, braking, and accelerating. Steering has possible range from -1 to +l, with negative values turning left, and positive values turning right, means to drive straight Accelerating and Braking both have values from to
When choosing the prototypes for PW-Nets, we broke steering down into two concepts "turning left" and "turning right", with prototype representing each concept. For accelerating and braking, we only chose one prototype each. For accelerating a state of a straight road was chosen, since this typically seen as safe situation accelerate, and for braking the motion of taking turn was chosen, since it is necessary for car to always brake when turning a corner in order to drive safely, these where seen being somewhat interpretable for people (see Fig. 3
Note that many more concepts could be used to dictate when the car brakes or accelerates. For example, if there obstacles the road, then could correspond prototypical example of these objects to the action "braking". That way, when the car sees such an obstacle, it will brake and explain why is is braking (i.e., due to its similarity to the prototype with the obstacle). However, here for simplicity (and to demonstrate multiple/single options), we only associate one concept with the actions braking and accelerating, whilst associating two with steering.
A.2 ATARI PONG
In this domain there are six action outputs, action 0 and are placeholders and do nothing, action & makes the racket go up. and action & makes the racket go down. The prototypes used in our tests are shown in Fig. with near identical ones used for the duplicate actions.
13

---

Published as a conference paper at ICLR 2023
A.3 SYMBOLIC DOMAINS
Although the focus of this paper is on deep RL, these domains were included to help prove the generality PW-Nets as proposed in Section 3.33 For Lunar Lander and Bipedal Walker we defined "ideal" prototypical examples of each concept in the action-space where the action of interest was or and all others 0, which we then mapped to the nearest training example. Lunar Lander had I for the action of interest, and 0 for the others (four prototypes in total). Bipedal Walker had two prototypes for each output action since they ranged from to +1 (so prototypes total), each representing the concept of one of its limbs moving forward backwards (e.g.. "Right Hip Moving Forwards").
B LOSSES IN PW-NET*
When learning all the prototypes in the latent space followed prior literature in Chen et al. 2019) and used their separation and clustering losses as descrihed below:
Clst
min jPpePs be (Epatchea([(t))
min
|| - -Ppll
(3)
Sep
min p (Epatchea([(xx))
min
||z -
(4)
71
where n. is the number of instances in a batch, is prototype for one of the classes in P, f(. is the encoder, and is the latent representation of some patch in test instance I₁.
However, in domains considering whole instances, and not parts of them, the losses become:
Accelerate
Brake
Turn Left
Turn Right
Figure 3: Hlmman-Friende Prototypes for Car Racing: The four prototypes chosen for Car Racing in our tests. The two prototype for "Tum Left" and "Turn Right" contribute to the output action steer- ing. The two other prototypes singularly contribute to the actions accelerate and brake, respectively.
14

---

Published as a conference paper at ICLR 2023
3
Move Down
Move Up
Figure 4: Human-Friendly Prototypes for Pong: The player is the green object which must hit the white ball back to the left side of the table. The red arrow shows the direction the white ball is travelling.
1
Clst
min min || P,ll jPjEP 2 ==(((z))
(5)
n
Sep =
min || - P2l12 j.p Ep ==(((z))
min
(6)
In the continuous actions spaces, the losses had to be modified to work, so they became:
Clst
min || 14--p11 2111 ((z))
n
Sep
|p: p311 s.t. itj
(8)
1=1
1=1
where simply represents the number of prototypes used. Intuitively speaking, the first loss forces all prototypes to be close to the data distribution, that when they are mapped to real instances they do not change much. The second loss forces all prototypes to be far apart from each other. Each loss had to be scaled in optimization with parameters A and X2, respectively. In discrete domains, set the values to 0.8, and 0.08 [following Chen et al. (2019)]. but in continuous domains we found 0.08, and 0.008 worked better.
C USER STUDY DESIGNS
Rather than include the user surveys in this manuscript, uploaded them in the accompanying zip file (and repo) in the interests of keeping this document to reasonable size (both in terms of file size and document length). However, we do show some screenshots here in Appendix E.
D TRANSFORMATION ARCHITECTURE
Each PW-Net model here used multiple transformations hs, to convert into a latent space which it could have its similarity measured against numan-defined prototypes. The architecture for each of these is specified Table 3 In practice, the prototype sizes were usually 50 in length, but as little 5 also worked occasionally.
15

---

Published as a conference paper at ICLR 2023
Table 3: The transformation architecture used in all experiemnts.
Transformations hi.j
Layer
Layer Parameters
Linear
(Latent size, prototype size) prototype size
Normad
ReLU
Linear
(prototype size, prototype size)
E USER STUDY: SURVEY DETAILS
Our study was designed to see if people can predict OOD>baaed failures and ID-based successes based on the explanations given. For the ease of readers, we have included all OOD/ID materials shown to users in Figure 5 Then, Figures [6 and 7 show an example of the actual survey questions asked in all four groups. Please consult the supplementary material or GitHub repo see the full surveys. Note we are happy to share the surveys so other researchers do not have to rewrite them from scratch they wish reproduce the results.
E.1 THE TRAINING DATA GROUP: OMITTED RESULTS FROM USER STUDY
Note that there was fourth group in our user study dubbed the "Training Data" group (see Fig. 7 However, due to concerns the Training- -aatinngg-Daaaaaa group here was not as well controlled the others (due to an additional text prompt reminding users that they are only seeing small sample of training data), we removed it from the main paper, but we show here in Figure X in case some readers are interested in the results nevertheless. This additional text was added an attempt to control for the fact users were not reading associated actions with each prototype in the group, so they needed additional cognitive load to be equal to the experimental group. However, it was ultimately decided to omit the group from the main paper as was seen as too likely to be possible confounding factor.
3).
Specifically, Fig. ( shows a significant difference in predicting OOD failures. In particular, when comparing participants who received "Human-Frienddl: Prototypes" (M=2.24, SD=0.68) to participants in the "Training- ability to predict when the system would nor drive safe, t(58) -3.32, p .002. This shows that prototypes give global understanding of the system's capabilities which allow people to confidently predict OOD-based system failures better than observing segments of training data. However, reiterate that these results may not reproduce if the instructions in Step 2 were more minimal (see Fig. BB.
control group (M=2.79, SD=0.57), the prior demonstrated better
F ADDITIONAL PW-NET RESULTS
To help further prove the generality of PW-Nets, we include some more case studies here of their performance in different domains.
F.1
CART POLE
Here, we trained a DQN model to perform in the cart-pole domain from Open Al's gym environment. symbolic domain with a four-dimensional state space corresponding to cart position, cart velocity, pole angle. and pole velocity at tip. The output actions are simply go left or go right. To succeed in the environment, the agent must remain balanced for 200 actions.
The DQN learned to perform perfectly in the domain, achieving average reward of 200 across 150 simulations. Following Algorithm I we first collected simulation input-output data for 1000 simulations, corresponding to 20,000 datapoints for training the PW-Net. To pick the prototypes, this data was partitioned into two groups corresponding to each action, and the mean centre of the cluster used get the prototype for each action which was the closest mapping in the real data.
16

---

Published as a conference paper at ICLR 2023
In Distribution Materials (ID)
Out-of-Distribution Materials (OOD)
Figure 5: User Study Materials: The and OOD materials shown to users the study. The materials were gotten from the domain the model trained in, whilst the OOD materials were gotten from Woodcock (2016).
(A) Human Friendly Prototypes Group
Black- Group
Step Note Stueton
Step to Nitte the Sttrtion
Step 2: Fiezal the car's lsannd concapts
Step explanation
Esolenatio:: The lvarned to drine well pracciiing milfiona mallioss simulations.
Step
Stop
Sieer giv
Stargg
Soongyy
Devigree
situsiiin
pwddidos
prediction
Figure 6: Example Question: (A) The Human-Friendly Prototype group, and (B) the Black-Box group.
We trained five PW-Nets on these data, for 30 epochs, and ran 30 simulations each time to collect the results. Overall, the PW-Net perfectly performed in the domain with an average reward of 200
17

---

Published as a conference paper at ICLR 2023
(A) Non-Human Friendly Prototypes Group
(B) Training Data Group
Step Note Stuuton
Step Nove the Shurtoon
Step 2o lacall the can's teemee concepts
nandam secampka car practiddig 1g but retmmbee share MILLIONS of altuation you are NOT being ahown.
Acaalonta
Step de
Desggroo
66100
Dnscpeoo
The car will drvo valdy
drice callly in dtublio
0
on conideet m.th
an conteeee in
Figure 7: Example Question: (A) The Non-Human-Frrendyy Prototype group, and (B) the Training- Data group.
5.0
4.0
(A)
0.4
(B)
02
1.5
0.0
Black-oss
Pronchipes
Doca
Priendly
Friendly Prototypes
Figure 8: Additional User Study Results: (A) The "Training Data" group from our user study shows how people are less able to confidently predict OOD errors where the car crashes into red obstacles or runs off the road. (B) The accuracy all groups are plotted to predict if the car will crash not. Standard Error bars are shown in (B). The results of (A) were omitted from the main study due to some concerns the extra text Step 2 of Figure confounded the results.
bar chart showing their ability
+ 0.00, and an accuracy of 98.81% 1,, showing that it approaches the same function as the black-box. but the constraints of using the prototypes means it inevitably learns to reason in different manner more intuitive for humans.
The other algorithms PW-Net* and VIPER also achieve similar results, showing that Cart Pole is relatively simple domain compared to the others tested in the main paper. Notably, VIPER achieves better reward than PW-Net*. but only slightly.
F.2 ATARI BREAKOUT
Here, we used a pre-trained model from Greydanus 2022) The model uses the Asynchronous Advantage Actor-Critic (A3C) approach. Like Atari Pong. the domain's input state space is raw pixels, and there are six output actions. The model achieves reward for rebounding the ball on screen to hit obstacles above it, and missing the ball. We limit the number of actions per episode
18

---

Published as a conference paper at ICLR 2023
Table 4: Additional Results: Various baselines are compared to PW-Net. Overall again, only PW-
Net consistently performs well.
Atari Breakout
Cart Pole
Method
Reward
Accuracy
Reward
Accuracy
PW-Net
28.14 + 0.20
92.44 + 0.26
200.0 + 0.00
98.69 + 0.001
VIPER
N/A
N/A
200.0 + 0.00
54.35
3.03
PW-Net*
23.41 + 0.10
88.94 + 0.06
197.4 1.40
98.69
+ 0,000
k-Means
00.00 + 0.00
00.00 = 0.00
20.11 9.68
14.40 7.77
Black-Box
26.00 + 0.81
N/A
200.00 + 0.00
N/A
to 1000 to (1) avoid the episode going too long, and (2) to which agent performs best as quickly as possible.
The A3C model learned to perform well in the domain, achieving average reward of around 26.00 across 30 simulations. Following Algorithm [I we first collected simulation input-output data for 30 simulations, corresponding 30,000 datapoints for training the PW-Net. To pick the prototypes, this data was partitioned into two six corresponding to each action, and the mean centre of the cluster was used to get the nearest prototype for each action.
We trained five PW-Nets on these data, for 50 epochs, and ran 30 simulations each time to collect the results, and the same done for PW-Net and k-means. Overall, the PW-Net performed well in the domain with an average reward of 28. 14, and an accuracy of 92.44, showing that it approaches the same function as the black-box, but the constraints of using the prototypes means inevitably learns to reason in different manner more intuitive for humans.
The other algorithms PW-Net* and k-means achieve 23,41, and 0.00 reward, respectively. This shows that again, PW-Net* performs well, but better performance is achieved using human-defined prototypes and multiple transformations (see Fig. T]. k-means performs poorly this domain, showing the necessity for using transformations in PW-Nets.
Interestingly, the agent here uses recurrent neural network based policy. So, this further proves the generality of PW-Nets into different architecture choices.
19


Reviews:
[{"review_id": "glGP82ELw8S", "reviewer": "Reviewer_7Kzz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method to explain a pretrained DRL policy. At a high level, it adds a wrap network on top of the original policy network and learns this wrap network by minimizing the difference between its output and the policy network output. Then, it generates explanations using this wrap network. The paper tried to prove the wrapped network could mimic the policy network outputs and empirically show it. Finally, the paper designs a user study to demonstrate the utility of the obtained explanation.  ", "summary_of_the_review": "The paper proposes an interesting and new explanation method for DRL policies. It seems to be more human-understandable. However, it is unclear what explanation is generated from the proposed model and how to generate it. This imposes challenges for readers to understand the benefits of the proposed method. Besides, the theoretical analysis is relatively trivial and non-rigorous. Therefore, I would rank this paper below the acceptance bar.   ", "strength_and_weaknesses": "Strengths:\n+ This paper proposes a new explanation method that encodes human-specified prototypes to improve its human explainability.   \n+ This paper well summarizes the existing literature. \n\nWeaknesses:\n- My first major concern about this work is the obtained explanation. After reading the whole paper multiple times, it is still relatively hard to understand what exactly the explanation is. The point I can get is the paper tries to explain the individual actions of an agent. But, It is unclear to me the paper explains the actions through what. For example, most existing works explain individual actions by **highlighting the regions in the input state that are most important to the output action.** In this work, I do not find such a definition. The proposed technique trains the network by predicting the original policy action. After the training, it has prototypical states $S_p$, the weight $W'$, and the encoder $h$. How to generate explanations from these ingredients is not specified. I would highly suggest the authors (1) Give a clear definition of the explanation in this work; (2) Specify how to generate the explanation; (3) Explain the functionality of each component in generating the explanation. (Why we need the prototypes and weight).   \n\n- Regarding the technique, I also have the following questions: (1) How to specify the weight $W'$. Should it be dynamically changed in different states and trajectories? (2) What is the influence of wight and prototypical states on the approximation accuracy (MSE) and explanation quality? \n\n- IMHO, the theoretical proof is relatively trivial. The final conclusion is if the similarity is proper, the predicted action is accurate. Since the model is actually learning the proper similarity, this is equivalent to saying if the model $h$ is well trained, the output is accurate. This is obviously true. \n\n- The evaluation is also not that comprehensive. The quantitative evaluation mainly focuses on measuring the difference between the original policy network and the predicted one. This is only the first step. Two networks give the same output may be based on different evidence. The paper misses a quantitative evaluation of the explanation of fidelity or faithfulness. That is, whether the given explanation truly reflects the decision process of the neural policy. The authors could refer to existing evaluation methods for how to design and conduct such an evaluation (e.g., [A Benchmark for Interpretability Methods in Deep Neural Networks]). \n\n- The user study is not that clearly stated. First, how to define in-distribution and OOD? Second, I would suggest providing an example to demonstrate what is presented to the participants and what are the survey questions. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GMvn_9fht-Q", "reviewer": "Reviewer_KNoV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on how to construct a wrapper model that is applicable to any arbitrary trained agents which leads to interpretability improvement in deep reinforcement learning. Their technique is called Prototype Wrapper Network (PW-Net) with the key concept of using human knowledge for prototypes and mapping between prototypes and actions.\n\nThe main contributions and findings of the paper:\n- Using prototype-based learning in deep reinforcement learning\n- Pre-hoc interpretability, forcing the agent to learn the human-designed prototypes makes the model inherently explainable\n- Performance of the agent doesn’t degrade by using the proposed wrapper\n- User trial to measure user’s predictions (failures and successes) of different models", "summary_of_the_review": "Overall, I vote for accepting. I like this paper because the proposed technique can provide inherent interpretability for deep reinforcement learning agents. My main concern is the extra work and time of creating human-defined prototypes and weight matrix that is needed for the wrapper network.", "strength_and_weaknesses": "Strengths:\n- First prototype-based deep reinforcement learning technique\n- Can be added to any pre-trained agent\n- Experiments fit the task by comparing the proposed method to the black-box agent\n- Evaluated the technique in a user study\n\nWeaknesses:\n- Experiments section lacks some explanations of the evaluation metrics and results (see questions Q2-Q4)\n- Needs hand-crafted prototypes: my concern is the ease-of-use of the method, as it needs extra work from humans to define the prototypes (considering all actions with multiple prototypes) and also the weight matrix. End-to-end learning is usually preferred because it doesn’t need extra hand-crafted elements.\n\nQuestions and suggestions for the authors:\nQ1: Figure 1.: indices of h, z and p. Even though it is highlighted in the text that the model uses separate mapping, the image of the network shows the same indices (i,j). \n\nQ2: What is the reason for the different evaluation metrics (MSE vs Accuracy) of discrete and continuous action spaces? \n\nQ3: There are no results of the VIPER in Car Racing and Atari Pong. Why?\n\nQ4: As your technique doesn’t give any advantage to the trained agent, how could the PW-Net achieve slightly better results in the experiments section?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "D8c4nQnjpm6", "reviewer": "Reviewer_EkdB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new post-hoc explainable method, called PW-Net, to provide insights into the workings of deep RL policies. The main advantage of the proposed method is to use human-friendly prototypes to make the decision-making process of an RL agent clear. Also, a user study is included to support the main claims of the paper.", "summary_of_the_review": "See above.", "strength_and_weaknesses": "Following are suggestions, questions, and concerns that I had while reading the paper.\n\n1- It is mentioned in the paper that “This paper builds upon recent work building prototype-based neural networks for interpretable supervised learning.” However, from [1] and [2] we know that borrowing such methods from other learning approaches could often lead to misunderstanding and faulty insights, since they’re not inherently designed for RL. How does this work address such issues?\n\n2- Related work section needs a better discussion. There’re a lot of works missing that could improve the narrative of the paper, and better describe its goals and motives. A few examples of such papers are: [1], [2], [3], [4]\n\n3- Since the proposed method is model agnostic, it’d be good to have results of other black-box pre-trained models in the paper for each particular environment. For example, in car racing, other than PPO, another policy could be studied. Also, providing results on more environment could help to verify the generalizability and robustness of PW-Net.\n\n4- Doesn’t PW-Net change the behavior of the original black-box pre-trained policy? If yes, how can its interpretations be trusted to reflect the original policy? If no, why there’s a performance difference according to the Table 1?\n\n[1] M. H. Danesh, A. Koul, A. Fern, S. Khorram. Re-understanding Finite-State Representations of Recurrent Policy Networks\n\n[2] A. Koul, A. Fern, S. Greydanus. Learning finite state representations of recurrent policy networks.\n\n[3] A. Mott, D. Zoran, M. Chrzanowski, D. Wierstra, D. J. Rezende. Towards interpretable reinforcement learning using attention augmented agents\n\n[4] A. Atrey, K. Clary, D. Jensen. Exploratory not explanatory: Counterfactual analysis of saliency maps for deep rl", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0UEDF5MPE2", "reviewer": "Reviewer_VRxv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new method, Prototype-Wrapper-Network (PW-Net) to offer explainability of pre-trained RL policies by training a new set of embedding functions on top of the existing feature representation for a learned policy. These embedding functions are each associated with a given prototype/exemplar for salient actions in a given domain. The fine-tuning process learns to map from the learned policy's representations into similarity functions for each prototype, and the final output action is chosen as a pre-defined weighted combination of similarity scores for each prototype. By having direct access to similarity scores for different actions in the domain, and by pre-defining the mapping from prototypes-to-actions, humans are given the opportunity to better understand actions taken in a domain. The PW-Net is evaluated to show that it does not reduce overall performance for pre-trained policies, and a user-study shows that humans can use the given prototypes to better understand how the policies work (and to trust the method more).", "summary_of_the_review": "The paper contributes an explainability method based on human labels and action prototypes, offering useful explanations for output actions that are verifiable through known prototypes and similarity scores. While the method is sound and the performance results are convincing (i.e., no performance drops with the PW-Net), the user study design is questionable and it is not clear that the trust and accuracy ratings from participants can be fully attributed to the PW-Net rather than simply to the instructions of the study. \n\nI advocate for acceptance on the basis of the method, but I would like to see a clearer description of the user-study in the main paper and a discussion of possible confounds for the results (such as the instructions telling participants that one method saw millions of other samples). If the study cannot be appropriately defended, I would even advocate removal of the study, on the basis that it is likely not studying explainability, but is instead studying the effects of a text prompt. With clearer explanation of the study or removal of the study, I will increase my score.", "strength_and_weaknesses": "Strengths:\n* The PW-Net is explained clearly and is easy to follow.\n* Experimental results verify that black-box policy performance is preserved with the PW-Net wrapper and output mappings.\n* Human evaluation shows that the prototypes and PW-Net explanations improve user trust enhance user's abilities to predict policy behavior.\n\nWeaknesses:\n* Much of the method must be hand-designed by humans. This is discussed in the paper as a positive, because prior work that simply learns prototypes does not always learn human-usable prototypes, but hand-designing mappings and prototypes is a non-trivial exercise, particularly for something like a 7-DoF robot arm.\n* The PW-Net assumes that learned representations are fixed, so any amount of fine-tuning for the policy would require additional fine-tuning for all learned prototypes and embedding functions.\n* The prompts and instructions in the user study vary significantly from the control to the proposed method. It is not clear whether human users are rating the control as less trustworthy because the PW-Net prototypes are significantly more useful or explainable, or because participants are being explicitly told that the control has seen \"MILLIONS of situations you are NOT being shown.\" Apart from that very bold text prompt at the top of every question, the two surveys appear to be identical. It therefore seems possible that the user study results are not the result of more interpretable prototypes or a useful mapping function, but just because users are being reminded that one method is less trustworthy than another. Intuitively, the method should improve trust and accuracy, as is reported here, though recent work on explainability methods with humans finds that this is not always the case [1, 2, 3, 4].\n\n[1] Poursabzi-Sangdeh, Forough, et al. \"Manipulating and measuring model interpretability.\" Proceedings of the 2021 CHI conference on human factors in computing systems. 2021.\n\n[2] Silva, Andrew, Mariah Schrum, Erin Hedlund-Botti, Nakul Gopalan, and Matthew Gombolay. \"Explainable Artificial Intelligence: Evaluating the Objective and Subjective Impacts of xAI on Human-Agent Interaction.\" International Journal of Human–Computer Interaction (2022): 1-15.\n\n[3] Peter Hase and Mohit Bansal. 2020. Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5540–5552, Online. Association for Computational Linguistics.\n\n[4] Andrew Anderson, Jonathan Dodge, Amrita Sadarangani, Zoe Juozapaitis, Evan Newman, Jed Irvine, Souti Chattopadhyay, Matthew Olson, Alan Fern, and Margaret Burnett. 2020. Mental Models of Mere Mortals with Explanations of Reinforcement Learning. ACM Trans. Interact. Intell. Syst. 10, 2, Article 15 (June 2020), 37 pages. https://doi.org/10.1145/3366485\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%