ProtoX: Explaining a Reinforcement Learning Agent via Prototyping
Authors: Ronilo Ragodos
Tong Wang
Qihang Lin
Xun Zhou
Venue: NeurIPS 2022 Accept (2022)
Keywords: policy explanation
interpretability
imitation learning

Abstract: While deep reinforcement learning has proven to be successful in solving control tasks, the ``black-box'' nature of an agent has received increasing concerns. We propose a prototype-based post-hoc \emph{policy explainer}, ProtoX, that explains a black-box agent by prototyping the agent's behaviors into scenarios, each represented by a prototypical state. When learning prototypes, ProtoX considers both visual similarity and scenario similarity. The latter is unique to the reinforcement learning context since it explains why the same action is taken in visually different states. To teach ProtoX about visual similarity, we pre-train an encoder using contrastive learning via self-supervised learning to recognize states as similar if they occur close together in time and receive the same action from the black-box agent. We then add an isometry layer to allow ProtoX to adapt scenario similarity to the downstream task. ProtoX is trained via imitation learning using behavior cloning, and thus requires no access to the environment or agent. In addition to explanation fidelity, we  design different prototype shaping terms in the objective function to encourage better interpretability. We conduct various experiments to test ProtoX. Results show that ProtoX achieved high fidelity to the original black-box agent while providing meaningful and understandable explanations.

ProtoX: Explaining a Reinforcement Learning Agent via Prototyping
Ronilo Ragodos
Tong Wang"
Department of Business Analytics
Department of Business Analytics
University of Iowa Iowa City, IA 52242
University of lowa Iowa City, IA 52242
ronilo-ragodossuuiona edu
tong wangwwanguuiww.... edu
Qihang Lin
Xun Zhou
Department of Business Analytics
Department of Business Analytics
University of lowa Iowa City, IA 52242
University of lowa Iowa City, IA 52242
qihang- liraann-iinuioow... edu
xun-zhoubuiowa. edu
Abstract
While deep reinforcement learning has proven to be successful in solving control tasks, the "black-bos nature of agent has received increasing concerns. We propose prototype-based post-hoc policy explainer, ProtoX, that explains a black- box agent by prototyping the agent's behaviors into scenarios, each represented by prototypical state. When learning prototypes, ProtoX considers both visual reinforcement learning
similarity and scenario similarity. The latter is unique context, since it explains why the same action is taken in visually different states. To teach ProtoX about visual similarity, we pre-train an encoder using contrastive learning via self-supervised learning recognize states similar if they occur close together in time and receive the same action from the black-box agent. We then add an isometry layer allow ProtoX to adapt scenario similarity to the downstream task. ProtoX is trained via imitation learning using behavior cloning, and thus requires access to the environment agent. In addition to explanation fidelity, we design different prototype shaping terms in the objective function to encourage better interpretability. We conduct various experiments test ProtoX. Results show that ProtoX achieved high fidelity to the original black-box agent while providing meaningful and understandable explanations.
Introduction
Deep reinforcement learning (DRL) has been widely used to solve various control problems such as autonomous driving (1. gaming (2), etc. Despite superior performance, DRL agents are often criticized for being "black-boxes" in nature and unable to provide human-understandable explanations for their actions. Popular solutions to interpreting a black-box agent include attention-bart methods [3] and saliency maps 6) These explanations, however, only inform users of where the agent "looks", but do not provide a more specific explanation for why an agent takes an action in given state. Some recent methods answer this question by formulating a classification problem and building a decision tree from state information to predict an action limitation of feature representation, that they can only work with meaningful features in order to
[7]. However, tree- rre-based models face the
corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

RIGHT JUMP
RIGHT SPEED
Similarity scores
Similarity scurcs
Why does the agent play RIGHT JUMP?
Protetypes
Weights Evidenco
Weights Evidence
0.73 15.7 11.46
0.73
X -8.3 -6.06
RIGHT JUMP
0.32 x 11.3 3.62
0.32 2.7 0.86
0.51 x 15.7 = 8.01
0.51
X 118.9
-9.64
8
Total evidence for
Total evidence for RIGHT ISPEED: -13.7
JUMIT:
27.8
Figure 1: Given an input state where Mario stands in front of a pipe and the agent plays the action RIGHT JUMP, ProtoX explains this action by collecting evidence from the similarities to the prototypical behaviors. The evidence is the sum of similarities between the input state and the prototypes, weighted by the action connections. An action connection represents how strongly prototype supports the prediction of an action and in which direction. Thus, the action with the largest evidence (RIGHT JUMP with evidence 27.8) is predicted for the given state.
make sense to humans. They do not naturally apply to unstructured data such as images and texts unless auman-understandablee information is manually extracted.
This paper aims explain black-box agent that works with unstructured data. To make the explanations understandable, we make use of the concept of prototypes from recent DNN models for image classification [8 9 and text classification [00] A prototype is an instance (e.g. an image) from data that is representative of prominent feature for certain class. For example, the red- bellied woodpecker class could have prototype for its red head and a prototype for its wings with zebra-like pattern, etc. Then, when there is a new input, the input can be compared against all prototypes trained from data. the input also has a red head and zebra-like pattern on the wings, the weighted of similarities in the red-bellied woodpecker class will be high, and thus the input will be predicted to be this class.
In this paper, we propose policy explainer called ProtoX, which explains an action by relating an input state to set of prototypical behaviors learned from demonstrations from the agent. Figure I shows example of ProtoX explanation for an agent playing the Super Mario game. ProtoX learns a set of prototypical states during training, which are most representative of the agent's behavior.
In summary, ProtoX prototypes an agent's behavior into small set of representative behavior patterns, each represented by prototype. and explains the agent's action based on these basic patterns. Therefore, key determinant of the success of ProtoX is the prototypes. An important consideration when computing the similarity between an input and a prototype is the visual similarity. Our definition of similarity must appropriately balance the aspects of visual and contextual similarity. Otherwise, e.g. if two states look entirely different and are assigned high similarity, human users may get confused and discredit the explanation. In ProtoX, we design self-supervised training step for the encoder to learn visual similarity. Usually, images that are closer to each other time and receive the same actions are more visually similar than those far away or receiving different actions. Thus, we create a siamese net M [2] that utilizes a modified form of the time-contrastive loss. Our pre-training routine yields encoders whose feature spaces cluster together visually similar states which occur closely in time and have the same corresponding actions.
In the supervised learning setting, visual similarity would be the only consideration, given the nature of image classification tasks. However, in reinforcement learning setting, two visually similar states could receive different actions (such as the flip points we define later), and two visually dissimilar states could represent a similar "scenario" that requires the same action. Therefore, the similarities also need to take into account scenario similarity. Here, scenario represents collection of states in which an agent performed the same action for similar reasons. For example, in Figure [ an action RIGHT+J JJMP has high similarity score prototypes that represent different scenarios for

---

jumping, which could be that Mario encounters pipe (the 1st prototype), a hole (the 3rd prototype), or preparing hit question mark brick (the 2nd prototype). In fact, the scenario similarity brings more value to the explanation, since it helps users to think about what is common between two visually dissimilar states that leads to the same action. To allow the feature representations to incorporate scenario similarity, we add a near-isometry layer after the encoder, which applies a linear transformation of the output of the encoder to allow some states to move closer to each other.
ProtoX is trained end-to-end via imitation learning, by learning from the demonstrations of the black-box agent to be explained. We choose behavioral cloning [T3] in this paper because requires minimal information from both the agent and the environment: only demonstrations from the agent and no access to the environment or the agent. Therefore, is most applicable in practice. Many other imitation methods, such as GAIL- based methods and inverse reinforcement learning methods, require more information from the environment or/and the agent, such as knowing the transition functions, executing a policy during training, etc.
We summarize the contributions of ProtoX. First, ProtoX is the first to explain the actions of reinforcement learning agent via prototypes. Most importantly, the reinforcement learning agent is used a black-box, which means ProtoX only needs learn from demonstrations and does not need to access the internals of the agent, which makes it easier to use in practice. Meanwhile, ProtoX also does not need access or interact with the environment. Second, we design an encoder to preserve visual similarity for better interpretabilityy via self-suuerrvi allows ProtoX to adapt to both visual and contextual notions of similarity. Third, while the existing interpretable baseline (VIPER) only works well on structured data, ProtoX can more naturally work with unstructured data.
learning and an isometry layer.
2
Related Work
RL interpretation can be categorized into two main groups: inherently interpretable RL models and post-hoc explanation approaches. Inherently interpretable RL models generate policies that intrinsically understandable, such as Neurally Directed Program Search [14], fuzzy RL policies [15]. hierarchical policies 17 [8] or extract time step importance within episodes
strategy-level explanations [19. Post-hoc methods, like the proposed ProtoX, aim to explain a given black- box agent to help users understand the agent's 's decision-making process towards an action. The goal not good task performance but high fidelity in explaining a given black-box agent, regardless of the own performance of the agent.
Based on the type of data they work with, post-hoc methods can also be split into two areas. The first class of methods only work with structured data, such as genetic programming 20). [IPER and
other rule-based methods 21 22 23 expected consequences (24), causal lens (25. complementary
RL (26). etc. These methods often adopt existing forms of interpretable models from supervised learning, such as rules 77 26). However, since the explanations need to directly use state features, they do not naturally work with unstructured data. To learn video games from pixels, they require meaningful features to be extracted manually. The other type of post-hoc methods work with reward decomposition [30], and interestingness elements [3I] The explanations, however, are either only at the global level or too abstract, such as saliency maps that only inform users of where the agent "looks" but do not directly explain the policy.
structured data. Examples include saliency-baa based methods (7 28 29 26
Another stream of work that aims to provide some interpretability stems from the family of imitation
learning. For example, inverse reinforcement learning (IRI.) methods (32 33 reward function under which the observed demonstrations of an agent is near-optimal. The interpretability, however, is only achieved through the learned reward function with a pre-defined structure (c.g.. linear combination of state features) rather than understanding the policy itself. Methods based on generative adversarial imitation learning (GAIL) (35 such as xGAIL (36) and infoGAIL [7 provide indirect explanations to the learned policy in terms of global and local feature importance or latent factors that govern the policy generation. However, none of them provide a direct explanation of the expert's 's actions under each state that are intuitive to humans. More importantly, all of the above methods require intensive interactions with the environment or the agent during training (such as executing policy), which are often not accessible most practical situations. On the other hand, our proposed ProtoX model is completely offline and requires no environment or
34] aim at learning
3

---

E-SNE plot for fal)
Simll
0.213
1.6
RIGHT
14.3
RIGHT + SPEED
0.842
gime trames
layer
24.1
Simd.
PX
0.765
Similurity Scores
JUMP
t-NE plot for Afr))
Figure 2: The model architecture of ProtoX.
agent interaction. also needs no additional information from the environment such as the transition function, reward function, etc. Another work that is relevant to ProtoX is VINN [38], which also learns from visual demonstrations. VINN uses BYOL [39 to train a self-supervised encoder for visual representation and then uses KNN based regression for action prediction. Since VINN was not designed as an explainer, it does not consider the visual similarity and the human-understene understandabll explanation ProtoX does.
3 Prototype-based Policy Explainer
We consider discrete-time partially observable Markov decision processes (POMDP). which in- clude video games such the OpenAl Gym classic control games [40]. We denote the obser- vation (e.g. frame of video game) at time by og and the C latest observations at time by We call the state at time and denote the set of all states across all times as a state space s Let A be the action space of the POMDP, and consider a policy that time based the C' latest observations, namely, mapping S A.
8, (ot, 0t-1..... 1...1......... ,t-.-.......-11
chooses action
Given policy by an expert n* - A, our goal is to build prototype-based policy explainer that is parameterized by o, denoted Tz S A, called ProtoX, such that (s(s) л (s), vs. We train Ty via behavioral cloning, which proceeds with two steps. First, we apply π to the POMDP to generate set of state-action trajectories and decompose all trajectories into a dataset of the expert's actions at different states, denoted by {(s,, n (s,)) Next, we train a classifier to predict actions from states.
We present the architecture of ProtoX, which consists of four main components, an encoder, near-isometry layer, prototype layer, and fully connected layer. See Figure for an illustration of architecture. Below, we describe its main components and then formulate the learning objective, which considers both the fidelity and the interpretability.
3.1 Feature Representations Preserving Visual and Scenario Similarity
key determinant of the success of ProtoX that the feature representation of the states that can respect the visual similarity and reflect scenario similarity. Therefore, ProtoX needs to recognize task-relevant features from the images benefit the downstream task, as well as understand what considered "visually similar". However, we do not have labeled data that supervises an encoder to determine which images are similar and which are not. Therefore, we use self-supervised learning approach. We construct an objective via contrastive learning and train an encoder fo parameterized by 0 using a siamese variational auto-encoder (VAB) 41 The contrastive portion of our objective similar to the time-contrastiv triplet loss from [11,, based on the idea that images that are closer to each other in time are more likely to be visually similar to each other. Our loss function differs slightly in that it uses time-contrastive quadruplet loss. We use quadruplet loss that the encoder learns that states are the most similar when they occur close time and are associated with the same action. Thus, this encoder also incorporates functional and temporal similarity in states' representations.
We have little abuse of terminology here S is different from the traditional state space of POMDP.

---

For a given state st, which we call an anchor state, a positive (visually similar) image si is one that is at most 8 away in time in an episode and receives the same action from T², i.e., x² (s²) = x (s{). ). We construct two types of negative (dissimilar) states. We find one negative state & among states that are within the same temporal neighborhood but receive different action than the anchor. We find a second negative state s, among states that are outside the temporal window. Since 8f" is far away from 8t in time, it is highly likely they look dissimilar. Thus, a quadruplet (Se. &5 ,,, St E 5t is con- structed. See Figure for an illustration of the idea in Super Mario.
snchor
st
102
104
107
102 5
11:5
the neighborhood of receiving the same action
the neighhorfood of 9 receiving diffeerent action
Ouaide neightorhood of
Figure 3: The model trains itself by trying to answer the following questions simultaneously: What is com- mon between the 8, and s frames? What is different between the similar-looking &, and 8, What is con- sidered visually dissimilar between 8, and 8, ?
The quadruplet loss is then defined onm (st, ,5 as
Lquadrupter (8t, st S7,
st SF max((]fo(ss) - fo(st) )12 [[ - fo )12 m1, 0) (|5((5))
(1)
+ max( - fo(ss) - - so )|2
2 m2,
mi and m2 are referred as margins. my is used that, by minimizing Laundruplet fo views anchors as at least my closer to positives than to the first type of negatives. m2 is used to ensure that the two types of negatives are pushed at least m2 away from each other. The full loss function for the siamese VAE is the quadruplet loss plus the conventional VAE loss function. Pseudocode and further details for our pre-training algorithm can be found in the supplementary material. Once fo trained, it will be frozen for the down-stream training, such that the property discussed above is preserved.
Output of the Encoder The encoder is able to map visually similar states close together in the latent space. In general, we find that the most similar states to a given state are those that almost look identical. The less similar states are generally those that could occur in small window of time before or after the given state. We show the efficacy of the encoder by aggregating the most similar 30 states together, i.e., the 30 states with the smallest [))) - fe(x=')l (222 See Figure 4 for an example that shows what states the pre-trained encoder views as similar in Super Mario Bros. level 1-1. This example shows that the states similar to the input are those where Mario is in the process of jumping over a hole in the ground.
Top 30 Most Sinilar States
Input
Figure 4: An input state x and top 30 states most similar to fo(x) overlaid.
Learning Scenario Similarity Note that the training of fo does not involve the downstream tasks, thus the feature representations, while respecting the visual similarity, may not consider the task relevance, i.e., the scenario similarity, which is task-specific and agent-specific. Similar failure modes in contrastive learning are explored in (42.. Therefore, to overcome the possibility of the fe failing to be useful to the downstream task, we add linear layer A x''''''xW - RC' H'W" after the encoder. Since A is learned during training, it will further adapt the representations of the input states to the downstream RL task. However, we also want to make sure this linear transformation of A does not undo ProtoX's ability to understand visual similarity. Thus, A needs be nearly isometric, such that, via 43 if | A
30 Similar
Input
Figure 5: An input state x and top 30 states most similar to A fox) overlaid. ProtoX identi- fies the green pipe being similar the hole, since they both acti- vate the action of JUMP.
T| < 5 (where
Therefore, we will include a penalty term in the objective such that the isometry layer allows some evolution of the encoder Afe while still retaining the way fo learned to separate dissimilar states during pre-training.
8 > 0), then for all data x, x'. (=) Afo(2 < (1+ f))l {1+))])fe(z))
Iso
AA
(2)
2'
5

---

where I is the identity mapping of the appropriate dimension.
The t-SNE plots in Figure why the isometry layer benefits the downstream task. We show the 30 states most similar to the same input in Figure this time after the isometry layer, i.e., the 30 states with the smallest (o) Afo(2') (=')l222 In contrast to Figure 4 in the example in Figure 5 ProtoX identifies scenario similarity as it considers jumping over a hole is similar to jumping over a green pipe. For more analysis of the isometry layer, please see the supplementary material.
3.2 Learning Prototypes
We then add prototype layer after the isometry layer that automatically learns K prototypes for each action in A. represented by { {= training. Each prototype has a shape of x C'''W', which is the latent representation of an actual image from training data. Given an input state æ, ProtoX computes the distances [[ pf ll for & = 1..... K and a E A. It uses these distances to compute similarity scores between the image and prototypes, following the transformation used in [4]]
Note that the prototypes are learned via end-to-end
sim sim(Afal (Afa(z). pt) exp( -
p l2)
(3)
where 3 controls the concentration of similarity scores and is set 0.05 in all our experiments.
The similarities are then fed into fully connected layer with weight matrix W E R|A]Kx|A| with
W (w2,Au M = a/,==i..,,,,EEEA The subscript represents the prototypes and the superseript represents the output actions, and there are |A[K protolypes in total. The linear layer outputs |A| evidence scores, expressed as weighted sum of similarity score contributions from all prototypes:
es (xx KE' wae Sim(Afo(=).
The linear model is chosen for its ease of explanation. It could be replaced by other interpretable models like decision trees, following the idea in (9] The evidence score is computed for every action and the final prediction (action) is made argmax ee" W). Given this decision rule, we next design an objective function that encourages the model to associate scenarios with an appropriate action perform.
3.3 Objective Function
Since To is post-hoc explainer for an expert T², we consider two desiderata when training T&. 1) fidelity of imitation, and 2) interpretability of the prototypes. We construct the following terms in an objective function for the two purposes.
Fidelity A cross-entropy term, denoted 1 E.= Lor(u, (e* ae 4). is included make sure ProtoX can reproduce the actions of the black-box expert as accurately as possible, which reflects the fidelity in the explanation. Here, e is defined in (4).
Separation and Clustering Terms The terms Clst (Clustering) and Sep (Separation) are taken from the ProtoPNet 8] The Clst term encourages prototypes be close to the encodings of training instances that have the same action label as the prototypes. The Sep term encourages prototypes to be far from encoded training instances with different action labels the prototypes. This term enforces consistency with our time-contrastive loss, since both states that image representations should be dissimilar if their corresponding actions differ. The Clst and Sep terms together encourage the model to learn prototype scenarios such that similar scenarios have an associated appropriate action and dissimilar prototype scenarios have different appropriate actions.
Sep = - Σi_ ming stt. aæy, [Af(())) - 12l12: Clst = Σ. ming [[Af(z;) -
(5) 11 -
Y/ represents the action for instance i and a represents an action that not y/. Thus, equation (5) means the distance between the encoding of should be far away from prototypes with a different action, which represented by the separation term. On the other hand, the encoding of x; should be close to at least one prototype for action y/.

---

Representability term The Rep (representability) term encourages prototypes to be similar to some encoded training instances, to ensure the representability of the prototypes. Because our pre-traiming routine already ensures that encodings [o())) are meaningful representations, we use this Rep term to further ensure that the prototypes are representative of the data. By continuously ensuring that prototypes are close to actual encodings of training data, we justify the success of prototype projection steps that will be explained later. This term is inspired by the Evidence term of [44].
Rep Σx= EaEA min; [[Af( - pill:
(6)
As discussed previously, the isometry term in formula ( allows some evolution of the encoder Afa while still retaining the way fo learned to separate dissimilar states.
Isometry Term
Therefore, the objective function
C(PPR)REA R. A, W)
1 Σ Lorlu, (2i, W))aeA) + A1Sep + A2Clst + ReR + Aa (7)
where X; 2 0, I,..., 4. are weighting parameters subject to tuning. The first term encourages high fidelity, and the last four terms are for prototype shaping.
3.4 Downstream Training Routine
We train our model with the standard behavior cloning (BC) algorithm, but with the aforementioned surrogate loss L. This training routine is inspired by that of the related ProtoPNet(8).
Prototype Training and Projection A prototype layer in ProtoX consists of K tensors of trainable parameters. Let's 's use one prototype tensor as an example, denoted by p. Since is trainable, as the model optimizes the objective in formula (7) via pack-propagation, p is automatically learned. How- ever, one cannot directly obtain the prototypical game states from p since is only representation in the embedding space. One needs to find out which actual state p corresponds to in the real game state space, represented by pixel values. However, p may not be able to map back onto an actual game state. Therefore, we do prototype projection, similar to the projection steps in ProtoPNet [8] every 25 epochs, we map all states in the training data into the embedding space via the encoder fo and isometry layer A, then find the one that is closest to p. i.c., x arg min [[Af(zz) p pl12- This becomes the prototype, we update p accordingly by setting p to Af)
Prototype Merging Once ProtoX model trained and we project prototypes into the space of encodings of real states, we often find that there is a significant amount of repetition among the prototypes. To eliminate the redundant prototypes, we create new prototype layer consisting only of the unique prototypes. We then also create a new fully connected layer whose weights capture the redundancy present in the previous prototype layer. Supposing that prototype i = in is the same as prototypes i2, is after projection, the connection between prototype i and action 7 in the fully connected layer is wa E/= Wiyl- We find that this prototype merging procedure effectively reduced the number of prototypes while retaining the performance of the network. This also means when we build ProtoX, we can over werrpaaameeerizz it by allowing it to learn many prototypes, but they will eventually be projected into smaller set of unique prototypes.
4 Experiments
We evaluate ProtoX on four Atari/NES game environments from OpenAl Gym and compare the results with two black-box and one interpretable baseline. We first simulate near-optimal black-box agent and then use different methods imitate the black-box. We then evaluate the fidelity of different methods, measured as the accuracy of their predictions compared to the agent. In addition, we evaluate the sensitivity of ProtoX on flip points, test whether ProtoX can capture important time steps in a reinforcement learning task. Then we investigate the interpretability of the explanations generated by ProtoX. Finally, we simulate a bad agent and use ProtoX diagnose its bad behaviors.
4.1 Experimental Setup
We use four video-game tasks from OpenAl Gym, namely, Pong, Seaquest and two levels from Super Mario Bros 45). For each game, generate a near-perfect black-box agent, denoted as n². Pong
7

---

and Seaquest use PPO models from the 3 47 package. The experts for Super Mario are also PPO models from 48]. Then we train ProtoX and other methods to imitate each agent. See the Appendix for hyperparameter settings and further details on our experimental design.
We compare ProtoX with three baselines. 1) VIPER 7.. which builds a decision tree model to output policies. Since VIPER only works with structured data and does not directly work with images, we extract features from the Pong and Seaquest games. Due to the complexity of Super Mario, we do not benchmark VIPER on the Super Mario games. 2) GAIIO 49), which also learns from observational data like ProtoX but a black- box model. Unlike ProtoX, GAIFO still interacts with the environment. The goal here is to compare ProtoX's 's fidelity to another imitation learning method with similar (slightly more) access to the environment. 3) ResNet-BC, which follows the same training procedure as ProtoX except using ResNet18 model to work with the game frames. ResNet-Bel BC is a black-bor -bxx model that does not explain an agent. We run ResNet-BC to evaluate how much fidelity is lost due to using prototypes; Both ProtoX and ResNet-BC are trained with the behavior cloning algorithm using 30,000 state-action pairs obtained via an expert trained with PPO.
4.2 Fidelity Evaluation
We first evaluate the fidelity of each method, i.e., how often each method agrees with the PPO agent x². For each game, we let the agent gener- ate test set of 10, 000 state-action pairs Diest = {s,, x (s:)).. We then apply each method to the states and record their predicted actions { (s:)). which is compared against (s:)) to compute the accuracy, i.e., E1 (*1) m(a)))
Table 1: Fidelity evaluation of different methods.
Datasets
Pong Seaquest Mario 1-1 Mario 8-3
ProtoX
95%
96%
93%
95%
VIPER
45%
31%
GAIFO
19%
11%
33% 99%
25%
ResNet-BC
95%
98%
99%
[Del recorded in [ ProtoX achieved much bet-
ter fidelity than the existing interpretable method, VIPER, since VIPER is a decision tree-based model and does not allow image state inputs. We set the maximum tree depth 10 for both games, which will produce a tree with thousands of nodes for both games. Increasing the maximum tree depth will slightly increase the fidelity by less than 3 percent, but it will significantly increase the number of nodes in the tree. Yet VIPER still performs much worse than ProtoX. In addition, as numerical features (coordinates of ball, or a submarine, etc) are more difficult understand than images in video games. GAIFO also performs poorly. GAlL-based methods are typically only applied to low dimensional problems like joint dynamics [35 49 When applied to high-dimensional Atari
problems, GAIL has been shown to achieve poor performance [50]. with our experiments.
We arrived at a similar conclusion
The comparison of ProtoX and ResNet-BC, even though ResNet- BC is not interpretable, shows that the interpretability design in ProtoX has small negative impact on fidelity, a 3% drop on average. When training ProtoX, purposely over-parameterize the network by using more prototypes than should be necessary. For example, when training ProtoX for Super Mario Bros. World 1-1, we use 350 prototypes initially. However, after the prototype projection and merging step, we are able to significantly reduce the number of prototypes in a trained ProtoX model, thus making each explanation low in complexity. In the case of Super Mario Bros, we can reduce the number of prototypes from 350 to only 17 while retaining relatively high fidelity. We report the number of prototypes for ProtoX models in Table The small number of prototypes ensures that the explanations are easy to follow.
Table 2:
The number of pro- totypes in ProtoX models
Datasets of prototypes
Pong
33
Seaquest
42
SM 1-1
17
SM 8-3
12
4.3 Sensitivity to Flip Points
For an RL agent, usually, the most important explanations are for time steps where the action changes from the previous action. This is because there's 's often no significant pixel change two consecutive game frames, and thus a change in the action would be worth investigating. Therefore, if a state receives a different action from its previous state, we call it a flip point. The set of flip points can be represented as JF {st SSix"(s # x (s)1))} We show an example of flip
8

---

point in Figure 6 At time step t - 1, the agent presses RIGHT + SPEED to run to the right. In the next time step, the agent presses RIGHT JUMP begin jumping over the green pipe. We evaluate the sensitivity of ProtoX to flip points, i.e, when the black-box agent changes its action,
Table Sensitivity analysis
can ProtoX catch up? We define sensitivity as the
percentage of successful flips, i.e., sensitivity(ra.
Dataset
ProtoX
VIPER GAIfO ResNet-BC
Exer ((="(x)==,(x)) We collect a total of 10,000
Pong
84%
44%
22%
80%
[F
Seaquest
87%
24%
6%
99%
flip points from the test set and report the fidelity of ProtoX this set in Table 3 This shows that even though flip points are visually similar, ProtoX realizes that they differ and is able reproduce the agent's 's action.
SM 1-1
94%
18%
97%
SM 8-3
89%
16%
99%
prototype
ddhl
action/IGHT
artion: JUMP
Prototype
Scenario
Prototype
Socnario
Fle point
divers enemiee
Most similar protatype
(a) A prototype for action FIRE
(b) A prototype for action UP
Figure 6: A flip point and the cor- responding prototype. ProtoX ex- plains the running action in time step 1-1 with a prototype of Mario running to the right, unimpeded by any obstacle. In contrast, Pro- toX recognizes the jumping action in time step as being similar to another scenario where the expert agent jumped over a green pipe.
Figure 7: Prototypes from Seaquest and their interpretation. The first prototype is for the action "FIRE", which means the submarine tries to shoot enemies to gain points. The prototype represents a situation where the player has shot missile to the right at an enemy coming in from the right screen edge.
The second prototype is for the action "UP", which means the submarine will move up for oxygen. This prototype represents
scenario where the submarine has saved divers, and there are enemies in the front, it is safe go up.
4.4 Prototype Interpretation
To investigate the interpretability of ProtoX, we show an example of the prototypes in ProtoX model learned from Seaquest. The game is an underwater shooter in which the player controls a submarine. Its three goals are 1) to shoot enemies, 2) to rescue divers, and 3) to surface for oxygen when needed.
In Figure | we show 2 examples of prototypes. To understand what scenario each prototype represents, we overlay 60 most similar states by plotting the full-color representations of each state with low transparency over one another. Thus, we can faintly see what each scenario represents. For example, for the first prototype in Figure 7 the submarine located in several places near the bottom center of the screen. But the common feature in this scenario is the enemies to the right of the submarine. The common feature of the second prototype that the submarine is in the same general position as in the prototype, but with varying air tank levels and a varying number of rescued divers.
Diagnosing Bad Behaviors The explanations provided by ProtoX can also be used to diagnose why an agent makes mistakes. To simulate this use case, we train a bad agent for Super Mario Bros. that never jumps and consequentially is always killed by the first Goomba it encounters. We then train a ProtoX on trajectory data from this agent see whether ProtoX can explain why the agent fails to jump to avoid the Goomba. Figure 8 (a) shows that Mario sprinted to the right into the Goomba. ProtoX explains this by relating this state to similar prototypes, and we show the most similar prototype in Figure (b), which is state in the beginning of the game when Mario could sprint to the right because there are no obstacles. To understand why the bad agent thinks the input similar to the prototype, we create an importance map over the bad agent's prototype. The importance map is generated by masking the prototype with patches and evaluating the change in the similarity between the masked prototype and the input state. A bigger change indicates higher importance.

---

(a) input state, Mario runs goomba
(b) The most similar prototype from had agent' ProtoX
(c) The similar prototype from the good agent" ProtoX overlaid by the importance map
overlaid by the importance map
Figure 8: For the bad agent, ProtoX views the prototype in (b) similar, without consideration of any strategic elements. In particular, only the sky seems to matter. In contrast, ProtoX for the good agent selects different prototype (c) and seems to view the immediate area of Mario hitting the brick to kill the Goomba as important, which explains why the good agent chooses the action "JUMP": he needs to jump kill the Goomba.
Then, to highlight the most important area, we keep only the top 95% of pixels with the highest values. The rest are set O. See the supplementary material for how to generate the importance plot. The importance map is overlaid on the prototype in Figure 8 (b). We deduce from the importance plot that the bad behavior is due to the agent only looks at the blue sky to determine the similarity.
To contrast with the bad agent, we then feed the same input state to another ProtoX trained on a good agent. The new ProtoX identifies different prototype for the input, shown in Figure In addition, the importance plot shows that the new ProtoX looks at the strategic elements of the games. Specifically, the importance highlights Mario jumping hit brick to kill the Goomba. ProtoX thinks this is similar to the input state, where Mario should also jump to kill the Goomba.
Conclusion
We propose a prototype-basee policy explainer, ProtoX, for explaining black-box reinforcement learning agents. ProtoX explains an agent by prototyping its behavior into different scenarios, each represented by prototype. ProtoX explains an action by comparing the corresponding input with its prototypes, computing the similarity scores between the inputs and prototypes, and then calculating accumulated evidence for each action. ProtoX takes into account both visual similarity and scenario similarity. We use self-supervised training and contrastive learning to learn the notion of visual similarity from expert demonstrations, and allow ProtoX to learn scenario similarity by using an isometry layer during its behavior cloning training. Results show that ProtoX achieved high fidelity compared to VIPER, a black-box model with similar network architecture, and also a GAIfO model whose policy network had similar architecture.
Reproducibility Our code is available at http:://gituub.oo/rrr///////
Discussion of Limitations The interpretation of the prototypes requires basic understanding of the task. For example, to interpret the prototype for UP in Figure]Jusee Jusers need to know the basic logic for playing the game, and thus can comprehend why the submarine goes up in that scenario. In addition, we have also noticed in the Seaquest game that the expert anticipates the arrival of enemies and shoots them before they even enter the screen. Thus, ProtoX learns prototypes showing the expert shooting at nothing. ProtoX's 's mode of interpretation does not inform the users of such anticipatory behavior. One would not be able to understand such prototypes without basic understanding the game. Also, ProtoX may learn similar and redundant prototypes. Some pruning techniques could be created to further improve the quality of prototypes.
Acknowledgements
We thank Nima Safaei for his help with running some baselines in earlier versions of this manuscript. We would also like to thank the anonymous reviewers for their comments and questions, which helped improve this paper. Finally, would also like to thank Katherine Ragodos for her mroof-reading.
10

---

References
[1] B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani. and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transporiation Systems, 2021.
[2] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):5295533 2015.
[3] Alex Mott. Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo J Rezende. Towards interpretable reinforcement learning using attention augmented agents, arXiv preprint arXiv:1906.00 02500. 2019.
[4] Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Femn Visualizing and understanding atari agents. In International Conference on Machine Learning, pages 1792-1801. PMLR, 2018.
[5] Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. Graying the black box: Understanding dqns. In International Conference on Machine Learning, pages 1899-1908 PMLR, 2016.
[6] Ziyu Wang, Tom Schaul, Mattco Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995-2003. PMLR, 2016.
Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. arXiv preprint
2018.
[8] Chaofan Chen, Oscar Li. Chaofan Tao, Alina Jade Barnett, Jonathan Su, and Cynthia Rudin. This looks like that: Deep lcarning for interpretable image recognition, 2019.
[9] Meike Nauta, Ron van Bree, and Christin Seifert. Neural protolype trees for interpretable fine-grained image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14933-14943, 2021.
[10] Dat Hong. Stephen Baek, and Tong Wang. Interpretable sequence classification via prototype trajectory. arXiv preprint 2xXiv:2007,01777 2020.
[11] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self-supervised learning from video, 2018
[12] Jinghuan Shang and Michael S. Ryoo. self-supervised disentangled representation leaming for third-person imitation learning. 2021.
[13] D. A. Pomerlcau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1 3(1)-88,97, 1991.
[14] A. Verma, V. Murali, R. Singh, P. Kohli, and S Chaudhuri. Programmatically interpretable reinforcement 2018.
learning. In PMLR, pages
[15] D. Hein, A. Hentschel, Runkler T., and S. Udluft. Particle swarm optimization for generating interpretable fuzzy reinforcement learning policies. Engineering Applications of Artificial Intelligence, 65:87-98, 2017.
[16] T. Shu, C. Xiong, and R. Socher. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. 2017.
[17] B. Beyret, A. Shafti, and A. A. Faisal. Dot-to-dot: Explainable hierarchical reinforcement learning for robotic manipulation. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5014-5019, 2019.
[18] G. Cideron, M. Seurin, F. Strub, and O. Pietquin. Self-educated language agent with hindsight experience replay for instruction following, 2019.
[19] Wenbo Guo, Xian Wu, Usmann Khan, and Xinyu Xing. Edge: Explaining deep reinforcement learning policies. Advances in Neural Information Processing Systems, 34, 2021.
[20] Daniel Hein, Alexander Hentschel, Thomas Runkler, and Steffen Udluft. Particle swarm optimization for generating interpretable fuzzy reinforcement learning policies. Engineering Applications of Artificial Intelligence, 65:87-98, 2017.
[21] Guiliang Liu, Oliver Schulte, Wang Zhu, and Qingcan Li. Toward interpretable deep reinforcement learning with linear model u-trees. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 414 429. Springer. 2018
[22] Youri Coppens, Kyriakos Efthymiadis, Tom Lenaerts, Ann Nowé, Tim Miller, Rosina Weber, and Daniele Magazzeni. Distilling deep reinforcement learning policies in soft decision trees. In Proceedings of the IJCAI 2019 workshop on explainable artificial intelligence, pages 1-6, 2019.
[23] Nathan Dahlin, Krishna Chaitanya Kalagarla, Nikhil Naik, Rahul Jain, and Pierluigi Nuzzo. Designing interpretable approximations to deep reinforcement learning with soft decision trees. arXiv preprint arXiv:2010.14885. 2020.
11

---

[24] Waa, J van Diggelen, K Bosch, and M Neerincx. Contrastive explanations for reinforcement learning in termss of expected consequences. In Proceedings of the Workshop on Explainable Al the IJCAI conference, Stockhulm, Sweden., 37, 2018.
[25] Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learning through a causal lens. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2493-2500, 2020.
[26] Jung Hoon Lee. Complementary reinforcement learning towards explainable agents. arXiv preprint 1.Xiv:1901.00188.. 2019.
[27] Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia Sycara. Transparency and explanation in deep reinforcement learning ncural networks. In Proceedings of the 2018 AAAT/ACM Conference on AI, Ethics, and Society, pages 144-150, 2018.
[28] Raghuram Mandyam Annasamy and Katia Sycara. Towards better interpretability in deep q-networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4561-4569, 2019.
[29] T Nathan Mundhenk, Barry Y Chen, and Gerald Friedland. Efficient saliency maps for explainable ai. arXiv preprint arXiv:1911.1129 2019.
[30] Zoe Juozapaitis, Anurag Koul, Alan Femn, Martin Erwig, and Finale Doshi-Velez. Explainable reinforce- ment learning via reward decomposition. In IJCAI/ECAI Workshop on Explainable Artificial Intelligence, 2019.
[31] Pedro Sequeira and Melinda Gervasio. Interestingness elements for explainable reinforcement learning: Understanding agents' capabilities and limitations. Artificial Intelligence, 288:103367, 2020.
[32] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceed- ings of the twenty-firt international conference on Machine learning, page 2004.
[33] Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd national conference on Artificial intelligence-Volle Volume 3. pages 1433-1438. 2008.
[34] Brian D Zicbart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. ICML, 2010.
[35] Jomathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29:4565-4573, 2016.
[36] Menghai Pan, Weixiao Huang. Yanhua Li. Xun Zhou, and Jun Luo. xgail: Explainable generative adversarial imitation learning for explainable human decision analysis. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1734-1343, 2020.
[37] Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 3815-3825, 2017.
[38] Jyothish Pari, Nur Muhammad Shafiullah, Sridhar Pandian Arunachalam, and Lerrel Pinto. The surprising effectiveness of representation learning for visual imitation, 2021.
[39] lean-Bastion Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self- supervised learning, 2020.
[40] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wajeiech Zaremba. Openai gym, 2016.
[41] Ross Goroshin, Joan Bruna, Jonathan Tompson, David Eigen, and Yann LeCun. Unsupervised learning of spatiotemporally coherent metrics, 2015.
[42] Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy, Understanding contrastive learning requires incorporating inductive biases, 2022.
[43] Roman Vershynin. High-dimensional probability: an introduction with applications in data science. Cambridge University Press., 2018
[44] Yao Ming, Panpan Xu, Huamin Qu, and Liu Ren. Interpretable and steerable sequence learning via prototypes. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, Jul 2019.
[45] Christian Kauten. Super Mario Bros for OpenAl Gym. GitHub, 2018
[46] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
12

---

[47]
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave. Anssi Kanervisto, and Noah Dormann.
Stable baselines3, https:///gtthu..commm// wwtp:///LLL-t///aaaeeeeaaaaaeeee
2019.
[48] Viet Nguyen. [pytorch] proximal policy optimization (ppo) for playing super mario bros. https
wwtnrttb..ooo/uuuppp////urraraaa www.bbroospppoe -ro--ooo-ytooorrtt 2021.
[49] Faraz: Torahi, Garrett Warnell, Peter Stone. Generative adversarial imitation from ohservation, 2018.
[50] Xingrui Yu, Yueming Lyu, and Ivor W. Tsang. Intrinsic reward driven imitation learning via generative model, 2020.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes] See the Section
Did you include the license the code and datasets? [No] The code and the data proprictary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5 Conclusion
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A] N/A
(b) Did you include complete proofs of all theoretical results? [N/A] N/A
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We include the code, data, and necessary instructions in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All training details provided in the supplementary material.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] Experiments were performed using single seed.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] All experiments were done on system with an RTX 3060Ti GPU, AMD Ryzen 7 3700X 8-Core Processor, with 32GB RAM
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We cite the models and libraries we used.
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We did not use such data.
13

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We did not use such data. All data are from OpenAl Gym
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "pWF0H91Fdyg", "reviewer": "Reviewer_m9zL", "summary": "The authors seek to explain black-box deep reinforcement learning agents using prototyping. Prototyping groups together agent behaviors and game-states into prototypical states. They achieve this using ProtoX, which is a self-supervised neural network that uses prototypical states in its forward pass. Prototypes are trained so that they represent groups of similar game states according to visual similarity and action similarity. They're used within ProtoX, which is trained to accurately predict the action taken by the agent given the observation. They demonstrate the method on Pong, Seaquest and two levels from Super Mario Bros. The first show that ProtoX has good fidelity compared with alternative methods (fidelity measure by whether the methods groups states with the same action together). Then they show that ProtoX is sensitive to flip points, which are states that are temporally adjacent but where the agent chooses a different action. Then they demonstrate ProtoX on seaquest and show that prototypes match the scenarios that activate them and show that prototypes of scenarios where the agent exhibits bad behaviours share a similar action to the true action used in the scenario. ", "questions": "I could be convinced to change my assessment if the authors could show how ProtoX can explain something about the agent's internals, which is necessary to explain why an agent takes an action in a given state. This could perhaps be achieved by augmenting ProtoX to identify feature-level prototypes. ", "limitations": "The authors have a good discussion of the method's limitations. They are non-obvious and I commend the authors for highlighting them. ", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors do a good job of presenting and explaining their method very clearly. I wasn't previously familiar with prototyping methods and have learned about them thanks to the authors. I am not aware of previous deep RL interpretation methods that take a similar approach; the approach is therefore likely original. \n\nI will judge the paper's quality and significance according to the stated goals of the paper, which are nicely emphasized in the introduction: \n\n(line 23) 'Popular solutions to interpreting a black-box agent include attention-based methods [3] and saliency maps [4, 5, 6]. These explanations, however, only inform users of where the agent “looks at”, but do not provide a more specific explanation for why an agent takes this action in a given state'\n(line 30) 'This paper aims to explain a black-box agent that works with unstructured data.'\nIn summary, the authors state in the introduction that the goal of the paper is an explanation of _why_ an agent takes an action in a given state. \n\nAt this point it's worth reflecting on what an explanation of an action (in a given state) actually is, since this is the whole point of ML interpretability work and of this paper. An explanation of an action in a given state should give us an understanding of why the state causes to the action; it should illuminate something about the black-box process that results in the agent taking a particular action in a particular state. Unfortunately, to this end, ProtoX does not succeed. ProtoX only tells us which states are similar to states where an agent took a given action. While prototyping is a valid though (arguably weakly informative) black-box interpretation method, this is only true if the prototypes are on the feature-level, i.e. they should tell us what parts of the state they are prototypes of. ProtoX provides protoypes that aren't on the feature-level; it provides us only with a global picture of the state. It reveals nothing about the black-box process that led from the state to the action. The authors justifiably criticise attention-based methods and saliency maps that tell us little about why an input leads to action; but even those methods tell us markedly more about the internals of the black-box than ProtoX, since those methods at least help us to identify features that the black-box decision making process is using.  \n\nValidation of the explanations provided by our interpretability methods is an important step, one that the authors perform in section 4.4 (Diagnosing bad behaviours). But I'm not convinced that this demonstration explains why the agent makes a mistake here (and my reason is in addition to the above criticism that ProtoX doesn't help us to explain the black-box decision making process): ProtoX is trained to group states together that are both visually similar AND that have similar actions. Rather than explain bad behaviour, figure 8 therefore shows only that both the prototype and the real scenario both involve the agent moving right and have similar visual features (because that is what the prototypes were selected to do), but doesn't explain anything about why the bad behaviour happened.  \n\nProtoX is an interesting network, and the ability to group together similar states (in both the visual and action modality) is a praise-worthy feat. But as an approach to help explain _why_ an agent takes an action in a given state (which was the stated goal of the paper) it does not succeed. \n\nMinor criticisms:\n\nThe authors write (line 280): \"We evaluate ProtoX on four classic control problems from OpenAI Gym and compare the results with two black-box and one interpretable baseline\". The environments the authors are referring to are Pong, Seaquest and two levels from Super Mario Bros. I'd encourage the authors not to call these \"classic control problems from OpenAI Gym\" because gym.envs.classic_control is a different set of environments than Atari games. Their conflation will be confusing to many readers. \n\n(Line 133, footnote). The authors can avoid the abuse of notation by instead describing their RL setup as an MDP: From Wikipedia for POMDP: \"an MDP can be reformulated as a POMDP by setting the observation set to be equal to the set of states and defining the observation conditional probabilities to deterministically select the observation that corresponds to the true state.\"\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "drMti5-fjhm", "reviewer": "Reviewer_XKWc", "summary": "The paper introduces a method for explaining black-box RL agents. This is accomplished by learning \"prototype\" states and identifying which prototype the current state corresponds to.\nOverall a good and potentially impactful paper and I'm recommending acceptance but conditional on some clarification.", "questions": "I've included most of my burning questions in the section above. Here's some minor stuff:\n\n- Q.1) In the checklist, you mentioned \"the code and data are proprietary\" - what data?\n- Q.2) You also mentioned all experiments were done with a single seed - did you do a preliminary test to verify that indeed your method isn't sensitive to random seeds?\n- Q.3) How sensitive is your method to the loss coefficients?", "limitations": "The authors adequately addressed limitations. I'd also add that maybe this method could in the future be combined with visual saliency methods to highlight what part of a prototype is responsible for the agent's decisions.", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "TL;DR I think the method introduced in this paper is powerful and desperately needed as a tool for the RL community. I just don't have an intuition on how the prototypes are discovered, even after multiple reads. I'm happy to raise my score if the authors address some of my concerns below.\n\n### Strengths\n\n- S.1) Very cool idea and very useful to RL folks.\n- S.2) General paper structure good and illustrations helpful.\n- S.3) Flip point and fidelity evaluation are good ideas. \n- S.4) Easy-to-reproduce Jupyter notebook included. \n\n### Weaknesses\n\n- W.1) I don't fully understand how prototypes are automatically generated, end-to-end. I've seen the different losses in sec. 3.3, I've looked at the pseudocode in the appendix and I had a very quick glance at the Jupyter notebook but I still don't have a good intuition on how they come to be. Maybe other reviewers got behind this more easily. If you could make another attempt at giving me a high-level intuition on the mechanism behind the prototype learning in the rebuttal phase, I'd appreciate that.\n- W.2) Analogous to this, there are a few expressions in the paper that are too unclear for me, like lines 61 (\"If two states...\"), 263 (\"During this step...\"), 272 (\"We then also create...\")\n- W.3) I think using prototypes for explainability calls for some more qualitative analysis. I'd like to see some more examples with annotations on what they indicate, like Fig. 7. The purpose of the paper is to provide a \"policy explainer\" and I don't have a feeling that a policy was explained, other than in the \"bad behavior\" example.\n- W.4) Using 30 most similar frames to a given frame and combining them transparently is conceptually a good idea but as a reader, I can't really make out a lot there. I'd recommend either using fewer frames or making a grid of frames or maybe even 5 images of 6 overlayed frames or something similar. \n- W.5) 2 minor writing things: (a) I think that figures should stand on their own in a paper and a figure + its caption should contain all the information needed to parse it. In your paper, a lot of times, the caption of the figure is actually in the main text of the paper, not in the caption. For example, the captions for Fig. 6 and 7 are in lines 350+ and 359+ respectively. (b) I like it when at the end of the introduction, there's a list of contributions, either in text or as an actual list but in a very clear and condensed form and highlighted as \"contributions\".\n- W.6) 2 minor experimental things: (a) Why did you choose GAIfO? Why is this a valid baseline? I don't see any similarity or comparability. It's not made to explain anything, it's trained differently, and its networks are different. The only similarity is that it's an imitation-learning method (which is not the focus of your method). (b) In the bad behavior example: you have a policy that always outputs \"RIGHT\", right? Then if I understand correctly, your ProtoX can't learn nuances between states because all states have the same action. Therefore the conclusion of \"it predicts that the danger (goomba) is equal to flat ground\" is by definition and a bit pointless, no?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "BZXpFTDIFby", "reviewer": "Reviewer_QULj", "summary": "This paper applies an interpretability method based on discovering prototypical inputs to the problem of interpreting reinforcement learning (RL) policies. The authors create a model that uses a time contrastive loss to encodes states that lead to similar actions and that are similar in time as close to each other in embedding space. They connect this encoding to a behavior cloning model that first applies an isometric transformation to the encoding, then extracts a set of prototypes. The similarity between an input image and the prototypes is computed, and these similarity scores are fed to a fully connected layer that outputs scores for each action. Thus, the model can not only predict which action the agent is likely to take from a given state, but show the most relevant prototype state to explain why it takes that action. Experimental results reveal that the model is highly accurate (although not as accurate as a ResNet + BC baseline), and provides intuitive prototypes that help interpret the behavior of the RL agent.  ", "questions": "Please explain the differences between this work and ProtoPNet [8] and clarify that explanation in the text.\n\nIs the process of merging prototypes manual? This seems time consuming. Is there a way to automate it? \n\nPlease add a reasonable baseline to the sensitivity to flip points experiment. \n\nClarify how time contrastive losses are used in pre-training vs when optimizing Eq. 7. \n\nThe caption in Figure 8 refers to the policy running Mario as \"he\", and makes a statement about what \"he thinks\", which is anthropomorphizing the RL policy and overclaiming. Please fix. \n\nSince GAIfO performed so poorly, improving its implementation or adding a more competitive baseline would improve the paper. \n\n", "limitations": "The authors acknowledge the limitations of their method with respect to manually merging prototypes, and the fact that interpreting prototypes requires knowledge of the task. \n\nThe societal implications of the work are not discussed. Better interpretability for RL actually could have a number of beneficial societal implications, since it would help with deploying RL to safety-critical human environments (e.g. for robots). This would be good to discuss. ", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Significance:** The results of this paper are likely to be significant to the RL community. As an RL researcher, I can confirm that I would find this method useful for interpreting the behavior of a trained agent and debugging why it is failing. Overall the idea for the paper is crisp and compelling. \n\n**Novelty:** The novelty of the paper may be somewhat limited. The authors state that the training procedure is \"essentially identical to that of the related ProtoPNet [8]\" (line 261). I recommend the authors clarify this (including line 240), since ProtoPNet is about image recognition and therefore it seems unlikely that it uses the same type of time contrastive loss. Nevertheless, the time contrastive loss used here is also similar to (but not identical to) that of Time Contrastive Networks [10]. Essentially the novelty seems to be in the combination and application of these existing techniques to the domain of RL. \n\n**Quality:** The experimental evaluation is relatively thorough, uses fairly complex environments, and compares to reasonable baselines. However, the performance of GAIfO is so poor, that it suggests the authors might need to debug their implementation of GAIfO, or choose a more competitive baseline. The authors should also provide some estimate of variance or error bars for their results in Table 1 and Table 3 (as this is part of the reproducibility checklist).\n\nThe \"diagnosing bad behaviors\" experiment is a compelling illustration of the usefulness of this method. \n\nThe sensitivity to flip points experiment was creative and interesting, but there is not much we can conclude from the results unless they include a comparison to some other baseline (such as ResNet-BC). Is ProtoX more sensitive to flip points than other techniques because of its focus on prototypes? If not, this is not a very interesting result. \n\nThe overview of related work is well written and complete, and clearly outlines the distinction between this paper and prior work. \n\n**Clarity:** Overall the paper is well written and clear. The examples provided in Figure 1 do a great job of framing the contributions of the paper and making them obvious. \n\nThere are a couple of clarity issues however. One is the discussion of the time contrastive (TC) loss as only focusing on \"visual similarity\". This happens in the intro (lines 60-68), and again in the method (Section 3.1, line 194). Because the TC loss forces states that have similar actions and occur at similar times to be similar, it is not only focused on visual similarity. In fact it incorporates functional and temporal similarity. So referring to it as visual similarity is confusing and misleading. This leads to some awkwardness in the intro, for example where it concludes \"Consequently, visually similar states are closer together in the latent space\"... the previous sentences have not described why *visually* similar states would be close together at all. \n\nA second clarity issue relates to how the encoder is trained. Section 3.1 and Eq 1 provides an explanation for training the time contrastive encoder. But Section 3.3 appears to use very similar losses, and the differences between how they are applied is not clear. Is the procedure to first pre-train the encoder using the TC losses in Section 3.1, and then continue training the whole network using contrastive losses, but with respect to prototypes, in Section 3.3? Is the encoder frozen after pre-training?\n\nIt would be helpful to explain $a$ and $y_i$ in Eq 5. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept