Explaining RL Decisions with Trajectories
Authors: Shripad Vilasrao Deshmukh
Arpan Dasgupta
Balaji Krishnamurthy
Nan Jiang
Chirag Agarwal
Georgios Theocharous
Jayakumar Subramanian
Venue: ICLR 2023 poster (2023)
Keywords: Explainable RL
Explainable AI
Offline Reinforcement Learning
Trajectory Attribution
Decision-Aware AI

Abstract: Explanation is a key component for the adoption of reinforcement learning (RL) in many real-world decision-making problems. In the literature,  the explanation is often provided by saliency attribution to the features of the RL agent's state. In this work, we propose a complementary approach to these explanations, particularly for offline RL, where we attribute the policy decisions of a trained RL agent to the trajectories encountered by it during training. To do so, we encode trajectories in offline training data individually as well as collectively (encoding a set of trajectories). We then attribute policy decisions to a set of trajectories in this encoded space by estimating the sensitivity of the decision with respect to that set.  Further, we demonstrate the effectiveness of the proposed approach in terms of quality of attributions as well as practical scalability in diverse environments that involve both discrete and continuous state and action spaces such as grid-worlds, video games (Atari) and continuous control (MuJoCo). We also conduct a human study on a simple navigation task to observe how their understanding of the task compares with data attributed for a trained RL policy.

Published as a conference paper at ICLR 2023
EXPLAINING RL DECISIONS WITH TRAJECTORIES
Shripad Deshmukh*) Arpan Dasgupta*?, Balaji Krishnamurthy', Nan Jiang', Chirag Agarwal', Georgios Theocharous", Jayakumar Subramanian'
Media and Data Science Research, Adobe
International Institute of Information Technology Hyderabad UUniversity of Illinois Urbana- Chaaaa-Chaamaiin *Adobe Research
ABSTRACT
Explanation is key component for the adoption of reinforcement learning (RL) in many real-world decision-making problems. In the literature, the explanation is often provided by saliency attribution to the features of the RL agent's state. In this work, we propose a complementary approach to these explanations, par- ticularly for offline RL, where we attribute the policy decisions of a trained RL agent to the trajectories encountered by it during training. To do so, we encode trajectories in offline training data individually as well as collectively (encoding a set of trajectories). We then attribute policy decisions to a set of trajectories in this encoded space by estimating the sensitivity of the decision with respect to that set. Further, we demonstrate the effectiveness of the proposed approach in terms of quality of attributions as well as practical scalability in diverse environments that involve both discrete and continuous state and action spaces such as grid-worlds, video games (Atari) and continuous control (MuJoCo). We also conduct a human study simple navigation task to observe how their understanding of the task compares with data attributed for a trained RL policy.
INTRODUCTION
Reinforcement learning (Sutton & Barto 2018 has enjoyed great popularity and has achieved huge success, especially in the online settings, post advent of the deep reinforcement learning (Mnih et al.] 2013 Schulman et 2017 Silver al. 2017 Haarnoja et al., 2018). Deep RL algorithms are now able to handle high-dimensional observations such as visual inputs with ease. However, using these algorithms in the real world requires i) efficient learning from minimal exploration to avoid catastrophic decisions due to insufficient knowledge of the environment, and ii) being explainable. The first aspect is being studied under offline RL. where the agent trained on collected experience rather than exploring directly in the environment. There is huge body of work on offline RI. (Levine jet al. 2020) Kumar et al. 2020 Yu et al., 2020; Kostrikov et al., 202T). However, more work is needed to address the explainability aspect of RL decision- making.
Previously, researchers have attempted explaining decisions of RL agent by highlighting important features of the agent's ss state (input observation) (Puri et al. 2019 Iyer et al. 2018 Greydanus al.) 2018). While these approaches are useful, we take a complementary route. Instead of identifying salient state-features, learn certain behaviours. We call this approach as trajectory-aware RI. explainability. Such explain- ability confers faith in the decisions suggested by the RI agent in critical scenarios (surgical (Loftus) ct al. 2020), nuclear (Bochnlein et al. (022). etc.) by looking at the trajectories responsible for the decision. While this sort of training data attribution has been shown to be highly effective in su- pervised learning (Nguyen et al. 202T). to the best of our knowledge, this is the first work to study data attribution-basee based explainability in RL. the present work, we restrict ourselves to offline RL setting where the agent is trained completely offline, i.e., without interacting with the environment and later deployed in the environment.
wish to identify the past experiences (trjectories) that led the RL agent to
Email for correspondence: shdes hondeshmuee @amuuadddbbe... con
Work done during summer research intersship Media and Data Science Research, Adobe.

---

Published as a conference paper at ICLR 2023
Contributions of this work are enumerated below:
1. A novel explainability framework for reinforcement learning that aims to find experi- ences(trajectories) that lead an RL agent learn certain behaviour.
2. A solution for trajectory attribution in offline RL setting based on state-of-ther -rrt--rrr sequence modeling techniques. In our solution, we present a methodology that generates a single embedding for a trajectory of states, actions, and rewards, inspired by approaches in Natural Language Processing (NLP). We also extend this method to generate a single encoding of data containing set of trajectories.
3. Analysis of trajectory explanations produced by our technique along with analysis of the trajectory embeddings generated, where we demonstrate how different embedding clusters represent different semantically meaningful behaviours. Additionally, we also conduct study to compare human understanding of RL tasks with trajectories attributed.
This paper is organized follows. In Sec. 2 cover the works related to explainability in RL and the recent developments in offline RL. We then present our trajectory attribution algorithm in Sec. Sec.[] The experiments and results are presented in Sec. 4 We discuss the implications of our work and its potential extensions in the concluding Sec.
BACKGROUND AND RELATED WORK
Explainability in RL. Explainable AI (XAI) refers to the field of machine learning (ML) that focuses developing tools for explaining the decisions of ML models. Explainable RL (XRL) (Puiutta Veith 2020) is a sub-field of XAI that specializes in interpreting behaviours of RL agents. Prior works include approaches that distill the RL policy into simpler models such as de- cision tree (Coppens et al. 2019) or to human understandable high-level decision language (Vermal al. 2018). However, such policy simplification fails to approximate the behavior of complex RL models. In addition, causality-basee to explain an agent's action by identifying the cause behind using counterfactual samples. Fur- ther, saliency-based methods using input feature gradients <Iyer et al. 2018) and perturbations (Puri
approaches (Pawlowski et al. 2020 Madumal et al. (0200) aim
al., 2019: Greydanus et al. 2018) provide state-based explanations that aid humans in understand- ing the agent's 's actions. To the best of our knowledge, for the first time, we explore the direction of explaining an agent's s behaviour by attributing its actions to past encountered trajectories rather than highlighting state features. Also, memory understanding (Koul et al. 2018 Danesh et al. [222]) a relevant direction, where finite state representations for interpretability. However, unlike these works, we focus on sequence embedding generation and avoid using policy networks for actual return optimization.
recurrent policy networks are analysed
Ofline RL. Offline RL (Levine et al., 2020 refers to the RL setting where agent learns from collected experiences and does not have direct access to the environment during training. There several specialized algorithms proposed for offline RL including model-free ones (Kumar al.)
2020 2019) and model-based ones (Kidambi et al. 2020 Yu et al. 2020). In this work, we use algorithms from both these classes train offline RL agents. In addition, recently, the RL problem of maximizing long-term return has been cast as taking the best possible action given the sequence of past interactions in terms of states, actions, rewards (Chen al. 202T; Janner et al., 2021 Reed let al., [222]] Park et al., 2018). Such sequence modelling approaches to RL, especially the ones based on transformer architecture (Vaswani al., 2017). have produced state-of-the-art results in various offline RL benchmarks, and offer rich latent representations to work with. However, little to no work has been done in the direction of understanding these sequence representations and their applications. In this work, we base our solution on transformer-based sequence modelling approaches to leverage their high efficiency in capturing the policy and environment dynamics of the offline RL systems. Previously, researchers in group-driven RL (Zhu al. 2018) have employed raw state-reward vectors as trajectory representations. We believe transformer based embeddings, given their proven capabilities, would serve better representations than state-reward vectors.
2

---

Published as a conference paper at ICLR 2023
&. Trajestory Encoding
Training Explanation Policis
Original Data
Dota wie
Data w/o
wit
Trajectory (r) from Offinn: Oata
Trajectory Lmbrdding
Sequence Encoder
Trajectory Custering
AL Aipo
Alpp
KLAbO
Norlg
7
2
π m
Trajectory Clutter Attribution
Comrsponding Clusters
Trajectory
Clastering Algoritim
Trajetory duaten measing
Data Embedding
on
Torig (s,a)
C
select polity which suggests difee ent action, with correspording complementtry clatter embedding ciose to or origina
Givon Embesdiing
Pormutationally Invoriant Set Encoding
Representathve Embedddnng Glien
Trained on Data w/on w/e
Figure I: Trajectory Attribution in Offline RL. First, we encode trajectories in offline data using
sequence encoders and then cluster the trajectories using these encodings. Also, we generate single embedding for the data. Next, train explanation policies variants of the original dataset and compute corresponding data embeddings. Finally. we attribute decisions of RL agents trained on entire data to trajectory clusters using action and data embedding distances.
3 TRAJECTORY ATTRIBUTION
Preliminaries. We denote the offline RL dataset using that comprises set of n, trajectories. Each trajectory, denoted by Tj comprises of sequence of observation (ox), action (ag) and per-step reward (rx) tuples with & ranging from to length the trajectory T₁. We begin by training an offline RL agent on this data using any standard offline RL algorithm from the literature.
Algorithm. Having obtained the learned policy using an offline algorithm, our objective now to attribute this policy, i.e., the action chosen by this policy, at a given state to set of trajectories. We intend to achieve this in the following way. We want to find the smallest set of trajectories, the absence of which from the training data leads different behavior the state under consideration. That is, we posit that this set of trajectories contains specific behaviors and respective feedback from the environment that trains the RL agent make decisions in certain manner. This identified set of trajectories would then be provided as attribution for the original decision.
While this basic idea is straightforward and intuitive, it is computationally feasible beyond few small RL problems with discrete state and action spaces. The key requirement to scale this approach group the trajectories into clusters which
large, continuous state and action space problems, can then be used to analyze their role in the decision-making of the RL agent. In this work, we propose to cluster the trajectories using trajectory embeddings produced with the help of state-of- the-art sequence modeling approaches.
Figure gives an overview of our proposed approach involving five steps: (i) Trajectory encoding, (ii) Trajectory clustering, (iii) Data embedding, (iv) Training explanation policies, and (v) Cluster attribution, each of which explained in the sequel.
(i) Trajectory Encoding. First, we tokenize the trajectories in the offline data according to the specifications of the sequence encoder used (e.g. decision transformer/trajcttrry transformer). The observation, action and reward tokens of trajectory are then fed to the sequence encoder to produce corresponding latent representations, which we refer to as output tokens. We define the trajectory embedding average of these output tokens. This technique is inspired by average-pooling tech- niques (Choi et al, 2021 Briggs 202I) NLP used to create sentence embedding from embeddings of words present in it. (Refer Algorithm
3

---

Published as a conference paper at ICLR 2023
Algorithm 1: Trajetooresss
/* Encoći ng given set of trajeccciies individually
Input : :fffine Data { Sequence Encoder
Initialize: Initialize array I to collect the trajectory embeddings
for Tj in { do
Using E, get output tokens for all the o,
73
+/
far + (1, 0. (T.) where 3T
#input tokens
Take mean outputs to generate Ty's embedding 1
L + (eo1,) + Cal.j + eni + Cog,s + eag. + tra, +... eor + eor, + (,,3)((3T)
Append 1j to
Output Return the trajectory embeddings = {4.)
(ii) Trajectory Clustering. Having obtained trajectory embeddings, we cluster them using X-Means clustering algorithm (Pelleg al. 2000) with implementation provided by Novikov (209)). While in principle, any suitable clustering algorithm can be used here, we chose X-Means as it a simple extension to the K-means clustering algorithm (Lloyd, [182); determines the number of clusters ne automatically. This enables us to identify all possible patterns in the trajectories without forcing ne as hyperparameter (Refer Algorithm 2}
Algorithm 2: clusterTrajectorie.
/+ Clustering the trajectories using their embeddings
Input : Trajectoory embeddings = fti), clusteringAlgo
C + clusteringAlgo(() // Cluster usi igg provided clustering algorithm
Output Return trajectory clusters C 1ic
(iii) Data Embedding. We need a way to identify the least change in the original data that leads the change in behavior of the RL agent. To achieve this, we propose a representation for data comprising the collection of trajectories. The representation has to be agnostic to the order in which trajectories are present in the collection. So, we follow the set-encoding procedure prescribed in (Za- heer et al. (017)) where we first sum the embeddings of the trajectories in the collection, normalize this sum by division with a constant and further apply a non-linearity, in our case, simply, softmax over the feature dimension to generate single data embedding (Refer to Algorithm 3
We use this technique to generate data embeddings for ne sets of trajectories. The first set represents the entire training data whose embedding is denoted by duiig The remaining n. sets are constructed as follows. For each trajectory cluster Cj. we construct a set with the entire training data but the trajectories contained in cj. We call this set the complementary data set corresponding to cluster ej and the corresponding data embedding as the complementary data embedding d,
Algorithm 3: generaateDaaaEmmeedidd
/+ Generating data embedding for a given set of trajecttriis Input : Trajectory embeddings {4). Normalizing factor M, Softmax temperature Ton
3 E.4 // Sum the trajectory embeddings and normalize them exp(s;/Tan)
M
d + (dyld, Σe expl*s/Tanl (umn)) ) Take softmax al along feature dimension
Output Return data embedding d
(iv) Training Explanation Policies. In this step, for each cluster Cj using its complementary data set, train offline RL agent. We ensure that all the training conditions (algorithm, weight initialization, optimizers, hyperparameters, etc.) are identical to the training of the original RL policy, except for the modification in the training data. We call this newly learned policy as the explanation policy corresponding to cluster Cy. We thus get n. explanation policies at the end of this step. In addition, compute data embeddings for complementary data sets (Refer to Algorithm 4
(v) Cluster Attribution. In this final step, given a state, we note the actions suggested by all the explanation policies at this state. We then compute the distances of these actions (where
4

---

Published as a conference paper at ICLR 2023
Algorithm 4: trainExxPPoliiess
/* Train exp1 anation policies compute related data embeddings
Input : Offline )ata(r:). Traj. Embeddings T, Traj. Clusters C, offlineRLAlg
for cy in do
( // Compute comp Lompeenen dataset corresp. to Cj T; F /T Gather corresp. 7 embeds
Explanation policy, и, f- +=offineRIAlgo((())))
Complementary data embedding, dj &- gennerateDataaEmbedddinnig M, Twa)
Output Explanation policies {mT Complementary data embeddings {d,}
assume metric over the action space) from the action suggested by the original RL agent at the state. The explanation policies corresponding to the maximum of these distances form the candidate attribution set. For each policy in this candidate attribution set, we compute the distance between its respective complementary data embedding and the data embedding of the entire training data using the Wasserstein metric for capturing distances hetween softmax simplices (Vallender) 1974). We then select the policy that has the smallest data distance and attribute the decision of the RI. agent to the cluster corresponding to this policy(Refer to Algorithm 5). Our approach comprised of all five steps is summarized in Algorithm
Algorithm 5: generaaeCCussterAttbbbiio
x Gencrating cluster attributions for Ocrig = Woria(s)
Input State Original Policy Wadgg Explanation Policies (My Original Data Embedding daig. Complementary Data Embeddings {dz}
Original action, Gorig f- [s)g(()
Actions suggested by explanation policies, a, + (s)
damina; calcAActionDistaaneeaadd a,)// Compute action distance
K t argmax(da.ggh) /// Get candi date clusters using argmax
Wh + Wasa (ongg dx)// Compute Wasserstein distance b/w complementary data
embedcings of candidate clusters & orig data embedding
Cimal argmin((xx)// /// Choose cluster with min data embedding dist.
Output : Ciinal
EXPERIMENTS AND RESULTS
Next, we present experimental results to show the effectiveness of our approach in generating trajec- tory explanations. We address the following key questions: Q1) Do we generate reliable trajectory explanations? (Sec. (4.) Q2) How does a human understanding of an environment align with trajec-
tories attributed by our algorithm and what the scope of data attribution techniques? (Sec. 4.3
4.1 EXPERIMENTAL SETUP
We first describe the environments, models, and metries designed study the reliability of our trajectory explanations.
RL Environments. We perform experiments on three environments: i) Grid-world (Figure which has discrete state and action spaces, ii) Seaquest from Atari suite which has environments with con- tinuous visual observations and discrete action spaces (Bellemare et al. 2013). and iii) HalfChee- tah from MuJoCo environments which are control environments with continuous state and action spaces Tooooov et al. 2012).
Offline Data and Sequence Encoders. For grid-world, collect offline data of 60 trajectories from policy rollout of other RL agents and train an LSTM-baser trajectory encoder following the procedure described in trajectory transformer, replacing the transformer with LSTM. For Seaquest, we collect offline data of 717 trajectories from the D4RL-Atari repository and use a pree-triiedd decision transformer trajectory encoder. Similarly, for HalfCheetah, collect offfine data of
5

---

Published as a conference paper at ICLR 2023
1000 trajectories from the D4RL repository (Fu al. 2020 and use a pre-trained trajectory trans- former as a trajectory encoder. To cluster high-level skills in long trajectory sequences, we divide the Seaquest trajectories into 30-length sub-trajectories and the HalfCheetah trajectories into 25- length sub-trajectorris. These choices were made based on the transformers' input block sizes and the quality of clustering.
Offline RL Training and Data Embedding. We train the offline RL agents for each environment using the data collected follows for grid-world, we use model-based offline RL, and for Seaquest and HalfCheetah, we employ DiscreteSAC (Christodoulou 2019) and SAC (Haarnoja et al. 2018).
respectively, using d3rlpy implementations (Takuma Seno 2021b. We compute data embedding of
entire training data for each of the environments. See Appendix A.33 for additional training details.
Encoding of Trajectories and Clustering. We encode the trajectory data using sequence encoders and cluster the output trajectory embeddings using the X-means algorithm. More specifically, we obtain 10 trajectory clusters for grid-world, 8 for Seaquest, and 10 for HalfCheetah. These clusters represent meaningful high-level behaviors such as 'falling into the lava', "filling in oxygen', 'taking long forward strides', etc. This discussed in greater detail in Section A.44
Complementary Data Sets. We obtain complementary data sets using the aforementioned cluster information and provide complementary data sets for grid-world, 8 for Seaquest, and 10 for HalfCheetah. Next, we compute data embeddings corresponding to these newly formed data sets.
Explanation Policies. Subsequently, we train explanation policies on the complementary data sets for each enviromment. The training produces 10 additional policies for grid-world, 8 policies for Seaquest, and 10 policies for HalfCheetah. In summary, we train the original policy on the entire data, obtain data embedding for the entire data, cluster the trajectories and obtain their explanation policies and complementary data embeddings.
Trajectory Attribution. Finally, attribute decision made by the original policy for given state to a trajectory cluster. We choose top-3 trajectories from these attributed clusters by matching the context for the state-action under consideration with trajectories in the cluster in our experiments.
Evaluation Metrics. We compare policies trained on different data using three metrics (deter- ministic nature of policies is assumed throughout the discussion)- - 1) Initial State Value Esti- mate denoted by (so)) which is measure of expected long-term returns to evaluate offline RL training as described in Paine al. (2020), 2) Local Mean Absolute Action-Value Difference: defined as E(IAQ=ml)) = B[ (Toiig
(Tarig (s)) Quee (T))))) that measures how original policy perceives suggestions given by explanation policies, and 3) Action Contrast Measure: a measure of difference in actions suggested by explanation policies and the original action. Here, we use ((I((Trigs)) (s) A πy (s)) for discrete action space and [E((Tarig( - म. (s))²) for continuous action space. Further, we compute distances between embeddings of original and complementary data sets using Wasserstein metric: Wass(dongg d,). later normalized to [0, 1]. Finally, the cluster attribution frequency is measured using metric P(Cnnal = =.)
4.2 TRAJECTORY ATTRIBUTION RESULTS
Qualitative Results. Figure 2 depicts a grid-world state (1, 1), the corresponding decision by the trained offline RL agent 'right', and attribution trajectories explaining the decision. As we can observe, the decision is influenced not only by trajectory (traj.-i) that goes through (1, 1) but also by other distant trajectories(trajs-ii,, iii). These examples demonstrate that distant experiences (e.g. traj.-iii) could significantly influence the RL agent's decisions, deeming trajectory attribution an essential component of future XRL techniques. Further, Figure 3 shows the Seaquest agent (submarine) suggesting action 'left' for the given observation in the context of the past few frames. The corresponding attributed trajectories provide insights into how the submarine aligns itself to target enemies coming from the left. Figure [00 shows HalfCheetah observation, the agent suggested action in terms of hinge torques and corresponding attributed trajectories showing runs that influence the suggested set of torques. This is an interesting use-case of trajectory attribution it explains complicated torques, understood mainly by the domain experts, in terms of the simple semantic intent of "getting up from the floor'.
Quantitative Analysis. Tables I 2 and 3 present quantitative analysis of the proposed trajectory attribution. The initial state value estimate for the original policy Worig matches exceeds estimates

---

Published as a conference paper at ICLR 2023
State:
Action: RIGHT
Trajectory (i)
Trajectory
Trajectory (iii)
Figure Grid-world Trajectory Attribution. RL agent suggests taking action 'right' in grid cell (1,1). This action is attributed to trajectories (i). (ii) and (iii) (We denote gridworld trajectory by annotated A.V,D,< arrows for 'up'. 'down', 'right'. 'left' actions respectively, along with the time-step associated with the actions (0-indexed)). We can observe that the RL decisions could be influenced by trajectories distant from the state under consideration, and therefore attributing decisions trajectories becomes important understand the decision better.
Trajectory II
Content
URIFT
DOWARIGHTTRE
BOVARGITTEE
THE
TBE
UPLEET
Trijeetory (i)
Irajectory (i)
Observation
RIGHT
DOWNRIGHT
DOWNLEFT
DOWNLEFT
DOWNLEFT
LEFT
Figure 3: Seaquest Trajectory Attribution. The agent (submarine) decides to take 'left' for the given observation under the provided context. Top-3 attributed trajectories shown the right (for each training data traj., we show 6 sampled observations and the corresponding actions). As depicted in the attributed trajectories, the action 'left' is explained in terms of the agent aligning itself to face the enemies coming from the left end of the frame.
for explanation policies trained on different complementary data sets in three environment set- tings. This indicates that the original policy, having access to all behaviours, is able outperform other policies that are trained on data lacking information about important behaviours (e.g. grid- world: reaching a distant goal, Seaquest: fighting in the top-right corner, HalfCheetah: stabilizing the frame while taking strides). Furthermore, local mean absolute action-value difference and action differences turn
be highly correlated (Tab. and 3) i.e., the explanation policies that suggest the most contrasting actions are usually perceived by the original policy as low-return actions. This evidence supports the proposed trajectory algorithm as want to identify the behaviours which when removed make agent choose actions that are not considered suitable originally. In addition, we provide the distances between the data embeddings in the penultimate column. The cluster attri- dependent bution distribution is represented in the last column which depicts how RL decisions on various behaviour clusters. Interestingly, in the case world, we found that only a few clus- ters containing information about reaching goals and avoiding lava had the most significant effect on the original RL policy.
We conduct two additional analyses 1) trajectory attribution Seaquest trained using Discrete BCQ (Sec. A.7)) and 2) Breakout trajectory attribution (Sec. A.8). In the first one, we find a sim- ilar influence of clusters across the decision-making of policies trained under different algorithms.

---

Published as a conference paper at ICLR 2023
Table 1: Quantitative Analysis of Grid-world Trajectory Attribution. The analysis is provided using 5 metrics. Higher the (s((s))), better is the trained policy. High B(l ()) along with high [B(1 (Targ F (s)) is desirable. The policies with lower Waa(d, d d,) and high action con- trast are given priority while attribution. The cluster attribution distribution given in the final column.
(so)) B(IQQ mu ()) B(1 (Torig (8) # xT(s)) Wais (d, dz) P(Cinal
orig
0.3061
0.3055
0.0012
0.0409
1.0000
0,0000
0.3053
0.0016
0.0409
0.0163
0.0000
0.3049
0.0289
0.1429
0.0034
0.0000
0.2857 4
0.0710
0,1021
0.0111
0.3750
0.2987
0.0322
0.1429
0.0042
0.1250
0.3057
0.0393
0.0409
0.0058
0.0000
0.3046
0.0203
0.1225
0.0005
0.5000
0.3055
0.0120
0.0205
0.0006
0.0000
8
0.3057
0.0008
0.0205
0.0026
0.0000
0.3046
0.0234
0.1429
0.1745
0.0000
Table 2: Quantitative Analysis of Seaquest Trajectory Attribution. The analysis is provided us- ing 5 metrics. Higher the (sV(s))). better is the trained policy. High TE( TE[ along with high E( oToig + π. (s)) is desirable. The policies with lower Wais(d, d,) and high action contrast are given priority while attribution. The cluster attribution distribution is given in the final column.
[B())s))) B( AQ Nww ()) B(1 (s) + TT(+)) Wain d,) P(Canal Cj)
orig
85.9977
50.9399 0
1.5839
0.9249
0.4765
0.1129
57.5608
1.6352
0.8976
0,9513
0.0484
66.7369
1.5786
0,9233
1.0000
0,0403
3.0056
1.9439
0,9395
0.8999
0.0323
4
58. 1854
1.5813
0.8992
0.5532
0.0968
5
87.3034
1.6026
0,9254
0.2011
0.3145
70.8994
1.5501
0.9238
0.6952
0,0968
89.1832
1.5628
0.9249
0.3090
0.2581
Secondly, in the breakout results, we find clusters with high-level meanings of 'depletion of a life' and 'right corner shot' influence decision-making lot. This is insightful the two behaviors highly critical in taking action, one avoids the premature end of the game, and the other is part of the tunneling strategy previously found in Greydanus al. 20188.
4.3 QUANTIFYING UTILITY OF THE TRAJECTORY ATTRIBUTION: A HUMAN STUDY
One of the key desiderata of explanations is to provide useful relevant information about the be- haviour of complex AI models, To this end, prior works other domains like vision (Goyal al. 2019) and language (Liu et al., 2019) have conducted human studies to quantify the usefulness of output explanations. Similarly. having established a straightforward attribution technique, wish to analyze the utility of the generated attributions and their scope in the real world through a human study. Interestingly, humans possess an explicit understanding of RL gaming environments and can reason about actions at a given state to satisfactory extent. I everaging this, we pilot human study with 10 participants who had a complete understanding of the grid-world navigation environment to quantify the alignment between human knowledge of the environment dynamics with actual factors influencing RL decision-making
Study setup. For the study, design two tasks: i) participants need to choose trajectory that they think best explains the action suggested in a grid cell, and ii) participants need to identify all relevant trajectories that explain action suggested by RL agent. For instance, in Fig, 4a| show one instance of the task where the agent is located at (1, 1) and taking 'right' and a subset of attributed trajectories for this agent action. In both tasks, in addition to attributions proposed by
8

---

Published as a conference paper at ICLR 2023
technique, add i) a randomly selected trajectory explanation and ii) a trajectory selected from a cluster different from the one attributed by our approach. These additional trajectories aid in identifying human bias toward certain trajectories while understanding the agents' behavior.
Results. On average, across three studies in Task 1, we found that 70% of the time, human partic- ipants chose trajectories attributed by proposed method the best explanation for the agent's action. On average, across three studies Task 2, nine participants chose the trajectories generated by our algorithm (Attr Traj 2). In Fig. 4b observe good alignment between human's 's under- standing of trajectories influencing decision-making involved in grid navigation. Interestingly, the results also demonstrate that not all trajectories generated by our algorithm are considered relevant by humans, and often they are considered as good as random trajectory (Fig. 4b; Attr Traj 1).
In all, per the Task 1 results, on average 30% of the time, humans fail to correctly identify the factors influencing an RL decision. Additionally, actual factors driving actions could be neglected by humans while understanding a decision as per Task 2 results. These findings highlight the neces- sity to have data cntribution-bbsee explainability tools to build trust among human stakeholders for handing over the decision-making to RL agents near future.
State: (1. 1) Action: RIGHT
Attributed Trajectories
Number of Humars
(a)
(b)
Figure 4: Column (a): An example of the human study experiment where identify the attributed trajectories that best explain the state-action behavior of the agent. Column (b): Results from our human study experiments show decent alignment of human knowledge of navigation task with actual factors influencing RL decision decision-making. This underlines the utility as well as the scope of the proposed trajectory attribution explanation method.
required to
5 DISCUSSION
In this work, proposed novel explanation technique that attributes decisions suggested by an RL agent to trajectories encountered by the agent in the past. We provided an algorithm that enables perform trajectory attribution in offline RI. The key idea behind approach trajectories using sequence modelling techniques, cluster the trajectories using these embeddings and then study the sensitivity the original RL agent's policy the trajectories present in each of these clusters. We demonstrated the utility of our method using experiments in grid-world, Seaquest and HalfCheetah environments. In the end, we also presented human evaluation of the results of our method to underline the necessity of trajectory attribution.
encode
The ideas presented in this paper, such generating trajectory embedding using sequence encoders, and creating encoding of the set of trajectories, can be extended to other domains of RI.. For in- stance, the trajectory embeddings could be beneficial recognizing hierarchical behaviour patterns studied under options theory (Sutton et al.) 1999). Likewise, the data encoding could prove help- ful in studying transfer learning (Zhu et al. 2020) in RL by utilizing the data embeddings from both the source and target decision-man makinn tasks. From the XRL point of view, wish extend our work to online RL settings where the agent constantly collects experiences from the environment.
While results highly encouraging, one of the limitations of this work is that there are established evaluation benchmarks as well as no other works to compare with. We believe that more extensive human studies will help address this.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We thank anonymous reviewers for their helpful feedback to make this work better. Moreover, NJ acknowledges funding support from NSF 11S-1112471 and NSF CAREER IIS-214178I.
Finally, we wish to dedicate this work to the memory of our dear colleague Georgios Theocharous who is not with us anymore. While his premature demise has left an unfillable void, his work has made an indelible mark in the domain of reinforcement learning and in the lives of many researchers. He will forever remain in our memories.
REFERENCES
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
Amber Boehnlein, Markus Diefenthaler, Nobuo Sato, Malachi Schram, Veronique Ziegler, Cristiano Fanelli, Morten Hjorth-Jensen, Tanja Horn, Michelle P. Kuchera, Dean Lee, Witold Nazarewicz, Peter Ostroumov, Kostas Orginos, Alan Poon, Xin-Nian Wang. Alexander Scheinker, Michael S. Smith, and Long-Gang Pang. icolloquium/i Machine learning in nuclear physics. Reviews of Modern Physics, 94(3), sep 2022. doi: 10. 1033/eemoodhys.44..33 94.03300.....003.... URL https:///oo. 94. 031003]
org 100.11103322eewmodoooo
James Briggs. How get sentence embedding using bert?, Oct 2021. URL https://
datascience st www.sstttcccacsteeee com/ wwccc/////ccce a/103569
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106.0/345, 2021.
Hyunjin Choi, Judong Kim. Seongho Joe, and Youngjune Gwon. Evaluation of bert and albert sentence embedding performance on downstream nlp tasks. In 2020 25th International conference on pattern recognition (ICPR), pp. 5482-5487. IEEE, 2021.
Petros Christodoulou. Soft actor-critic for discrete action settings. arXiv preprint 2019.
Youri Coppens, Kyriakos Efthymiadis, Tom Lenaerts, Ann Nowé, Tim Miller, Rosina Weber, and Daniele Magazzeni. Distilling deep reinforcement learning policies in soft decision trees. In Proceedings of the IJCAI 2019 workshop on explainable artificial intelligence, pp. 1-6, 2019.
Mohamad H Danesh, Anurag Koul, Alan Fern, and Saeed Khorram Re-understanding finite-state representations of recurrent policy networks. In International Conference on Machine Learning. pp. 2388-2397. PMLR, 2021.
Justin Fu. Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau. Benchmarking batch deep reinforcement learning algorithms. arXiv preprint arXiv:1910.011 2019a.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 8052-2062 PMLR, 2019b.
Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. In International Conference on Machine Learning. PMLR, 2019.
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. In International conference on machine learning, pp. 1792-1801. PMLR, 2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- 2018
cations. arXiv preprint arXiv:
10

---

Published as a conference paper at ICLR 2023
Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia Sycara. Transparency and explanation in deep reinforcement learning neural networks. In Proceedings of the 2018 AAAI/ACM Conference AI, Ethics, and Society, pp. 144-150, 2018.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021.
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-14, 1998.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-21822 2020.
Iya Kostrikov, Rob Fergus, Jonathan Tompson, and Otir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 9774-5783. PMLR, 2021.
Anurag Koul, Sam Greydanus, and Alan Fern. Learning finite state representations of recurrent policy networks. arXiv preprint 11X1v: 2018.
Aviral Kumar, Justin Fu, G. Tucker, and Sergey Levine. Stabilizing off off-policy q-learning via boot- strapping error reduction. In NeurlPS, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 1191, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. CoRR, abs/2005.01643, 2020. URL
Hui Liu. Qingyu Yin, and William Yang Wang. Towards explainable NLP: A generative explanation framework for text classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5570-5581, Florence, Italy, July 2019. Association for Com- putational Linguistics. doi: 10.18653/v1/P11 1560. URL https: ptrtpss//aclannthl...o orgg
P19- P19-1560
Stuart Lloyd. Least squares quantization in pem. IEEE transactions on information theory, 28(2): 129-137, 1982.
Tyler J Loftus, Amanda C Filiberto, Yanjun Li, Jeremy Balch, Allyson C Cook, Patrick Tighe, Philip Efron, Gilbert R Upchurch Jr. Parisa Rashidi, Xiaolin Li, et al. Decision analysis and decision-making making. Surgery, 168(2):253-266, 2020.
reinforcement learning in surgical decision
Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learn- ing through causal lens. In Proceedings of the AAAI conference on artificial intelligence, vol- ume 34, pp. 2493-2500, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Giang Nguyen, Daeyoung Kim, and Anh Nguyen. The effectiveness of feature attribution methods and its correlation with automatic evaluation scores. Advances in Neural Information Processing Systems, 34:26422-26436, 2021.
Andrei Novikov. PyClustering: Data mining library. Journal of Open Source Software, 4(36): apr 2019. doi: 10.21105/joss.01220 012300 URL
01130
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna. Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.0905: 2020.

---

Published as a conference paper at ICLR 2023
Seong Hyeon Park, BycongDo Kim, Chang Mook Kang, Chung Choo Chung, and Jun Won Choi. sesuencc-to-sequeeee prediction of vehicle trajectory via Istm encoder-decoder architecture. In 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1672-1678. IEEE, 2018.
Nick Pawlowski, Daniel Coelho Castro, and Ben Glocker. Deep structural causal models for tractable counterfactual inference. Advances in Neurul Information Processing Systems, 33:857- 869, 2020.
Dan Pelleg, Andrew W Moore, et al. X-means: Extending k-means with efficient estimation of the number of clusters. In leml, volume I, pp. 727-734, 2000.
Erika Puiutta and Eric Veith. Explainable reinforcement learning: A survey. In International cross- domain conference for machine learning and knowledge extraction, pp. 77-95. Springer, 2020.
Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishna- murthy, and Sameer Singh. Explain your move: Understanding agent actions using specific and relevant feature attribution. arXiv preprint 1rX1v:1912.121111,000 2019.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley Sons, 2014.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.0611 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXivv 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):35-339,, 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181- 211, 1999.
Michita Imai Takuma Seno. d3rlpy: An offlin deep reinforcement library. In NeurlPS 2021 Offline Reinforcement Learning Workshop, December 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi: 10. 201.://00000002..366666666
SS Vallender. Calculation of the wasserstein distance between probability distributions the line. Theory of Probability & Its Applications, 18(4):784-786, 1974.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz. Kaiser, and Illia Polosukhin. Attention you need. Advances in neural informa- tion processing systems, 30, 2017.
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning, pp. 5045-5054. PMLR, 2018.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14142, 2020.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander J. Smola. Deep sets. CoRR, abs/1703.06.. 14, 2017. URL htttt://arxiv.ooo
abs/ 1703. 06114
12

---

Published as a conference paper at ICLR 2023
Feiyun Zhu, Jun Guo, Zheng Xu, Peng Liao, Liu Yang, and Junzhou Huang. Group-driven reinforce- ment learning for personalized mhealth intervention. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 590-598. Springer. 2018.
Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. arXiv preprint arXiv:2009.0788 07888, 2020.
13

---

Published as a conference paper at ICLR 2023
A
APPENDIX
A.1 OVERVIEW OF PROPOSED TRAJECTORY ATTRIBUTION
The following an overview of our proposed 5-step trajectory attribution algorithm.
Algorithm 6: Trajectory Attribution in Offline RL.
Input Offline Data {T;), J. States needing explanation Sexp. Sequence Encoder E, offlineRLAlgo, clusteringAlgo, Normalizing constant M, Softmax Temperature soft
/* Train original offline RL policy
Torig f- offlineRLAAgo((7)))
/x Encode individual trajectories
encoodeTrajectories(((}} E) Algo.
Cluster the trajectories
C + clusterTrajectoriess// clusteringAlgo) // Algo.
Compute data embedding for the entire dataset
doig gemeriieDataEnmeeddigg, s // Algo. 3
Generate explanation ion policies and their corresponding
compl complementary data embeddings
trainExpPolicies( T. C, offlineRAAlo)/// Algo. {m,}, {dz} +
Attributing policy decisions for given set of states for s E Sap do
Ctinal generateClusterrAtrbbuuiio Torig: { dong: d/)// Algo. *Optionally, select top N trajectories in the cluster Crinal using pre-defined criteria.
A.2 GRID-WORLD ENVIRONMENT DETAILS
Agent
Action:
4
Goal
Lava
Wall
Figure 5: Overview of the Grid-world Environment. The aim of the agent is to reach any of the goal states (green squares) by avoiding lava (red square) and going around the impenetrable walls (grey squares). The reward for reaching the goal is +1; if the agent falls into the lava, it is -1. For any other transitions, the agent receives -0.1. I. The agent is allowed to take up, down, left right as the action.
A.3 ADDTIONAL. TRAINING DETAILS
1. Seaquest Atari Environment We employed Discrete SAC to train the original policy along with explanation policies, where the training was performed until saturation in the performance. We used the critic learning rate of x 10-4 and the actor learning rate of 3 x 10-4 with a batch size of 256. The trainings were performed parallelly on a single Nvidia-A 100 GPU hardware.
2. HalfCheetah MuJoCo Environment- - We used SAC train the original policy well as explanation policies where we trained the agents until training performance saturated. We again used the critic learning rate of 3 x 10 and the actor learning rate of 3 10 with batch size of 512. The policy trainings were performed parallelly on a single Nvidia-A100 GPU.
14

---

Published as a conference paper at ICLR 2023
A.4 CLUSTERING ANALYSIS
20
-26
-04
0.8
8.2
0.4
66
-40
-32
-20
-20
28
(a) Grid-world
(b) Seaquest
(c) HalfCheetah
Figure 6: PCA Plot depicting Clusters of Trajectory Embeddings for a) Grid-world, b) Seaquest, and c) HalfCheetah. We find that these clusters represent semantically meaningful high-level be- haviours.
We observe that the trajectory embeddings obtained from the sequence encoders when clustered together demonstrate characteristic high-level behaviours. For instance, in the case of grid-world (Refer 7. the clusters comprise semantically similar trajectories where the agent demonstrates be- haviours such as 'falling into the lava', "achieving the goal in the first quadrant', 'mid-grid journey to the goal', etc. For Seaquest 8 we obtain trajectory clusters that represent high-level behaviours such 'filling in oxygen', 'fighting along the surface', 'submarine bursting due collision with enemy', etc. and for HalfCheetah in Fig. S we obtain trajectory clusters that represent high-level actions such "taking long forward strides "jumping hind leg', 'rumning with head down'. etc.
Although these results look quite promising, in this work we mainly focus on trajectory attribu- tion that leverages these findings. In future, wish analyse the trajectory embeddings and the behaviour patterns in greater detail.
XJ
Achieving Goal in Top right corner Cluster 1
Mid- grid joumey goal - Cluster 3
Falling into lava Cluster 9
Figure 7: Cluster Behaviours for Grid-world. The figure shows 3 example high-level behaviours along with the action description and id of the cluster representing such behaviour.
15

---

Published as a conference paper at ICLR 2023
Filling Oxygen Cluster
Submarine Bursts Cluster
1100
1168
1160
Fighting with Out Cluster
Figure 8: High-level Behaviours found in elusters for Seaquest formed using trajectory embed- dings produced using decision transformer, The figure shows 3 example high-level behaviours along with the action description and id of the cluster representing such behaviour.
Jumping Hind Leg Cluster
Running with Head Down Cluster
Forward Stride Cluster
Figure 9: High-level Behaviours found in clusters for HalfCheetah formed using trajectory beddings produced using trajectory transformer. The figure shows 3 example high-level behaviours along with the action description and id of the cluster representing such behaviour.
16

---

Published as a conference paper at ICLR 2023
A.5 HALFCHEETAH TRAJECTORY ATTRIBUTION RESULTS
Due space constraints, we present the qualitative and quantitative results for the HalfCheetah environment here.
Trajectory (iii) Trajectory (ii) Trajectory (i)
Observation
Action Torques
bthigh: 0.9477, bshin: 0.9183,
bloot: -0.8425, fthigh: -0.9621,
fshin: -0.9704, ffoot:-0.0850
HalfCheetah Trajectory Attribution. The figure shows agent suggesting torques on different hinges for current position of the cheetah frame. The decision is influenced by the runs of cheetah shown on the right (we show 5 sampled frames for one trajectory). The attributed trajecto- ries explain the torques in terms of the cheetah getting up from the floor. (Context is not shown here because unlike Seaquest environment, HalfCheetah decisions are made directly on a given observa- tion (Puterman, 2014 Kaelbling et 1998).
Figure 10:
Table 3: Quantitative Analysis of HallCheetah Trajectory Attribution. The analysis is provided
using 5 metries. Higher the (VV(s))), better is the trained policy. High B(IAQ+M,))) I)) along with high IE( ((Forrg (s) , (+)) is desirable. The policies with lower Wast d,) and high action trast are given priority while attribution. The cluster attribution distribution is given in the final column.
Performance Metrics
π
((V(s)))
())
(TTari (8)
Fz (*)) Wais dz) PP(Cinal = Cg)
orig
131.5449
0
127.1652
0.5667
0.6359
0.2822
0.0143
1
118.5663
0.4796
0.5633
0.0396
0.0214
2
122.0661
0.6904
0.9366
0.0396
0.1464
3
133.4590
0.5360
0.6611
0.0396
0.1250
118.3447
0.5622
0.6194
1.0000
0.0964
5
138.7517
0.6439
0,8262
0.0316
0.0893
6
120.7088
0.4740
0.4803
0.8813
0.0214
7
135.6848
0.5154
0.5489
0.0394
0.2036
8
113,3490
0.7826
1.0528
0.7067
0.1214
83.6211
0.9702
1.3453
0.0264
0.1607
17

---

Published as a conference paper at ICLR 2023
A.6 ADDITIONAL ATTRIBUTION RESULTS
Context
Observation
State:(3, (3, Action: DOWN
LEFT
Attributed Trajectory
Attributed Trajectory
URLEFT
DOMMRIGHTFIRE DOWNRGGHFRRE
FIRE
FIRE
UNLEFT
Obsenvaion
Attributed Trajectory
(a)
(c)
Action Torques
bthigh: 0,9477, bohin: 0.9183, bloot: ftrigh:
Figure 11: Additional Trajectory Attribution Results. Here we show randomly chosen trajectory
from the top-3 attributed trajectories.a) Grid-world agent suggests taking 'DOWN' in cell (3,5) due to the attributed trajectory leading to lava. b) Seaquest agent suggests taking 'LEFT" and the corresponding attributed trajectory. c) HalfCheetah agent suggests particular set of torques and the run found responsible for the same is shown side-by-side
A.7 TRAIECTORY ATTRIBUTION ACROSS ALGORITHMS
Context
Observation
UPLEFTFIRE
Attributed Trajectory
RIGHT
UPLEFTFIRE
UPLEFTFIRE
DOWNFIRE
DOWNFIRE
Figure 12: Trajectory Attribution for DiscreteBCQ-trraine
Seaquest Agent. The action 'UP-
LEFTFIRE' is explained by our algorithm in terms of corresponding attributed trajectory. The action helps align agent to face enemies from left (akin to Fig. 3
18

---

Published as a conference paper at ICLR 2023
we perform trajectory attribution Seaquest environment trained using Discrete SAC algorithm. Here, we show results of our attribution algorithm in identifying influential trajectories for agents trained on same data but with different RL algorithm. Specifically, choose Discrete Batch Constrained Q-Leaannng (Fujimoto et al. 2019h a) to train Seaquest policy.
In Sec. 4.2
Fig. 12 depicts qualitative explanation generated using algorithm for DiscreteBCQ trained agent. 4 gives quantitative numbers associated with attributions performed in this setting. is that our proposed algorithm assigns similar importance to various clusters as done in T That is, we find that certain behaviours in the data agnostic to the algorithm used for training play similar role in determining final execution policy. Thus, we find that our algorithm is generalizable and reliable enough to provide consistent insights across various RL algorithms.
quite interesting
Table 4: Analysis of Trajectory Attribution for DiscreteBCQ-trriinee Seaquest Agent.
(so))
B(|QQ ())
(Forii)) * (s))))
(ais(d,d)) d,) P(otial = cj)
orig
1.3875
0.9619
0.1309
0.9249
0.4765
0.1025
0.5965
0.1380
0.8976
0.9513
0.0256
2
1.0157
0.1325
0,9233
1.0000
0,00854
3
1.1270
0.1323
0,9395
0.8999
0.0769
1.2243
0.1280
0,8992
0.5532
0.1025
1.2143
0.1367
0,9254
0.2011
0.3248
0.9752
0.1334
0.9238
0.6952
0.1196
7
1.1229
0.1352
0.9249
0.3090
0.2393
A.8 TRAJECTORY ATTRIBUTION ON ATARI BREAKOUT ENVIRONMENT
We present attribution results on additional environment of Atari Breakout [Bellmaae al. (2013)
trained using Discrete BCQ. Fig. 13 shows an instance of qualitative result and table 5 gives the
number for overall attributions performed on Breakout.
Observation
DDI
RIGHT
Attributed Trajectory
DOB N
D02 41
BU2
RIGHT
RIGHT
NOOP
FIRE
Figure 13:
Trajectory Attribution for DiscreteBCQ-trainee Breakout Agent. The agent pro- poses taking 'RIGHT' in the given observation frame. The corresponding attribution result shows how the ball coming from left would be played if moved to right.
19

---

Published as a conference paper at ICLR 2023
9
10
13544 004
20
feature
-20
-10
10
feature
044 144 145
Figure 14: Breakout Trajectory Clusters. The figure shows a PCA plot of breakout trajectories clustered into clusters. Our method identifies cluster 2 ('corner shot from the right') and cluster 3 ('depletion of life") as the most important high-level behaviours in the data for learning.
Table 5: Quantitative Analysis of Trajectory Attribution for DiscreteBCQ-ttrained Breakout Agent. We identify that clusters 2 and representing 'corner shots from right' and 'depletion of life', impact the mocision-making significantly [4] This is insightful given how important these behaviours are in general, the first one shows how to avoid ending the game prematurely, and the second one is well-known strategy in Breakout for playing at the end of the right frame to break the walls the top left for creating a tunnel.
((V(s))) B(IQQm. mew ()) E(1 (Forig (s) F xs(s)) Wais(d, d,) P(Cinal Cj)
orig
1.4570
1.1877
0.0972
0.7469
0.8828
0,0000
1.5057
0.0990
0.7317
0.2046
0.1428
1.2107
0.0983
0.7405
0.1676
0.2619
1.3946
0.0930
0.6687
0.1417
0.3095
4
1.4533
0.1043
0.7225
0.3827
0,0476
1.4678
0.1030
0.7310
0.5339
0.0000
1.1719
0.1022
0.7322
1.0000
0,0000
1.3493
0.1092
0.7225
0.3935
0.0000
1.2775
0.0916
0.7604
0.6999
0.04761
9
1.3773
0.0956
0.7496
0.5700
0.04761
10
1.4351
0.0998
0.7520
0.3005
0.1428
20


Reviews:
[{"review_id": "rJH5gZcn0N", "reviewer": "Reviewer_CPZK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors try to explain decision making of an RL agent from another perspective, and that is to highlight which trajectories in the training data are salient for the taken action. That is similar to prior saliency methods except that instead of highlighting input features, in this paper the trajectories from the training data are getting salient.", "summary_of_the_review": "See above.", "strength_and_weaknesses": "Following are suggestions, questions, and concerns that I had while reading the paper. \n\n1- Discussing how the current work aligns with findings from [4] and [5] could improve the paper.\n\n2- Combining the proposed method with salient feature extraction [1, 2, 3] and memory understanding [4, 5] could lead to significant insights.\n\n3- According to Section 3, any RL algorithm can be used. If so, it would be worthwhile if authors could provide results for other well-known RL algorithms to verify the proposed method's generalizability and reliability.\n\n4- How the role of memory is studied? Because of the complex update mechanisms in the RNN, we do not really know how removing a set of trajectories from the training data would affect it during the training. So how can the results be trusted?\n\n5- This work assumes that the training process of RL agents is deterministic. But we often see that due to randomness involved in the training, it’s possible to fall into a local minima so the distance between a given policy and an explanation policy could not necessarily be due to lack of cluster_i, but could be due to the stochasticity involved during training. How can we be assured that it is not the case?\n\n6- Providing results for more environments could be beneficial to check if the method is generalizable and robust.\n\nMinor issues:\n\nIn Section 2, “explain an action’s agent by identifying” -> “explain an agent’s action by identifying”\n\n[1] Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishnamurthy, and Sameer Singh. Explain your move: Understanding agent actions using specific and relevant feature attribution\n\n[2] Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia Sycara. Transparency and explanation in deep reinforcement learning neural networks\n\n[3] Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents\n\n[4] Anurag Koul, Sam Greydanus, Alan Fern. Learning Finite State Representations of Recurrent Policy Networks\n\n[5] Mohamad H Danesh, Anurag Koul, Alan Fern, Saeed Khorram. Re-understanding Finite-State Representations of Recurrent Policy Networks", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pKxTP_MpQ8-", "reviewer": "Reviewer_XJVh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper discusses the problem of explaining decisions made by an RL policy. The authors propose learning trajectory embeddings, followed by clustering, and then learning different policies for datasets constructed by removing different clusters. Candidate clusters are then chosen based on which policies lead to most deviation in chosen actions compared to the original dataset policy. Finally, the cluster with the minimum embedding distance from the original dataset is chosen to be most attributable to explaining the behavior at the given state. ", "summary_of_the_review": "The paper is written well, explains most concepts in detail and motivates the problem well. The experiments are done well, although I am not sure what kind of tasks one should really be testing explainability on. As I am not well aware of the relevant literature (and it’s quite possible there is minimal literature around RL explainability), it is hard for me to comment how effective the current evaluation setup really is. My current hunch is that the experiments considered are too simplistic in terms of gathering understanding on explainability. One minor question I have on this: The PCA plots for Seaquest do not seem to have a clear structure. Any comments on this? The mean action-value difference might not be a suitable metric if the action chosen by the other policy is out of distribution for the Q-value trained on the original policy. Thought on this? Why do you use a MSE as the action contrast measure for the continuous case (given you’re comparing probability distributions).", "strength_and_weaknesses": "Strength \n\n- Well motivated problem\n- Simple method\n- Fairly well done experiments\n\nWeaknesses\n\n- Limited scope in terms of experiments", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Q3nBWYGQ0F", "reviewer": "Reviewer_UWVh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the Explainable RL. To identify which trajectory among offline dataset contributes most to the behavior of the learnt policy, the authors first discretize the offline trajectory dataset into different groups, and train a separate policy for each group, representing the counterfactual outcome if that group is excluded from the training set. To explain a new action at a given state, the policies which are very unlikely to take that action would be chosen as the candidates for explaination, among which the one with training data mostly similar to the original data would be identified as the final explainer.", "summary_of_the_review": "This paper needs more comparison reuslts and more detailed analysis to enhance its persuasion.\nSome questions and suggestions:\n\n1. Fig.2 shows the clusters of trajectory embeddings, this paper said these clusters represented semantically meaningful high-level behaviors, but there are no specific behavior demonstration for each corresponding cluster to support this claim, and Fig.2b didn't show a perfect clusterd results.\n2. Fig.3 shows the action suggestion is influenced by trajectories distant from the cell under consideration, but this is concluded by the qualitative results, in my opinion, I can't see any attribution from traj.ii&iii to the corresponding decision at position (1,1).\n3. About the quantitative analysis, this paper said thet the relatively large initial state value estimation for the original policy indicates the original policy can have access to all behaviors, I don't think this explanation is reasonable, maybe it's just an overestimated value.", "strength_and_weaknesses": "Strength:\n1. This paper tries to learn the explainability of RL methods from a more complete perspective(trajectory information) instead of local information(current state information). And it's interesting to conduct a human study to verify the rationality of the learned explanation.\n2.  The proposed method is reasonable.\n\nWeaknesses:\n1. The descriptions of the experimental results should be improved, especially, more detailed annotations on the visual results are necessray to help the readers understand better.\n2. Without comparison methods, it's hard to evaluate the effectiveness of the proposed method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "K28-WGdltR", "reviewer": "Reviewer_NGMa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about explanations in reinforcement learning (RL). The idea is to focus on trajectories in offline settings (in a dataset) and find a cluster of trajectories that best explains a particular decision/behavior. The approach finds embeddings of trajectories, then clusters them, and verifies which of the clusters most contributes to a decision, by looking at the original dataset \"minus\" each of the clusters to see the influence (aka in terms of how close the behavior is to the original when trained with the restricted set). The approach is evaluated on three domains, and the paper contains a small user study to verify whether generated explanations match human thoughts about which trajectories contribute to decisions.\n", "summary_of_the_review": "Very clear paper with a practical approach that could be extended in many ways. The conceptual usefulness of this type of explanations is a bit unclear to me, and the evaluation does not help here. Positioning of the work is done ok, but it misses some relevant directions, for example clustering-based approaches in RL.", "strength_and_weaknesses": "The main strengths of the paper are that it is a very clear paper (both in terms of technical content, and in terms of structure/narrative) and that it provides a very simple, yet practical approach to work on explanations, trajectories and RL.\n\nThe first (clarity) is not yet visible in the abstract (which is not too clear I think) but the rest of the paper is highly clear about motivations, formalizations, choices, experimental setup and outcomes. All formalizations are at the right level; for example, Markov decision processes (etc) are not even defined, since all relevant matters are at the level of trajectories, and the text is doing a very good job to explain exactly what is needed. The five steps in Section 3 are very clear. Maybe one could explain a little more about the choice for embeddings (and technical details and choices), and I do not think Fig 1 is even necessary, but overall this is done well. The experimental setup is well motivated and the outcomes clear. I like the small \"shopping list\" in Section 4.1. which gives all details. Maybe a bit more detail on (for example) the attribution (and the matching) could be given here though.\n\nAnother big strength seems to lie in the simplicity of the (technical) approach. I think that it is very easy to (fore)see lots of variations on instantiations of parts of the algorithm to experiment with, and the general approach of trajectory \n\nA small weakness is the related work section, which could give more detail on explainable RL, which is quite an active subarea already, and this section does not provide enough state-of-the-art yet, I think. I think the offline RL paragraph is less important since it is merely a setting/context for this paper. I also think that the paper lacks detail and positioning in the context of trajectory clustering in RL, for example when it comes to personalization. There's quite some work on that, see for example as a starting point:\n\nZhu, F., Guo, J., Xu, Z., Liao, P., Yang, L., & Huang, J. (2018, September). Group-driven reinforcement learning for personalized mhealth intervention. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 590-598). Springer, Cham.\n\nA bigger weakness is that I am not sure the explanations actually \"work\". Technically I see what the method is doing, but conceptually I am not (yet) convinced this type of explanation is very useful. The basic idea is to look at clusters of trajectories and see whether we can match a decision/behavior to whether we have seen traces in the dataset that somehow provide a hunch on whether we have seen something like this before. It seems natural (also in the illustrations) that if a decision appears in one or more trajectories, it may be more likely that these trajectories can be used in an explanation, but if the dataset containst all kinds of trajectories (good and bad) then this does not sound logical. Technically clustering trajectories can always be done, but the actual \"meaning\" of it for explanations is less clear to me from reading the paper. The computational experiments only look at \"matching\" the behaviors (which are a bit in line with what I wrote in the above) but the human evaluation is even less clear about the usefulness and meaning. That is, in 70 per cent of the cases the best explanation coincided with the generated one but it also involves random trajectories (for which the probability that the decision was on them feels much lower). Overall, the evaluation seems to be very short, and not enough analyzed. Also, in the computational experiments (maybe also for the humans, but I could not find out) the clustering is probabilistic (as in: the attribution is a distribution, in the last row of Tables 1 and 2); how does this factor in the explanations? This is unclear to me, since in the end (alg 5, line 6) a particular cluster is assigned, but what if the distribution is very uniform? What happens with the results, what drives these results, and how will the (conceptual power of the) explanations be affected?\n\nSmall things:\n- I did not understand the motivation for the \"average of these output tokens\" (page 3) nor \"how\" this is done.\n- \"These choices.... of clustering\" at the end of page 5: can you elaborate on these choices?\n- The numbers of trajectories seems small (60, 717, etc). How does this influence the results? Is there any motivation here? Also, can you motivate more the choice for the 30 and 35 sublength trajectories?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster