Reproducibility Study of "Explaining RL Decisions with Trajectories"
Authors: Clio Feng
Colin Bot
Bart den Boef
Bart Aaldering
Venue: Accepted by TMLR ()

Abstract: This paper reports on the reproducibility study on the paper `Explaining RL Decisions with Trajectories'  by  Deshmukh et al. (2023). The authors proposed a method to elucidate the decisions of an offline RL agent by attributing them to clusters of trajectories encountered during training. The original paper explored various environments and conducted a human study to gauge real-world performance. Our objective is to validate the effectiveness of their proposed approach. This paper conducted quantitative and qualitative experiments across three environments: a Grid-world, an Atari video game (Seaquest), and a continuous control task from MuJoCo (HalfCheetah). While the authors provided the code for the Grid-world environment, we re-implemented it for the Seaquest and HalfCheetah environments. This work extends the original paper by including trajectory rankings within a cluster, experimenting with alternative trajectory clustering, and expanding the human study. The results affirm the effectiveness of the method, both in its reproduction and in the additional experiments. However, the results of the human study suggest that the method's explanations are more challenging to interpret for humans in more complex environments. Our implementations can be found on GitHub.

Published in Transactions on Machine Learning Research (04/2024)
Reproducibility Study of "Explaining RL Decisions with Tra- jectories"
Clio Feng University of Amsterdam
clio. froofeen@@student. uve.ml
Colin Bot University of Amsteriam
colán. botssstudnnnt.cmmmmmmmm uva.nl
Bart Aaldering University of Amsteniam
bart. wett.alldering@ttuueennn.. uva.nl
Bart den Boef
University of Amsterdam
bort.den. der.ddde.boogg. boort.den.boef@ssuuuddt...u
Reviewed on OpenReview: https: // operr/veew.ewe.... net/ forum? id-
Abstract
This paper reports on the reproducibility study on the paper 'Explaining RL Decisions with Trajeclories" by Deshmukh et al. (2023). The authors proposed method to elucidale the decisions of an offline RL agent by attributing them to clusters of trajectories encountered during training, The original paper explored various environments and conducted a human study to gauge real-world performance. Our objective is to validate the effectiveness of their proposed approach. This paper conducted quantitative and qualitative experiments across three environments: Grid-world, an Atari video game (Seaquest), and contin- uous control task from MuJoCo (HallCheetah). While the authors provided the ccode for the Grid-world environment, vironments. This work extends the original paper by including trajectory rankings within a cluster, experimenting with alternative trajectory clustering, and expanding the human study, The results affirm the effectiveness the method, both in its reproduction and in the additional experiments. However, the results the human study suggest that the method's explanations are more challenging to interpret for humans more complex environments. Our implementations be found on Hub
re-implemented it for the Seaquest and HalfCheetah en-
1 Introduction
Reinforcement. Learning (RI) demonstrates remarkable performance in dynamic settings, enabling real- time decision-making through direct engagement with the environment. However, the application of RL in practical contexts poses significant challenges. Offline RL research, which relies on pre-collected data rather than real-time interaction, attempts solve issues of learning efficiency and environmental risks. An important issue that remains is the lack of explainability of the RL docision-making processes.
Previous studies have focused on explaining RL agents' by highlighting important aspects of observations leading agent decisions (Gupta et al. 2019 Iyer et al. 2018 Greydanus et al.) 2018). Seeking to tackle this issue from a different perspective, Deshmukh et al. (2023) introduced an approach that highlights which past experiences ("trajectories' from pre-collected data) are responsible for an RL agent's 's decision. This concept of 'attributing' an algorithm's is decisions to training data has previously demonstrated effectiveness in supervised learning settings by Nguyen et al. (2021). In RL, trajectorics are sequences of states, actions, and rewards from an environment that an agent traverses. The paper proposes a trajectory attribution solution in offline RL scenarios using sequence modelling networks.

---

Published in Transactions on Machine Learning Research (04/2024)
This work aims to examine the results by reproducing and extending the experiments presented in the original paper. To assess the approach's 's efficacy, the original paper conducted experiments across diverse environments, including: i) Grid-world, à puzzle game where the objective is to navigate from a starting point to a winning position while avoiding obstacles, characterised by discrete state and action spaces. ii) Seaquest, an Atari game where the player controls a submarine that must shoot enemies while saving divers, without running out of oxygen. has a continuous state space where observations are screen frames and discrete action space (Bellemare al. 2013). iii) HalfCheetah, a continuous control task where the goal is to make 'cheetah' figure move forward by manipulating its joint angles, featuring continuous state and action spaces (Todorov et al. 2012). Figure shows example states for all three environments.
In this paper, we successfully reproduce the explanation framework and experiments in the three environ- ments above. To provide a comprehensive overview, Section 2 outlines its scope, and Section 3 details the proposed method and experiment implementation details. Replicated qualitative and quantitative results are discussed Section 4.1 with additional experiments and human study Section 4.2 The process of reproducing the experiments is discussed in Section 5 concluding in Section i
(a) Grid-world
(b) Seaquest
(c) HalfCheetah
Figure 1: Visual representations of example states in the Grid-world, Seaquest, and HalfCheetah environments. In the Grid-world visualization, winning states are depicted in green, while losing states are shown in red. The HalfChoctah state visualization utilized the built-in renderer.
2 Scope of Reproducibility
The scope of reproducibility in this paper is centered around the author's explainability framework for deep reinforcement learning. The main claim made by the author is follows:
"The proposed approach is efficient in lerms of allribution qualily and practical scalability across a spectrum of environments, including grid worlds, video games (Atari), and continuous control scenarios (MuJoCo).
Supporting claims and observations derived from the original paper include:
Interpretable Trajectory Clustering: Trajectory clusters generated by various suitable algo- rithms demonstrate consistent semantically meaningful high-level behavior.
2. Qualitative Performance: The method effectively explains agent actions with semantic intent by attributing relevant trajectory clusters.
3. Quantitative Performance:
The method achieves consistent quantitative results for multiple
relevant metrics outlined in Section 3.2
Human Study Insights: Humans predominantly choose trajectories attributed by the method the best explanation and sometimes fail correctly identify the factors influencing an RL decision.
By reproducing the experiments and analyses conducted in the original paper for the Gorddworld, Seaquest, and HalfCheetah environments, aim to validate the main claim by affirming these Sub-claims.
2

---

Published in Transactions on Machine Learning Research (04/2024)
Trajectory Encodingg
d. Training Explanation Palicies
Q
(tue))
Dadaa
C4 Rg
Q
Original Data
Data w/o C₁
Data w/o
Data w/o C.r
Trajectory (tz) from Offiine Data
RL
Trajectory Embedding
Encoder
b. Trajectory Clustering
AL Algo
RL Algo
Algo
RL Algo
orig
πC 1
2
T
FL
f
Trajectory Cluster Attribution
Explanation
Corresponding Clusters
Trajectory Embeddings
Clustering Ahgorthm
Trajectory clusters w/ semantic
1
Data Embedding
Trained on w/o
T orig (s, a)
π' Trained on Data w/e
Select policy which suggests different action, with corresponding complementary cluster embedding close original
Given Embeddings Trajectories
Permutationally Invariant Encoding
Representative Embedding Given
m
Trained on
Data
Figure 2: Trajectory Attribution in Offline R.. Figure from original paper Deshmukh et al.
(2023
3 Methodology
3.1
Description of methods
This section will detail the proposed trajectory attribution process illustrated Figure g Initially, a set of trajectories, representing sequences of environment states, actions, and rewards, is acquired. step a each trajectory is encoded into latent sequence using seq2seq model, which differs for each environment. A trajectory's embedding Tj is defined as the average of the vectors from its encoded sequence.
In step b, the encoded trajectories T- {Ty} are grouped into clusters C' by clustering algorithm, which is set to the X-means algorithm (Pelleg & Moore 2000) by default. This algorithm automatically determines the number of clusters, identifying diverse patterns in trajectories without imposing a fixed cluster number as a hyperparameter. The original authors suggest the potential applicability various algorithms for trajectory clustering, a claim that we examine in this paper. The explanations generated by the framework thus attribute decisions to clusters of trajectories, rather than individual trajectories. This addresses the computational constraints that come with large trajectory datasets.
In step a 'complementary dataset' is created for each cluster Gj by removing it from the original dataset. For each complementary dataset as well as the entire dataset, an embedding d, is computed. This is done by applying a softmax with temperature Toon. to the sum of trajectories. Note that the original paper mentions dividing the sum of embeddings by a normalizing factor before applying the softmax with temperature, but that dividing by Tana achieves the same results.
In step d, RL agent is trained every cluster's complementary dataset, can be seen in Figure 21. These 'explanation policy' agents will be used for the cluster attribution. This process, along with the generation of all cluster data embeddings, is detailed in Algorithm 1
In step e, actions from the original policy are attributed to a cluster by explanation policy agents. They select an action for the observed state and the algorithm chooses candidates based on the highest distances to the original agent's action. The metric for this distance not specified in the original paper. The candidate cluster with the smallest Wasserstein distance the original dataset's cluster embedding is selected as the responsible cluster for the state-action pair, outlined in Algorithm Essentially, this means that the
3

---

Published in Transactions on Machine Learning Research (04/2024)
Algorithm 1: ExpPolices
Data: Offline Data {r.), Trajectory Embeddings T, Trajectory Clusters C, Offline RL Algorithm
offlineRAAlgo
Result: Explanation Policies {my), Complementary Data Embeddings {dj} for Cj in C do
+ cj
// Compute complementary dataset corresponding to 9j
T heaaeTTacccttrynbeddnnnee (ri)) ;// Gather corresponding trajectory embeddings
Explanation policy, < olllineLLAlgo((()))
Complementary data embedding, d, - geneadteDaatllbmbbldiigg Tsott)
Output: Explanation Policies { Complementary Data Emboddings {d,}
algorithm identifies the cluster whose exclusion significantly alters the action while minimally affecting the data embedding.
Algorithm 2: CenerateCCusseerAAttbiuuu
Data: State 8, Original Policy Forig, Explanation Policies {π²}, Original Data Embedding dorig:
Complementary Data Embeddings {dj}
Result: Final Cluster Attribution Crinal
Original action, Gorig (s);
Actions suggested by explanation policies, aj + ,s(s);;
damin:a; &- CaleeAceiooDsssaneeeeee. az):
// Compute action distance // Get candidate clusters using argmax // Compute Wasserstein distance between da and darig // Choose cluster with the minimum data embedding distance
K + argmax((dammed4
Wk <- Wasserstein(dorig,. dz):
Cinal & argmin(wk)
Output: Cfinal
Every environment's algorithm implementation uses a different seq2seq trajectory encoder. The Grid-world implementation uses a tokenizer to encode the states and actions as numbers, followed by a trajectory transformer as introduced by Janner et al. (2021) modified by replacing the transformer module with a Long Short-Term Memory (LSTM) cell (Hochreiter uses decision transformer as introduced by CCen et al. (2021). The HallCheetah implementation uses unmodified trajectory transformer. The deep RT agents used for the explanation policies differ per environment implementation. The Grid-worke implementation uses a simple model-based algorithm using table of transition probabilities and action values computing using the Bellman equation (Bellman, 1957) The Seaquest implementation uses Discrete Soft Actor- (SAC) agents (Christodoulout 2019 The HalfCheetah implementation uses regular SAC agents (Haarnoja et al. 2018). For Grid-world and Seaquest, a binary function determined action distance, while the actions with the top 3 Euclidean distances were chosen candidates for HalfCheetah.
Schmidhuber) 1997). The Seaquest environment
To gain insight into the semantic meaning of a cluster, select one trajectory that is most representative. This is done by ranking trajectories within cluster in terms of how similar they are to the observation to be explained. By comparing all the observations in trajectory to the observation using similarity metric, we measure the relevance the trajectory. This allows us to choose the most similar trajectory as the individual attribution. For Seaquest, we compare mean squared error with structural similarity, which should more representative of similarity (Wang et al.] 22004; Wang Bovik 2009). The HallCheetah observations are vectors, for which the use of the Euclidean distance satisfactory.
3.2 Evaluation metrics
To quantify the effects of the proposed algorithm, the authors introduced five metrics. First, shown in Equation 0 Initial State Value Estimate (ISVE) Paanee et al. 2020) represents the expected rewards
4

---

Published in Transactions on Machine Learning Research (04/2024)
after completing full trajectory, based on initial state so by some value function V. A higher ISVE indicates that a policy is trained well.
(V((()))
(1)
The second metric is the Local Mean Absolute Action Value Difference (LMAAVD) in Equation The action values, cumulative rewards of taking a certain action, are calculated using the action-value function Q. LMAAVD considers the absolute difference between actions suggested by an explanation policy Tj for cluster perceived by the original policy Torig-
E(IQQ Toris 1) E (Torig())) - Trviie (T)()))))
(2)
The Action Contrast Measure (ACM) in Equation 3 measures the likelihood that an action suggested by Tj is different than the action suggested by Torig in state s. A higher ACM value for a cluster j means that Ty is likely to suggest a different action than Worig.
((I(Forig()) + (s))))
(3)
The Normalized Wasserstein Difference (NWD), shown in Equation is a way to represent the difference between complementary dataset embedding dj for cluster and the original dataset embedding dorig, normalized to the range [0, 1]. A lower NWD mcans that a complementary dataset resembles the original dataset closely, which is preferable for the candidate clusters. lower NWD means that a comple- mentary dataset resembles the original dataset closely. This is preferable for the candidate clusters because we want to find the cluster with the smallest change to the original dataset that leads to a different action. Intuitively, this cluster is likely to be the most responsible for the original decision.
Wdist (dorigg d.))
Cluster Attribution Frequency (CAF) in Equation represents the probability distribution P that the j-th cluster oj gets assigned as the responsible cluster Canal for a decision made by Torig.
P(Cinal =cz)
(5)
3.3 Datasets and Models
Grid-world. The original paper uses 5 Dyna-Q agents placed at random start locations to obtain trajectories of lengths 1 to 15, which resulted in a dataset of 60 trajectories (Sutton 1990). Trajectories with no final reward were ignored. 10 trajectories with a negative total reward were included as well as 50 trajectories with a positive total reward. This dataset was used to train the trajectory embedding LSTM and the agents.
Seaquest. We used 'scaquest-mixod-v4"" from d4rl-Atari (uu al. 2020), as the original paper does not mention specific dataset version. It consists of 1M pairs of obobervations, actions, and rewards. One observa- tion is an 84x84 grayscale image. Corresponding to the original paper, extracted 717 trajectories, divided into sub-trajectories of length 30. We used model weights for the decision transformer from Huggingtace
HalfCheetah. We used halffcheetah-mmedium-=22 from d3rlpy (Seno & Imail 2022). as the specific version was not mentioned the original paper. The dataset consists of 1000 trajectories length 1000, which are divided into sub-trajectorics of size 25 to make the trajectories less complex, as described the original paper. We used model weights for the trajectory transformer from its original implementation which was trained the same dataset.
5

---

Published in Transactions on Machine Learning Research (04/2024)
3.4 Hyperparameters
Grid-world. The modified trajectory transformer trained the same dataset the offline agents. The Dyna-Q agents trained for 2 episodes with 5 evaluation episodes per epoch, where training episode is one full run through the environment, with learning rates 0.1 and gamma value 0.95. The modified trajectory transformer with LSTM hidden layer size 32 was trained for 25 epochs with a learning rate 1, clipping gradionts to a maximum norm of 10. The X-means algorithm was run with a cluster range between and 10 clusters and Tooft was set to 10. The offline agents had a minimum action value and transition probability of 10-9.
Seaquest and HalfCheetah: Seaquest's transformer ran with vocab size of 18, a block size of 90, and 2719 timesteps, utilizing the 'reward-condttonedd model type. HalfCheetah's transformer used default parameters with sliding window of size 10. The X-means algorithm operated with cluster range of 2 to 8 for Seaquest and 2 to 10 for HalfCheetah. Discrete SAC and regular SAC agents from d3rlpy (Seno & Imai, 2022) were employed for Seaquest and HalfCheetah respectively, with hyperparameters consistent with the original paper: actor, critic, and temperature learning rates x 10 14-4, batch size of 256 for Seaquest and 512 for HalfChootah. Tooft was set 103 and 10 for Seaquest and HalfChoctah respectively.
3.5 Experimental setup
To gain insight into the amount of variability in the results, we conducted five runs with different random soeds for cach of the environments and reported the mcans and standard deviations. The clustoring was kept the same for each run because otherwise cluster labels would be shuffled.
Grid-world. We trained the RL policies until convergence, defined by maximun changes between iterations with a threshold of 10-4. We calculate the metrics for each agent and attribute individual trajectories from the responsible clusters. Additionally, we experimented with different clustering algorithms instead of X- means to verify their impact on the results.
Seaquest and HalfCheetah. We trained the RL policies for 10 epochs of 10a steps. We compute the expected action values for each action for 1000 random observations with the original policy and all of the explanation policies. Using this information
attribute responsible clusters and calculate the metrics. After that, we attribute individual trajectories from the responsible clusters. To evaluate the clustering effectiveness, we compare the Principal Component Analysis (PCA) of the clustered embeddings to the original paper. We also do qualitative and quantitative analyses of the trajectory attributions.
Human study. Similarly to the original paper, we included two types of questions where the participant needs to select trajectories they think best explain an RT agent's action given an observation. Question Type 1 has two options with one being correct. Question Type 2 has four options with two being correct. To extend the original study, broadened the scope to inchude Seaquest and HalfCheetah, increased the number of participants from 10 to 18 who had good understanding of the RL field, and added two new question types. Question Type 3 asks the participants to rank trajectories belonging to the same cluster by relevance. Question Type is an open-ended question, allowing the participant to explain previous answers or clarify what is unclear for each question. Lastly, we increased the number of questions for each type of questions per environment from to 6 (for Type and 2). Example questions can be found Figure 3
3.6 Computational requirements
The computational requirements are listed in Table [ Carbon emissions are included and calculated using Machine Learning Impact calculator (Lacoste et al. 2019). Experiments were conducted using a private infrastructure, which has a carbon efficiency of 0.432 kgCOzeq/kWh. The scale of our experiments was relatively small the environmental impact was minimal.

---

Published in Transactions on Machine Learning Research (04/2024)
This is an instance the task where the agent is located at (1. 1) and taking the righh' action.
State: (1, 1) Action: RIGHT
2
3
4
Trajectory
Trajectory 2
Please explain what makes you consider your choice best explains the action suggested in grid cell you de not know, why unclear?
7
2
3
4
6
7
(a) Responsible Trajectory
(b) Type 1 & 4 question
Trajectory
Trajectory
Trajectory
Trajectory
Trajectory
Mark only
one oval per row.
Trajectory
Trajectory
2
Trajectory
Trajectory
Trajectory
Please explain what makes you consider your choices are relevant in explain the action. you do not know, why unclear?
Pleese explain your reasoning, you do not know, why unclear?
(c) Type & question
(d) Type 3 & 4 question
Figure 3: Human Survey Question Examples for the Grid-world part of our extended human study.

---

Published in Transactions on Machine Learning Research (04/2024)
Table I: Computational requirements. Enviroument-speiinnn requirements are listed, as well as the esti- mated kg Emissions. Estimations were calculated using MachineLeaariing Impact calculator
Lacoste
et al. 2019
Spec
Gridworld
Seaquest
HalfCheetah
Ran on OS CPU
Jupyter Notebook
Python script
Python script
64-bit Ubuntu 22.04
Windows 11 Pro Intel Core i5-12400F
64-bit Ubuntu 22.04
6-core Ryzen 4500u
6-core Ryzen 4500u
at 2.3 GHz
at 4.4 GHz
at 2.3 GHz
GPU
Radeon Graphics 16GB
NVIDIA GeFFrce GTX 960
Radeon Graphics 16GB
RAM
16GB
Total compute time (h)
1
20
40
kgCO2eq Emissions
0.13
1.56
2.59
4 Results
4.1
Results reproducing original paper
In this section, reproduce the quantitative and qualitative results from the original paper along with extension of the original human study detailed in Section 4.2.2
4.1.1 Effectiveness of trajectory embedding cluster generation
We atlempled to prove that our reproduction of the trajectory embedding and cluster generation works similarly to the original paper's 's implementation. To achieve this, we generated the same figures as the original paper displaying a 2-D PCA the trajectory embeddings per cluster. The plots for Seaquest and 4 Both environments show similar separation clusters and PCA value ranges to the plots the original paper. When inspecting the clusters by hand, they appear to encompass similar semantic behaviour (e.g. 'lining up the player with an enemy' for Seaquest and 'landing from a high jump' for HalfCheetah). The Grid-world PCA plot generated by the provided code is identical to the original paper, which is shown in Appendix 5b These abservations support Sub-claim made in Section 2
HalfCheetah are shown in Figure
4.1.2 Qualitative analysis of trajectory attributions
All three environment implementations behave similarly to the observations made in the original paper. The attributed trajectory clusters include trajectories that are both semantically related and distant to the observation that is to be explained. As mentioned in Section 4.1.1 clusters tend to contain trajectories with a similar semantic meaning. These observations support Sub-claim made in Section 2 For example, an attributed cluster from the Crrd-world implementation may include trajectorics that pass through the same state that to be explained, well trajectories that don't. Similar observations were made for the Seaquest. and HallCheetah implementations.
4.1.3 Quantitative analysis of trajectory attributions
The quantitative experiments measured the metrics described in Section 3.2 for the three different envi- ronments' implementations. The code for the Grid-world experiments provided by the authors successfully reproduced quantitative results, see Appendix A.2 The quantitative results of the Seaquest experiments are shown in Table [a] These results are different. from those in the original paper. The ISVE and LMAAVD metrics mentioned in Section 3.2) are lower in all policies suggesting that they are poorly trained. The use of reward scaler in the agents of the original paper could explain the large differences in ISVE values. However, there is no way verify this without accessing the code of the original authors. In addition, one of the clusters' complementary datasets has NWD of 0, which is caused by the normalization process from
8

---

Published in Transactions on Machine Learning Research (04/2024)
2
8
Trajectory Clustering Seaquest
Trajectory Clustering HalfCheetah
10
feature 2
feature
2
-5
-10
1
-10
-5
10
15
20
25
feature
feature
(a) Seaquest
(b) HalfCheetah
Figure Principal Component Analysis (PCA) plot depicting Clusters of Trajectory Em- beddings for Scaquest and HalfChectah. These clusters represent semantically mcaningful high-level behaviors and the trajectory embeddings are similar to those in the original paper.
the original paper. The non-normalized Wasserstein distances are not zero. As a result of this, the policy trained on this complementary dataset always gets picked whenever it has a different action, in the case of discrete action spaces. This undermines Sub-claim discussed in Section 2 since one cluster can not always be the relevant one. Sub-claim does hold since we were able to achieve quantitative results for the relevant metrics. The quantitative results of the HallCheetah experiments are shown in Table 2b The results are comparable to the ones presented in the original paper for all metrics. This also supports Sub-claim 3 in Section 2
4.2
Results beyond original paper
To further test the claim of the original paper, we performed additional experiments. The results will be discussed in the section below.
4.2.1 Ranking Trajectories Within a Cluster
The use of mean squared error for Seaquest. resulted in trajectories that were not consistently similar to the given observation, which in line with the findings of Wang & Bovik (2009). However, we found that Structural Similarity provided an accurate ranking of the similarity the trajectories, agrecing with Wang et al. (2004). We find that trajectories suggested through the Euclidean distance metric in the HalfCheetah implementation returned trajectories that look similar to the trajectory leading up to the observation that is to be explained, more than returning random trajectory from the attributed cluster.
4.2.2 Human study results
Analysis of the Type questions reveals that simpler environments, participants predominantly favour trajectories identified by our algorithm as the optimal explanation for the agent's actions, supporting Sub- claim 4 in Section 2 However, environmental complexity increases, participants struggle to discern the factors inffuencing the agent's decision, unlike our algorithm. The accuracies for question type 1 are 77.92%, 71.48%, and 29.55% for Grid-world, Seaquest, and HalfCheetah respectively.
Shuffling the clusters feasible as this would also shuffle the policies thus making averages per index include different clusters.

---

Published in Transactions on Machine Learning Research (04/2024)
Table 2: Quantative analysis of Seaquest & HalfCheetah Trajectory Attribution. higher Initial State Value Estimate (ISVE) means better-trained policy. Higher Local Mean Absolute Action Value Difference (LMAAVD) and Action Contrast Measure (ACM) mean that explanation policies suggest more contrasting actions. Normalized Wasserstein Distance (NWD) represents the difference between the comple- mentary and original dataset of the given cluster. The Cluster Attribution Frequency (CAF) measures how often each cluster gets recognized as the responsible cluster. Clusters with low NWD and high ACM and LMAAVD are desirable. For each metric, the highest scoring cluster denoted in bold. The means and standard deviations are calculated from five runs. Standard deviations of 0.000 have been denoted as "-! Note the lack of standard deviations for the NWD metric, there was no difference between experiment runs because clusters were kept constant
Table 2a: Quantitative analysis results for the Seaquest environment implementation.
ISVE ((V(s)))
LMAAVD E(IAQ
ACM
NWD
CAF
([Iorr (8) + (T()))
Waist (d, dj)
P(cenal = Cj)
orig
1.8286 +
1.9277 + 0.3790 =
0.9240
0.7831
0.0030
1
1.9551
+ 0.3818 +
0.9330
0.2261
0.0010 +
2
1.8403
+ 0.3897 =
0.9330
0,0000
0.9329 +
3
1.9595
= 0.3822 =
0.9320 =
0.0406
0.0090 +
4
1.9703
+ 0.4047 +
0.9440 +
1.0000
0.0000 +
5
1.9964
+ 0.3721 + 0.9160 =
0.2368
0.0000
1.9421
+ 0.3835
+ 0.9270 =
0.8785
0.0000
1.7458
=
0.4029
=
0.9320
=
0.0358
0.0541
Table 2b: Quantilative analysis results for the HallCheetah enviromment implementation.
ISVE
LMAAVD B(IAQ momi
ACM B(1 (Torig (8) F (s))
NWD
CAF
(sV())))
Waist (d. d,)
P(CAnna = Cy)
orig
151.222 6.2500
0
158.8762 + 7.3547
0.7462 = 0.0892
0.8527 = 0.0751
0.5290
0.0048 + 0.0057
1
158.3154 + 5.4562
0.6234 = 0.1238
0.7296 = 0.1428
1.0000
0.0000
2
189.3018 + 8.3362
0.7567 + 0.1399
0.7888 + 0.0907
0.3199
0.1434 +: 0.0430
3
159.4749 + 9.4689
0.6945 = 0.0925
0.7537 = 0.1102
0.4184
0.0146 + 0.0140
4
203.7592 21.2017
0.7509 = 0.1129
0.8067 = 0.0556
0.9267
0.0000
5
122.0764 - 3.2605
1.4960 = 0.3245
1.2818 = 0.2074
0.3910
0.1194 + 0.0369
6
162.9265 21.6331
1.5264 + 0.6830
1.1268 0.2711
0.1901
0.3502 0.0922
7
147.2848 4.0694
0.6449 0.0636
0.6890 = 0.0789
0.0000
0.1678 + 0.0557
8
178.0963 = 4.0394
0.6301 = 0.0249
0.7031 = 0.0308
0.3480
0.0896 + 0.0204
158.9716 + 10.1533 9
0.6606 + 0.2046
0.7106 + 0.1449
0.2130
0.1 + 0.0421
In response to Type 2 questions, becomes evident that not all trajectories generated by our algorithm are deemed relevant by humans; some are rated as no better than random, further confirming Sub-claim Specifically, 76.56%, 66.95%, and 36.1% of participants selected trajectories generated by our algorithm for Grid-world, Seaquest, and HalfCheetah, respectively. Moreover, the accuracy decay follows similar trend to the one observed in Type questions.
To further investigate the factors that humans consider relevant, we look al the results of Type 3 questions. People tend to prioritize trajectories with the shortest path to the goal in Grid-world and visually similar trajectorics for Seaquest and HalfChoctah, ranking them highest. Type question responses support these observations. In Grid-world 90% of participants cited 'reaching the goal state', 80% mentioned 'shorter trajectories', and 20% referenced 'exhibiting similar actions as influential factors in their decisions.
10

---

Published in Transactions on Machine Learning Research (04/2024)
4.2.3 Alternative Trajectory Clustering
To validate Sub-claim 1 in Section beyond the X-means algorithm, alternative clustering methods were investigated. The goal is assess if these alternatives, when applied to Grid-world trajectory data, generate similar clusters and high-level semantic behaviour.
If the clustering made by a potential algorithm is topologically and semantically similar to X-means, the algorithm is suitable for use in the framework. To validate the similarity, we used two metrics: Solf-Organizing Map (SOM) (Kohonen 1990) and Normalized Mutual Information (NMT). SOM valuable for prescrving the topology of bigh-dimensional data and is particularly effective in capturing spatial relationships. NMI quantifies the shared information between true and prodicted clustering assignments. Both metrics yield scores between 0 and 1. A score of 1 indicates perfect agreement and score of 0 indicates no mutual information, comparable to random chance. If clustering scores high on both metrics, it is suitable for use in the framework.
Table 3: Clustering Evaluation Results. Self-Organizing Map (SOM), Normalized Mutual Information (NMI), and number of clusters compared to X-means for various clustering algorithms applied to the Grid- world trajectory dataset. More information on the tested algorithms listed Appendix A.3
Method
X-means
K-means
K-medians
Dbscan
Agglo
Optic
Claran
Cure
Rock
SOM
0.819
0.819
0.561
0.851
0.561
0.879
0.851
0.842
NMI
0.924
0.924
0.761
0.942
0.761
0.891
0.942
0.933
# Clusters
10
10
10
4
10
4
6
10
Looking at the two metrics in Table 3 it is clear that the clusters generated by most algorithms similar to X-mcans. This supports the claim made by the author that there are alternative clustering methods that could elfectively capture similar high-level semantic behaviour in trajectories.
5 Discussion
5.1
Reproducibility experiment experience
What was easy. The code was not publicly available when the reproducibility study was conducted, so contact was made with the authors to request the implementation code. They kindly provided the Grid-world experiment code on short notice. We thank the authors for their fast responsc and clear communictiinn. The provided codebase produced the same results as presented in the paper. The authors provided references pretrained decision and trajectory transformers for the Seaquest and HalfCheetah environments respectively, The authors mentioned librarics used in their implementations. The proposed algorithm is intuitive, which helps with reproducibility.
What was difficult. Version numbers for the libraries utilized in the experiments are missing from the original paper. This resulted in dependency issues which took time to solve. It is not entirely clear from the paper how metrics such as the ISVE and action distance were calculated for the Seaquest and HalfCheetah environments. The provided Grid-world code didn't help with clarity as the state and action spaces are different to the other two environments. This made comparing quantitative results difficult. To explore differences between ISVE values in the Seaquest environment with the original paper, trained Discrete SAC agents with multiplication reward scaler set to 40. This resulted in action values in the range of [0.1691, 70.4030), which is much closer to the values from the original paper. Therefore, suspect that the original paper used an unmentioned reward scaler in the Seaquest environment. No softmax temperature values were disclosed in the original paper. For Seaquest and HalfCheetah, the softmax inputs were very large, so a high temperature was needed return softmax outputs other than and In the original paper no experimental reruns were reported. Thus, the robustness of the quantitative results presented in the original paper is uncertain.
11

---

Published in Transactions on Machine Learning Research (04/2024)
5.2 Utility of explanations
The qualitalive analysis and human study showed that the explanations generated by the proposed method are only interpretable to humans if the trajectories themselves are interpretable. As environment complexity increases, becomes difficult to distinguish trajectories from one another. For example, the HalfCheetah environment trajectories tend to look similar. This means even though the method might attribute the perfect trajectory cluster an action, the explanation can still be unclear. This due to the inherent difficulty of understanding RL trajectories for humans in complex environments like HalfCheetah.
6 Conclusion
This paper has demonstrated our successful reproduction of the study on 'Explaining RL decisions with trajectories' (Deshmukh et al. 2023). Despite implementation difficulties, in Section 4 achieved results that generally align with the original work qualitatively and quantitatively, achieving interpretable trajectory clustering. In addition to this, we build upon the work in the following ways. In Section 4.2. we proposed methods to rank individual trajectories within a cluster to get a clearer represenlation of the framework's 's explanations. In Section 4.2.2 the human study was extended. The results show that more complex environ- ments lead to significantly worse interpretability for humans. Scction 4.2.3 alternative clustering methods were lested, which showed that multiple clustering algorithms suitable for the explanation framework. The results were made more robust by running the experiment multiple times.
In general, our results reaffirm the authors' claim that the approach is efficient regarding attribution quality and practical scalability. However, trajectory interpretability is still an important factor in the usefulness of the explanations. For future work, the human study results from Section 4.2.2 suggest that improvements in terms of the explanations' interpretability could be made to increase the usefulness of the explanation framework. For more complex environments, the trajectory embeddings and corresponding behaviour pat- terns could be analysed further, leading to more semantically meaningful clusters. This would improve the interpretability of the explanations because the clusters would be more distinct.
References
Mihacl Ankerst, Markus M. Breunig, Hans-Peter Kricgel, and Jörg Sander. OPTICS: ordering points to identify the clustering structure. pp. 49 60, 1999. doi: 00.1445/041182.3441877. URL https::/doi.org 10. 1145/304182.30415
M. G. Bellemare, Y. Naddal, J. Veness, and M. Bowling. The arcade learning enviromment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253 279, June 2013. ISSN
1076-9757. doi: 211613/jir.3911 URL http://dx.do.: doi. wwwpoooo......o..ooo 1613/
Richard Bellman. Dynamic programming. Princeton University Press, 1957.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Adilya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Tgor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. pp. 15084 15097, 2021. URL
TT835442000600777665116666666 Abstract.htm
Petros Christodonlon. Soft actor-critic for discrete action scttings. CoRR, abs/1910.07207, 2019. URL
http://arxiv.rrr/aaa/900..
Shripad Vilasrao Deshmukh, Arpan Dasgupta, Balaji Krishnamurthy, Nan Jiang, Chirag Agarwal, Georgios Theocharous, and Jayakumar Subramanian. Explaining RT decisions with trajectories. Tn The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. Open- Review.net, 2023. URL httpps///ppaarreeeeeeeoooodddd//
Martin Ester, Hans-Peter Kricgel, Jörg Sander, and Xiaowei Xu. density-basod algorithm for dis- covering clusters in large spatial databases with noise. In Evangelos Simoudis, Jiawei Han, and Us- ama M. Fayyad (eds.), Proceedings of the Second International Conference Knowledge Discovery
12

---

Published in Transactions on Machine Learning Research (04/2024)
and Data Mining (KDD-96), Portland, Oregon, USA, pp. 226-231. AAAI Press, 1996. 7/www.and aaai www..aaa.a..aaa...... htt..///bbrrr/////////////0---00 php
URL http:
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data- driven reinforcement learning, 2020. URL https://arxiv.org// wttp://g//aab....././/o
Samuel Greydamus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, TCMT, 2018, Slocklotmsmässsn, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1787 1796. PMLR, 2018. URL http::/ mlr.press ptr.spres/vv0/ greydanus18a. Fa html
Sudipto Guha, Rajeev Rastogi, and Kyuscok Shim. CURE: an efficient clustering algorithm for large databases. pp. 73 84, 1998. doi: 20.14452266344.26631.. URL httpp////ooo..r/1111111../...
276312
Sudipto Guha, Rajeev Rastogi, and Kyuseok Shim. ROCK: A robust clustering algorithm for categorical attributes. Inf. Syst., 25(5):345 366, 2000. doi:
URL https://doi. org/
Piyush Gupta, Nikaash Puri, Sukriti Verma, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishnamurthy, and Sameer Singh. Explain your move: Understanding agent actions using focuse feature saliency. CoRR. abs/1912.12191, 2019. URL
wttpaa/www....r/aaa/1///....1. 12191
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018. URL http://arxi: http:///axxvvv.or//aa.. abg//1b/1801. 01290
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 988):1355-178) 1780, November 1997. ISSN 1530-888X. doi: 10. 10.1162/neco.1999 1997.998..7788 URL http://dx doi 011/10.11///
neco. 1ww.1999..... 3.8.1735
Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia P. Sycara. Transparency and explanation in deep reinforcement learning neural networks. CoRR, Abs/1809.06061. 2018. URL
httt:://axxi.orr//bbb//// orooorgg/aoo abs//809,,.0///////// 06061
Anil K. Jain and Richard C. Dubes. Algorithms for Clustering Data. Prentico-Hall, 1988.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Marc' Aurelio Ranzalo, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Newral Information Processing Systems 34: An- nual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 1273 1286, 2021. URL httra://proceeiiggseerrrppp
Abstract. html
Teuvo Kohonen. The self-organizing map. Proc. IEEE, 78(9):1464-148) 1990. doi: 10.1109/5.58355, URL 1119//00.1119558888
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. CoRR, abs//190000900, 2019. URL htll:http:// http:://arxiiv..rr/////////////
James MacQueen et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1, pp. 281 297. Oakland, CA, USA, 1967.
Raymond T. Ng and Jiawei Han. CLARANS: A method for clustoring objects for spatial data mining. IEEE Trans. Knowl. Data Eng., 14(5):1003- 2002. doi: 2000://TTEEE22022133333030 URL https:
7/doi. 1tt.:/oog/01...1////... 11t.:1109/TTTDE 2002. 2032770
13

---

Published in Transactions on Machine Learning Research (04/2024)
Giang Nguyen, Daeyoung Kim, and Anh Nguyen. The effectiveness of feature attribution methods and its correlation with automatic evaluation scores, 2021. URL
14944
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning, 2020. URL
https://arxiv, orggabgg//..eooo...e..o ha///b///2000 200b//2000.oo 09055
Dan Pelleg and Andrew W. Moore. X-means: Extending k-means with efficient estimation of the number of clusters. In Internalional Conference Machine Learning, 2000. URL http:httss//ppp..eaattcccoooooo
org/CorpusID : 11243672
Takuma Seno and Michita Imai. d3rlpy: An offline deep reinforcement learning library. Journal of Machine Learning Research, 23(315):1-20, 2022. URL http://jmlr wttp:/paper//////22000111111 .tmll
Richard S. Sutton. Integrated architectures for learning, planning, and reacting based approximating dynamic programmiing In Bruce Porter and Raymond Mooney (eds.), Machime Learning Proceedings 1990, Рp. 216 224. Morgan Kaufmann, San Francisco (CA), 1990. ISBN 978-1-55860-1411 doi: https: /doi. org 10.16 1-78----5-60--11111--1---------1 141-3.5003044. p11/897815580011150338
URL
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, IROS 2012, Vilamown, Algarve, Portugal, October 7-12, 2012, pp. 5026 5033. IEEE, 2012. doi: N0.1109/IROSS.202..65332 2012.
URL https://doi: /tot::://doioor/11... 10. 1110/IRROSS 2012 6386109
Zhou Wang and Alan C. Bovik. Mean squared error: Love or leave it? A new look at signal fidelity measures. IEEE Signal Process. Mag., 26(1):98-117, 2009. doi: 10. 10.1109/MSP.... 2000/MMSP2008.03006 200.20008..444 URL https:
1tooi.org//0.1109//10
2008. 9306499
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., 13(4):600-612, 2004. doi: 10.1109/TIP. 2003.819861. URL
1109/TIP 20033 819861
A Appendix
A.1 Evaluation Metrics
A clear overview of the evaluation metrics used for quantitative analysis can be found in Table 4
Table 4: Metrics used for quantative analysis. In these formulas, represents a value function, that returns reward based on input state 80 represents state 0 of given trajectory; the initial state. Q is action-value function that returns the estimated cumulative reward of taking action decided by policy, either the original policy Torig that returns the Wasserstein distance between its inputs, which are the original dataset embedding d and the complementary dataset embedding dj of cluster j. Finally, P(Cinal cj) denotes the probability distribution that the j-th cluster is the attributed cluster Canal for a given decision.
explanation policy πj of cluster in stale Waist is function
Abbreviation
Name
Formula
ISVE
Initial State Value Estimate
E(((s)))
IMAAVD
Local Mean Absolute Action-Value Difference
E( E(IAQ Focte 1) E[[ T (Torig Q (Tz()))1) E (Torig(s) [T; (s)))
ACM
Action Contrast Measure
NWD
Normalized Wasserstein Distance
Waist (d, dg)
CAF
Cluster Attribution Frequency
P(Canal Cj)
14

---

Published in Transactions on Machine Learning Research (04/2024)
4
7
7
0.75
0.75
0.50
0.25
fostuee 2
2
0.00
0100
-0.25
-0255
-0.50
-0.50
-3
10
Scature
fentuee
(a) Ours
(b) Original paper
Figure 5: PCA Plot depicting Clusters of Trajectory Embeddings of the Gridworld environment for a) our results and b) the original paper. The clusterings are almost identical to the ones presented in the original paper.
A.2 Gridworld Trajectory Attribution Results
The results of the Gridworld trajectory attribution were very similar to the original since were provided with the original code by the authors. Thercfore, we chose to present the results of Table 5 here.
Table 5: Quantative analysis of Gridworld Trajectory Attribution. A higher Initial State Value Esti- mate (ISVE) means better-trained trained policy. Higher Local Mean Absolute Action Value Difference (LMAAVD) and Action Contrast Measure (ACM) mean that explanation policies suggest more contrasting actions. Nor- malized Wasserstein Distance (NWD) represents the difference between the complementary and original dataset of the given cluster. The Cluster Attribution Frequency (CAF) a measure of how often each cluster gets recognized the responsible cluster. Clusters with low NDD and high ACM and LMAAVD are desirable. For each metric, the highest scoring cluster is denoted in bold. The means and standard deviations calculated from five runs. Standard deviations of 0.000 have been denoted -_
ISVE B(V(so))
LMAAVD B(IQQ
ACM
NWD Waist (d, (d,d))
CAF P(Canal cj)
π
B(1 (Torig (s) # (s()))
orig
0.3061 +
0
0.2990 +
0.0313 +
0.0408 +
0.0009 +
0.2000 +
1
0.3053 +
0.0395 +
0.0408 +
0.0020 +-2
0.0000 +
2
0.3049
0.0309
0.1224
0.0001 +
0.8000 +
0.3055
0.0015
0.0204
1.0000
0.0000 +
0.3054
0.0224
0.1224
0.0428 +
0.0000 +
5
0.3057
0.0275
0.0204
0.0011 + 0.0000 +
6
0.3046
+
0.0137
0.1224 +
0.0008 + 0.0000 +
7
0.3055
+
0.0119 +
0.0204 +
0.0003 + 0.0000 +
8
0.3057
+ 0.0008 +
0.0204 +
0.0003 + 0.0000 +
9
0.3046
0.0291
0.1428
0.0005 + 0.0000 +
In addition the similar results of Table the clusters found are also identical those of the original paper. For that reason, present. Figure
here.
15

---

Published in Transactions on Machine Learning Research (04/2024)
A.3 Clustering Algorithm Information
Table 6 lists the sources for the different clustering algorithms.
Table 6: Clustering algorithm sources.
Algorithm
Source
K-Mcans
MacQueen et al.
1967
K-Medians
Jain & Dubes
1988
Dbscan
Ester et al.
1996
Agglo
Jain Dubes
Optic
Ankerst al.
1999
Claran
Ng & Han
2002
Cure
Guha et al.
1998
Rock
Guha ct al.
2000
16


Reviews:
[{"review_id": "ihfbZImTz8", "reviewer": "Reviewer_JUYP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n* The paper is attempting to reproduce the results of another paper. Thus, the exact settings it used to do so is important. The paper has gone into great detail about what the experimental setup and the hyperparameters used.\n* It gives a clear overview of the paper it is trying to reproduce.\n* The extension of the experimental results give additional value to the paper beyond just a validation of a previous paper\n* The computational requirements and environmental impact section was particularly noteworthy\n* The possible reasons for why reproducing the quantitative results was difficult was also appreciated\n\n## Weaknesses\n* The paper reproduces the results of the original paper in the same domains as the original paper. While this setup helps to validate if the original results can be reproduced, there is a question of how general the insights of the original paper are could also have been answered by evaluating on an additional domain.\n* It is unclear what the scientific insight this study has provided. The additional experiments give us some deeper insight on the original paper. But it is unclear how valuable this insight is.\n* Some of the results, such as the ones in Table 2, would benefit from clearer descriptions of what a reader should take away from it.\n* In section 3.2, the first sentence could be expanded on to reduce murkiness and establish clearly what the algorithm is trying to do.\n\n### Questions:\n* In Section 4.2.1, what anecdotal evidence are the authors referring to? Their own?\n* In section 4.2.2, how is the fact that the algorithm is not picking relevant trajectories a point confirming sub-claim 4? If I understand correctly, sub-claim 4 indicates that humans sometimes fail to correctly identify the correct factors influencing the RL decision. But Section  4.2.2 only indicates that humans do not pick some of the trajectories that the algorithm proposes. Perhaps the proposes trajectories are not relevant?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is attempting to reproduce the results of a paper published at ICLR 2023, focused on explaining the behavior of an agent trained with a static dataset (offline reinforcement learning, RL). The original paper clustered trajectories and then explained decisions by finding the cluster that had the most impact on a particular decision. This paper additionally tries to rank particular trajectories within a cluster as well, extends human studies, and tests some alternative clustering techniques.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In references, the reference for the main paper being reproduced, \"Explaining RL decisions with trajectories\", is not cited appropriately. No mention of where it was published is made.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QB6mqdPL6Q", "reviewer": "Reviewer_SnGL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. The manuscript is relatively easy to follow, and it is packed with experimental results.\n2. I was able to understand the algorithm and generally the original work without having to look at its manuscript (even though later I did skim through that).\n3. The manuscript's authors seem to have done a great job at being fair with the original work's authors, and have largely reproduced the original results.\n\n## Weaknesses\n\nMy main concern with the paper is that the additional content on top of the original work is extremely limited. The paper fundamentally adds two things:\n\n1. A way to rank trajectors against different explanation policy datasets -> I think this would be useful in the context of choosing hyperparams such as numbers of clusters and generally understanding the policy space better from a distributional perspective (e.g. bottleneck states, space variance). By itself, I'm not sure how much value it adds on top of the original work, and it's not clear to me whether we can make any strong claim from the data gained within the current experimental setting.\n2. The human study extension -> I don't find the one major conclusion from the additional human experiments is not well defended: it's not clear to me whether these environments are \"complex\", and good ALE / simple mujoco environments are generally well interpretable in the literature, given how common it is to make qualitative analysis on top of policies trained in such environments. I would expect bad policies to be hard to interpret however, and there might be other factors at play here.\n\nSo, overall I find the manuscript useful, but I'm not sure about its added value outside of the efforts to reproduce the original work's findings.\n\nAlso, as a nit: I think the problem statement could be better introduced. See below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript attempts to reproduce the algorithms and experiments of \"Explaining RL decisions with trajectories\", a work that introduced a method for explaining RL policies learned using an offline dataset by comparing state action transitions with trajectories held in a training dataset. The authors expand that work by carrying out further human evaluation, and by doing some analysis on the trajectory clusters used in the method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "What follows is a bunch of feedback and questions per page:\n\n- p1: \"While offline RL research...\" -- I don't think offline RL as a framework has solved the issues you mention, and it's not sensible to imply that the list of RL-issues-in-real-life-scenarios is small. I can see what the manuscript is trying to say here, but adjustment is welcomed.\n- p1: \"...by highlighting critical state features\" -- would be good to add a non-dense explanation here so that readers unfamiliar with the explanability literature is not forced into holding black boxes already. Same idea for what it means to \"attribute\" past experiences to agent decisions. Without a simple explanation here, the sentences don't have much unambiguous meaning (NB: things like the meaning of \"trajectories\" is instead well known in the RL community, so no need to clarify if you have verbosity constraints).\n- p3: \"...thus point to clusters of trajectories\" -- I'd maybe rewrite this: \"point to\" is a little handwavy / confusing.\n- p3: \"...is trained on every cluster's complementary dataset\" -- I would maybe point the reader to Figure 2d, as it's otherwise ambiguous.\n- p4: \"..why a cluster is attributed as responsible\" -- reading the manuscript top down, it's not clear what this means. The paragraph feels also a bit roundabout-y about wanting to find the most similar trajectory in the selected cluster for analysis (because the hypothesis is that ultimately that's what matters the most here? Maybe worth clarifying the reason you are doing this analysis here?).\n- p4: \"A higher LMAAVD...\" -- the explanation seems like a repetition of its previous sentence. Is it just trying to clarify the sign? Might be worth shortening.\n- p5: \"A lower NWD means...\" -- why is it preferable?\n- p6: \"... because otherwise cluster labels would be shuffled\" -- I understand why this is a problem in your analysis, but in your view if you had a way to fix this inconsistency, would you have been able to gain further insight in the results?\n- p6: \"... we experimented with different clustering algorithms...\" -- this is only done in the grid-world setting. How do you think the higher state dimensionality for the other environments would affect the conclusion of generality wrt. clustering algorithms? Also, how did the new experiments pick a well functioning set of clusters? (e.g. a poorly trained / regularized set could massively influence later results)\n- p6: \"Human study\" -- it would be helpful to have a single example of each type of question here, rather tha in the appendix.\n- p6: \"MachineLearning\" -> \"Machine Learning\"\n- p7: for section 4.1.2, I think it is necessary to qualify a little bit more how the inclusion sets work for each environment. Do the sets tend to have bottleneck states for instance, or any other clear structure?\n- p8: \"... NWD of 0\": is this a problem with the experimental setting? Is the manuscript trying to argue that care is necessary when choosing clustering hyperparams or doing regularization? I would love to see some hypothesis (or better, detailed explanation) for why this happened.\n- p9: \"Anecdotal evidence\" -- it would be great if the manuscript discussed any visible failure modes.\n- p9: \"... topologically and semantically similar to X-means\" -- what's meant specifically here? Can you give examples of some that don't apply?\n- p10: \"..this is due to the inherent difficulty of understanding RL trajectories for humans\" -- that's a somewhat surprising takeaway. Aren't these environments simple enough that good policies should be naturally interpretable? Is sub-optimality wrt. a clear reward function maybe the real factor?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GDWGZVU8oC", "reviewer": "Reviewer_4piC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- **Comprehensive Methodology:** The paper carefully details the methods used to reproduce and extend the original work, including the introduction of new evaluation metrics and experimental setups for environments not covered by the original authors.\n\n- **Extended Analysis:** By incorporating trajectory ranking within clusters, alternative trajectory clustering, and an expanded human study, the paper significantly builds upon the original findings, offering deeper insights into the explainability of RL decisions.\n\n- **Quantitative and Qualitative Validation:** The study confirms the effectiveness of the original method through both quantitative and qualitative analysis, demonstrating the reproducibility of results across multiple environments and metrics.\n\n- **Insightful Human Study:** The expanded human study provides valuable insights into the interpretability of the explanations generated by the method, highlighting the challenges faced by participants in more complex environments.\n\n- **Environmental Impact Consideration:** The paper includes a section on computational requirements and carbon emissions, reflecting an awareness of the environmental impact of machine learning research.\n\n\n## Weaknesses\n\n- **Complexity in Interpretability:** The results suggest that while the method is effective in generating explanations, the complexity of those explanations increases with the complexity of the environment, potentially limiting its utility for end-users without domain expertise.\n\n- **Implementation Challenges:** The paper discusses difficulties encountered due to missing version numbers and unclear calculation methods for certain metrics, which could have been mitigated with better documentation from the original authors.\n\n- **Dependence on Clustering Quality:** The utility of the explanations seems highly dependent on the quality of trajectory clustering. While the paper explores alternative clustering methods, the dependence on this step underscores a potential fragility in the method's applicability across diverse RL tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides a comprehensive examination of the original work by Deshmukh et al. (2023), which introduced a method for explaining reinforcement learning (RL) decisions through trajectory clustering. This reproducibility study extends the original research by including additional experiments, extending the human study, and experimenting with alternative clustering methods.\n\nThis reproducibility study significantly contributes to the field by not only validating the original findings but also extending them in meaningful ways. It underscores the potential of trajectory-based explanations in RL while highlighting the challenges in making these explanations accessible and interpretable to a broader audience. The careful and detailed methodology, along with thoughtful analysis and suggestions for future research, make this paper a valuable resource for researchers interested in explainable AI.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Required Changes\n\nThe paper does not require any further changes for acceptance.\n\n***\n\n## Recommendations to Strengthen the Paper\n\nIf desired, the following points can further strengthen the paper.\n\n### 1. Clarification and Expansion of Methodological Details\n\n- **Metric Calculation Clarification:** The methodology section should include a clearer explanation of how the metrics such as ISVE and action distance were calculated, especially for the environments like Seaquest and HalfCheetah, where the state and action spaces differ significantly from Grid-world.\n\n### 2. Enhanced Discussion on Limitations and Challenges\n\n- **Interpretability in Complex Environments:** The paper could delve deeper into the challenges of interpretability in complex environments. Suggestions for making the trajectory explanations more accessible to users without specific domain expertise would be valuable.\n\n- **Dependence on Clustering Quality:** Discuss the potential limitations and implications of the dependence on trajectory clustering quality. This should include a discussion on how different clustering methods might impact the utility and scalability of the proposed explanation framework across various RL tasks.\n\n### 3. Additional Experiments and Analysis\n\n- **Robustness to Clustering Methods:** Conduct further experiments with a broader range of clustering algorithms to assess their impact on the quality and interpretability of the explanations. This could help in identifying more robust and generalizable approaches to trajectory clustering within the context of RL decision explanation.\n\n- **Automated Evaluation of Explanations:** Explore and possibly develop automated metrics for evaluating the quality and utility of the generated explanations. This could provide a more objective basis for comparing different explanation methods in RL and facilitate a deeper understanding of what constitutes a \"good\" explanation.\n\n### 4. Revision of Human Study\n\n- **Broaden Participant Diversity:** To ensure the human study results are generalizable, consider including participants with varying levels of expertise in RL, not just those who have a good understanding of the field. This could provide insights into how intuitive the explanations are to a broader audience.\n\n- **Detailed Analysis of Human Study Results:** Expand the analysis of the human study results to explore why participants struggled with interpreting the explanations in more complex environments. This could involve a qualitative analysis of the open-ended responses to understand the specific aspects of the explanations that were difficult to interpret.\n\n### 5. Environmental Impact\n\n- **Detailed Discussion on Computational Efficiency:** While the paper mentions the computational requirements and carbon emissions, it would be beneficial to discuss strategies for minimizing the environmental impact of such studies. This could include recommendations for more efficient computational practices in ML research.\n\n### 6. Conclusion and Future Work\n\n- **Explicit Recommendations for Future Research:** The conclusion should include explicit recommendations for future research directions, based on the findings and limitations of the current study. This could involve suggestions for alternative models or methods for explaining RL decisions, as well as potential improvements to the existing framework to enhance its interpretability and applicability.", "broader_impact_concerns": "The paper poses no ethical concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a reproducibility study, reexamine the results of a previously published paper, adds new experiments, and deepens the analysis of the original work. 

(+) The paper is well-written. 
(+) The methodology is reasonable. It validates the original findings and expands them in meaningful ways (although the expansion is limited).
(+) The changes made by the authors after the reviews addressed some of the reviewers' concerns.  

(-) The new experiments are rather limited, and thus, the paper does not significantly expand our understanding of the original work.
Decision: Accept (poster)