'Explaining RL Decisions with Trajectories’: A Reproducibility Study
Authors: Karim Ahmed Abdel Sadek
Matteo Nulli
Joan Velja
Jort Vincenti
Venue: Accepted by TMLR ()

Abstract: This work investigates the reproducibility of the paper "Explaining RL decisions with trajectories“ by Deshmukh et al. (2023). The original paper introduces a novel approach in explainable reinforcement learning based on the attribution decisions of an agent to specific clusters of trajectories encountered during training. We verify the main claims from the paper, which state that (i) training on less trajectories induces a lower initial state value, (ii) trajectories in a cluster present similar high-level patterns, (iii) distant trajectories influence the decision of an agent, and (iv) humans correctly identify the attributed trajectories to the decision of the agent. We recover the environments used by the authors based on the partial original code they provided for one of the environments (Grid-World), and implemented the remaining from scratch (Seaquest and HalfCheetah, Breakout, Q*Bert).
While we confirm that (i), (ii), and (iii) partially hold, we extend on the largely qualitative experiments from the authors by introducing a quantitative metric to further support (iii), and new experiments and visual results for (i). Moreover, we investigate the use of different clustering algorithms and encoder architectures to further support (ii). We could not support (iv), given the limited extent of the original experiments. We conclude that, while some of the claims can be supported, further investigations and experiments could be of interest. We recognize the novelty of the work from the authors and hope that our work paves the way for clearer and more transparent approaches.

Published in Transactions on Machine Learning Research (05/2024)
'Explaining RL Decisions with Trajectories': A Reproducibility Study
Karim Abdel Sadek, Matteo Nulli, Joan Velja and Jort Vincenti
University of Amsterdam
Reviewed on OpenReview: http:://poparrieeeeeeecc/cccccca
Abstract
This work investigates the reproducibility of the paper Explaining RL decisions with trajectories " by Deshmukh et al. (2023). The original paper introduces a novel approach in explainable reinforcement learning based on the attribution decisions of an agent to specific clusters of trajectories encountered during training. We verify the main claims from the paper, which state that (i) training on less trajectories induces lower initial state value, (ii) trajectories in a cluster present similar high-level patterns, (iii) distant trajectories influence the decision of an agent, and (iv) humans correctly identify the attributed trajectories to the decision of the agent. We recover the environments usod by the authors based on the partial original code they provided for one of the environments ( (Grid- World), and implemented the remaining from scratch (Seaquest and HalfCheetah, Breakout, Q Q*Bert). While we confirm that (i), (ii), and (ili) partially hold, we extend on the largely qualitative experiments from the authors by introducing a quantitative metric to further support (iii), and new experiments and visual results for (i). Moreover, we investigate the use of different clustering algorithms and encoder architectures to further support (ii). We could not support (iv), given the limited extent of the original experiments. We conclude that, while some of the claims can be supported, further investigations and experiments could be of interest. We recognize the novelty of the work from the authors and hope that work paves the way for clearer and more transparent approaches.
1 Introduction
Reinforcement Learning (RL), formalized in the pioncering work of |Suttnn & Barto (2018). focuses on learning how to map situations to actions, in order to maximize reward signal. The agent aims discover which actions are the most rewarding by testing them. This addresses the problem of how agents should learn policy that takes actions to maximize the cumulative reward through interaction with the environment. A recent pivotal focus in RL is the increasing attention on the explainability of these algorithms, a factor for their adoption in real-world applications. Precedent work in the field of XRL include Puiutta & Veith (2020).
Korkmaz (2021) and Coppens et al. (0019). This reprodueibility report focuses on the work of Deshmukh et al. (2023). which proposes an innovalive approach to enhance the transparency of RL. decisionnmmaking
processes. Given the rising interest and applications of Offline RL (Levine et al. (2020) Kumar et al. (2020)), ) obtaining explainable decision is important desideratum. Deshmukh et al. (2023) introduces novel
framework in the oflline RL landscape. This new approach is based altributing the decisions of an RL agent to specific trajectories encountered during its training phase. It counters traditional methods that predominantly rely on highlighting salient features of the state of the agent(lyer et al. (2018)).
Despite proposing a novel approach, no subsequent work has built upon it. We believe this is primarily due to the absence of code available online and the limited number of environments on which this method was tested, which reduces its practical applications. Therefore, we intend to not only validate the original results
"(karim.aaur sadeksadek, matteo.null, nulli, joaj.velja, jort.vincent Srshhcenti(Ssstuuent..aaaa

---

Published in Transactions on Machine Learning Research (05/2024)
by Deshmukk et al. (2023) but also explore the robustness and generalizability of the proposed methodology across different environments and settings. Our reproducibility study is step toward ensuring that advance- ments in this domain are both transparent and trustworthy, paving the way for a better understanding of RL systems.
2 Scope of Reproducibility
Explainability and interpretability have recently become of great interest for the adoption of Al systems in real-world applications. In particular, understanding and explaining the behavior and decisions of RT. agents is crucial task considering the plausible large-scale adoption of these systems. On top of the aforementioned
ones in Section 1 other examples of Explainable Reinforcement Learning (XRL) studies include high-level
decision language approach by Puri al. (2019] Pawlowski al. (2020) and Madumal et al. 2020 The goal of this report is to analyze the reproducibility of the work by Deshmukh et al. (2023). Given the novelty of the work, it follows that there is no existing benchmark to compare the results claimed by the authors. Our contribution lies mostly in the implementation, verification, and interpretation per se of these results. We proceed by verifying the claims made by the authors, which we summarize and re-state as follows here below:
Removing Trajectories induces lower Inétial State Value: Including all relevant trajectories in the training data will result in higher equal initial state value estimates compared to training sets where key trajectories are omitted. This holds also for other metrics may consider. The definitions be found in Section 3.3
Cluster Hive- Behaviours: High-level behaviours are defined patterns within trajectory which lead to the same result and repeat over multiple trajectories. We aim to verify that different embedding clusters represent different meaningful high-level and interpretable behaviors.
Distant Trojectories influence Decisions of the Agents: Decisions performed by RL agents can be influenced by trajectories distant from the state under consideration. In such scenarios looking only at the features in the action space may not provide full understanding of the behaviour of an agent.
Human Study: Humans may accurately identify the determinant trajectories that influenced the decision of an RL agent.
3
Methodology
The original paper code is not yet publicly available. However, obtained part of the code from the authors: we were given the Grid- World environment, together with part of its related experiments. We followed their code and expanded upon it, in order to verify the claims. On the other hand, we wrote from scratch the implementation for Senquest, HalfChetoah. Additionally, we added two environments to the analysis of Deshunukh el al. (2023), Breakoul and Q*Bert.
3.1 Environments
The investigations made in the paper regard three different types of Reinforcement Learning environments:
Grid- World, a grid-like environment in which the agent has a discrete state and action space. The game consists of an agent starting from a point the grid and moving inside it. The goal reach ''grenn' cell while avoiding entering a 'red' cell, while making the smallest number of steps possible. The default grid has size of 7x7.
Seaquest, a vidco-gamo-like environment in which the agent has discrete state and action spacc. The game consists in a submarine moving underwater. Here are more information on the Atari Seaquest environment.
2

---

Published in Transactions on Machine Learning Research (05/2024)
3. HolfCheetah,
environment where the agent has continuous state and action space. The game consists of 2-dimensional robot having number of joints. The goal is to get the cat shaped robot to run, by turning its joints using rotational forces (torque), Here are more information on the HalfCheetah environment.
4. Breakout, one of the most played atari-produced video-games. The game consists of paddle and hitting a ball into brick wall at the top of the screen, where the goal is to destroy all the bricks. Here are more informations on Breakoul environment.
5.
Beerr is video-ge rideo-game game like environment. The game consists of agent which hops each cube of a pyramid one at time, and the goal is to change their color of into specific one. Here are more informations on Qert environment.
3.2 Datasets
Datasets used in the analysis have the same composition between the five environments. Each dataset D comprises of a set of n, trajectories. Each Ty a k-slep trajectory and each trajectory step is tuple Tj T5. where Jj = (o;, az rt). Here o, is the observation in that step, az is the action taken in that step and r; is the per-step reward. However, collecting data depends on the environment in which the experiments are made. Regarding Grid- World, agents are trained specifically to generate data trajectories. For Seaquest data is instead downloaded from Adtrl-Atari Repository for Breakoul and Q*Bert from Expert-offline RL Repository whereas in the case of HalfCheetah from d4rl Repository of Fu et al.) 2020
3.3 Model Description
Across all three environments, trajectory attributions and explanations are made through the following 5 steps, also summarized in Figure
Trajectory Encoding
Training Explanation Polices
Origina Bata
Data w/jn
Daca web Fu
Défine
Trajectory
Trejectory Cintaring
Algo
Algo
Horig
T
12
1'm
e. Trajectory Cuoter Attribution
Exolanation
Comesponding
Inajectory Embedding
Clustering Aggrithrs
Trajectory dusters w/ semantic measing
Data Embeddingg
Torig (5,
Select polley which suggests action, commopending complementary cluster embrrdin dose to origiral
Given Embeddings of set-of Trijectoiiss
Permutattonall meariott Set Enouding
Representative Embedding of Cleen Trap.
Trained T'm
Cy.
Figure 1: Trajectory attribution process by Deshmukh et al. (2023)
a. In Grid- World trajectories are generated by training different agents using Model-based offline RL through the Dyna-Q Algorithm (Appendix A.1). Trajectorics are then encoded. In Grid- World the authors define Seq2Seq LSTM based encoder-decode architecture, After training, only the output of the encoder which corresponds to the trajectory embedding of Figure kept. On the
3

---

High values are Published in Transactions on Machine Learning Research (05/2024)
other hand, in all others (Seaquest, Breakout, Qe and HalfCheetah) the trajectories encoders are pre-trained. For the former, the model is obtained following the instructions on pre-trained decision transformer) For the latter, the pre-trained model is downlosded from the GitHub repository pre- trained trajectory transformer from Janner et al. (2021). Both architectures are GPTs. Last but not least, these encodings are then embedded.
b. The embeddings are passed through the XMeans clustering algorithm introduced by Pelleg et al. (2000). The implementation used by the authors is the one from Novikov (2019). Using XMeans is an arbitrary choice and in Section 4.5 will investigate other options.
The cluster representations are embedded obtaining the representative embedding of given trajecto- ries.
d. The so-called complementary datascts are obtained. That is, for each cluster we create different dataset where for each cluster j we retain all the data but those trajectories belonging to cluster j itself. We obtain then 10, 8, and 10 complementary datasets for the three environments respectively, and train for each complementary dataset new explanation policies and actions. In particular for Seaquest, Breakout and B*Bert we use DiscreteSAC Christoooool (2019), whereas for HalfCheetah we employ SAC HHarroja et, al. (018)). They are state-of-the-arl Reinforcement Learning algorithms merging Q-Learning with policy-optimization, used following the d4rl implementation by Seno &
Imai 2022).
In the end, the decision made in a given state is attributed to trajectory cluster.
Evaluation of new policies and actions is done through 5 metrics:
Inâtial State Value Estimate (ISV) ((V(s)))
This metric is measuring the expected long-term returns in evaluating oflline Reinforcement Learning training. The higher the ISV value, the better our trained policy.
Local Mean Absolute Action Value Difference a Estimating how much the original policy differs from the new calculated one. desirable.
Aclion Contrast Measure (Toori))) # [s)))
Quantifying the disparity between recommended actions coming from the new explanation policies and original actions. Higher values are associated with better policies.
Wasserstein distance Waiet dz)
It measures the distance over a metric space between the original data embedding set and the complementary data embedding sets.
Cluster attribution frequency P(Crnal = c2)
Computes the probability of a cluster being responsible one for an RL decision.
A low Wasserstein distance and high Action Contrast Measure values correspond with higher frequency attribution.
3.4 Hypee--arameters
In order to reproduce the experiments of the paper we strictly used, when available, the same hyperpa- rameters used by the authors. This was the case for Grid- World. Regarding Seaquest, Breakout, Q* Bert and HalfCheetah we developed the code from scratch. Hence, we cannot be certain about the exact hyper- parameters used by the authors. In all instances, we retained the default settings provided by the libraries. If certain essential values were absent, chose those that aligned with the settings used in Grid- World.
4

---

Published in Transactions on Machine Learning Research (05/2024)
3.5 Experimental Set up and Code
Our experimental setup follows the approach of Deshmukh et al. (2023) in proving their claims. For claims Clustering High Level Behaviour and Distant Trajectories inffuence Decisions of the Agent we visually inspect the trajectories by plotting them, together with additional experiments. Removing Trajectriis induces a lower IVS claim is carefully taken care of by inspecting the 5 different metrics previously introduced. Human Study claim verified by replicating the analogous human study.
4 Results
In order to verify the previously stated claims, we proceed with an empirical evaluation, aiming to reproduce the results obtained by the authors. The result from Grid-Word presents some variance with respect to the ones reproduced in this article. Concerning the four other environments, they lack reproducibility due total absence of original code. Additionally, we further investigate each claim by conducting additional experiments. Most of these are done using the proposed number of trajectories by Deshmukh et al. (2023) (60 in Grid- World 7x7, 717 in Seaquest, 1000 in HalfCheetah), unless specified otherwise.
4.1 Removing Trajectories induces a lower Initial State Value
Reproducibility: In the Grid- World environment, the authors introduced different metrics to show that trajectorics play an important role in obtaining high-quality policies. The values obtained by the authors are reported in Table of the original paper. We present our findings in Table ! providing evidence for the claim of the authors. The results are in general reproducible, with small variation. The original policy, trained on all the trajectories, achieves the highest ISV among all. We report the reproduced results for the Seaquest environment in Table 2 We observe that the results are not similar to the ones in the original papers. This discrepancy could be altributed to several factors. The setup process, involving the installation of numerous packages and the use of outdated libraries, likely introduced minor computational variances. Additionally, changes in game versions, such as upgrading Seaquest from v4 to v5, affected the game-play dynamics, increasing the available action space. The choice of the difficulty level of the game also influenced the dataset, as easier versions had shorter trajectories due quicker game terminations. Another motive for this is given by the limited amount of training carried for agent. In fact, given computational limitations, we are not able to train for long horizon of time. Moreover, the settings of the experiments from the authors are unclear. There is no notion or explanation for what it means to train an agent until saturation, and no further details on the hyperparametors of the experiments (for additional details, see Appendix Dj. On the other hand, even given the differences and limitations explained above, the Claim Removing Trajeclories induces a lower Initial State Value still holds. The policy trained on the whole dataset achieves a higher ISV than any of the other ones. At the same time, the other metrics are consistent in terms of conclusions we can draw. Summarizing, the difference in reproducibility is due to the limited details given by the authors, together with the limitation on our computational resources. Results for HalfCheetah are reported in Table 6
We perform a further analysis to highlight the reproducibility of the results. To have a better comparison of our results with the one obtained by the authors, we report Table 3 the average values for the metrics we are investigating. We report an average across the clusters. We found (minimal) differences in our results. The ISV for the original policy coincides to one of the original paper. We neglect the last column since all probabilities sum to one. This further supports the claim of the authors.
Additional Experiments: We further analyze if trajectories are important to obtain a good ISV. Specifi- cally, we see if the clusters more often present in the altribution set hold a larger importance in influencing the ISV. Figure 2 shows an inverse correlation between the number times a cluster was responsible for a change in the decision and the ISV of the policy trained without that cluster. This further validates claim Removing Trajectories induces a lower Initial State Value, providing additional insights. Extra details can be found in Appendix C.4
5

---

Published in Transactions on Machine Learning Research (05/2024)
(su)
E
AQ
Morls (s)
E [ (s) # xz (s))
Waist d,)
P(CAnnal cj)
orig
0.307 = 2e-04
0.305 + 9e- 12
0.011 + 0.001
0.041 = 0.047
0.241 = 0.227
0.000 = 0.000
0.303 + 2c-03
0.007 + 0.011
0.041 = 0.042
0.371 = 0.348
0.025 = 0.050
0.304 - 0e-+00
0.002 + 0.001
0.122 = 0.049
0.924 + 0.150
0.0000 + 0.000
0.304 + 0e++00
0.022 + 0.001
0.0000 + 0.049
0.111 = 0.094
0.0000 + 0.000
4
0.305 + 1e-11
0.041 + 0.004
0.122 = 0.049
0.142 = 0.096
0.0000 + 0.000
0.300 + 3e-03
0.029 + 0.005
0.041 = 0.055
0.036 = 0.017
0.175 = 0.231
0.287 + 5e-03
0.061 + 0.015
0.163 + 0.049
0.040 + 0.051
0.500 +: 0.237
0.301 + 7e-03
0.030 + 0.016
0.020 = 0.059
0.099 = 0.178
0.150 = 0.183
0.305 + 6e- 13
0.008 + 0.010
0.021 + 0.049
0.022 = 0.026
0.100 = 0.200
0.304 + le-11
0.017 + 0.011
0.143 + 0.043
0.061 + 0.095
0.05 + 0.061
Table 1: Quantitative Analysis and reproducibility study of Claim Removing Trajectories in-
duces a lower Initial State Value for Grid- World. Results show the mean + standard deviation over 5 different seeds.
EsV(s
E
A Q Torix
[8)
[[
(8) TT (8))
Waist (d, d,)
IP(Cinal Cj)
orig
3.569 + 1e-04
0
3.537 + 2c-02
0.217 + 0.020
0.125 = 0.050
0.008 - 0.005
0.050 = 0.000
2.679 + 7e-9
0.869 + 0.002
0.375 + 0.280
0.933 + 0.130
0.000 = 0.000
2
2.858 + 30-01
0.880 + 0.020
0.000 = 0.000
0.071 + 0.010
0.025 = 0.050
3
3.061 + 1c-02
0.625 + 0.003
0.125 = 0.163
0.190 = 0.090
0.000 = 0.000
4
2.310 + 0e-+00
1.700 + 0.044
0.000 = 0.000
0.005 + 0.001
0.175 = 0.231
5
2.701 + 5c-06
0.650 + 0.051
0.125 = 0.183
0.002 - 0.040
0.600 + 0.256
6
2.010 + 4e-02
1.697 = 0.016
0.125 = 0.163
0.010 = 0.020
0.000 = 0.000
7
2.664 + 7e-03
1.018 + 0.001
0.000 = 0.000
0.001 + 0.00
0.150 0.100
Table 2: Quantitative Analysis and reproducibility study of Claim Removing Trajectories in-
duces a lower Initial State Value for Seaquest. Results show the mean + standard deviation over 5 different seeds.
Additionally, we report the ISV tables for the Bert and Breakout games in Appendix section B.1 Tables andd] For Q*Bert, the highest ISV value assigned to cluster 1, which invalidates the initial claim. On a brighter note, for Breakout, the claim holds. In the following three experiments, due to the lack of evidence for the other games, the experiments were discontinued for those two games. However, we report the shapes of the clusters in Appendix section B.3 for the sake of completeness.
4.2 Cluster High-Level Behaviours
Reproducibility: In Grid- World, this claim can be verified by either observing their shared high-level behavioural patterns or by using some quantitative metric. We deemed the latter to be a more appropriate approach. Thus, we proceeded to define this starting from inspecting trajectories belonging to that cluster and calculating the percentage of such manifesting certain pattern. We show one trajectory for each of the three analyzed clusters in Figure 3. The following high-level behaviours are retrieved: 'Achieving Goal in Top right corner', "Mid-grid journey to goal' and 'Falling into lava'. A practical explanation of this self-defined metric can be done analyzing the trajectory behavior of 'Falling into lava'. This is spotted by looking for a -1 reward value in the last but one state, and then calculating the percentage of trajectories that have this wanted characteristic within each cluster. We repeat this for every cluster and pick those with a percentage value greater then 90%. The procedure then iterated for the other two categories, by changing the charactoristics to look for. In the 'Achieving Goal in Top right corner' we look for a +1 reward in the last but one state and going towards position Whereas in the 'Mid-grid Journey to gool' category look for trajectories starting in the middle of the grid and having positive reward in the last but

---

Published in Transactions on Machine Learning Research (05/2024)
(su)
E AQ Karic (s)]
E (Torig
* (
Wdise (d. d,)
Mean Clusters (Original Paper)
0.3027
0.0231
0.0821
0.0301
Mean Clusters Reproduced)
0.3029
0.0230
0.0714
0.1098
0.0002
0.0001
0.0107
0.0797
Table 3: Additional Quantitative Analysis on claim Removing Trajectories induces lower Initial State Value for Grid- World.
Conteing
Figure 2: Correlation between Action Value and the Cluster Attribution Frequency. (i) The plot obtained using the DBSCAN algorithms shows (weak) correlation of the action value with the attribution frequency of a cluster. We clearly observe that Cluster 1, which was the one attributed more often, is of crucial importance. (ii) The plot obtained using XMeans clearly shows the phenomena of Claim 2. There is a clear negative correlation between the two quantities, which highlights the importance of data trajectories. Again, the cluster attributed most agent docisions, i.c. Cluster 7, constitutes a fundamental portion of the training data that leads to high-value policy.
state. These align with the behaviours found by the authors. The claim is thus supported. Note that cluster labels vary from those highlighted by the authors.
6
Geal right corner Cluster
Mid-Grid
Figure 3: Reproducing and verifying claim Cluster High- Behaviours in Grid- World. Clus- ter showcases the presence of behaviour 'Achieving Goal in top right corner'. Cluster of 'Mid-grid journey to goal' and cluster 2 of 'Falling into lava?. Three High-Level Behaviours found match those highlighted by the authors.
In Seaquest analysis, tried to replicate the cluster findings from the original study in Figure 1 We noticed differences in the number of data points and their distribution. Converting 717 trajectories into around 24,000 sub-trajectories for the XMeans algorithu revealed more data points than shown in the original graph of the authors. This discrepancy could be due to two reasons: (i) the choice game mode and data source might affect the length of observations, which was detailed by the original authors, and (ii) the authors might have used more complex method aggregate data post-encoding than the simple averaging they described.

---

Published in Transactions on Machine Learning Research (05/2024)
2: 81 2
000
a. Author Seaquest
b. Our Seaquest
G. Author HalfCheetah
d. Our HalfCheetah
Figure 4: Clustering differences in Seaquest and HalfCheetah: This figure contrasts the clustering outcomes between our study and the original paper. Figure (a) and (c) illustrate the clusters of the authors for Seaquest and HalfCheetah, while figure (b) and (d) reflect our observations, revealing significant differences in distribution and amount of data points. These discrepancies may highlight the influence of game mode choices, dataset specifics, and data aggregation techniques on clustering outcomes.
Additionally, when trying to interpret the high-level meaning of those clusters, we obtained some discrep- ancies. Results in Figure show strong link between the 'Filling Orygen' behavior and cluster 7, while the other behaviors remained unclear, questioning the specific claims of the authors. However, this does not undermine the broader notion of ''meaningfll clusters', but, given the scope of this paper, finding those interpretations was deemed excessively time-consuming and beyond our current objectives. More details can be found in Appendix B.2
the context of the HalfCheetah environment, assessing the significance of clusters proved challenging due the high- digh-dimmesional and intricate nature of the observation space.
(a) Average time spent by each cluster filling their oxygen tanks
(b) Average number of submarine de- (c) Average surface combat encounters structions per cluster
per cluster
Figure 5: This figure evaluates high-level player behaviours in the clusters of the Seaquest game. Sub-figure (a) shows the average time spent on filling oxygen, sub-figure (b) details the average submarine explosion, and sub-figure (c) counts surface combat encounters. These insights collectively enhance our understanding of the meaning within each cluster.
Additional Experiments: The three initial patterns explained in 4.2 are found both when using XMeans and DBSCAN and also when training with a higher number trajectories (250). In this section, we investigate whether other meaningful high-level behaviours exist. We successfully identified an additional pattern. That is, each trajectory belonging the same cluster has the same length. However, this last behaviour emerges only when using 60 trajectorics with both XMeens and DBSCAN. It is not found when the number of trajectories used increases to 250. This phenomenon may arise due to the increased granularily of each cluster, a scale varying with the number of trajectories. Namely, by keeping the number of clusters fixed, less granularity is obtained. Thus each cluster can obtain worse cluster representations by grouping a larger number of trajectories together. However, this does not negate the claim of the authors because the original behaviours are also spotted with a higher number of trajectories.
4.3 Distant Trajectories influence Decisions of the Agents
Reproducibility: We start by analyzing claim Distant Trajectories influence Decisions of the Agents in the discrete Grid- World environment. The authors perform a qualitative analysis to support their claim. We try reproduce the results and the plots given the code provided by the authors. The hyperparameters are set equal to the default values. The results of Figure 2 in the paper by Deshmukh al. (2023) reproducible
8

---

Published in Transactions on Machine Learning Research (05/2024)
using the code given by the authors. Note that it may take more than attempt to reproduce these results. This is due to the possible variation of clusters from each iteration of the code. Nevertheless, we found that these results were easily reproducible with little effort. Results are shown in Figure The trajectories (i).(i), (iii) are equivalent to the oncs indicated in the paper by the authors. We plot an additional trajectory which is part of the attributed cluster. are considering. This investigation confirms the claim of the paper. However, given the highly qualitative justification provided by the authors, we seek a more structured and quantitative way of analyzing this claim. We defer these experiments to Section
important stress that also (iv) is distant from the state (1, 1) we
and la Cown
Papelirg
X2
FA
TH
Figure 6: Plot of grid and trajectories reproducing results and verifying Claim DTAG. The original optimal action is 'right' in state (1,1). When removing the trajectories belonging to the attributed clusters, all decisions are equally optimal, i.c. 'right', 'left', up', or 'down'. This decision is attributed to 8 different trajectories, of which 4 plotted here above.
Additional Experiments: Reprodueibility section highlighted how trajectories far from our state of interest can influence the decision of our agent. However, it is not explicitly clear to what extent this is true. We strive to perform more rigorous analysis, considering each state with attributed trajectories responsible for a decision change. For each of these attribution sets, we compute the average distance from the state to its trajectories. A rigorous definition of how we calculate the average and the distances can be found in Appendix C.3 together with a detailed pseudo-code in In our experiments, two different cluster algorithms are employed. For DBSCAN we set = 2.04. Ten final clusters are obtainod. No seed is needed given the deterministic nature of DBSCAN. The other cluster method is XMeans. The seed is set to 0 and 99 respectively for the initialization of the centers and for the XMeans algorithm. We perform the Grid- World Four Room Environment introduced by Sutton et al. (1999). Its size is 11x11. Given the larger grid and the scope of our experiment, we generate higher number of trajectories. Namely, we produce 250 trajectories that end in a positive terminal and 50 trajectories that achieve a negative reward.
experiments
Results: Figure 7 shows the Bar-plot these trajectories. We can observe that the average lengths are mostly larger than 6 in both scttings. Intorestingly, we note that using the XMeans algorithm we have no state which is explained only by trajectories passing through it. From the results of the plot, we conclude that indeed distant trajectories are important to explain the decision of RL agent, further confirming claim Distant Trojectories influence Decisions of the Agents.
4.4 Human Study
In order to verify this claim we reproduce the experiments as well the study setup of the authors. Deshmukh et al. (2023) study is conducted on 10 people, which may not itself be sufficient to support the claim. In trying to improve this, we doubled the interviewees lo 20 people, each of whom first. received an explanation of how Grid- World navigation works. Following this explanation, they all gained a full understanding of the navigation process. 40% are university graduates mathematics and computer science. 45% are student graduates in engineering. The remaining 5% come from different study backgrounds. We begin by showing two Attributed Trajectories (Attr Iraj and Allr traj 2), one Random Trajectory, and

---

Published in Transactions on Machine Learning Research (05/2024)
DéSoi Quater'
Kmram Clanering
Figure 7: Distances between states and their attributed trajectorics. On the x-axis, we have bins, respectively in the range 0-3, 3-6, 6-9, and 94. (i) shows the results using the DBSCAN algorithm. We see that we have many trajectories with a length bigger than 9. The states that fell in the blue bin were explained only by 1 or trajectories. This is due to how DBSCAN constructs the clusters, which favors big variance in size wet them. We had then bigger chance of having clusters with few Irajectories that were all passing through an attributed state. (ii) shows the results using the XMeans algorithm. We note again that the majority of the states had large average distance to their attributed trajectories.
Trajectory belonging to Alternate cluster for each state. We investigate two questions. (i) Which single trajectory do you believe best explains the action suggested? (ii) Can you point out all the trajectorios you believe explain the action suggested?
Averaging between both states, the results on Question (i) highlight how almost ~ 72.5% the interviewees correctly identify one of the two attributed trajectorics. For Question (ii), Figure S shows that in 63% of the cases, humans can correctly identify all the attributed trajectories. The results obtained are similar to the ones of the authors. However, given the very limited sample size of our experiments, do not have enough evidence to support the claim.
State
200
12.-
300
A3h13]] Amrro
Abtras
Asrra]
Arrma
Abmma
Figure 8: Human Study. The plot represents human answers to the two questions introduced. Attr traj 1 and Attr traj 2 are trajectories belonging to the cluster attributed for the decision of the agent in the specific state. Random stands for a randomly selected trajectory from the whole set. Whereas Alternate a randomly selocted trajectory from the whole set without those belonging the attributed cluster. The results presented are for states (1,1) and (5,2). In both states we notice a decent level of human understanding, This suggests meaningful understanding of which trajectories influence the agent's 's decision decision-making process.
4.5 Results beyond original paper
In this section we go beyond the author claims and try to experiment with the authors design choices like the employed encoder method as well as cluster algorithm.
Improving clustering algorithm: XMeans has proven to be useful in determining almost accurately the correct clustering trajectorics, we propose different approach by using DBSCAN algorithm. Introducod in Ester al. (1996). DBSCAN is nonpaarmit density- clustering method that groups together sets of points packed together. The density-based characteristic of DBSCAN, differently to X-Means, is
10

---

Published in Transactions on Machine Learning Research (05/2024)
not relying on the assumption that clusters are spherical-shape. It is rather able to find clusters with any kind of variance between the points. This new algorithm could lead to new clusters, and possibly different insights from which we may be able to extract new information. Results: Figure 9 shows a better visual clusters representation computed by DBSCAN. This outcome is particularly visible in the reduced amount of overlaps which are instead present in XMeans. Additional metric results are available Appendix C
020
Clustering through XMaars
Clustaring through DBSCAN
Figure 9: Clustering Methods: XMeans vs DBSCAN. Through DBSCAN obtain a lower number of clusters which eliminates overlaps between XMeans clusters 2, 3 and 0, 1. This beller visual represenlation is mainly due to difference in the algorithmic process.
Different encoder techniques: In Grid- World Environment the T.STM-based Seq2Seq encoding used by the authors has proven to efficient. However, in this section we set out to experiment with different encoding techniques. Our hope is that they could provide a better hidden representation for trajectories. We employed two kinds of pre-trained encoders: [Trajectory Transtormer originally proposed in Janner et al.) (2021]. The model takes input data in the form (state, action, reward) matching perfectly with the one provided by the authors. BERT base model Devlin et al. 2201)) Pre-trained using Masked-Language Modelling, its capability of being adapted to many scenarios made it good candidate to replace the LSTM in the paper. Results: Experiments are performed over 250 trajectories. We defer the table of results to I0] as we obtain no notable increase in performance across all metrics. Additionally, an inspection of high-leve! behaviors of clusters, as section 4.2 highlights similar results.
5
Discussion
Across this work, we performed several experiments aimed reproducing key findings of Deshmukh et al. The outcomes of this reprodueibility study partially confirm their claims.
2023
WorddWorld
Seaquest
Half-Cheetah
Breakout
Q*Bert
Removing trajectories
Cluster behaviours
x
?
Distant trajectories
?
7
Human study
?
?
?
Reproduced Results per Game for Each Claim. /represents validated results, a Xdenotes an invalidated statement. for the specific game, and ?indicates that we cannot confirm or deny the claim of the authors for this specific game. This may arise from time constraints or because the claim itself lacks sufficient precision, making it impossible to definitively confirm refute even with additional experimentation.
Table 4: Summary:
Reproducibility: As reported in table 4 we can sustain claim Removing Trajectories induces lower Initial State Value for Gorl-World, Seaquest and Breakoul. Our results for HalfCheelah and B do not sustain this claim. The claim Cluster High-Level Behaviours is accepted only in Grid- World. In Seaquest we identified only one out of three high-level behaviors. This is not enough able to obtain any high-level behaviour for HalfCheetah, Breakout and Q Bert.. On claim Distont Trajectories in influence Decisions of the Agents, obtained results consistent with the original ones, supporting the claim
support the claim. We were not
11

---

Published in Transactions on Machine Learning Research (05/2024)
for Grid-World. Our additional experiments further show that distant trajectories have significant impact on the decision of the agent. The Human Study was carried out only in Grid- World. While we obtained similar results, the claim was superficially investigated by the authors. Their original experiments were not sufficient to support Human Study. Our survey, although more extensive, was limited due to time and resources constraints. As a consequence, we can not confirm Human Sludy.
Code: Across environments, reproducibility varies significantly due to code availability. The Grid- World code provided by the authors lacked proper sceding mechanism. Despite this, similar reproducibility was not highly jeopardized. All of our experiments are completely reproducible. First, in Seaquest and HalfCheetah the code was unavailable. Relevant implementation details, hyperparameters, and training techniques were not mentioned in the paper. We coded everything from scratch. This includes the whole trajectory attribu- tion process, environment requirements, library dependencies, training and evaluation loops, and many more. In spite of these difficulties, we were able to perform the original experiments, supporting most of the claims of the authors when obtaining relevant evidence. We provide our complete code implementation. Additional training and implementation details can be found in Appendix D The complete code implementation is available in our GitHub Repository
What was easy: The authors shared the code for the Grid- World environment directly with us, although it is not publicly available online. This facilitated our ability to try to reproduce some of the results of the paper for this environment. In addition, the code provided was relatively easy to understand and adapt in order to expand our reproducibility study further. This allowed us to extensively produce additional experiments.
What was difficult: As mentioned throughout previous sections of the paper, the implementations for the Seaquest and HallCheetah environments were not provided to us, thus leading to our implementation from scratch. Although very relevant to the results, the paper only briefly mentions Additional Training Details in the appendix, lacking any other explanation about the Python environment being used, any data pre- processing stage, or tweaks required for compatibility. With regards to the environment issues, it is worth noting that, being the field very novel and active, some of the libraries implemented do not get developed any longer. This may be the case of ''mujoco-yy', , the library on which HalfChetaah relies on: because of this, we had to backtrack compatibility between dependencies of librarics that are in constant development and some whose support has been interrupted for years. On lop of this, we also encountered compatibility issues between different Operative Systems: MacBooks produced post-2020 abandoned Intel-based processors to adopt natively built Apple Silicon processors. This required the community to rebuild packages to support a new architecture (arm64 vs x86/x64), in order for these libraries to be able to run on these devices. Co-occurrently, support for Mujoco-py has been halted, thus requiring a somewhat, convoluted installation procedure, consisting of manual pointers definition, third-party compilers requirement and many other details we decided not to bother the reader with. Needless to say, the lack of code in this delicate instance, required extensive trial and error experimentation with dependencies and installation procedures, thus slowing down the overall process for housckceping operations rather than actual development. We believe the lack of transparency with regards to the aforementioned environments led to such striking differences in absolute value results between our reproduced results and the original ones: again, details on the training procedure and model implementation should have been made public, given the rather complex nature of the task at hand, along with hyporparameters sctup, which we go over in further detail in Appendix D
Communication with the authors: Communication with the authors was carried out by the course coordinators themselves at the beginning of the course. We received the aforementioned subset of the original implementation halfway through the course, thus leading to the issues discussed in the previous section. No further communication with the authors has been conducted by us.
Key takeaways and original paper limitations: In our investigation we find that the proposed method- ology is of questionable effect, as is not yet generalizable to many environments. Another key limitation is allowing only one cluster per attribution. We believe that allowing the method to consider trajectories from more then one cluster could lcad to more comprchensive analysis. Nonetheless we believe that this work Deshmukk et al. (2023) and its proposed altribution process have the possibility of unlocking new streams of research into a better understanding of Reinforcement Learning systems.
12

---

Published in Transactions on Machine Learning Research (05/2024)
References
Petros Christodoulou. Soft actor-critic for discrete action settings, 2019.
Youri Coppens, Kyriakos Efthymiadis, Tom Lenaerts, Ann Nowé, Tim Miller, Rosina Weber, and Daniele Magazzeni. Distilling deep reinforcement learning policies in soft decision trees. In Proceedings of the IJCAI 2019 workshop explainable artificial intelligence, pp. 1-6, 2019.
Shripad Vilasrao Deshmukh, Arpan Dasgupta, Balaji Krishnamurthy, Nan Jiang, Chirag Agarwal, Georgios Theocharous, and Jayakumar Subramanian. Explaining rl decisions with trajectories, 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
Martin Ester, Hlans-Peeer Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pp. 226222262222222 231, 1996.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data- driven reinforcement learning, 2020.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data- driven reinforcement learning, 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement lcarning with a stochastic actor. CoRR, abs/1801.01290, 2018. URL http:
//arxiv. /larxiv.or//abeel wwt./o/////../////..1//1///////. 11/./1011..00 01290
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Pi- cus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, (35777882)377778 362-362, September 2020. doi: 14.1088//416886-0002266666 URL. https: //doi org/10.1111 1038/ 000115666--20----22
Rahul Tyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia Sycara. Transparency and explanation in deep reinforcement learning neural networks. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Sociely, pp. 144-150, 2018.
Michael Janner, Qiyang Li, and Sergey Levine. Offiine reinforcement learning as one big sequence modeling problem. In Advances Neural Information Processing Systems, 2021.
Ezgi Korkmaz. Investigating vulnerabilities of deep neural policies. In Cassio de Campos and Marloes H. Maathuis (eds.), Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pp. 1661 1670. PMLR, 27 30 Jul 2021. URL
https://prroceeingg mlr.
html
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative qleamiing for offline reinforce- ment learning. CoRR, abs/2006.04779, 2020. URL
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR, abs/2005.01644 2020. URL https::/arxiv..t http:/awwwiv.orraa/////aa
01643
Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learning through causal lens. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 2493 2500, 2020.
Andrei V. Novikov. Pyclustering: Data mining library. Journal of Open Source Software, 4(36):1230 2019. 01230
doi: 10.21 00.21105/joss.01230 URL https:///ot:iit..
13

---

Published in Transactions on Machine Learning Research (05/2024)
Nick Pawlowski, Daniel Coelho de Castro, and Ben Glocker. Deep structural causal models for tractable counterfactual inference. Advances in Neural Information Processing Systems, 33:857- -697, 2020.
Dan Pelleg, Andrew W Moore, et al. X-means: Extending k-means with efficient estimation of the number of clusters. In Teml, volume 1, pp. 727-734, 2000.
Erika Puintta and Fric M. S. P. Veith. Explainable reinforcement learning: A survey. CoRR, abs/2005.06247, 2020. URL https://arxiv.. wwtpsiii.org//bbba wtt./os////a.....oee 06247
Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishnamurthy, and Sameer Singh. Explain your move: Understanding agent actions using specific and relevant feature allribution. arXiv preprinl 1912 12191, 2019.
Matthias C Rillig, Marlene Agerstrand, Mohan Bi, Kenneth A Gould, and Uli Sauerland. Risks and benefits of large language models for the environment. Envirommental Science & Technology, 4444)33444-3 3464-34666 2023.
Takuma Seno and Michita Imai. d3rlpy: An offline deep reinforcement learning library, 2022.
Richard $ Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990, pp. 216 224. Elsevier, 1990.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard $ Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181 211, 1999.
14

---

Published in Transactions on Machine Learning Research (05/2024)
A Methodology
A.1 Model description
Dyna-Q Algorithm
Introduced in [Sutton (1990). Dyna-Q Q is a Model based Reinforcement Learning Algorithm. Conceptually it is an algorithm that illustrates how real and simulated experience can be combined in building a policy, Dyna-Q algorithm (10) introduced in Section 3.3 starts by initializing so-called table. A table made of all possible states vs all possible actions. The model also contains slate, action, nexl stale, and reward tuples. This way the model can be both improved and queried to get the next state in the planning part. The process begins by obscrving state S (a), and then selecting the next action A, in a greedy manner (b). After taking the action A, we observe a reward R and a state S'. These two values are then used in the formula in (d) to update the Q table cell corresponding with state s and action a. After these classic Q-Learning steps we perform a loop (f) which consists of the added Dyna-Q part. First, we randomly select a state S and an action A and then we deduce new state S' and new reward R which will then be used to update the Q-table as before.
Tabular Dyna-Q
Initialize Q(x.a) and Modell(s, for all s and A(s()
Loop forever:
(a) S +- current (nonterminal) state
(b) A E-greedy(S, Q)
(c) Take uction A; observe resultant reward, R. and state, S'
(d) Q(S, A) < Q(S. A) 1 a[R max, Q(5", e) Q)5,4)]
(e) M (Sdel(S,A)) A) +- R. (assuming deterministic euviromment)
(f) Loop repest n times:
S random previously observed state
A - random action previously taken in S
R, S' + oddel A)
Q(S, + Q(S, A) + a [8 + max, QiS" a) Q(5
Figure 10: Dyna-Q algorithm
A.2 Computational Requirements & Environmental Impacts
The experiments were performed using a MacBook with Apple M2 Pro silicon chip with 10 CPU cores (1), MacBook with Apple M1 silicon chip with 8 CPU cores (2), and a Microsoft Windows 11 Pro with Intel(R) Core(TM) 17-10710U with 6 CPU cores (3). Most of the experiments in Grid- World run easily and in scconds on our local machines. On the other hand the running time for Seaquest and HalfCheetah with machine (1), can vary between 50 minutes (with 10 steps per epoch) and hours (with 100 steps per epoch). We employed pre-trained models both for Seaquest and HalfChestah as explained in Sections 3.3 and 3.2. The same is done for some additional experiments on Grid- World (Section C.2 Discussion on the environmental impact of these models has been addressed in previous literature (Rillig et al. (2023)). In addressing our own ecological footprint, we used the Code Carbon Tool (on machine (1)) to estimate our total energy consumption in obtaining cluster attributions. In Grid- World the estimated consumption approximately 0.000170 kWh of electricity. Whereas in Seaquest we 0.005133 kWh of electricity. Similarly, in HalfCheelah we consume 0.004783 kWh. We then use the CO2e equation obtain the corresponding CO2 emissions. The formula
COze= = P.E. P.t
is comprised of CI Carbon Intensity (fixed value of 0.954), PUE Power Usage Effectiveness (also fixes 1.58), P Power required (estimated through Code Carbon Tool) and the training time t in hours. Our final emissions available in Table 5
15

---

Published in Transactions on Machine Learning Research (05/2024)
Grid- World
Seaquest
HatCChostah
CO2e lbs (10 steps per epoch] COye stops per opoch)
0,0000021
0,0064470
0,0080070
0.0576760
Table 5: Emission levels in training
models from top to bottom. The results highlight our CO2 equivalent levels in training the models. Seaquest and HalfCheetah were both trained using two different number of iterations per epoch. However, Grid- World was not trained with a different configuration as a good performance was attained when using the standard
employed by the authors.
B Results reproducing original paper and verification of the claims
B.1 Removing trajectories induces lower ISV
Despite the challenges in replicating the exact clustering outcomes for Half Cheetoh as highlighted earlier, reveals some interesting patterns. The table provides quantitative analysis that, despite reproducibility
issues, still shows consistent trends across different metries.
[*o]
E AQuang (8)
1 (orrig (#) [y(s))
Wass dy)
F(Crnal cy)
orig
3.3615
3.4558
0.0942
0.0038
1.0000
0.1000
3.4691
0.0028
0,3000
3,2958
0.0656
0.0035
0.8730
0.0000
3
3.3621
0.0006
0.0017
0.8483
0.0000
3.3621
0.0009
0.0022
0.2986
0,6000
3.5280
0.1665
0.0041
0.5340
0.0000
6
3.3444
0.0170
0.0016
0.5245
0.0000
3,3206
0.0408
0.0052
0.6162
0.0000
3,3745
0.0602
0.0028
0.0000
3,3826
0.0337
0,0013
0.7855
0,0000
Table 6: Quantitative Analysis and reproducibility study of Claim Removing Trajectories in- duces a lower Inétial State Value for HalfCheetah
(soo)
E AQ
[+)
1 (8) * (s()))
Wdist (d, dz)
P(Canal = Cy)
orig
1.075
0.962
0.116
1.000
0.333
0.000
1.118
0.045
0.000
0.501
0.933
1.104
0.031
0.000
1.000
0000
1.006
0.0945
0.933
0.158
0.067
1.059
0.029
0.986
0.167
0.000
Table 7: Quantitative Analysis and reproducibility study of Claim Removing Trajectories in- duces a lower Inétial State Value for Q *bert.
(soo)
E AQTami
[s)
E 1
(s) F (s))
Waist (d, dz)
P(Cinal C2)
orig
0.812
0.797
0.018
0.512
0.579
0.000
0.811
0.001
0.000
1.000
0.000
0.798
0.013
0.000
0.954
0.000
0.764
0.048
1.000
0.159
0.000
0.791
0.021
0.000
0.254
1.000
Table 8:
Quantitative Analysis and reproducibility study of Claim Removing Trajectories in- duces a lower Initial State Value for Breakout.
16

---

Published in Transactions on Machine Learning Research (05/2024)
B.2 Meaningful Clusters
In order to oblain Figure 5 specifically analyzed the oxygen lank indicator at the bottom of the screen, recognizable by its unique color. An increase in the bar is interpreted as the submarine refilling its oxygen tank, while a empty tank resulted in a submarinc explosion. This might occur cither from running out of oxygen or sustaining damage from enemies. For the 'Fighting with head out" behavior, we monitored the position of the submarine within the top 30 pixels of the screen. We would consider engaging in surface combat it remained in this for more than out of the 30 frames in a sub-trajectory
The analysis highlights distinct behaviors: Cluster 7 is linked to 'Filling Oxygen', and Clusters 2 and 3 to 'Submarine Burst', with no clear trend for Fiihhiin with Head Out. 11 shows overlapping behaviors across clusters, complicating the attribution of specific actions. Consequently, the most representative cluster appears to be the one of refueling oxygen: two frames depict the submarine at the surface (directly implying oxygen refueling), while the remaining three suggest imminent game resets, indirectly associated with oxygen refill.
(a) Filling Oxygen Cluster 9
(b) Submarine Bursts
Cluster
(c) Fighting with Head Cluster
Figure 11: Reproducing Figure of the authors paper. High-level Behaviours found in clusters for Seaquest formed using trajectory embeddings produced using decision transformer. The figure shows 3 example high-level behaviours along with the action description and id of the cluster representing such behaviour
B.3 Clusters for Bert and Breakout
Displayed in Figure 12 are the clusters for Q-Bert and Breakout. The choice of five clusters was made for both games, as they independently feature significantly more rewards within the trajectories as well as fewer action states (only 6 and 4, respectively), compared to their Seaquest counterpart. Consequently, opted
17

---

Published in Transactions on Machine Learning Research (05/2024)
to reduce the length of the sub-trajectories to 15 and to decrease the number of clusters to account for these differences.
Trajectory Lmbeddiing for initial centers
Fejectory Embedicings inttill centers
Teature
-2-
(a) Clusters Qert
(b) Cluster Breakout
Figure 12: Clusters for the new games Qr and Breakout
C Results beyond original paper
C.1 Improving clustering algorithm
Table is mirroring the results produced in Table [
In this section however produce metrics results
using DBSCAN clustering algorithm, instead XMeans. As introduced before the values are similar to those attained in the original table.
[so]]
AQ (e)
[x)))
Wawe(d,d)
P(ctnal c2)
orig
0.3059
0.3056
0.0018
0.0613
1.0000
0.0000
0.2980
0.0326
0. 0.1429
0.0009
0.1250
0.30415
0.0406
0.1225
0.0000
0.3058
0.0281
0.0000
0.00007
0.0000
0.3045
0.0026
0.1021
0.0003
0,5000
0.3045
0.0288
0.1225
0.0005
0.0000
0,2859
0,069
0.0817
0.3750
0.3058
0.0204
0.0205
0.0457
0,0000
Table 9: Quantitative Analysis of DBSCAN algorithm
C.2
Implementing different encoder techniques
(su)
AA (s)]]
E [ (s) 전 πT (s))
Waist Waise(d, d,)
Mean Clusters (Original Paper)
0.3451
0.0224
0.9035
0.1821
Mcan Clusters (Traj. Transformers)
0.3413
0.0325
0.039
0.0723
Mean Clusters (BERT)
0.3427
0.04074
0.8645
0.1098
Table 10: Quantitative comparison of LSTM, Bert and Trajectory Transformers
C.3 Are distant trajectories really important?
Distance State-Trajecto TrajeeTrajccory and its importance Let us formally define the following variables:
S: set containing all states with at least attributed trajectory
18

---

Published in Transactions on Machine Learning Research (05/2024)
Ty set of all the attributed trajectories for the state S
i i-th trajectory in the attribution set T.. Each trajectory i has length 4
a (b;c): distance from point to point in our grid. It calculated by implementing the A search
algorithm.
d((a,s)): distance from state 8 to trajectory tis- Mathematically, for each point E 1i,0
10
if 3 Py ti, s.t, Pj =s
d(tr,a:)) 8) = ( E a" (pj, 8) otherwise
d(Ta,8) 8): Average distance of the attribution set T3 from its respective state 8. We implement it as:
m
d(T., 8)
d(ln,a86)
T
Given the elements introduced above, we can calculate the average distance of a state from its attributed trajectories, denoted by s). Note that in the formulation above, if trajectory ti passes through the state s, we d(tr,,; s) O. This is design choice, which can be further justified. In fact, we are interested in considering 'far' only the trajectories where there is interaction with the state itself.
We designed and implemented Algorithm ! We provide a high-level pseudo-code for better understanding of the steps we perform.
Algorithm 1: Algorithm for calculating the average distance State Attributed Trajectories
Inputs: S, T, for s e
foreach E S do
D empty list
foreach li,a E T. do
5
= M- empty list
forcach Pj E a do point distance (p).
7
append point distance M end 10
9
end
11
m Average of the list M
12
Append to D
Set 0
end
return the list D. It contains the average distances of each state from its attributed trajectories.
We introduced clear metric, together with algorithm that provides details how compute it.
C.4 Are data trajectories important to obtain a good action value? Are some more important than others?
In this section, aim to provide further details experiments the assumptions of Claim Removing Trajectories induces lower Inátial State Value. Values from both the original paper and from Table that data trajectories are important to obtain good ISV for our state. We are interested to see if some of the clusters are more important than others in determining this value. The setting of the experiments is equivalent to the one introduced in Section 4.3
Figure shows that data trajectories play factor in obtaining a satisfying action value. For simplicity, the original policy is not plotted, but its value higher than any other policy in both cases. This again connects
19

---

Published in Transactions on Machine Learning Research (05/2024)
with and proves claim Removing Trajectories induces lower Initial State Value, even with higher grid size and number of trajectories. The plot illustrates the action value for each of the policies ..,.... Am illustrated in Figure ! For each cluster C₁ we show the action value obtained with the policy тy, plotted against the number of times this cluster C₁ has been the responsible cluster for a change in the decision of the agent (on the x-axis). In Figure we observe clear inverse correlation between the two. Clusters that. have been attributed more often to change in decision are important in obtaining a high ISV. In fact, keeping this data out of our policy training induces lower action value for our state. This shows that the higher the importance of the cluster, the higher the gap in performance. We believe this is interesting result, which further indicates the conceptual importance of the trajectory attribution method. In fact, it is not only a matter of the number of trajectories we train on. We found that specific clusters can hold larger weight in the decision of an agent. This suggests that some trajectories are more fundamental than others.
C.5 Additional Hyper-Paramen Parameters Experiments
In this section we investigate the change in metrics when hyperparameters regarding the agent training are changed. We play with the values of alpha, gamma and number of evaluation epochs. We proceed to generate offline data for each combination of the above mentioned hyperparameter and successively Irain the authors Seq2Seq model.
The best results in terms of loss value are shown in Table 11
However while we reach better loss results we
Table 11: Experimental Results
Alpha
Gamma
Eval. Epochs
Loss Value
0.1
0.95
15
0.0678
0.1
0.5
5
0.0965
0.1
0.5
10
0.0965
0.01
0.01
5
0.0369
0.001
0.1
15
0.0815
do not necessarily obtain better overall metrics.
D Training setting for Seaquest and HalfCheetah
As mentioned throughout the paper, we implemented from seratch the code for the Seaquest and HalfCheetah environments. Due to the lack of details provided in the original study, we provide our own setup for the training of the aforementioned environments.
For Seaguest, we train Discrete SAC model based on the original work by Christodoulou (2019). developed by [enoo & Imail (2022). Observations for the game were in the form of 84x84 greyscale frames, which we stacked, forming for each observation a 4x84x84 array. This allowed the model to incorporate some degree of temporal awareness, also referred to as context in the original work. Subsequently, in order to preserve spatial information, we implemented a custom encoder for the model, in the form a Convolutional Neural Network. We are not aware whether the authors pursued this approach their study, but due the nature of the data itself, we are sure this implementation helped the training and performance by a significant margin. Once more, due to the nature of the data (images), we implemented 'pixel' scaler for preprocessing purposes to act pixel value normalizer.
For HalfCheetah, on the other hand, we implemented SAC model based on the original work by Haarnoja et al. 22018.. We kept the standard pyperaaraeeterss provided in the documentation of the library by Seno & Imai (2022) One critical missing piece of information we extensively discussed upon implementation was given by the notion of difference in action between models: due to the continuous nature of the actions this environment, becomes almost impossible for two policies predict the same set of actions. For this purpose, decided to implement way of comparing actions based through nummpyiselose((.bb) by Harris
20

---

Published in Transactions on Machine Learning Research (05/2024)
al. 2020 The similarity formula given by the above method is
a bl S (abs. tolerance + rel. tolerance (b])
Unlike the built-in 'sclosisclose', the above equation is not symmetric in and b: it assumes b is the reference value that "isclose(a, b)' might be different from 'isclose(b, a). Furthermore, the default value of 'abs. tolerance' is not zero, and is used to determine what small values should be considered close to zero. Once again, we are not aware of what the authors did on this end, but we have reasons to believe this approach has grounds for a correct interpretation. We acknowledge potential scnsitivity to the results based on the choice of the hyperparamelers of the method, for which we kept the default ones.
Finally, although the authors mention a training schedule until saturation without further explanation, we followed the guidelines provided in the DR3RLpy framework outlined by Fu et al. (2021). training both our models for 10 epochs, each taking 100 steps. We are not aware of whether our approach matches the one suggested by the authors, but results, although not in absolute value, are relatively consistent with those provided in the original study. Thus we can conclude that our hyperparameters setting is sufficient for reproducing the results.
21


Reviews:
[{"review_id": "q4LQ0UK3Wo", "reviewer": "Reviewer_HzA9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- This study places a lot of effort in comprehensively reproducing the original Deshmukh et al, 2023, including running a human study. The authors of this submission clearly placed a lot of attention to faithfully reproduce every detail of the prior work.\n\n- The authors seek to improve the method of analysis for some of the phenomena of study, such as investigating the use of an alternative clustering algorithm, DBSCAN in place of X-Means, as well as seeking to understand how varying the amount of trajectories used in clustering impact the resulting clusters. Moreover, the authors also provide a more quantitative measure on which to base the finding on the idea that an agent's behavior can be attributed to distant states, by computing an average distance of each attributed state to the trajectory states in its attributed cluster.\n\n- The authors convincingly reproduce several of the empirical results attained by Deshmukh et al, 2023.\n\n### Weaknesses\n- It is not clear what can be learned from the observation that training on all trajectories leads to an improved policy. Can this be due to the higher-return trajectories being more useful for learning? Ablations on the specific data mixture would be more informative here.\n\n- While the authors make a commendable effort in describing exactly the differences in their reproducibility study from that of the original, it would benefit the paper to include a skimmable summary of these differences, as well as which empirical results were reproduced and which were not. \n\n- The biggest weakness is that the authors are not clear on the main takeaways from their work. Given the choice to reproduce any offline RL paper, why choose this one? \n\n- Related to the above, the authors should elaborate on the conclusions that can be drawn from the results of their reproducibility experiment. What are the limitations of this approach for trajectory attribution in general, e.g. when might there be false positives or false negatives in attribution, how might the constraint on single-cluster attribution limit the nature of the resulting attributions? What are the possibilities opened up by such interpretability methods, should they be consistently reproducible?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents a reproduction study of the paper \"Explaining RL Decisions with Trajectories\" (Deshmukh et al, 2023). The original study seeks to attribute the policy decisions of train offline RL policies to specific data subsets. These subsets are computed by clustering the trajectory embeddings—generated by pretrained encoders—and training explanation policies on each cluster, defined as the policy trained on the complementary dataset w.r.t. to each cluster. Given an offline RL policy trained on the full dataset, its action choice at a given state is then attributed to a specific cluster by finding the cluster corresponding to the explanation policy whose own action choice is furthest from the offline RL policy's action choice, in terms of some action-space metric. This original study and this reproduction study both investigate this attribution approach in three environments: Gridworld, Seaquest, and HalfCheetah. \n\nIn particular, these studies seek to find support for four phenomena:\n- Removing clusters of trajectories from the training set leads to lower Initial State Value.\n- Trajectory embedding clusters correspond to meaningfully different behaviors.\n- Distant trajectories influence the agent decisions\n- Humans can correctly identify the attributed clusters among a set of trajectories including random ones\n\nThe study finds many of these results are reproducible, though not consistently so across all environments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Following the points raised in the weaknesses section, I would suggest the following changes:\n- Add more motivation in the intro for why the authors believe it is particularly important to reproduce Deshmukh et al, 2023.\n- Add a deeper discussion on the key takewaways from this reproducibility study, as well as on the limitations of the attribution method and the settings studied. \n- Include a skimmable summary (e.g. a table) of the key results: Which original results were reproduced and which were not, as well as methodological differences. The latter might be a separate listing in the appendix.\n- A brief description for why DBSCAN is a sensible alternative method to try in addition to the original X-Means method for clustering would improve the legibility of the experiment design.\n- The carbon footprint calculations break the flow of the paper significantly. I suggest moving this to the appendix.\n- The abbreviations for the main phenomena under study, e.g. RTISV, are not memorable and make the paper harder to read. I suggest renaming these to short terms more naturally indicative of each phenomenon.", "broader_impact_concerns": "The conclusions of this paper somewhat strengthen the claims in the original Deshmukh et al, 2023 paper, which introduces a method for attributing the decisions made by an offline RL policy on specific clusters of trajectories. This work could lead to such attribution methodologies be adopted more broadly. This poses a risk if such attribution methods result in misleading or spurious attributions in real-world settings. This is a general concern for many interpretability methods.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BWO6bgdoQp", "reviewer": "Reviewer_NCRA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nUnderstanding the influence of the data for agents behaviour is the key aspect of RL.\n\nWeaknesses:\nThe relevance of the paper is limited. While the reproduction of result is valuable it is of limited scope. I would much love to see more environments. It is very interesting if the conclusion hold for other training methods etc. The authors barely scratch the surface of interesting questions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper attempts to reproduces 'Explaining RL Decisions with Trajectories'. They partially confirm findings and extend analysis with different metrics and analysis.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Perhaps the easiest way to make the paper more attractive significantly increase the empirical content. This could include:\n- more enviroments\n- more RL training algorithms\n- study of design choices (akin to the on in Sec 4.2.1 but in a much broader scope)\n\netc.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "l7O7QzOlyQ", "reviewer": "Reviewer_7zst", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n\nThe paper clearly explains in detail all the hyperparameters and/or provides the github links to the used code sources.\n\nThe authors conducted additive studies regarding the impact of the choice of the clustering method. \n\n*Weaknesses*\n\nThe experiments of the original paper were particularly poor: only 3 environments including a very small griworld, only one seed for the quantitative study, only 10 subjects in the human study. Hence, we would expect a reproducibility study to explore larger scales and counterbalance this issue. But the only “correction” was to add 10 more subjects to the human study. Consequently, this reproducibility study inherits the limitations of the initial paper.\n\nAnother problem of this paper is the redaction. I found it hard to read, with:\n\n\nA lot of concepts used before being explained \n(for ex: what does 'removing trajectories' means in the abstract, or what does 'high level behaviour' means in section 2), \n \nRedundant structures that repeats several time the same sentences \n(for ex: the story of the seed).\n\n\nThe “RTISV”, “CHLB”, “DTDA” and “HS” notation, that forces the reader to go back to p.2 in order to remember what claims are addressed every time these letters are appearing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tries to reproduce the result from 'Explaining RL Decisions with Trajectories'. \n\nThe original paper's method consist in encoding trajectories from a training set and then to applying clustering algorithm to extract clusters of explored behaviors, and finally explains a decision made by the trained RL agent with the cluster that, by comparing with an agent trained without its trajectories, has the largest impact on that decision. Then, 3 studies are conducted to analyze these clusters:\n\n1) A quantitative study, that measures the influence of each cluster by removing its trajectories from the training set, with many divergence metrics. (\"RTISV\" claim). \n\n2) A qualitative study that displays representative trajectories of the different clusters and show how they influence a specific decision at a specific state. (\"CHLB\" and \"DTDA\" claims)\n\n3) A human study, where they ask subjects to guess the representative trajectory that best explains a decision of the trained RL agent, to see if this corresponds to the cluster that the algorithm attributed to the decision. (\"HS\" claim)\n\n1) and 2) were conducted on 3 envs: a gridworld, Seaquest, Halfcheetah, while 3) is only conducted on a gridworld.\n\nThe general observations of the reproducibility study are:\n- results of 1) 2) and 3) on the gridworld are approximately reproduced (up to a seed), and claims are validated.\n- results of 1) and 2) on Seaquest and Halfcheetah diverge from the original paper (which is expected since the env are slightly different and the algorithms differ a bit, but the claims can still be validated.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The most important changes would be to add more environments, to repeat the quantitative experiment with multiple seeds (at least 5), to report both means and standard deviations.\n\nRegarding the redaction, I would strongly suggest to split the paper in 4 section, one for each claim, in order to avoid the “RTISV”, “CHLB”, “DTDA” notation. \n\nAlso, the fact that the seed used in the original paper was not communicated is not relevant and is not an issue. What we expect from reproducibility is not to recover the exact values but to validate the claims.\n\nDetails:\nThe carbon footprint section is irrelevant and should be moved to the appendix.", "broader_impact_concerns": "No ethical concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: NCRA: "Solid empirical work with applicational relevance."
7zst: "The new version of the paper addresses my most important concerns, and this time brings significative gains to interpret the extent of the initial paper."

Reviewers 7zst has concerns about the novelty and the significance of the work. However, I believe that the paper still provides a valuable companion to the original paper, in particular since it finds insufficient evidence for one of the main claims in the original paper (iv).

Another reproducibility of the same paper appeared in TMLR: https://openreview.net/forum?id=JQoWmeNaC2 at the same time. The two studies are complementary, and given that two independent groups went on to reproduce the paper, provides an evidence of the community's interest.
Decision: Accept (poster)