Behaviour Discovery and Attribution for Explainable Reinforcement Learning
Authors: Rishav Rishav
Somjit Nath
Vincent Michalski
Samira Ebrahimi Kahou
Venue: Accepted by TMLR ()

Abstract: Building trust in reinforcement learning (RL) agents requires understanding why they make certain decisions, especially in high-stakes applications like robotics, healthcare, and finance. Existing explainability methods often focus on single states or entire trajectories, either providing only local, step-wise insights or attributing decisions to coarse, episodelevel summaries. Both approaches miss the recurring strategies and temporally extended patterns that actually drive agent behavior across multiple decisions. We address this gap by proposing a fully offline, reward-free framework for behavior discovery and segmentation, enabling the attribution of actions to meaningful and interpretable behavior segments that capture recurring patterns appearing across multiple trajectories. Our method identifies coherent behavior clusters from state-action sequences and attributes individual actions to these clusters for fine-grained, behavior-centric explanations. Evaluations on four diverse offline RL environments show that our approach discovers meaningful behaviors and outperforms trajectory-level baselines in fidelity, human preference, and cluster coherence. Our code is publicly available.

Published in Transactions on Machine Learning Research (09/2025)
Behaviour Discovery and Attribution for Explainable Rein- forcement Learning
Rishav Rishav University of Calgary, Mila
mai.riishav@@maal.com com
Somjit Nath
McGill University, Mila
Vincent Michalski
Université de Montréal, Mila
Samira Ebrahimi Kahou
University of Calgary, Canada CIFAR AI Chair, Mila
Reviewed on OpenReview: https: // opwnreveew... net/ forum? id- JoHtp01H91
Abstract
Building trust in reinforcement learning (RI.) agents requires understanding why they make certain decisions, especially in high-stakes applications like robotics, healthcare, and fi- nance. Existing explainability methods often focus on single states or entire trajectories, either providing only local, step-wise insights or attributing decisions to coarse, episode- level summarics. Both approaches miss the recurring strategies and temporally extended patterns that actually drive agent behavior across multiple decisions. We address this gap by proposing fully offline, reward-free framework for behavior discovery and segmentation, enabling the attribution of actions to meaningful and interpretable behavior segments that capture recurring patterns appearing across multiple trajectories. Our method identifies coherent behavior clusters from stale-action sequences and attributes individual actions to these clusters for fine-grate grained, behavior-centric explanations. Evaluations on four diverse offline RI. environments show that our approach discovers mcaningful behaviors and out- performs trajectory-level basclines in fidelity, human preference, and cluster coherence. Our code is publicly available
1 Introduction
Explaining the decisions of RT agents is increasingly important as these agents are deployed in high-stakes domains such as robotics, healthcare, and finance (Sutton & Barto 2018 Arulkumaran et al. 2017 Fatemi
et al. 2019). Interpretability is critical for building user trust, assessing safety, and diagnosing failure modes. While many existing explainability methods RL focus on input features, individual states or entire episodes, they often overlook a crucial abstraction: behavioral contert. Agent decisions are frequently driven by recurring strategies that unfold over multiple timesteps and appear repeatedly across episodes. This temporal structure, which reffects what the agent is doing rather than just where it is, remains underexplored in current interpretability frameworks.
One line of work focuses on identifying influential features within individual observations. These include saliency-based techniques (Greydanus et al. 2018). attention heatmaps, and attribution methods adapted from supervised learning, such as LIME (Ribeiro et al. (016)) and SHAP (Lundberg Lee 2017), which have boen applied in RL highlight input relevance (He al. 2021
Carbone 2020
Beechey al. 2023
https://rish-as www.gg..uub... github.io/bexr!

---

Published in Transactions on Machine Learning Research (09/2025)
Figure 1: Representative trajectory from the MiniCriddTloCCossLaaa environment, annotated with behavior cluster assignments shown via colored backdrops (cO, cl, etc, best viewed at 200%). The full trajectory is shown left-to-right, top-to-bottom, with segments assigned to distinct behavior clusters. For example, c2 captures exploration, c0 corresponds to lava traversal, and c3 involves goal approach. This segmentation reveals temporally extended patterns in agent behavior. Descriptions for all clusters are summarized Table and visual samples from each cluster along are provided Appendix C
Although effective for local analysis, these methods operate at the level of single timesteps and provide only snapshot-level insight. As a result, they fail to capture long-term structure in agent behavior.
Other approaches expand the scope by examining how past events influence agent decisions. Causal graph models (Madumal et al., 2020 aim to explain actions by modeling dependencies between decisions and their outcomes. Counterfactual techniques such as EDGE Guo al. 2021) identify causal features by intervening stale representations. StateMask (Cheng et al. 2023 takes a different approach, learning lo mask parts of the input sequence that are less relevant for predicting returns. While these methods can highlight important moments or compress decision histories, they do not capture highor-level behavioral patterns or recurring strategy components. Their focus remains on local explanations rather than structured descriptions of what the agent is doing over time.
Trajectory-based methods aim to capture longer-term context when explaining agent decisions. For example, Deshmukh et al. (2024) attribute decisions to full trajectories retrieved from offiine data that may have influenced the agent. While this provides more temporal information than single-step explanations, it treats each trajectory as a whole, without distinguishing between different phases of behavior. As result, actions taken during exploration, goal-seeking, or recovery may all be attributed to the same trajectory, making it hard to tell which specific behavior was responsible for a given decision.
In this work, we propose behavior-centric framework for post-hoe attribution in RL. Rather than assign- ing decisions to individual features, single states, or entire trajectories, segment rollouts into temporally coherent clusters that capture recurring patterns in agent behavior, shown in the annotated MiniGridTT- woGoalsLava rollout in Figure 1 Agent decisions are then attributed to these discovered behavior segments, which are identified in an unsupervised manner and validated through visualization and human evaluation. This behavior- attribution enables analysis how different strategies influence outcomes. For example, in an autonomous driving scenario, can reveal that unsafe lanc changes consistently occur during a specific merging behavior. Such insights support systematic identification of policy failures and can inform safety auditing, strategy refinement, and monitoring of policy drift.
2

---

Published in Transactions on Machine Learning Research (09/2025)
Our main contribution is behavior-centric explanation framework that attributes agent decisions to unsu- pervised behavior segments, rather than isolated states full trajectories. The method operates entirely offline and does not require access to rewards or the underlying policy model. By clustering sequences of state-action pairs, we produce behavior-level interpretations of agent decisions that reflect recurring exe- cution patterns. To our knowledge, this is the first approach explain RL policies through unsupervised discovery of behavior segments. We evaluate the framework on four offline RL benchmarks and show that it identifies coherent behavioral clusters, enables faithful and interpretable attributions, and outperforms trajectory-level baselines in both quantitative metrics and human studies.
Graph Segmentation
Tohantred Tralictoriia
Palis
Tranybrrerr Enooder
Belasior Cluster
Training transformer based sea2seq model with VQ bottleneck
Graph Segmentation to obtain 'behavior' clusters
3.
4.
MSEUCAT) MOSIetad)
IT1
diossal to
State Action
Tall
T22
IT2
Plaka
ITa
Actor
Training policies on 'behavlor' clusters and full data
Attributing
Figure 2: Overview of the proposed behavior attribution framework. A transformer-based Vector Quantized Variational Autoencoder (VQ-VAE) is trained on state-action sequences to produce discrete latent codes, which are used to tokenize trajectories. These tokenized trajectories then segmented via graph cluster- ing to discover distinct behavior clusters. Policies trained each cluster well the full dataset. During attribution, actions taken by the original policy are compared to those from each cluster policy, and attribution is made based on minimal mean squared error (MSE), assigning actions to the most behaviorally aligned cluster. causal mask ensures that encoding respocts the temporal structure trajectorics. The transformer architecture described in detail A Appendix B.
2
Related Work
Behavior Discovery in R... Structured behavior discovery has been studied extensively RT support temporal abstraction, planning, and policy reuse. The Options framework (Sutton et al. 1999 Precup
2000
formalized temporally extended actions. Later works (Bacon al. 2017 Harb et al. 2017 Jin
et al. 2022 proposed learning such abstractions from data using intrinsic or extrinsic signals. U Unsuperviss
skill discovery approaches (Eysenbach et al. 2018 Achiam et al. 2018 Villecroze et al. 2022) maximize
diversity or rely on nonparametric clustering to identify latent behaviors. In the oflline setting, OPAL (Ajay
et al. 2021 and Diffuser (Janner et al. 2022 utilize learned behavioral embeddings to support planning or
imitation.
LAPO (Schmidt & Jiang 2024) and Genie (Bruce al. 2024] extend these ideas by discretizing behavior via VQ-based latent spaces for downstream control simulation. However, such methods designed for learning compact action spaces or optimizing policies, not for attributing observed behavior. In contrast, we apply sequence-level discretization to state-action trajectories for the purpose of discovering coherent behavior segments. These are subsequently segmented via a graph-based approach to yield interpretable behavior units that support post-hoe explanation in a reward-free, offline setting.
Explainability in RL. A key objective in Explainability in Reinforcement Learning (XRL) is to under- stand why agent takes particular action. Feature-level methods attribute decisions to salient. input
3

---

Published in Transactions on Machine Learning Research (09/2025)
regions using gradient-based techniques
GGryydanus et al. 2018 Zahavy et al 2017) or certurbation-basse
importance (Puri et al. 2019 Tyer et al. 2018). Others adopt techniques like LIME (Ribeiro et al. 2016)
and SHAP (Lundberg & Lee 2017) to explain decisions in reinforcement learning by attributing importance
to input features (Beechey et al. 2023 Zhang et al. 2022).
Other works focus on attributing importance individual transitions. EDGF (Guo et al. 2021| uses coun-
terfactuals to identify which state features caused a particular action. StateMask (Cheng al. 2023 learns to mask non-critical observations without degrading performance, enabling step-level saliency. RICE CCheng
et al. 2024) extends this idea by identifying critical states and reinitializing the agent from them to improve training. AIRS (Yu et al. 2023) and Liu et al. (Liu et al. 2023) attribute decisions to specific moments using reward gradients or visual cues. These methods effectively highlight pivotal decisions, but typically focus on isolated steps or episodes and do not uncover structured behavioral patterns that persist across trajectories. Morcover, many of them rely on reward signals or environment access, whereas our approach is fully offline and rewerd-free, identifying behavioral structure from trajectories alone.
Several approaches instead operate at the level of trajectory segments or full rollouts. HIGHLIGHTS (Amir & Amir 2018) selects representative snippets to summarize policy behavior but does not explain individual actions. AOC (Sun et al.) 2023) retrieves similar decisions from a corpus to justify actions, but formulates explanation as policy construction rather than post-hoc analysis. Deshmukh et al. (Deshmukh et al. 2024) are most closely related to our work, as they attribute actions to full trajectories retrieved from offline data. However, their method treats each trajectory as a single unit, which can bundle together multiple distinct behaviors and reduce attribution granularity. In contrast, we decompose trajectories into temporally coherent, recurring behavior modules and associate each action with its corresponding behavior. This allows for structured, behavior-level attribution grounded in patterns that recur across episodes.
Some approaches aim to explain agent behavior by modeling its internal structure or learning simplified surrogates. Causal frameworks (Madumal et al.] 2020; Pawlowski et al. 2020) build structural models to answer contrastive or counterfactual queries, revealing why specific actions were taken. These methods can offer deep insight into decision rationale, but often rely on access to predefined or learned causal graphs which may be hard to obtain in complex domains. Distillation- based approaches (Coppens et al.) 2019 Verma et al. 2019) approximate policies using decision trees or symbolic programs. While interpretable, they focus on replicating outputs rather than exposing the underlying decision process, offering limited insight into the factors or abstractions that drive decision- maaiii..nmaking.....
Latent Discretization in RL Disercte representations have proven useful for structuring policies and simplifying planning in RL. VQ-VAEs (Oord et al. 2017) have been adopted for compressed modeling (Hafner ct al. 2020), action quantization (Luo al. 2023), and hicrarchical decision-making (Nachum et al. 2018).. LAPO (Schmidt & Jiang] 2024] recovers symbolic latent actions via inverse dynamics, and Genie [Bruce et al.) 2024) learns behavior tokens for autoregressive simulation. Unlike these methods, we do not use discretization to build a control policy. Instead, we leverage it to recover graph-structured segmentation over trajectories, cnabling post-hoc analysis and modular attribution of learned behaviors.
Evaluation in XRL Surveys on explainable reinforcement learning (Cheng et al., 2025 Vouros 2022
emphasize the lack of unified evaluation standards. While fidelity, interpretability, and human alignment are commonly used, most work focuses on fcature saliency or trajectory summarics. Few methods address behavior-level explainability. Our approach introduces structured, temporally grounded explanations via unsupervised behavior discovery. We evaluate it through fidelity scores, cluster coherence, and human preference, offering distinct contribution to the XRL landscape.
3 Method
We propose a three-stage framework for post-hoc behavior discovery and attribution in offline reinforcement learning. First, we train a transformer-based VQ-VAB to discretize state-action trajectorics into latent codes that capture temporally extended behavioral motifs. Next, we construct a behavior graph using these codes and apply spectral clustering to identify coherent. behavior segments. Finally, we attribute actions from
4

---

Published in Transactions on Machine Learning Research (09/2025)
pretrained policy to the discovered behaviors by comparing them with cluster-specific behavior models. This framework enables structured, interpretable analysis of agent behavior directly from offline data.
3.1 Behavior Discovery
Behavior discovery is at the core of our framework, where we identify and segment meaningful sub- sub-trajectories from offline RL data. The process begins with transformer-based Vector Quantized Variational Autoencoder (VQ-VAE) trained in sequence-to-sequencc fashion, producing discrete latent codes for each time step, which are later used for segmentation.
The encoder processes sequences of state-action pairs {ss,, as), (51+1, (e-1),........... (St4 dc+k)), where 8 E R" (or s e R= for image-based observations). and a R". Positional encodings and causal masking ensure temporally valid and autoregressive representation learning. The encoder outputs latent vectors 2 E Rd. which are then discretized via learned codebook C {(i, C2..,,N by computing:
g(2)) arg min 2 c/ll² c
(1)
The VAE loss combines codebook and commitment objectives:
Lrg = [1 sg((,))l + [[lsg(z)) - -gll²]]
(2)
where sg(+) denotes stop-gradient. The decoder reconstructs future states {8441, S242....) using only past
information. The total training objective is:
k
Crecon + aLvy where Lrecon =
E
|s - $el1
Using state-action pairs instead of only states is key discovering behavior-centric latent representations. Prior works like LAPO (Schmidt & Jiang 2024) and Genie (Bruce et al.] 2024) have shown that when only states used imput, the learned latents tend to capture action-like primitives, effectively forming latent action space. In contrast, by including actions alongside states, the model learns representations that reflect what the agent actually did in each context, leading to latents that correspond to temporally extended behaviors rather than individual action decisions. Similar insights are seen in Deshmukh et al. (2024), which also incorporate action and reward information to better isolate the behaviors underlying agent decisions.
Although the VQ-VAE provides per-timestep discretization, the autoregressive attention in transformers leads delay in token shifts relative behavior change. Further, minor input perturbations can yield different latent codes for the same behavior. To address this, apply graph-based segmentation that smooths the raw ixation into coherent behavior segments.
3.2 Behavior Segmentation
To uncover structured behaviors from offline RI. data, segment trajectory sequences using graph-based approach over latent tokens. Each token is discrete representation of state-action chunk, obtained via vector quantization in the behavior discovery stage. The segmentation graph captures both how frequently these tokens transition into one another and how similar they are in latent space, capturing both temporal and structural coherence.
Formally, we define a graph G (V, E), where each node i V corresponds a codebook token G; and edge weights encode a hybrid of transition dynamics and latent similarity:
way - X) Count(q + +,) + ||c. - Cy 112
(4)
Here, Count(c: - cÿ) is the normalized frequency of transitions in the dataset, while the Euclidean term penalizes large jumps in latent space. The hyperparameter X [0, 1] controls the trade-off between temporal
5

---

Published in Transactions on Machine Learning Research (09/2025)
continuity and spatial clustering. We empirically ablate À in Appendix A and show that moderate values consistently yield better segmentation. This hybrid weighting is necessary: transition-only graphs may over- segment due to noise or repeated transitions between distinct behaviors, while distance-only graphs may merge temporally distinct behaviors with similar representations. Combining both allows the method to recover temporally extended, recurring behavioral motifs.
We apply spectral clustering segment the resulting graph. First, we compute the symmetric similarity matrix S as Sig Wij Wjes and construct the unnormalized Laplacian:
= D- S, where Da = Σ Sky 3
(5)
The top & eigenvectors of L embed the graph nodes into lower- swver-diensional space, where connected compo- nents become geometrically separable. The number of clusters & is determined via eigengap analysis. Each cluster corresponds to distinct behavior, and each timestep in the dataset is labeled accordingly.
We also compare this segmentation approach with alternatives, such clustering directly in latent space (e.g., k-means) or over raw state-action pairs. As discussed in Appendix A spectral clustering consistently outperforms these alternatives by yiclding more temporally stable and semantically meaningful clusters.
Clnster #
HalfCheetah- medium-v2
MiniGridTwoo
Seaquest-
Pen-
GoalsTava
mixed-v0
expert-vi
Jump prep using hind legs Running on hind legs
Crossing lava
Exploration variety Idle Gameplay
Initial Grasp
Running into obstacle/war /wala
Final Pose Adjustment
Leaping forward
Exploration varicty
Transition phase
Manipulating pen
Gait transition
Approaching goals
Moving downwards
Stable hand
Bounding with accel.
Forward exploration
Agent killed
Noisy cluster
Bounding
Stuck at wall
Random firing
Pose adjustment
Fall recovery
Contimuous motion
Moving right
Accel. leap
Transition phase
Moving left
Rumning on front leg
Grid exploration
Exploration variety
Running on hind legs
Transition phase
Filling oxygen
10
Corner exploration
11
Clean exploration
Table 1: Interpretable behavior clusters discovered across four benchmark environments. Descriptions are based on visual inspection of representative samples randomly chosen per cluster. Visual samples and detailed descriptions for each cluster across all environments available in Appendix C
3.3 Attributing Actions to Behaviors
We attribute policy actions to discovered behavior clusters to explain agent decisions in terms of learned behavioral motifs. policy æ is trained on the full dataset. For each behavior cluster k, we train a behavior cloning model M2 using its trajectory segments.
In continuous action spaces, attribution uses MSE:
d Σ
arg min MSE(a, az), MSE(a, ax) = d 1=1
(a: - مي,ي
(6)
In discrete domains, altribution uses cross-entropy:
C Pay log(ps.))
k- = arg min (P)(Pa.p)), Lce(Pe.ps) = -
Pos
(7)
j=1
This process yields behavior-level explanations for each action, helping clarify which learned behavior motif best accounts for given decision.

---

Published in Transactions on Machine Learning Research (09/2025)
Figure 3: A sample sequence from HallCheetah with cluster labels visualized through colored backdrops for each frame. For instance, Cluster involves hind-leg loading before a leap, Cluster captures running on front leg, and Cluster 2 corresponds to high speed forward Icaps. Full cluster definitions are in Table Sample sequences from each cluster can be found in Appendix C
4 Experiments
We evaluate the effectiveness of our framework for behavior discovery and altribution using three benchmark environments from [4RL((Fu et al., 2020) hullcheetab-medimm-22 pen-expert-vl-as and seaquest-mixed
vO from D4RL-atari repository (Takuseno 2025 as well as a custom environment, ThnnGGridTwooollsavv.
based on the MiniGrid suite. These environments span a broad spectrum of RL challenges: halfcheetah- medium-v2 involves high-dimensional continuous locomotion; seaquest-mixed--v is discrete-action Atari domain with dense visual input and long- horizon objectives; pen-expert-vl requires precise dexterous manip- ulation with high-dimensional proprioceptive and object-centric observations; and MiniGriidTwoGollsl..e supports goal-conditioned navigation with clearly interpretable behavior switches. The datasets for the D4RL environments were collected using partially trained SAC (Haarnoja et al., 2018) (hafcchettth) DQN (Mnih et al. 2015; (seaquest), and scripted expert policies (pen), as described in Fu et al. (2020), and reflect varying levels of sub-optimality to simulate realistic offline conditions. The dataset for MiniGridTwooooal-- sLava was generated by us using a PPO policy trained to approximately 30 40-40% task success. Our pipeline uses transformer- based VQ-VAE model to discretize statc-action sequences into latent codes, which are used to construct behavior graph over time. Spectral clustering is then applied to obtain segmented tra- jectories, as described in Section 3.2 To assess the robustness of our method, we conduct broad ablation study varying the codcbook size, number of clusters, graph construction parameters (c.g., À for balancing transition vs. latent similarity), and architecture components. Full results are presented in Appendix A with hyperparameters listed in Appendix B.1
4.1 Behavior Discovery and Segmentation
We evaluate whether the behavior clusters discovered by our framework reflect temporally coherent and semantically meaningful segments across four offline RL environments. The segmentation is based on latent codes obtained from VQ-VAE trained on state-action sequences, post-processed using spectral clustering algorithm over a graph constructed from transition frequency and latent similarity. This hybrid formulation enables the discovery of segments that are both structurally consistent. and behaviorally distinct.
As summarized Table D our method isolates a range of interpretable behaviors across environments. In MiniGrridIVoooalLLava clusters include exploring near walls, crossing lava, and approaching goals. In seaquest-mixed-vv,, observe clusters corresponding to oxygen refilling, directional movement, and repet-
7

---

Published in Transactions on Machine Learning Research (09/2025)
itive firing. For halfchoetah-modiume the clusters capture different locomotion modes such bounding. leaping, and gaits involving instability or braking.
c5
Figure 4: Visualization of sample trajectory from pen-expert-vl, displayed with frameskip of 2. Cluster assignments are indicated by the colored backdrops for each frame, with corresponding cluster IDs (cO, cl, c2, c3, c5) shown below the trajectory. Full details of cluster definitions can be found in Table and representative samples from each cluster can be found Appendix C This image best viewed at 200% zoom.
Figures 1 provide rollout-level visualizations of the discovered clusters, highlighting how behavior boundarics align with high-level shifts in strategy or movement. Cluster assignments typically remain stable within consistent behaviors and change sharply clear behavioral transitions, often at action-level granular- ity. Across episodes, observe that similar behavior types consistently grouped together suggesting that the framework recovers reusable, high-level behavior patterns rather than overfitting to surface-level differences in input.
While these visualizations offer qualitative validation, we also conducted a human study (described next) to quantitatively assess the interpretability of the discovered behavior clusters. Participants described sample segments from different clusters and provided confidence scores, allowing us to measure agreement and semantic coherence.

---

Published in Transactions on Machine Learning Research (09/2025)
C3
C7
Figure 5: Sample from Seaquest-mixen mixed-vv with cluster labels visualized through colored backdrops for each frame. Cluster moving down, Cluster 6: moving right, Cluster oxygen refill. See Table for more. We further present sample sequences from each cluster in Appendix C
4.1.1 Human Study: Behavior Interpretability
To evaluate the semantic coherence of our discovered clusters, we conducted a two-part human study. Part A was designed to assess whether clusters correspond to behaviors that are interpretable and describable by humans.
For each of the four environments, we randomly sampled three behavior clusters. For each cluster, three representative trajectory segments were selected and shown to 15 participants who had prior familiarity with the tasks. Participants were asked to (1) describe the cluster behavior in natural language, and (2) rate their confidence on a 1-5 scale. We computed the average pairwise cosine similarity between the textual descriptions using MiniLM sentence embeddings, after subtracting shared embodding representing the general euvironment context, Higher intra-cluster similarity indicates consistent human interpretations of the behavior.
Environment
Cluster1
Cluster2
Cluster3
AvgCosine
AvgConfidence
soaquest-mixed-w
0.68
0.69
0.63
0.67
5.00
MiniGridTTooGolllle
0.68
0.64
0.58
0.63
4.33
halfcheetah-medium222
0.61
0.65
0.61
0.62
5.00
pen-expert-vl
0.64
0.61
0.57
0.61
4.33
Table 2: Intra-cluster cosine similarities and average confidence scores for human--witter descriptions of sampled behavior clusters across four environments. Higher cosine values indicate greater semantic agreement among annotators, reflecting stronger behavioral coherence within clusters. For comparison, the average cosine similarity between randomly selected descriptions from different clusters was 0.42.
The consistently high cosine similarities and confidence scores across environments suggest that our clusters are scmantically meaningful and interpretable for most participants. Participants often used similar phrases (e.g., "approaching goal", "crossing obstacle", "leaping forward") describe different samples from the same cluster, supporting the behavioral consistency captured by framework.

---

Published in Transactions on Machine Learning Research (09/2025)
4.2 Behavior Attribution
To validate the relevance of discovered behaviors, we perform action-level attribution by training separate behavior cloning (BC) model for each cluster. Details about the BC models and their training can be found in Appendix B.2 These models serve as local approximators for how each behavior typically responds in given states. For any observed action, we identify the most likely behavior by comparing the output of these models to the agent's actual action.
In continuous control environments (e.g., halfcheetah-meddumn (MSE) between the action predicted by each cluster model and the ground truth action. The cluster min- imizing this error is selected the behavior attribution. In discrete domains (e-g. seaquesl-mixed-x MiniCriddTwoCoalsLarr)..
we compute the mean squared error
compare action probability distributions using cross-entropy.
This attribution mechanism consistently assigns decisions to plausible behavior clusters across environments, For example, in seaquest-mixedd-v., sequences involving upward swimming and rapid shooting are reliably attributed to distinct clusters. haviors like high-speod bounding and slower gait transitions. These results suggest that the segmented behaviors not only reflect semantic structure but also support faithful explanation of policy behavior. AL- tribution examples visualized in Figure
ietthteetah-mddum---V the attribution clearly distinguishes between be-
4.2.1 Human Study: Attribution Quality
To evaluate whether our behavior-level attributions align with human intuition, we conducted Part B of the human study. In this selting, participants (n = 15) were shown a short context. window consisting of preceding and following frames surrounding a specific agent action. They were then asked to select which of four video segments best explained that decision.
The four options were: (1) the behavior segment attributed by our method, (2) trajectory segment retrieved using the method of Deshmukh et al. (2024), and (3 4) randomly sampled segments of similar length from other behavior clusters. To avoid trivial selections, none of the presented options contained the query segment itself. The goal was to assess which independent segment best captured the behavioral pattern underlying the given action.
Importantly, all vidco options were presented without labels, metadata, or trajectory statistics to minimize cognitive bias arising from cluster identity or video length. This setup encourages participants to rely solely the semantics of the action context when judging which segment provides the most plausible and behaviorally aligned explanation.
Environment
%0urs
Deshmukh et al. (2024)
%Randdm
pen-expert-vl
85.7
14.3
0.0
MiniGrriTTwoCCoallaayy
61.9
28.5
9.5
seaquest-mixed--v0 balfcheetab-meddim-y
68.4
31.6
0.0
81.3
13.9
4.7
Table 3: Human preference scores across environments for the segment attributed by method, the trajectory-level bascline from Deshmukh et al. (2024). and random segments. Our method was most fre- quently selected as the best explanation in all environments. The largest margins were observed in pen- expert-vl and halfcheetah maneettaaa-mdimm2222 highlighting the benefit of fine grained, behavior-centric attribution.
These results suggest that human evaluators consistently find our behavior-level allributions to be more plausible and semantically aligned with the observed action than those derived from full-trajectory retrieval methods. This supports our intuition that behavior sogments serve as more targeted and reusable explanatory units, especially in domains where agent behavior varies continuously or includes overlapping strategies. Additional details regarding this study provided in Appendix [
10

---

Published in Transactions on Machine Learning Research (09/202))
Attributed Cluster: CT
Attiibuted Cluster: C4
Attributed Cluster: C3
Attributed Cluster: C9
Attributed Cluster:
Attrbbted Cluster: C2
Figure 6: Attribution results for varions environments. The attribution, as described in Section 3.3 is performed for the middle frame in each sequence (indicated with a dashed border). To provide better temporal context, both preceding and succeeding frames are shown. The assigned behavior cluster is noted the right. Across all environments, the assigned clusters align well with the observed behavior. For example, in HalfChectah, cluster 1 corresponds to hind-leg running, while cluster 4 captures bounding accelerating motion. Similarly in Sequest, cluster corresponds to filling oxygen and cluster corresponds to downward motion. Similar consistency is also seen for MiniGrid and Pen.
4.3 Quantitative Evaluation: Cluster Soundness and Structural Quality
To complement the qualitative and human studies, quantitatively evaluate discovered behavior clus- ters using two perspectives: (1) how consistently each cluster captures policy-aligned behavior (cluster soundness), and (2) how structurally coherent and well-separated the clusters are in latent space (structural quality). These evaluations aim to assess whether the clusters are not only interpretable but also statistically meaningful.
Cluster Soundness via Fidelity Score. To assess whether our discovered clusters represent distinct and consistent behavioral modes, compute the Average Fidelity Score (AFS): the mean prediction error
11

---

Published in Transactions on Machine Learning Research (09/2025)
between behavior cloning (BC) model trained per cluster and the original policy. A low AFS implics that the cluster encapsulates a coherent policy fragment. To assess the distinctiveness of each cluster, we compare this to random baseline where actions are assigned to clusters uniformly at random (Rand-Ours), maintaining the same cluster sizes.
Table shows that the AFS gap between our clusters and the random assignment substantial, indicating that each of our clusters captures unique behavioral pattern. In contrast, for Deshmukh et al, (2024), the fidelity scores of their clusters and their random baseline (Rand-Deshmukh et al. (2024) often very close. This suggests their segments are not strongly aligned with distinct decision-making behavior, and that their clusters may be less bchaviorally mcaningful.
Environment
Ours
Rand-Ours
Deshmukh et al. (2024)
Rand Slaud-Dehmuuh et al. (2024)
hallchootal-mediune
0.36
2.29
0.05
0.09
seaquest- mixed-v0
veaquest-mixed-v0
0.59
0.98
0,13
0.17
MiniGriddWooooaleee
0.47
0.68
0.09
0.10
pen-expert-vl
0.16
0.49
0.13
0.20
for our method, random cluster assignments, and the trajectory-level baseline from Deshmukh et al. (2024) AFS is computed as the error between the main policy (trained on the full dataset) and the output of the behavior model corresponding to a given cluster. Scores are averaged over 200 randomly selected actions from 20 episodes. In our method, the larger gap between attributed and random cluster scores (Ours vs. Rand-Ours) indicates that the discovered clusters capture clearly distinct behaviors. In contrast, the small gap observed for the baseline suggests its clusters are less behaviorally differentiated.
Table 4: Average Fidelity Score (AFS)
Structural Cluster Quality. In addition to fidclity, evaluate the structural propertics of the discovered clusters in the latent space using two slandard unsupervised clustering metrics. The Silhouelle Score (SS) captures how well each latent token fits within its assigned cluster versus the next-best alternative-hise higher values indicate better scparation. The Davies-Bouldin Score (DB) comparcs intra-cluster dispersion to inter- cluster separation lower values are better. Together, these metrics reflect whether the latent structure of the clustering is compact and distinct.
As shown in Table 5 our method generally produces clusters with better scparation (higher SS) and lower within-chuster variance (lower DB) than Deshmukh et al. (024). particularly in continuous and high- dimensional settings such as halichestah-medium-- and seaquest-mixed-vo. In MiniGridTwoGoals... the baseline scores slightly better due to its LSTM-based encoder specialized for discrete grid settings, whereas our model uses a uniform transformer backbone across all environments. Despite this, our approach yields more interpretable segments (as seen AFS and human studies), suggesting that structural metrics alone may not fully capture behavior-level coherence.
Silhouette Score
Davies-Bouldee J Deshmukh et al. (2024)
Environment
Ours
Deshmukh et al. (2024)
Ours
lalfchoetab-mmediumyy
0.24
0.14
1.68
2.13
mixed-0
0.21
0.12
1.88
2.19
MiniCrid/Twoooolllaaas
0.37
0.48
1.44
1.19
pen-expert-vl
0.20
0.20
1.47
1.32
Table 5: Structural cluster quality using Silhouette and Davies-Bouldie scores across four environments. Our method yields more compact and separated clusters in continuous control Chalfcheetah-mediums and high- dimensional image-based tasks (seaquest-mixed-v0). In grid-based domains such as the LSsM-based encoder of the baseline offers better numerical clustering scores, though these do not always translate to semantically meaningful behaviors (see Table and human studes)..
12

---

Published in Transactions on Machine Learning Research (09/2025)
5 Limitations
The proposed framework, while effective in discovering and attributing behaviors, has certain limitations. In highly stochastic environments such as malfcheetab-medium--2 noisy or frequent behavior transitions can result. in overlapping segments that are less clearly delineated. While most clusters correspond to interpretable behaviors, a few cxhibit ambiguity, making it difficult to precisoly characterize their somantic content. Moreover, our graph-based segmentation relies on hand-crafted similarity measures and traditional spectral clustering, which may not optimally adapt diverse environments, Future work could investigate graph neural network-based segmentation methods that learn task-specific similarity structures and provide more robust behavior partitioning,
6 Conclusion
We introduced post-hoc framework for discovering and attributing behavior-level patterns in offline rein- forcement learning. By combining soquenco-level discretization through transformer-based VQ-VAE with graph-based segmentation via spectral clustering, our method extracts interpretable, temporally extended behaviors from raw trajectories. Autribution is performed by matching policy actions to behavior-specifee models, enabling fine-grained explanations of decisions. Evaluations across diverse domains, including loco- motion, manipulation, and Atari-style control, demonstrate that the discovered segments align with semanti- cally meaningful behaviors and offer improved interpretability over trajectory-level baselines. While current results are limited offline settings, future extensions could explore applications in debugging, anditing, and safe deployment of RL agents in real-time systems.
7
Acknowledgment
We sincerely Uhank NSERC, CIFAR, Digital Research Alliance of Canada Deuvr Datacloud for providing resources to complete this project.
References
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algo- rithms, 2018. URL httphttts:/aaae //tp:v///xxivv///a hwwpi//orrabbbs////eeae/ /
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. In International Conference on Learning Repre- sentations (ICLR), 2021. URL https://openrreviex wttp://www......eee.oo.or///oaas
Dan Amir and Ofra Amir. Highlights: Summarizing agent behavior to people. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '18, pp. 1168 1176, Richland, SC, 2018. International Foundation for Autonomous Agents and Multiagent Systems.
Kai Aralkumaran, Mare Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. TEFE Signal Processing Magazine, 34(6):26-33 2017.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAT conference on artificial intelligence, volume 31, 2017.
Danicl Beechey, Thomas MS Smith, and Özgür Şimşck. Explaining reinforcement learning with shapley values. In International Conference on Machine Learning, pp. 2003-2014 PMLR, 2023.
Jake Bruce, Michael Dennis, Ashley Edwards, Jack Porker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal Behbahani, Stephanic Chan, Nicolas Hoess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Singh, and Tim Rocktäschel. Genie: Generative interactive environments, 2024. URL bittss///ares wwtp:/wwwrr/abb/////.00/ 15391
15391
13

---

Published in Transactions on Machine Learning Research (09/2025)
Nicolas Blystad Carbone. Explainable ai for path following with model trees. Master's thesis, NTNU, 2020.
Zelci Cheng, Xian Wu, Jiahao Yu, Wenhai Sun, Wenbo Guo, and Xinyu Xing. Statemask: Explaining deep reinforcement learning through state mask. In Shirty-seventh Conference on Neural Information Processing Systems, 2023. URL /atzps://openrevieww
Zelei Cheng, Xian Wu, Jiahao Yu, Sabrina Yang, Gang Wang, and Xinyu Xing. RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanation, June 2024. URL http://arxix: brg/abs/2400 0/////222000000444 03064 arXiv:2405.0306 [cs].
Zelei Cheng, Jiahao Yu, and Xinyu Xing. A survey on explainable deep reinforcement learning, 2025. URL 06869
https://arxivy wwwp:/orggaaba...aa.
Youri Coppens, Kyriakos Efthymiadis, Tom Lenaerts, Ann Nowé, Tim Miller, Rosina Weber, and Daniele Magazzeni. Distilling deep reinforcement learning policies in soft decision trees. In Proceedings of the IJCAI 2019 workshop on explainable artificial intelligence, pp. 1 6, 2019.
Shripad Vilasrao Deshmukh, Arpan Dasgupta, Balaji Krishnamurthy, Nan Jiang, Chirag Agarwal, Georgios Theocharous, and Jayakumar Subramanian. Explaining rl decisions with trajectories, 2024. URL https:
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function, 2018. URL http:////arrav../aaaa///////////
Mchdi Fatemi, Shikhar Sharma, Harm Van Seijen, and Samira Ebrahimi Kahou. Dead-ends and secure exploration in reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Pro- ceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1873 1881. PMLR, 09 15 Jun 2019. URL htttps:/pprocedinng mlr. presa/ vS7/fatemi19a.htm
Justin Fu, Aviral Kunar, Ofir Nachum, George Tucker, and Sergey Levine, D4rl: Datasets for deep data- driven reinforcement learning, 2020.
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. In Proceedings of the International Conference on Machine Learning (ICML), 2018
Wenbo Guo, Xian Wi,, Usmann Khan, and Xinyu Xing. EDGE: Explaining deep reinforcement learning policies. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, 2021. URL
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-poliey maximum entropy deep reinforcement learning with a stochastic actor, 2018. URL
01290
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations (ICLR), 2020.
Jean Harb, Pierre-Lue Bacon, Martin Klissarov, and Doina Precup. When waiting is not an option Learning options with deliberation cost, 2017. URL htttp::///axxiv.orr/bbbbee
04571
Lei He, Aouf Nabil, and Bifeng Song. Explainable deep reinforcement learning for uav aulonomous naviga- tion, 2021. URL /tttps://arxiv..oo http//bbs/2000/ 14551
Rahul Tyer, Yuezhang T.i, Huao Li, Michael Lewis, Ramitha Sundar, and Katia Sycara. Transparency and explanation in deep reinforeement. lcarning neural networks. In Proceedings of the 2018 AAAT/ACM Conference on AI, Elhics, and Society, pp. 144 150, 2018.
Michael Janner, Yilum Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. Tn International Conference on Machine Learning, 2022.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Muyang Jin, Zhihao Ma, Kebing Jin, Hankz Hankui Zhuo, Chen Chen, and Chao Yu. Creativity of ai: Automatic symbolic option discovery for facilitating deep reinforcement learning. ArXin, abs/2112.09836, 2022. URL sr.p org/CorppsID: 24533352222
Haozhe Liu, Mingchen Zhuge, Bing Li, Yuhui Wang, Francesco Faccio, Bernard Ghanem, and Juergen Schmidhuber. Learning to identify critical states for reinforcement learning from videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Newral Information Processing Systems (NeurIPS), pp. 4765 4774, 2017.
Jianlan Luo, Perry Dong, Jeffrey Wu, Aviral Kumar, Xinyang Geng, and Sergey Levine. Action-quantized
oflline reinforcement learning for robolic skill learning. 2023. URL htt.aagss/ppenreriiieee///oooous
n91ew97SAn
Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learning through a causal lens. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Volodymyr Maih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Toannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):522 533, 2015. doi: 00.1038/nature1422
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hicrarchical reinforcement learning- In International Conference on Learning Representations (ICLR), 2018.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukeuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 6306-6315, 2017.
Nick Pawlowski, Daniel Coelho de Castro, and Ben Glocker. Deep structural causal models for tractable counterfactual inference. Advances in neural information processing systems, 33:857-869, 2020.
Doina Precup. Temporal abstraction in reinforcement learning. University of Massachusetts Amherst, 2000.
Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishnamurthy, and Sameer Singh. Explain your move: Understanding agent actions using specific and relevant feature attribution. arXiv preprint ar Xiv::19122. 12191, 2019.
Marco Tulio Ribciro, Sameer Singh, and Carlos Guestrin. "why should trust you?": tions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 1135-1144. ACM, 2016.
Explaining the predic-
Dominik Schmidt and Minqi Jiang. Learning to act without actions. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
Hao Sun, Alihan Hüyük, Daniel Jarrett, and Mihacla der Schaar. Accountability in offline reinforcement learning: Explaining decisions with corpus of examples, 2023. URL https://arxiv wttps/iwwwrgrr/////////os wtgg/bs///////////o 07747
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181 211, 1999.
Akira Takuseno. d4rl-atari: Datasets for data-driven deep reinforcement learning with atari (wrapper for 2025.
datasets released by google). httt::///gghhhhh.ccaaatssseee//
15

---

Published in Transactions on Machine Learning Research (09/2025)
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Program- 02477
matically interpretable reinforcement learning, 2019. URL
Valentin Villecroze, Harry J. Braviner, Panteha Naderian, Chris J. Maddison, and Gabriel Loaiza-Gani Ganem. Bayesian nonparametrics for offline skill discovery, 2022. URL http:://aaaxivvoorr/aa//////2///
04675
George A. Vouros. Explainable deep reinforcement learning: State of the art and challenges. ACM Compuling Surveys, 55(5):1 (115):3333 39, December 2022. ISSN 1557-7341. 10. 1 1145/3527448
doi: 10.1145/3527448. URL http://dx. www.dooiooooo org/
Jiahao Yu, Wenbo Guo, Qi Qin, Gang Wang, Ting Wang, and Xinyu Xing. AIRS: Explanation for Deep Reinforcement Learning based Security Applications. 2023.
Tom Zahavy, Nir Ben Zrihem, and Shie Mannor. Graying the black box: Understanding dqns, 2017. URL
https://arxiv: wttp:v/aaxxvvvvvmm http:///aaaaa..ooo//////2200000 02658
Ke Zhang, Jun Zhang, Pei-Dong Xu, Tianlu Gao, and David Wenzhong Gao. Explainable ai in deep reinforcement learning models for power system emergency control. IEEE Transactions Computational Social Systems, 9(2):419-423 427, 2022. doi: 10.1109/CCSS.2221.30066
A Appendix A: Ablation Studies
A.1 Alternative Clustering Strategies
We evaluate the effectiveness of our clustering approach by comparing it against simpler alternatives. Specif- ically, we assess whether learned representations are necessary for high-quality segmentation and whether spectral clustering offers benefits over standard methods like K-means. These comparisons help isolate the contribution of each component representation learning and clustering algorithm to the overall quality of behavior segmentation.
A.1.1 Clustering Raw State-Action Pairs
To evaluate the necessity of learning representations, compare our method against a baseline where clustering is performed directly on raw state-action pairs, without any VQ-VAE encoder. As shown in Table this results in significantly worse clustering performance across all environments. The clusters tend to be noisy and inconsistent, with poor separation between distinct behaviors and no temporal coherence. These results demonstrate the importance of using a represcntation model like VQ-VAE to transform trajectories into discrete, behavior-centric codes that capture higher-level structure.
Table 6: Clustering performance when applied directly to raw state-action pairs. reprosentation results in weak and noisy clusters.
Absence of VQ-based
Environment
Silhouette Score t Davies-Bouldin Score
Ialfcheetal-meedin--22
0.20
1.97
pen-expert-vl
0.13
2.23
MiniGrridDooGoussLLaaa
0.10
2.14
scaquest-mixod-vv0
0.03
8.88
A.1.2 Spectral Clustering vs K-Means
We also compare spectral clustering, our default. segmentation method, with standard K-means applied to the learned VQ- embeddings. As shown in Table 7 spectral clustering outperforms K-means in both Silhouctte Score and Davics-Bouldh Score across environments. Unlike K-means, spectral clustering leverages the global structure of the transition graph and better suited for identifying temporally smooth and behaviorally meaningful clusters.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Table 7: Comparison between spectral clustering and K-means applied to latent embeddings. Spectral clustering produces more temporally coherent and behaviorally meaningful clusters.
Environment
Method Silhouctte Score 1
Davics-Bouldin Score +
halfcheetab-meehum--2
Spectral
0.24
1.75
halfcheetal--mediu--22
K-Means
0.20
1.97
MiniGridTTwoGonlsLaava
Spectral
0.37
1.44
MiniGridTwoGoalll...e
K-Means
0.30
2.05
pen-expert-vl
Spectral
0.20
1.47
pen-expert-vl
K-Means
0.17
1.82
seaquest-mixod-v00
Spectral
0.22
2.55
seaquest-mixed-v00
K-Means
0.21
2.65
A.2 Lambda Factor
Silhouette
Davies-Bouldin vs
hallcheetah-medium-/ minigrid twogoals
0.350
3.00
seequest-mixed--0
0.325
Dovies-Boull din Score
Sihouette Scare
0.300
hulfcheetah-medium-22
0.275
minigrid-twogoals-lve
seaquest
2.25
0.250
1.75
0.175
1.50
0.0
0.2
0.0
1.0
0.0
0.4
0.5
0.8
1.0
Figure 7: Clustering quality metrics across values. uses only transition frequency; À uses only latent-space proximity. Best performance typically occurs intermediate values.
We study the effect of the À parameter that controls the trade-off between transition frequency and latent- space proximity in edge weighting. Figure 7 shows clustering quality metrics (Silhouette and scores) across À values. A balanced weighting (A s 0.3 to 0.5) typically yields the best results, confirming that both temporal and semantic structure are important for stable segmentation.
A.3 Number of Codebooks
We evaluate the sensitivity of our method to the number of codebook entrics in the VQ-VAE, as this hyperparameter governs both representational capacity and interpretability. Using too few codebook entries limits the expressiveness of the latent space, resulting in poor reconstruction and insufficient coverage of behavior diversity. On the other hand, very large codebooks can lead to sparsely used or entirely inactive (dead) codes, reducing interpretability and increasing latent fragmentation.
To select an appropriate codebook size, we consider two metrics: (1) normalized reconstruction loss, which assesses how well the latent space encodes input sequences, and (2) codebook occupancy, defined as the proportion of codebook entries used at least once over the final 10% of training steps. This smoothing avoids instability from early-phase underutilizatio..
17

---

Published in Transactions on Machine Learning Research (09/2025)
Effect Codebook Size on Reconstructio Less and Occupancy thafthhetahh medium v2)
Occupancy
1.00
1.2
1.0
69
3.40
Norred Recerghadee
0.8
foom tox fecorol wwt.erttrrrooooooo
0.5
3.50
0.64
0.50
0.-
-0.75
-0.00
0..00
6:54
0.00
641
Number Cucktook
Occupeniyy (pen-expert-<1)
on
Occupency (seequent-mixee-x0)
1.0
1.0
1.0
1.0
48
Nemolized Reoeddructin
0.7
6.4
9.
6.5-
-.4
-33
0.5
0.4
52
0.10
72
24
13B
216
0.36
32
12B
255
Number Codebook Extries
Mumoer of Caceboss
Figure 8: Effect of codebook size normalized reconstruction loss (orange, lower is better) and codebook occupancy (blue, higher is better) across four environments. Increasing the number of codebooks improves expressiveness but may reduce code utilization. Our method achieves stable performance when balancing these two metrics. In most domains, codebook sizes in the 64-128 range yield good trade-offs. Notably, in TaniGriaTooGoalsavaaa a smaller codebook size 16 offers both low reconstruction loss and high occupancy, suggesting that optimal codebook size can be environmeent-deeedden..
B Hyperparameters & Training
B.1 Transformer-based VQ-VAE Architecture
B.2 Behavior Cloning Models
To support attribution and post-hoc analysis, we trained separate Behavior Cloning (BC) models on the clustered statc-action data. Each cluster had a sufficient number of statc-action samples to allow reliable model fitting. However, due to noise in the clustering process, some individual state-action pairs were occa- sionally isolated from their surrounding behavior segments. To improve consistency, we applied a smoothing step that reassigned such outliers the most frequent cluster label among adjacent steps in the trajectory.
For training, used 3-layer multilayer perceptron (MLP) for environments with state-based observations, and a 3-layer convolutional neural network (CNN) for environments with imago-based inputs. The learning rate and other training details match those in Table g
C Cluster Visualization
To better interpret the discovered behavior clusters, we randomly sample trajectory segments assigned each cluster and analyze them based on visual inspection. For each cluster, we then describe the most common recurring patterns observed across the sampled segments. This process helps illustrate the typical
18

---

Published in Transactions on Machine Learning Research (09/2025)
Causal Mask
5tv2
Used for reconstruction loss
5,
Graph segmentation post training
Transformer Decoder
Opt
Cer2
Graph Clustering
Behavior Clusters
0p2
State
Transformer
Codebook
Action Encoder
Positional Encoding
Figure 9: Overview of the behavior discovery and segmentation pipeline. A transformer-based VQ- "Q-VAE, is trained to encode sequences of state-action pairs into discrete latent codes using codebook. The encoder applies causal masking to ensure autoregressive prediction, and the decoder reconstructs future states to optimize a reconstruction loss. After training, the resulting latent codes are used to construct a transition graph, which segmented using spectral clustering to produce semantically coherent behavior clusters. Refer Table for full details on the architecture and training.
Hyperparameter
halfcheetah- medium-v2
MiniGrid- LwoGoalsLava
seaquest- mixed-v0
pen-expert-vi
Learning Rate (LR)
10 10-4
x 10
1 x
x 10
Sequence Length (seq len)
50
Variable (max 40)
30
30
Batch Size
64
32
64
32
Number of Codes
128
16
64
64
Embedding Dimension
128
128
128
128
Combination Param (X)
0.75
0.45
0.6
0.6
Num Epochs
50
50
50
50
Optimizer
Adam
Adam
Adam
Adam
LR. Scheduler
Linear decay
Linear decay
Linear decay
Linear decay
Teacher Forcing
Linear decay to 0
Linear decay to 0
Linear decay to 0
Linear decay to
Transformer Heads
4
4
4
Encoder/Deccode Layers
4
2
4
Transformer Hidden Dim
128
128
128
128
Frame Skip Hardware
4
A100 GPU
A100 GPU
A100 GPU
A100 GPU
Table 8: Hyperparameter settings for all four environments. Note that sequence length is truncated
padded for environments with short episodes, and GPU used for all experiments is NVIDIA A100.
behaviors captured by each cluster and highlights the consistency of behavioral motifs discovered by approach.
Human Study Details
To assess the interpretability and plausibility the discovered behavior clusters, we conducted a two-part human study covering both cluster semantics (Part A) and behavior attribution (Part B). The design for our attribution study (Part B) was inspired by the human evaluation protocol in Deshmukh et al. (2024), where participants asked to choose the trajectory that best explains agent's action. This appendix provides
19

---

Published in Transactions on Machine Learning Research (09/2025)
Table 9: Number of state-action pairs per cluster used for BC training across environments.
Environment
CO
C1
C2
C3
C4
C5
C6
C7
C8
C9
C10
C11
hallchectab-modiun-22
11393
10510
10815
10780
5183
9646
6153
11951
2891
3122
seaquest-mixedl-v)
9710
3321
2894
7471
6692
9980
7081
7763
3510
7398
MiniGridIIoGGoalLLavaa
942
743
1217
932
936
746
938
432
1215
481
1114
1316
pen-expert-vl
8264
4213
9711
4232
2677
8819
Clamrd altte
Chered Keriigho paece Aith seiitagee rovemer
redar Se Chamri.reeeeeee
Set of all pen-expert-vl clusters with their detailed descriptions. Behaviors exhibit different stages of pen manipulation. Cluster 4 is skipped since there are no representative samples for as its quite noisy.
Figure 10:
additional details about the design, instructions, and interface used in each part. Figures and 14 show example questions presented to participants for the interpretability and attribution studies, respectively. All participants had prior familiarity with the environments and were not informed of the clustering method any ground truth labels.
20

---

Published in Transactions on Machine Learning Research (09/2025)
Cluster0: Cheetah is proparing to jump
Cluster1: Cheatah running hind legs
Cluster2: Cheetah leeping forward
Cluster3: Gait Transitions
Cluster4: Cheetah is exhibiting bounding motion
Cluster5: Stable bounding motion
Cluster6: Cheetah falls front legs and recovers
Cluster7: Cheetah Exhibits accelerating leaping motion
Cluster8: Cheetah running leg
Cluster9: Cheetah running hind legs
Figure 11: Set. of all malfchheeah--mediun--e clusters and their detailed descriptions. Most discovered be- haviors show some form of motion. Some clusters are easy to inspect and understand visually but some are difficult given the nature of the environment. The numbers in the images velocitics of the choetah which provides broader context.
21

---

Published in Transactions on Machine Learning Research (09/2025)
Cluster0: Crossing/landing on Lava
Cluster1: Running into obstacle/wall
Cluster2: Exploration Variety
Cluster3: Approaching/landing on Goals
Moving Forward and exploring (usually no-op action)
Cluster4:
Cluster5: Stuck at the wall
Continuous motion (generally without no-ops)
Cluster6:
This typically contained last states of various behaviors representing 'transition state'
Cluster7:
Cluster8: Exploration around the goals
Cluster9: Similar to Cluster 7
Cluster10: Exploration very close to goal, similar to Cluster
Cluster1 1: Exploration around the initial position.
Figure 12: Set of all MiniGridTTwoGolld... clusters with their corresponding description. Behaviors have good diversity, exploratory behaviors of distinct varieties are also discovered using this mechanism.
22

---

Published in Transactions on Machine Learning Research (09/2025)
Cluster0: Exploration, moving around the same location and firing
Cluster1: Idle gameplay
Cluster2: Initial state (s), this cluster typically has 1-2 frames
Cluster3: Downward Motion
Cluster4: Agent killed
Cluster5: Agent is firing in randomly from the same location
Cluster6: Agent is moving right
Cluster7: Agent is moving left
Cluster8: Exploration
Cluster9: Filling Oxygen
Figure 13: Set of all ceaquest-mixed-e clusters and their detailed descriptions. There some meaningful behaviors well as some exploratory clusters where it is hard to exactly define the behavior. This is expected since the data is from a suboptimal agent, thus exploratory behavior is expected.
23

---

Published in Transactions on Machine Learning Research (09/2025)
6. Observe the videos and the sequence closely.
Mark only one oval.
Video1
Video2
Video
Video
Which sequence best describes the key steps involved in learning this action?
Please rank the sequences from (most responsible) to 4 (least responsible) based on how directly they pinpoint to the specific action.
Mark only one oval per row.
1
2
Video1
O
Video2
Video3
Video4
Figure 14: Sample question from Human Study Part B (Attribution Quality). Participants are shown a short context window centered on specific agent action (top row) and are asked to rank four candidate video sogments (bottom) based on how well they explain the observed action. The query context is excluded from the options, encouraging participants to generalize behavioral patterns. The four options include the segment attributed by method, trajectory retrieved via Deshmukh et al. 2024 and two random segments.
24

---

Published in Transactions on Machine Learning Research (09/2025)
2.
Describe this in 1-2 lines. There are multiple rows, you can explain each independently jointly.
100
3.
What is your confidence about the description?
Mark only one oval.
1
4
5
Not
O
Highly Confident
4.
How similar do you think are each of these sequences?
Mark only one oval.
4
Not
Highly Similar
Figure 15: Sample quest ion from Human Study Part A (Behavior Interpretability) Participants are shown a set of trajectory segments sampled from the same behavior cluster and asked to describe the behavior in 1 2 lines. They are then asked to rate their confidence the description and judge how similar the sequences appear. These ratings are used to assess semantic coherence and human alignment of the discovered clusters.
25


Reviews:
[{"review_id": "5zIJNVrpjN", "reviewer": "Reviewer_qhiW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents extensive quantitative and qualitative analysis on the efficacy of the proposed algorithm. Also, it is well-written and easy to understand. However, justification for using a state-prediction loss is missing. As the goal is to discover behaviours, wouldn't it be more natural to predict actions instead of states? Also, the probability distribution of the next state given the current state and action is the transition function, which is the characteristic of the MDP and is independent of the agent's behaviour. In fact, the encoder does not even need to encode the whole segment due to the Markovian property of the MDP. Only attending to the last state-action pair of the segment will suffice for predicting the next state. Yet, the experimental results seem to imply that the encoder is indeed attending to the entire segment and is encoding segments from different behaviours into different codes. Why does it work?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a behaviour discovery algorithm for explainable reinforcement learning. The algorithm consists of two phases. In the first phase, it trains a transformer-based Vector Quantized Variational AutoEncoder (VQ-VAE) to predict the next state based on a sequence of states and actions. Then, it applies graph segmentation to cluster the behaviour segments. Extensive quantitative and qualitative studies were conducted on various offline RL benchmarks to demonstrate the efficacy of the proposed algorithm.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "All of the following are non-critical.\n1. The paper does not cite the D4RL paper Fu et al. (2020).\n2. Fu et al. (2020) states that the medium-level datasets were collected using partially trained SAC policies, not PPO policies.\n3. The original D4RL benchmark does not contain the SeaQuest environment.\n\n### Reference\nFu, Justin, et al. \"D4rl: Datasets for deep data-driven reinforcement learning.\" arXiv preprint arXiv:2004.07219 (2020).", "broader_impact_concerns": "I do not have any concerns on the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5HrTL3gQgP", "reviewer": "Reviewer_CtAc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros: \n\n1.The research problem in interesting and important. While I am not an expert in RL, the introduction about the applications of explainability in Robotics, healthcare sounds interesting. With more RL algorithms being deployed in the real world, understanding them will play an increasingly important role. \n\n2.The limitations are clearly discussed as a separate section. Potential future directions are also provided to address the limitations. \n\n3.Most figures are well-designed, with good illustration that help the readers to understand the main contributions of this paper. \n\n\n\nCons: \n\n1.Baseline models. It seems the baselines compared in the experiment section are quite limited. Could you explain the rationale for this? Also, why these methods are selected. The reviewer understands the specific challenges in terms of comparison and I am not asking for more experiments. More justifications will be appreciated. \nAt the beginning of section 4, please add references to these benchmark environments. \n\n2.In figure 3, is the goal only for example demonstration? Or what messages this figure tries to deliver. While this figure is clear but seems lacks enough readable information. \n\n3.There are 4 sub-components in figure 2. The reviewer suggests the authors to label them with 1234 or abcd, which probably will improve the readability when referred in the main paper. \n\n4.The reviewer is also curious about how the explainable results will benefit specific applciaitons such as in healthcare. Could you provide some examples? Who will use the results and how to benefit from them.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors addresses the limitations of current explainability methods in reinforcement learning, which often overlook recurring strategies and temporally extended behavior patterns. The authors propose a fully offline, reward-free framework that segments agent behavior into interpretable clusters across trajectories, enabling fine-grained attribution of actions to coherent behavior segments. Unlike prior methods that focus on isolated states or entire episodes, this approach captures and explains recurring decision-making patterns. In experiment, four offline RL environments demonstrate that the proposed  method produces more faithful, human-preferred, and coherent explanations than existing SOTA baselines. Further, the reviewer thanks the authors for making their code publicly available.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see weaknesses", "broader_impact_concerns": "The impact is good, potentially has lots of applications in healthcare and other domains", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CY6mbbMKjz", "reviewer": "Reviewer_Ttp7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The proposed method is novel and a reasonable approach for explaining the behaviors of RL policies. \n2. Qualitative visualization and human ratings validates the effectiveness of the proposed method. \n\n**Weaknesses**\n1. The figures in the paper are hard to interpret. Figures 1, 3, 4, 5, and 6 are supposed to show trajectories segmented into behavior clusters, but just by looking at the figures without any context, I can't tell the clusters apart. The captions help, but they induce confirmation bias.\n2. The main experiments involve qualitative visualizations and human studies, which are not quite convincing. \n3. The problem setting of behavior attribution has limited usefulness for the community.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a three-stage pipeline for explaining the behaviors of offline RL policies: (1) fit a transformer-based VQ-VAE to trajectories in the offline dataset, encoding trajectories $s_0, a_0, \\dots, s_t, a_t$ into discrete codes $c_i$ by reconstructing future trajectories ${s_{t+1}, a_{t+1}, \\dots, s_T, a_T}$ (2) build a transition graph using discrete codes as vertices and edge weights determined by the transition probability and similarity, then apply spectral clustering to obtain $K$ clusters, (3) train $K$ behavior-mode polices using data from each cluster. This pipeline can be used to (1) segment trajectories into clusters, and (2) behavior attribution by comparing the predicted action of a policy to the output of behavior-mode policies. Experiments show more preferred human ratings and higher clustering scores compared to prior trajectory-level behavior attribution methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Provide clearer visualizations in figures 1, 3, 4, 5, 6 (e.g. color-code the clusters and add colored backdrops to the frames) or use more conspicuous examples.\n2. Explain the human studies in more detail. The second human study is not very clear. For example, what does it mean for a human to select a video to explain a behavior? Wouldn't the exact same video be the best explanation?\n3. Why limit the setting to RL? This method doesn't make assumptions about the policy learning method. You can similarly apply this to behavior cloning policies and datasets.", "broader_impact_concerns": "The paper does not have ethical implications that warrant a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)