Enhancing Interpretability in Deep Reinforcement Learning through Semantic Clustering
Authors: Liang Zhang
Justin Lieffers
Adarsh Pyarelal
Venue: NeurIPS 2025 poster ()
Keywords: Interpretability
Reinforcement Learning
Clustering
Visualization
Neural Networks
Primary area: reinforcement_learning

Abstract: In this paper, we explore semantic clustering properties of deep reinforcement learning (DRL) to improve its interpretability and deepen our understanding of its internal semantic organization. In this context, semantic clustering refers to the ability of neural networks to cluster inputs based on their semantic similarity in the feature space. We propose a DRL architecture that incorporates a novel semantic clustering module that combines feature dimensionality reduction with online clustering. This module integrates seamlessly into the DRL training pipeline, addressing the instability of t-SNE and eliminating the need for extensive manual annotation inherent to prior semantic analysis methods. We experimentally validate the effectiveness of the proposed module and demonstrate its ability to reveal semantic clustering properties within DRL. Furthermore, we introduce new analytical methods based on these properties to provide insights into the hierarchical structure of policies and semantic organization within the feature space. Our code is available at https://github.com/ualiangzhang/semantic_rl.

Enhancing Interpretability in Deep Reinforcement Learning through Semantic Clustering
Liang Zhang
Justin Lieffers
College of Information Science
College of Information Science
University of Arizona Tucson, AZ 85721
University of Arizona
Tucson, AZ 85721
liangzhearizona.c edu
1 lieffers@arizone zona. edu
Adarsh Pyarelll
College of Information Science
University of Arizona
Tueson, AZ 85721
adarsh@arizona edu
Abstract
In this paper, we explore semantic clustering properties of deep reinforcement learning (DRL) to improve its interpretability and deepen our understanding of its internal semantic organization. In this context, semantic clustering refers the ability of neural networks cluster inputs based on their semantic similarity in the feature space. We propose a DRL architecture that incorporates a novel semantic clustering module that combines feature dimensionality reduction with online clustering. This module integrates seamlessly into the DRL training pipeline, addressing the instability of t-SNE and eliminating the need for extensive manual annotation inherent to prior semantic analysis methods. We experimentally validate the effectiveness of the proposed module and demonstrate its ability to reveal semantic clustering properties within DRL. Furthermore, we introduce new ana- lytical methods based these properties provide insights into the hierarchical structure of policies and semantic organization within the feature space. Our code is available https:: thub. com/ ual lngzhang/ swapt///emmman.........
Introduction
Deep reinforcement learning (DRL) has been widely applied in domains such robotics, autonomous systems, game playing, and healthcare, due to its ability to solve complex decision-makinn tasks [1-3]. However, the black-box nature of DRL models obscures the decision-making process, potentially leading to unforeseen risks. In this study, we explore semantic clustering properties to improve the interpretability of DRL models. We the term semantic clustering refer to the process of grouping states eliciting similar agent behaviors under comparable environmental contexts (e.g.. approaching target, jumping
higher platform in Procgen). Studying semantic clustering in DRL helps reveal the model's 's internal knowledge structure and semantic relationships between states, enhancing the interpretability and transparency of DRL models.
Although semantic clustering has been thoroughly investigated in natural language processing (NLP) [4, 5] and computer vision (CV) [6-9], it remains underexplored in DRL due to the complexity introduced by temporal dynamics and the absence of direct supervised signals. The sequential nature of decisions in DRL further complicates the task of capturing evolving semanties. Early work introduced external constraints e.g., bisimulation [10, 11] and contrastive learning [12-15]- to
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

shape feature spaces conducive to semantic clustering. In contrast, we focus on investigating whether DRL can intrinsically exhibit semantic clustering capabilities.
Mnih et al. [16] and Zahavy, Ben-Zrihem, and Mannor [17] analyzed the semantic distribution of the DRL feature space for Atari games using I-distributed Stochastic Neighhor Emhedding (t-SNE) [18]. However, these studies are limited in multiple ways: (i) they are limited to a small set of Atari games with fixed scenes, making it difficult distinguish whether clustering arises from pixel similarity or semantic understanding, (ii) Zahavy, Ben-Zrihem, and Mannor [17] manually define features for specific games, imposing substantial human effort, and (iii) both studies rely on t-SNE visualization for semantic analysis, which tends to produce unstable results and lacks an automated clustering mechanism. Thus, these approaches require significant manual effort for feature space annotation and analysis, hindering comprehensive semantic analysis and integration into downstream tasks.
Specifically, we make the following key contributions in this paper:
We comprehensively explore the semantic clustering properties of DRL, advancing the understanding of the black-box decision-makee processes. Unlike prior work that uses fixed-scene Atari games, we use Procgen' [19], which offers rich semantic diversity and dynamic, procedurally generated environments.
We introduce a novel end-to-end architecture that integrates feature dimensionality reduc- tion with online clustering, overcoming the limitations of prior t-SNE-based analyses and providing a more stable, effective means to study semantic properties in DRL.
We present new analysis methods to reveal the internal semantic structure, uncover the hierarchical organization of policies, and identify potential risks in DRL models.
2
Related Work
Semantic Clustering in NLP and CV Prior work in NLP has shown that the spatial arrangement of word embeddings reflects semantic similarities, with emmanticall--reltee terms forming clusters in the embedding space [4, 5]. Similarly, in computer vision, images with similar content are positioned closely in the learned feature space [6-9].
Semantic Clustering in DRL Mnih al. [16] and Zahavy, Ben Ben-Zrihem, and Mannor [17] have previously explored visualizing the DRL feature space using t-SNE. In these studies, t-SNE visu- alizations show that features of states with close pixel distances tend to cluster together. However, due to the fixed nature of the scenarios they used (Atari games), semantic clustering could not be conclusively verified. This limitation motivates our use of Procgen to validate our approach.
Interpretability of DRL DRL interpretability research often focuses on video games due to their controlled environments and clear rules, which make analyzing (ecision-making processes casier. PW-Net [20] uses human-friendie prototypes to explain the model's decision-makinn DIGR [21] generates saliency maps tlat highlight the most relevant features influencing the agent's decisions. Concept policy models integrate expert knowledge into multi-agen aulli- RL, enabling real-time interven- tion and interpretation of agent behavior [22]. MENS-DT-RL [23] applies decision trees provide rule-based explanation of the learning process. Furthermore, attention mechanisms and symbolic reasoning frameworks have also been applied to enhance interpretability [24-26). Our work explores how DRL, models internally organize information, offering new insights into the structure of learned representations.
VQ-VAE VQ-VAE [27] is family of generative models that combine classic VAE with discrete latent representations through posterior parameterization. Recently, VQ-VAE has been applied to various tasks, including high-resolution image generation [28], video generation [29], and speech coding [30]. It has also been used in model-based DRL to train transition models [31, 32).
3 Method
Our proposed architecture with a novel semantic clustering module is presented in Figure 1.
Detailed environment instructions are available here, in Appendix A of Cobbe et al. [19], and in the repository.

---

Background The VQ-VAE workflow begins with an encoder network E that maps imput then quantized by mapping it to the nearest
latent representation E (x). This representation
embedding in codebook (exlk E (1, (1,2.... ..... K) K)) The quantized representation is then passed into decoder network Ó reconstruct the input x. The loss function for VQ-VAE defined as:
LvQ-VAE = ||x 11x-Dree D(e 112 |[sg (É(x)) - exl1² +B Bl(sg(ex)) E
(I)
where sg is stop-gradient operator and / weights the distance reduction between the encoded output f (x) and its closest embedding eg.
In this paper, we modify VQ-VAE to (i) assign features to the nearest VQ embedding for clustering, (ii) seamlessly integrate with DRL training, enabling simultaneous clustering and policy learning, and (iii) enhance clustering and interpretability through joint training with additional losses, Further details provided in 3.2.
Classic DRL Training Pipeline
Extrator
Net
DRL
Fipul Bark
Foature
Oode
Semantic Clustering Module
Vector
FDR
VO
Festure
Embeddinng
Figure Overview of our architecture. The upper segment represents the classic DRL training pipeline, while the lower segment introduces the semantic clustering module. The Feature Di- mensionality Reduction (FDR) net reduces the dimensionality of state features, resulting in FDR features, which the vector quantizer then processes to generate discrete VQ codes (denoted w which
represent states associated with clusters clusters-along with the closest VQ embeddings. Subsequently, k is integrated into the state feature by element-wise addition after being expanded to match the state feature dimensions, enabling conditional policy training that better supports the integration of downstream tasks.
3.1 Semantic Clustering Module
To overcome the limitations of previous t-SNE-based semantic analyses (see 1), we propose a novel semantic clustering module, which includes dimensionality reduction and online clustering.
Dimensionality Reduction Given the complexity of states in most DRL applications. their features are often nigh-diimnsional.. For example, Mnih et al. [16] use a 512-dimensional feature vector when training DQN on Atari games. Clustering high-dimensional features is challenging due to the curse of dimensionality [33, 34]. To mitigate these issues, we reduce feature dimensionality before clustering, resulting more robust clustering outcomes. This not only simplifies the clustering process but also enables human-interpretabl visualizations, typically in 2D.
The instability of t-SNE arises from its non-convex objective function, making it highly sensitive to initialization and leading to varied and unstable visualization outcomes [18, 35]. To overcome these challenges, we propose the Feature Dimensionality Reduction (FDR) network. This network remaps high- singh-dimnnsional features to 2D using policy training data for online training, ensuring stable and efficient mappings after training. The FDR network's loss function is designed preserve the consistency of distance relationships between high-dimensional and 2D feature spaces, measured by pairwise similarities as described in $ 3.2.
Online Clustering
b-SNE-based analyses (e.g., [16, 17]) require per-state inspection and manual grouping because the plots lack clear cluster boundaries and often split semantically similar states across disconnected regions, making human curation time-consuming. To reduce such extensive annotation and facilitate

---

downstream integration, we introduce an online clustering approach implemmenedd via modified VQ-VAE V-VV-Ethaa that automatically segments the feature space and supports semantic analysis. This removes manual grouping: annotators instead watch few short clips per discovered cluster and provide semantic summary (see Table 1), typically within = minutes per environment. Details of the modified VQ-VAE design are provided in $ 3.2.
3.2
Loss Function Design
The loss function for our proposed framework given by
Liotal
LDRL + Aeul WWFRR<FDR + NVO-VAEL'VQ.VAE)
(2)
The DRLloss, function LDRL comes from the original DRL, model. WFDR and wyQ VAE the weights of the FDR loss (LeDR) and the modified VQ- VA-VAE loss (o'vo-vAE). respectiell.. Actl represents the control factor. We explain each of these components below.
FDR Loss LFDR is based on state features from the DRL training batch and FDR features generated by the FDR network. We use the Student's r-distribution for pairwise similarities as captures nonlinear structures and efficiently measures pairwise relative positions of features within batch without requiring the entire feature set, making it ideal for online clustering.
The pairwise similarities of state features Pij are given by
j(i,j) Pij = Ekst d(k,I)' where d(m, n) (1 (fm) f((s))
Here, f' is the feature extractor, s; is the th state in a batch, is the Student's- lfggrees-of-freedon parameter, and || is the la norm. The pairwise similarities for FDR features, qij are computed using the same expression (3), but with f replaced by g o where g is the FDR net. In contrast to other deep clustering studies, e.g., Xie, Girshick, and Farhadi [36] and Li, Qiao, and Zhang [37], same degree of freedom is selected for both high- and now-dimensional similarities, ensuring that the original distance relationship between features is maintained in the oow-dimensional space.
The FDR loss is given by
LFDR = -E Pt log (41))-
(4)
Minimizing LFDr encourages the low-dimensional mapping to preserve the pairwise neighborhood structure of the high-dimensions features.
Modified VQ-VAE Loss To perform clustering, we the second term of LvQ- from (1), which moves VQ embeddings closer to neighboring FDR features. These embeddings function similarly to centroids in online k-means [38] clustering. Since the other terms are unnecessary for our model, we only retain and modify the second term to define the modified VQ-VAE loss:
L' = ||sg Ig (f (s)) - -xx12
(5)
where eg is the closest embedding in the codebook to the FDR feature g(f(s)).
Control Factor Since effective semantic clustering relies a clear and distinguishable semantic distribution that is often difficult to achieve in the early stages of training, we propose an adaptive control factor (Aem) strategy updated according to training performance (see Appendix A).
Improved Clustering Our loss design not only achieves dimensionality reduction and cluster- ing but also enhances clustering properties, making the states within each cluster more compact (smaller intra-cluster distances) and the cluster boundaries more separable. This is crucial for clearly distinguishing the semantics of states at the cluster boundaries, further enhancing the model's inter- pretability. Because of the stop-gradien in L'VO-VAE it does directly pull FDR features toward their nearest codebook embeddings. However, when the FDR features become denser during joint training with LEDR. L'V-VVEE and thus decreases, yiclding tighter clusters. Morcover, since

---

LeDR aligns the affinity matrices and q, this densification in the low-dimensional FDR space is reflected and propagates into the high- dimensional state features. We demonstrate the improved clustering in 4 and provide more evidence of this enhanced clustering property and the intrinsic nature of semantic clustering in DRL in $ C.2.
Algorithm 1: PPO with Semantic Clustering Module (SCM)
Input: PPO network parameters 0, FDR network parameters é SCM hyperparameters, and PPO
hyperparameters such as value loss weight Wvalue, entropy loss weight Weatropy-
for each training iteration = 1.2.... do
Collect N trajectories D; {Ti, .... TNY using policy ng:
Trajectory collection
for each epoch j 1, 1,2, .... do
for each minibatch M S D) do for each state E M do fm < (Sm): PEDR + 8 (Im);
Extract state feature // Extract FDR feature // Assign to nearest VQ embedding
7
km - arg ming ||PER exll:
8
km hexpand expand( km dfm(fm)):
// Broadcast state feature dim.
10
In - fm km
// Apply element-wise addition // Compute policy outputs
x x(a(sm) c R (Pfuood)
11
V(sm) (Pffsoe))
// Compute value outputs
13
Lrro - Lpolicy Evalue- Wentropy Lentropy:
// PPO loss
14
LsCM + WFDR + wVQ-vAE
SCM loss
15
Liotal - Lppo + demn
// Total loss
16
Update 0, o, and {ex) K- by minimizing Litall
// Parameter update
Advantages of Online Training Online training offers several advantages: (i) it enhances clustering by incorporating the training of Lioaal (ii) training the VQ code & with a latent-conditioned policy (als, k) (where is the action) supports extension downstream tasks, such macro action selection in hierarchical learning, and (iii) it improves memory efficiency by eliminating the need to store large number of states during model training.
Training Process The training process our framework builds upon the structure of the original DRL algorithm while incorporating the semantic clustering module (SCM) by using (2) for total loss calculation. We take PPO [39] as an example, and the training procedure is outlined algorithm 1.
Simulations
In this work, we primarily study the intrinsic characteristics and black-box decision decision-making of DRL, and address the instability of t-SNE visualizations used in prior studies. Therefore, this section aims to: (i) compare t-SNE to validate the stability and effectiveness of the proposed clustering method, (ii) assess the semantic clustering properties of DRL improve interpretability, and (iii) introduce new methods to analyze policies and internal model characteristics, identifying issues in DRL fecision-making The integration of our module has minimal impact on performance (see C.1 and Appendix D).
4.1 Clustering Effectiveness Evaluation
We demonstrate the clustering effectiveness of our proposed approach using the CoinRun game from Proegen an example. Similar results can be easily extended to other games using the code and checkpoints provided in the supplementary material. We use trained model collect states, where the agent selects actions randomly with probability of 0.2 and follows the trained policy with probability of 0.8 to ensure diverse state coverage. States are sampled with a probability of 0.8, and 64 parallel environments collect states over 500 steps, resulting in =25,000 25,000 states for visualization.
5

---

The t:SNE space
The L-SNE space
The t-SNE space
75
300
Cluster
Cluster
25
Cluster
Cluster 5
Cluster
-30
Cluster
-75
-100
-50
50
-100
t-SNE
(a) I-SNE space of PPO.
(b) t-SNE
space of our
(c) Same as (b), but with 50% fewer
method.
states.
The FDR space
The FDR space
The L-SNE space
100
Cluster 0
Glstter
-2.5
-2.5
50
F3R
5.90
Claster
Cluster
-7.5
Cluster 6
Cluster
-10.0
-10.0
-50
-12.5
-225
-10
FDR
(d) FDR space of our method.
(c) Same as (d), but with 50% fewer states.
(f) Same (c), but with different ran- dom seed.
Figure 2: Visualization of features in t-SNE and FDR spaces using PPO and our method. To enable comparison, feature colors in the t-SNE visualizations of our method correspond to the cluster colors in the FDR space, while PPO features are shown in orange due to the absence of clustering. Unlike 1-SNE, which fails produce clearly separable clusters and exhibits sensitivity the number of states and random seeds, our method yields well-separated and stable clusters under varying conditions.
Note that the cluster colors (indices) in the t-SNE plots are assigned by our method and are used solely to facilitate comparison of spatial relationships.
Cluster Separation and Improved Clustering The t-SNE visualization of PPO (Figure 2a), spreads features across the space without forming clear clusters, limiting its utility for clustering analysis and requiring detailed manual examination of certain areas, in previous studies. In contrast, the t-SNE visualization of our method (Figure 2b), reveals numerous distinct, small clusters. States within each of these clusters originate from the same semantic group identified by our method. This dispersion into multiple smaller clusters is due to t-SNE's focus on local structures and its tendency avoid crowding, causing complete semantic clusters scatter. The visualization in the FDR space (Figure 2d). displays clear and separate complete clusters, which are identified by VQ codes. $ C.2 presents stop-gradient ablation that disables our proposed module while keeping the backbone and training protocol fixed; visualizations show fuzzier, less separable clusters than the full model, further supporting our gains in sharpness and coherence.
Sensitivity to Number of States Our method's stability is showcased in Figures 2c and 2e, where the number of processed states is reduced by 50%. Unlike the drastic changes in feature distribution seen in the t-SNE space (Figure 2c), the FDR space (Figure 2e) exhibits a stable mapping, merely reducing the quantity of features without altering their spatial distribution.
Sensitivity to Random Seed While the t-SNE representation is sensitive to randomness, as demon- strated by the significant difference between Figures 2c and 2f, the FDR space's 's mapping remains unchanged even when the random seed altered, maintaining the distribution in Figure 2c. t-SNE's randomness primarily stems from its random initialization and non-convex optimization process, leading to significantly different visualizations with different random seeds. In contrast, our model produces stable feature mapping after training, which does not vary with random seeds.
These clear contrasts highlight the robustness of FDR over the instability of t-NE, addressing prior limitations and enabling stable semantic clustering and analysis. In Appendix E, we further present a statistical comparison of common dimensionality reduction methods across multiple clustering metrics, demonstrating that method achieves superior clustering performance.

---

4.2 Semantic Clustering in DRL
Mean Images
The FDR space
23
Cluster
Clustar
Cluster
Cluster
Clusger
Cluster S
Cusster
Clisgar 7
-100
-125
3.00
25
$25
FDR
Cluster
Cluster 5
Figure 3: State examples in the Ninja FDR space and the mean images of clusters. Each dashed box contains a sequence of consecutive states assigned to the same cluster, with dotted arrows indicating their corresponding FDR feature positions. These examples demonstrate that semantically similar and temporally adjacent states are grouped into the same cluster, highlighting the learned semantic coherence. Descriptions of the state sequences in the clusters are provided in Table 1.
Table 1: Cluster descriptions and mean image outlines for the Ninja game
Cluster
Description
Mean image outlines
The agent starts by walking through the first platform and then performs high jump to reach higher ledge.
Essential elements are outlined, c.g., left-side wall, the current position of the agent on the first platform, and the upcoming higher ledges.
The agent makes small jumps in the middle the scene.
We can observe the outlines of several ledges below the agent.
Two interpretations are present: 1) the agent starts from the leftmost end of the scene and walks the starting position of Cluster and 2) when there are no higher ledges to jump to, the agent begins from the scene, walks over the first platform, and prepares to jump to the subsequent ledge.
The scene prominently displays the distinct out- line of the left wall and the first agent's current position close both of them.
platform. The
The agent walks the ledge and prepares jump higher ledge.
The agent is standing the outline of the rent ledge and the following higher ledges.
After performing high jump, the agent loses sight of the ledge below.
The agent is performing high jump.
The agent walks the ledge and prepares jump onto ledge at the same height or lower.
The agent is standing the outline of the rent ledge the following ledges the same height or lower.
The agent executes a high jump while keeping the ledge below in sight.
The agent is performing a high jump and the outline of the ledge below is visible.
The agent moves towards the right edge of the scene and touches the mushroom.
The outlines of the wall and platform on the far right are visible.

---

In this section, we illustrate semantic clustering analysis using the Ninja game, in which the agent goes from left right, jumping over various ledges and scores points by touching the mushroom on the far right. In Appendix G, we analyze additional games, reaching similar conclusions.
Mean Image Analysis We performed a qualitative analysis of the mean images of states within each cluster. Figure 3 presents state examples from the FDR space of Ninja along with the mean images of each semantic cluster, and Table contains natural language descriptions of the clusters well as notable features of the mean images corresponding to each cluster. Corresponding videos can be found in the supplementary material.
Unlike static semantic clustering in some CV and NLP tasks, where clustering is based on a single image or word, DRL's semantic clustering is dynamic in nature state sequences with similar semantics are grouped into the same semantic cluster. Notably, this semantic clustering goes beyond pixel distances and operates on semantic understanding level of the environment, as illustrated in figures 3 and 4. This generalized semantic clustering emerges from the DRL model's inherent ability learn and summarize from changing scene dynamics, independent of external constraints like bisimulation or contrastive learning, and without the need for supervised signals. The neural network's 's internal organization of policy-relevant knowledge indicates clustering-basee spatial organization based on semantic similarity. Furthermore, we find that video sequences within clusters can be summarized using natural language, akin the 'skills' humans abstract during learning processes.
Cluster 2
Cluster 4
Cluster 3
Cluster 7
Figure 4: Three episodes from the Ninja game. States within colored dashed boxes correspond to clusters of the same colors in Figure 3. Solid gray arrows indicate omitted intermediate states from the same cluster, while ellipses represent other omitted states. These visualizations illustrate consistent semantic alignment in cluster assignments across different episodes.
Table Human evaluation results
Mean Score (SEM)
No. Statement
Jumper
FruitBot
Ninja
The clips of each cluster consistently display the same
4.24 (0.15)
4.10 (0.11)
4.30 (0.15)
skill being performed
The clips of each cluster match the given skill description
4.36 (0.16)
4.16 (0.11)
4.20 (0.17)
The identified skills aid in understanding the environment and the Al's ss decision-making process
4.50 (0.22)
4.10 (0.18)
4.20 (0.20)
Human Evaluation In addition to qualitatively analyzing the mean images, we hired 15 human evaluators to validate the semantic clustering properties. Evaluators were adults (18+), native or highly proficient English speakers, with basic video game experience and a brief training session. Specifically, video sequences from each episode are segmented into multiple clips based on the cluster each frame belongs to, and these clips are grouped by cluster for evaluators to review. Each evaluator watched these grouped clips and responded to three interpretability- set of three games (Jumper, Fruitbot, and Ninja). The response for each question was chosen from five-point Likert scale with the options: Strongly Disagree (1), Disagree (2), Neutral (3), Agree (4), and Strongly Agree (5). Further details the evaluation procedure provided in Appendix I.
statements for two out of
8

---

The statements and the results of the human evaluation are provided in Table 2. The mean scores for all statement- eeniinnmenniroommnntt combinations are greater than with the exception statements and for FruitBot, for which the lower bounds on the mean set by the standard error of the mean (SEM) are 3.99 and 3.92 respectively. The slightly lower score on FruitBot may caused by the behavior description of clusters, the agent's 's relative distance to the wall ahead (far/near) and the agent's s relative position on the screen (left/center/right) require a higher degree subjective judgment. In contrast, Jumper has a clear radar for direction and position information, and Ninja has more explicit behavioral reference objects, e.g., ledges and mushrooms. Overall, these results suggest that humans generally agree that model possesses semantic clustering properties and supports interpretability.
4.3
Model and Policy Analysis
108 space
space
FORC
F3R
(a) Overall perspective.
(b) First example point
(c) Second example point (d) Third example point
Figure 5: Hover examples in the FDR space of Ninja. We observe sub-cluster in the FDR space as an example from an zoomed-ou perspective (a) and zoomed- in perspectives (b), (c), and (d). The agent standing on the edge of ledge. Although the scenarios of (b), (c), and (d) are different, proposed method effectively clusters semantically consistent features together in the FDR space.
To better explore the knowledge organization within the internal space of DRL models, we developed visualization tool (see Figure 5 for example). The tool supports 'statically' analyzing the semantic distribution models models-speciffee specifically, (i) when the mouse cursor hovers over a specific feature point, the corresponding state image displayed, and (ii) the tool includes zooming functionality to observe the semantic distribution of features in detail within clusters.
(a)
Cluster
(b) Cluster
(c) Cluster
(d) Cluster 5
(e) Cluster
(f) Cluster 3
Figure 6: Policy analysis examples in Ninja, showing states assigned different clusters.
In addition, we propose more 'dynamic' analysis method the VQ code enables us to determine the cluster to which the current state belongs, which allows for the semantic segmentation of episodes, as exemplified in Figure Our model excels at breaking down complex policies, thereby shedding light on their inherent hierarchical structures. Moreover, this segmentation is based on semantics, making it understandable to humans and likely to improve interpretability downstream hierarchical learning tasks. Consequently, this method introduces 'dynamic' strategy for dissecting policy structures.
We present policy analysis examples in Figure 6, leveraging clustering results from our method. Figures 6(a) and 6(b) show consecutive states assigned to clusters 5 and 7, respectively (see Figure 3 and Table 1). In 6(a), the right-side wall is absent and the agent walks along the ledge (cluster 5); in 6(b), right-side wall appears and the agent transitions to cluster 7, approaching the mushroom. Because the mushroom is visible in both frames, this cluster change is driven by detecting the ridh wall rather than by the mushroom's 's presence. This finding is confirmed in 6(c), which shows another state from cluster without any mushroom present, where agent continues along the ledge, incorrectly perceiving the conditions for cluster-7 behavior. Figures 6(d) and 6(e) depict states within the same episode: in 6(d), the agent initially plans to jump onto a lower ledge (cluster 5), but upon seeing a higher, safer ledge in 6(c), it shifts its strategy accordingly (cluster 3). Similarly,

---

6(f) shows state that has been assigned to cluster 3 by our model. helping us anticipate the agent's 's future behavior of jumping onto the higher ledge. These analyses demonstrate how our model helps clarify policy behaviors, uncover decision-making structures, and identify potential issues.
5 Limitations and Future Work
Our approach has several limitations to be addressed future work. First, it relies on clear semantic distributions, which can become unstable when policies deviate significantly from optimal behavior, resulting in ambiguous clusters. More robust clustering methods may be needed to improve stability. Second, the method is unsupervised, selecting appropriate number of clusters is crucial- too few clusters reduces clarity, while too many clusters causes semantic fragmentation. We used eight clusters to balance interpretability and granularity, but future work could explore adaptive techniques that adjust the number of clusters based on task complexity, the elbow method, silhouette- score optimization, etc. For further analysis of the impact of cluster numbers on performance and interpretabilityy see Appendix D. Furthermore, policy interpretations are manually described. In future work, we aim to automate behavior summarization and explanation (e.g.. using GPT-4V). In addition, since the FDR module optimizes pairwise similarities, a natural extension is to replace the current affinity with alternative measures (c.g., cosine similarity or bisimulation metries). Lastly, we plan to extend this method to other DRL algorithms, benchmarks, and settings.
Conclusion
In this paper, we investigated the semantic clustering properties of DRL. Using a novel approach that combines dimensionality reduction and online clustering, analyzed the internal organization of knowledge within the feature space. Our method provides a stable mapping feature positions and enhances semantic clustering, revealing meaningful structures in continuous sequences of video game states. We demonstrate that semantic clustering in DRL arises dynamically as the agent interacts with its environment. As the agent explores diverse states during reinforcement learning, naturally clusters semantically related states based on spatial and temporal relationships. This dynamic clustering exploits regularities in the environment, offering a unique approach compared to the static clustering observations NI.P and CV.
Acknowledgments
Research was sponsored by the Army Research Office and was accomplished under awards W91INE 20-1-0002 and I911NF-24--00034 The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. We gratefully acknowledge the anonymous reviewers and the program committee for their thoughtful feedback and constructive suggestions, which helped improve this work. We also thank Huy Le and Robert Lopez for their valuable assistance with the human evaluation.
10

---

References
[1] Vincent François-Lavert al. "An Introduction Deep Reinforcement Learning", In: Found. Trends Mach. Learn. 11.3-4 (2018), pp. 219-354. DOI: 10. 1511112202200771.. URL: https:
//doi.org// 1561 /2200000071
[2] Shengbo Eben Li. Reinforcement Learning for Sequential Decision and Optimal Control. Springer, 2023. ISBN: 978-981-19-7783-1. DOI: 10. 1007 / 978 -
981 - 19- 8. URL:
https://doi.org: org/10. 1007/978-981-19-778---
[3] Ashish Kumar Shakya, Gopinatha Pillai, and Sohom Chakrabarty. "Reinforcement learning algorithms: A brief survey". In: Expert Syst. Appl. 231 (2023), p. 120495. DOI: 10. 1016/J. ESWA. 2023. 120495. URL: https:httt/:/oorrroooooo htg.:/o...ggggg/ 1016/j wa16//.eswwa eswa. 2023. 1823.1204955
[4] Xin Rong. "word2vec Parameter Learning Leariigg Explained", In: CoRR abs/1411.2736 (2014), arXiv:
1411. 2738. URL: http:://axxi...t///////////// http//arxii..rrrabb//1111 2738.
[5] Jeffrey Pennington, Richard Socher, and Christopher
Global Vectors for Word Representation". In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, Special Interest Group of the ACL. Ed. by Alessandro Moschitti, Bo Pang, and Walter Daelemans. ACL, 2014, pp. 1532-1543. DOI: 10. 311151111111111111 https://doi. www.g//..000 1tt.:/r///...11/11111111444 3115/v1/d14-116.
Manning. "Glove:
URL:
[6] Fuchen Long et al. "PointClustering: Unsupervised Point Cloud Pre-training using Trans- formation Invariance in Clustering". In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. IEEE, 2023, 2023. 02090. URL: https: / //ooi.ii.///////////../ org/10.
pp. 21824-21834, DO1: 10.
1109/CVPR52729. 2023. 02090.
[7] Avinab Saha, Sandeep Mishra, and Alan C. Bovik. "Re-IQA: Unsupervised Learning for Image Quality Assessment in the Wild". In: IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. IEEE, 2023, pp. 5846-5855 DOI: 10. 1109/CVPR527299 2823, 00566. URL: https: http://o...og/111111011
CVPR52729 2021772222222....222 2023. 2023.00566.
[8] Sungwon Park et al. "Improving Unsupervised Image Clustering With Robust Learning". In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. Computer Vision Foundation / IEEE, 2021, pp. 12278-12287, DOI: 10. 1109/ CVPR46437 2021 01210. URL: /ttttp::///ppnecccessss thecvf com/ content /CVPR20212 Lttp:/uprrriisssoooooooeeeee...a ttt. Robustt5__eerrnnng5__.CCC ht__r//XSS._.2222___epeerr..... html.
[9] Lei Zhou and Weiyufeng Wei. "DIC: Deep Image Clustering for Unsupervised Image Segmen-
tation". In: JEEE Access 8 (2020), pp. 34481-34491 DO1: 10. 1100/009CCC / 2020 2974496.
URL: htttps://dot: org/ httg///...1///E.....001 1109/ACCESS. 2020. 2974496.
[10] Mete Kemertas and Tristan Aumentado-Armstronng "Towards Robust Bisimulation Metric Learning.. In: Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual. Ed. by Mare' Aurelio Ranzato et al. 2021, pp. 4764 4777. URL: https / proceedings neurips. cc/ pwpp:/pppp///// 2021 www.hh/mhhhheeeeeeee /ttp:hh/256ff88222225222 dadd 148-1488 Abstract. html.
[11] Amy Zhang et al. "Learning Invariant Representations for Reinforcement Learning without Reconstruction" In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL:
net net/forum?id==255
[12] Benjamin Eysenbach et al. "Contrastive Learning as Goal-Conditioned Reinforcement Learn- ing". In: Advances in Neural Information Processing Systems 35; Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022. Ed. by Sanmi Koyejo et al. 2022.
[13] Rishabh Agarwal et al. "Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning", In: 9th International Conference on Learning Representations, ICLR 2021, Viruaal Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL: https: //openreview

---

[14] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. "CURL: Contrastive Unsupervised Representations for Reinforcement Learning". In: Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event. Vol. 119. Proceedings of Machine Learning Research. PMIR, 2020, pp. 5639-5650, proceedings, mlr presss/v119/asskn222. htmm..
URL: http://
[15] Vihang Patil et al. "Contrastive Abstraction for Reinforcement Learning.. In: CoRR abs/2410.00704 (2024). DOr: 10. 48550 / ARXIV 2410 00704. arXiv: 2410. 00704. URL:
https://doi.oon http://oooooooo/////// 7.48550/arXis
007004.
[16] Volodymyr Mnih al. "Human-leve control through deep reinforcement learning.. In: Nat. 518. 758. (2015), pp. 529-533, DOI: 10. 008/nature 14266. URL: httpps//doo http///oo.or///..../// 1038/nature14236
[17] Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. "Graying the black box: Understanding DQNs In: Proceedings of the 33nd International Conference Machine Learning. ICML 2016, New York City, NY, USA, June 19-24, 2016. Ed. by Maria-Florina Balcan and Kilian Q. Weinberger. Vol. 48. JMLR Workshop and Conference Proceedings. JMLR.org, 2016,
pp. 1999-1908. URL: http: hatp://proceedinge. mlr. wwt.:pees/////aaaeaa//
html.
[18] Laurens van der Maaten and Geoffrey Hinton. "Visualizing Data using t-SNE". In: Journal of Machine Learning Research 9.86 (2008), pp. 2579-2605, URL: http://jmll //wrr. org/ hrrgpapers/
/
html.
[19] Karl Cobbe al. "Leveraging Procedural Generation to Benchmark Reinforcement Learning.. In: Proceedings of Machine Learning Research 119 (2020), pp. 2048-2056. URL: http:
//proceedings. mlr wwwpspelssleeeeeeee 19/11//oo/ h1t/////c/eaa.../e./// html.
[20] Eoin M. Kenny, Mycal Tucker, and Julie Shah. "Towards Interpretable Deep Reinforcement Learning with Sluman-Friendly Prototypes". In: The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
URL: https: //ppnreviiee net/ fert/form? i d=hWWWY55C_JJ@xx
[21] Jinwei Xing et al. " "Achieving efficient interpretability of reinforcement learning via policy distillation and selective input gradient regularization", In: Neural Networks 161 (2023), DOI: 10. 1816/J.NEUNET. NEUNET 2023. .22.. URL: httpp:://oot org/10. 1016/j. neunet. 2023.01.025.
pp. 228-241.
[22] Renos Zabounidis et al. "Concept Learning for Interpretable Multi-Agent Reinforcement Learning". In: Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand. Ed. by Karen Liu, Dana Kulic, and Jeffrey Ichnowski. Vol. 205. Proceedings of Machine Learning Research. PMLR, 2022, pp. 1828-1837. URL: https: /ptp:////ooccce..... html.
mlr.
[23] Vinícius G. Costa et al. "Evolving interpretable decision trees for reinforcement learning". In: Artif. Intell. 327 (2024), p. 104057. Dor: 10. 1016 J. ARTINT 2023. 104057. URL: https://doi. wrg/10..111 1ww11006..... wttp:/..attn..... 202.. 104057.
[24] Wenjie Shi al. "Self-Supervised Discovering of Interpretable Features for Reinforcement Learning". In: IEEE Trans. Pattern Anal. Mach. Intell. 44.5 (2022), pp. 2712-2724. Dor:
10. 1099119/TP 2020 3037898. URL: htwp::ttp/sseeo http::://doi.ooo org///..rr//01 10. 1109/TPAMI, 20t.222220.3737 3037898. [25] Alexander Mott et al. "Towards Interpretable Reinforcement Learning Using Attention Aug- mented Agents". In: Advances in Neural Information Processing Systems 32: Annual Confer- ence on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14 8-14, 2019, Vancouver, BC, Canada. Ed. by Hanna M. Wallach et al. 2019, pp. 12329-12338.
[26] Daoming Lyu et al. "SDRL: Interpretable and Data-Efficient Deep Reinforcement Learn- ing Leveraging Symbolic Planning". In: The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, Jamuary 27- February 1, 2019. AAAI Press, 2019. pp. 2970-2977999 DOI: 10. 1609/aaai v33101. 0301 2970. 1609/aaai. v33ie1. 33012970.
URL: https: //doi. org/10. 10
[27] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. "Neural Discrete Represen- tation Learning". In: (2017). Ed. by Isabelle Guyon et al., pp. 6306-6315, URL: https: / //proceedings.neuu nwwrips.ccccmmm httpp////.7.aaeeeeaee0o000000000 Abstract. html.
12

---

[28] Ali Razavi, Aäron van den Oord, and Oriol Vinyals. "Generating Diverse High--idelitg Images with VQ-VAE-2". In: Advances in Neural Information Processing Systems 32: Anmual Confer- ence Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada. Ed. by Hanna M. Wallach et al. 2019, pp. 14837-14847. URL.: https: / poroceedings.neuriie cttppsccc//ooo wwwc/papp///eeeecceeceeeeeeeeeee 2219 htte//ahhhhhhheaeeeeeeee 81 cd977.54DB Abstract. html.
[29] Wilson Yan et al. "VideoGPT: Video Generation using VQ-VAE and Transformers". In: CoRR
abs/2104. 10157 (2021). arXiv: 2104. 10157.. URL: https: /ttp///aaivvvvvava www.ariww...ooiaaoeeeeeeooeeeeee www.oo/g//////o/eeeee 10157.
[30] Cristina Gârbacea al. "Low Bit-rate Speech Coding with VQ- VAE and WaveNet Decoder".
In: IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019. IEEE, 2019, PP- 735-739. DOr: 10. 1109/ URL: https: //to://doiioooo org/10. 1 T09/ICASSP..... 2019 8683277.
ICASSP 2019. 998882777.
[31] Sherjil Ozair al. "Vector Quantized Models for Planning". In: Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event. Ed. by Marina Meila and Tong Zhang. Vol. 139. Proceedings of Machine Learning Research. http:tttc//pcceediinngge pww.eedince...../ mlr www.ppreessvv33 ozair21a
PMLR, 2021, pp. 8302-8313 URL: http: : html.
[32] Jan Robine, Tobias Uelwer, and Stefan Harmeling. "Diserete Latent Space World Models for Reinforcement Learning". In: CoRR abs/2010.05767 (2020). arXiv: 2010 05767. URL:
wwterii..org/ab//... wrg.:bs///1bbaa/oo 05767.
[33] Kevin S. Beyer et al. "When "Nearest Neighbor" Meaningful?" In: Database Theory ICDT '99, 7th International Conference. Jerusalem, Israel, Jamuary 10- 1999, Proceedings. Ed. by Catriel Beeri and Peter Buneman. Vol. 1540. Lecture Notes in Computer Science. Springer, URL:: https://doi. http://......gg//// org/10. 1007/
1999, pp. 217-235. DOI: 10. 1007/00033333 000-0--9227---0 - 7V_15. 1-540-49257-755C-
[34] Charu C. Aggarwal, Alexander Hinneburg, and Daniel A. Keim. "On the Surprising Behavior of Distance Metrics High Dimensional Spaces". In: Database Theory ICDT 2001, 8th International Conference, London, UK, Jamuary 4-6, 2001, Proceedings. Ed. by Jan Van den Bussche and Victor Vianu. Vol. 1973. Lecture Notes in Computer Science Springer, 2001, pp. 420-434. DOT: 10. 1000000-55500 44503-4400300000 -00003-X-220 _7_ URL: https: / /ww.///oo.../ /ttp///o....//111...... 540 1000444503xxx___22
[35] Martin Wattenberg, Fernanda Viégas, and Ian Johnson. "How to use t-SNE effectively". In: Distill 1.10 (2016), e2.
[36] Junyuan Xie, Ross B. Girshick, and Ali Farhadi. "Unsupervised Deep Embedding for Cluster- ing Analysis". In: Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016. Ed. by Maria-Florina Balcan and Kilian Q. Weinberger. Vol. 48. JMLR Workshop and Conference Proceedings. JMLR.org, URL: http: http://prrceeen...eee/ mlr press/v48/xieb1e html.
2016, pp. 478-487,
[37] Fengfu Li, Hong Qiao, and Bo Zhang. "Discriminatively boosted image clustering with fully convolutional auto-encoders' In: Pattern Recognir. 83 (2018), pp. 161-173. DOI: 10. 1016/j. patcog. 2018.0128000 8. 05.019. URL: https::/ org/10 /00 1016/j j. patcog. 2018.05.000 05.019.
[38] J MacQueen. "Some methods for classification and analysis of multivariate observations". In: Proceedings of 5-th Berkeley Symposium on Mathematical Statistics and Probability/UUivversst of Califormia Press. 1967.
[39] John Schulman et al. Proximal Policy Optimization Algorithms. 2017. arXiv: 1707. 06347
Lcs. URL: http: / www.wwwwwogw..///////aaopo 063447.
[40] Lasse Espeholt et al. "IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Leamer Architectures". In: Proceedings of the 35th International Conference on Ma- chine Learning, ICML 2018, Stockholmsmässsan, Stockholm, Sweden, July 10-15, 2018, Ed. by Jennifer G. Dy and Andreas Krause. Vol. 80. Proceedings of Machine Learning Research. PMLR, 2018, pp. 1406-1415. URL: http://proccedinggs mlr vrress/v80/esphholll html.
13

---

Appendix Overview
A. Architecture, Hyperparameters, and Computational Costs
B. Theoretical Analysis of Loss Design
C. Ablation Study Performance and Interpretability
D. Impact of the Number of VQ Embeddings Performance and Interpretability
E. Clustering Quality under Different
caaansiionnntt-Reducttt Methods
F. Semantic Formation in Clusters
G. More Examples and Mean Images in the FDR Space
H. Hovering Examples
I. Human Evaluation Details
J. Potential Societal Impacts
A Architecture, Hyperparameters, and Computational Costs
The training of the proposed method is consistent with the Impala architecture [40], the PPO algorithm [39], and the hyperparameters used in the Procgen paper [19]. The FDR net is composed of two fully connected layers with 128 and neurons, respectively. The codebook in the vector quantizer has eight embeddings, and the degree of freedom the FDR loss 20. The random seeds employed in Figure are 2021 and 2031, while the seeds used in Figure C.I are 2021, 2022, and 2023. We train all models on one NVIDIA Tesla V100S 32GB GPU. The operating system version is CentOS Linux release 7.9. 2009. Each runs takes around six hours.
In Equation 2 of the main paper, WFDR and WAQ-VAE are 500 and respectively, Acr is updated every 50 iterations according to the following expression:
S Smean
Acrl = min
(A.1)
0.8 Shighest
where Smean the mean score of the last 100 episodes in training, and Shighest the highest score of the environment.
All hyperparameters introduced in our method, except for the number of embeddings, through performance tuning to optimize the model's overall performance. The number of embeddings in the vector quantizer determined by ensuring that each cluster maintained singular semantic interpretation. During hyperparameter tuning, we found that performance is primarily influenced by WFDR, WAE-VAE, and deut, and is more robust the number of VQ embeddings and the degrees of freedom in Equation 3.
chosen
B Theoretical Analysis of Loss Design
The two auxiliary losses in our framework serve distinct theoretical purposes. The LFDR term intended to preserve relative geometry when mapping high-dimensional state features to 2-D space, whereas the modified VQ term should behave like a standard online k-means step so that the codebook converges meaningful cluster centroids. We formalize both claims below.
B.1 Batch-wise Distance Similarity Preservation
We first prove that driving LFDR - 0 guarantees that pairwise similarity orderings are preserved between the original feature space and the FDR space, thereby retaining semantic neigh- borhood structure.
Goal.
Notation.
Let
n be the mini-batch size.
(xi) be the batch of high-dimensional state features.
(yx Vi be the corresponding low-dimensional embeddings produced by the FDR network.
14

---

Q > 0 be the degrees of freedom of the Student-r kernel.
The Student-/ kernel
d1u,, v) = =(1+ ||w-v111
d(u,v) E (0.
Define for all 1 s $ the normalized pairwise similarities
d((xxxx))
d(yi.)))
Pij Ekse de(xxxxe))
qij Exse dc(yk.
and the Kullback-Leibe divergence
FFR Σ in
Pij
Pij
log
0.
qij
Theorem (Similarity Preservation). LFDR 0, then there exists a constant
Exsr de(k,,0))
Exsr di(xx.xe)
such that for all i
d(yi,y)) = xd((xixxj).
Consequently, the ordering of squared distances lly: y1ll and ||x; - x,ll² is identical. Moreover, if 1, then y())..))) xx)x;xx)) and hence lly; 1-1-X0 yjll ||x;-xyl1 xxyl1?
Proof. By the non-negativity of KL. divergence, LFDR = 0 iff Pij qij for every i # j. Equating
d(yrN1)) Exet de(yk.ye)
d((x)1j) Exst d((xe,xe)
d(yr.,y) = dd(xxxx))))
Since the Student-r kernel decreases strietly with ||u - v1l2, scaling by K > O preserves rank order. When strict monotonicity forces equality of squared distances.
Minimizing LEDR therefore enforces batch-wise isometry up to global scale factor k, which is exactly the property needed for semantic clustering the 2-D FDR space.
B.2 Modified VQ-k-Means Equivalence
Goal.
Next we show that the gradient update used for the vector-quantizer codebook is algebraically identical an online k-means step.
Notation. Let
y, = (x)) be the FDR feature for state Xi.
{e kml be the learnable codebook entries.
k arg sin ||y; exll2 be the nearest code index.
my [i k2 k}] be the cluster size.
ja - my Ei yy be the empirical cluster centroid.
The modified VQ loss Liv = -(=llsg(>r)) - exil?.
Theorem 2 (Equivalence to Online k-Means). Updating each codebook vector via
((+1) =e - -BVe L'vor
B 8>0 O.
&
produces
(++)) =++l)==d = + - (())) which is the online k-means update with learning rate Yk-
Yk 2B ml
15

---

Proof. Because stop-gradde aop-grrdient sg();) blocks gradients with respect to yis
Ver Live = Σ (ex yy).
i:k==
Therefore
lr+1)
=k
22
-x) Σ - = + +B m2 (9x - e
Cly=k
which matches the standard incremental k-means rule.
C
Ablation Study on Performance and Interpretability
C.1 Performance Impact of the Semantic Clustering Module
Considering the cost of time and computational resources, we opt for training our model on the full distribution of levels in the "easy" mode. In Figure C.I, comparison of performance curves between the proposed method and the baseline is presented, where 'SPPO' denotes 'semantic' PPO, i.e., PPO integrated with our proposed semantic clustering module. Consistent with the Procgen paper [19], given the diversity of episodes during training, single curve represents both training and testing performance. Across these environments we observe that the proposed method closely aligns with the baseline performance, indicating minimal impact on performance from the introduced module. This expected, as the module only performs dimensionality reduction and clustering based on existing features, without introducing external information. The discrete code k reflects only the position of feature the learned space and is expanded and added element- wise to preserve the original feature dimensionality, ensuring the policy receives no additional information beyond what is already contained in the state feature.
coinrun
fruitbot
SPPO
25
6-
Score
jamper
ninja
20
Timesteps (Millions)
Figure C.I: Performance curves on 'easy' difficulty environments using three random seeds, trained and evaluated the full distribution of levels.
C.2 Semantic Clustering as an Intrinsic Property of DRL
We conducted stop gradient experiment further investigate whether semantic clustering is an inherent property of DRL. In this experiment, we applied stop gradient operation to Equation and removed the connection between the VQ codes and the original state features. This was done to prevent the semantic clustering module from influencing the feature space and to observe whether semantic clustering would still occur. The results, as shown Figures C.2 and C.3, demonstrated that states within the same semantic cluster continued to exhibit similar semantic interpretations, even without the influence of the semantic clustering module. However, the boundaries between clusters became less clear, making it more difficult to distinguish the semantics of states near the edges
16

---

of clusters. Notably, we also observed that this modification had minimal impact on performance, consistent with the trend shown in Figure C.I.
80
Gane
Oaiz
Oxi
Ounee
20
20
-40
-60
-60
-50
-20
Figure C.2: Visualization Features the t-SNE Space. The training eliminates the impact of the proposed semantic clustering module on the original feature space. Feature colors correspond to cluster colors in the FDR space of Figure C.3, facilitating the comparison of spatial relationships and feature distribution changes. Compared Figure 2b in the main paper, the absence of the semantic clustering module's enhancement makes sub-clusters less distinct.
Oaner g
Gamar
Ouster
-10
-8
-6
2
2
Visualization of Features in the FDR Space. The training eliminates the impact of proposed semantic clustering module on the original feature space. Compared to Figure 2d in the main paper, the cluster boundaries the FDR space are less distinct.
Figure C.3:
the
These observations suggest that semantic clustering is indeed an intrinsic property of DRL, driven by the agent's 's interaction with its environment during training. The proposed semantic clustering module enhances this natural clustering behavior by increasing the density of clusters, thus improving the separability between them. To fine-tune the influence of the module, we introduced control factor. At the beginning of training, the control factor kept low, allowing DRL training to shape the feature space independently. As the policy becomes more optimized and the semantic distribution of states becomes more organized, the control factor is gradually increased to further enhance the clarity and separability of clusters.
D Impact of the Number of VQ Embeddings Performance and Interpretability
To analyze the effect of the number of VQ embeddings (K) on both model performance and in- terpretability, we conducted experiments using the Jumper environment an example. Similar conclusions can be extended to other environments.
17

---

D.1 Performance Analysis
Figure D.I shows the performance of our model with varying numbers VQ embeddings. The results demonstrate that the number of embeddings does not affect model performance. This is expected, as our proposed method primarily focuses on feature dimensionality reduction and clustering. Combined with the overall performance results in Figure C.I, we observe that the model maintains consistent performance.
jumper
K-8
Timesteps (Millions)
Figure D.I: Performance comparison of models with different numbers of VQ embeddings in the Jumper environment.
D.2 Interpretability Analysis
Figures D.2a and D.2b illustrate the FDR results for K = 4 and K = 12, respectively, Our method effectively produces clusters that are clearly separable, regardless of the VQ embedding number.
However, interpretability influenced by the choice of K. When K = 12, the semantic clusters become overly fragmented, making difficult form coherent semantic explanations for clusters. Conversely, when K = 4, Table 3 shows that clusters contain multiple distinct semantic explanations, which negatively impacts interpretability.
space
space
5.0
rnily
Carar
-23
-50
-7.
-73
-3.0
-2.5
0.0
5.0
K 4
(b) K
12
Figure D.2: Visualization of the FDR spaces for different numbers of VQ embeddings in the Jumper
environment.
Clusters with incomplete or incoherent semantic descriptions hinder interpretability by introducing ambiguity in understanding the agent's behavior. This lack of clarity complicates policy analysis and makes it challenging to draw meaningful insights. Conversely, when single cluster contains multiple interpretable behaviors, it increases the cognitive load for users who must disambiguate
18

---

Table 3: Cluster descriptions for the Jumper game with
Cluster
Description
1) The agent is touching the carrot the upper left.
2) The agent is touching the carrot the right.
The agent is touching the carrot the bottom right.
4) The agent moving the left lower-left part scene.
The agent touching the carrot above.
2) The agent is touching the carrot on the left.
3) The agent is moving in the right or lower-right part of the scene.
2
1) The agent is touching the carrol below. The agent is moving in the upper part of the scene.
3
1) The agent is moving at the bottom of the scene. The agent approaching the carrot above.
between these behaviors. Such many-to-one mapping between behaviors and clusters undermines the straightforward identification of the agent's 's current strategy, reducing the utility of clustering as a tool for decision-making. To address these challenges, is essential to ensure one-to-one mapping between clusters and explanations. When each cluster is associated with single, coherent explanation, eliminates the need for further distinctions within clusters, facilitating clear policy analysis and enhancing human understanding of the agent's behavior.
Clustering Quality under Different Dimensionality-Reduue Reducttoo Methods
Silhouette Score (1)
Davies-Boulie Index (4)
0.6
0.6
0.4
0.4
0.0
Homap
PCA
MDS
ISAC
USAP
Hommp
PCA
M05
ISRE
(a) Silhouette score (higher is better)
(b) -ouuldn Index (lower is better)
Calinskk-Horabasz Index
30000
10000
0
UMM
homap
RCA
MD5
ISM
(c) Calinski Harabasz score (higher is better)
Figure E.1: Clustering metrics averaged over three Procgen games (Ninja, Jumper, and Fruitbot), each with three random seeds (nine in total). Bars show mean values; error bars denote the standard error of the mean.
To quantify how well various dimennioonaliy-reduccii (DR) techniques preserve the semantic clusters discovered by our model, we project the same set high-dimensional state ceatures-colleao following the procedure described in $ 4. 4.1-using five popular DR baselines: UMAP, Isomap, PCA,
19

---

MDS, and t-SNE. These metrics were selected to evaluate both the compactness within clusters and the separation between clusters the reduced feature space.
Silhouette - cohesion vs. separation each point's cluster.
-Buulin Index (DBI) average cluster similarity (lower indicates tighter, more separated clusters).
Calinski-Harbaasz (CH) ratio of between- within-cluster dispersion.
Figure E. shows that our learned 2-D FDR space achieves the best scores on all metrics: Silhouette the highest, DBI the lowest, and CH an order of magnitude larger than any baseline. These results confirm that our dimensionality- redductin module preserves- and even sharpens- the intrinsic semantic clustering properties uncovered in the high-dimensional feature space.
Semantic Formation in Clusters
To analyze how semantic clusters form in the feature space, sample 50,000 states per environment and compute: (i) the per-cluster mean image and the mean (# std) pixel distance from each state to its cluster mean (averaged over K=8 clusters), and (ii) the probability of cluster transitions along trajectories. We evaluate three models: Trained (ours), Stop-Grad (the ablation in $ C.2 that removes the effect of our modules by stopping gradients), and Raw (untrained). Results shown in Table 4. Compared to Raw, both Trained and Stop- Grad reduce transition probability, indicating policy- induced structure. However, Stop-Grad lacks fully distinct boundaries: it exhibits higher transition probability and lower intra-cluster pixel distance than Trained, whereas our method achieves the lowest transition probability and the highest intra-cluster pixel distance.
Table 4: Cluster transition probability and intra-cluster pixel distance (mean with std) over 50k states.
Environment
Model
Cluster transition probability Pixel distance mean (Std. Dev.)
FruitBot
Trained
0. 0011
100.00 (71.33)
FruitBot
Stop-Grad
0.1520
93.21 (68.09) 09)
FruitBot
Raw
0.2834
77.10 (49.94)
Jumper
Trained
0.2224
110.29 (62.29)
Jumper
Stop-Grad
0.3015
108.38 (61.22)
Jumper
Raw
0.5829
104.57 (58.31)
Ninja
Trained
0.2680
141.57 (67.88)
Ninja
Stop-Grad
0.2705
132.46 (66. 12) 87.61 (62.43)
Ninja
Raw
0,2712
We further assess temporal coherence with episode-level metrics: (i) Episode Cluster Entropy (ECE))the entropy of each episode's cluster distribution (lower is better, indicating more focused semantic grouping), and (ii) Temporal Cluster Agreement TCA @ k-e fraction of frame pairs lag & assigned to the same cluster (higher is better, indicating smoother, more stable semantics). As summarized in Table 5, our method consistently achieves lower ECE and higher TCA @ 3/6 than Stop-Grad across all three games. Note that Jumper episodes are shorter, yielding fewer clusters episode and thus lower ECE values overall.
Table 5; Episode- and frame-wise metrics (averaged over episodes).
Environment
Model
ECE
TCA@3
TCA@6
FruitBot
Stop-Grad
1,9068
0.7050
0.5309
FruitBot
Ours
1,8886
0.7302
0.5502
Jumper
Stop-Grad
0.6987
0.7040
0.6715
Jumper
Ours
0.5572
0.7911
0.7089
Ninja Ninja
Stop-Grad
1,5505
0.5798
0.3923
Ours
1.3086
0.6787
0.4985
20

---

G
More Examples and Mean Images in the FDR Space
G.1 CoinRun
To augment the exploration of semantic clustering as discussed in the main paper, this section analyzes two additional games characterized by distinct dynamics, CoinRun's gameplay mechanism is similar to Ninja's, s, requiring the agent to traverse from the far-left to the far-right, scoring points by interacting with coins the fär-right end of the scene, illustrated Figure G.I.
Figure G.I: A episode in CoinRun. Ellipses represent the omitted states.
The observations and insights obtained closely mirror those derived from the analysis of Ninja. Interested readers leverage the provided code and checkpoint for further exploration of similar findings. Consequently, for brevity, we refrain from extensively elaborating on analogous conclusions.
G.2 Jumper
In Jumper, the agent navigates cave to locate and touch carrots by interpreting a radar displayed in the upper right corner
screen. The radar's pointer indicates the direction of the carrot, while bar below the radar shows the distance between the agent and the shorter bars imply closer proximity, and vere-versa.
Mean Images
The FDR space
Cluster
Cluster
Custe:D
Cluste
Cluster2 2
Elgite
5.0
Clsiter
Cluste
Custer
Uluster
Claste
110.0
-12.5
10.0...75
Custers 8
Clussrr
Figure G.2: Examples and mean images from the Jumper FDR space.
The state examples and mean images from the clusters the FDR space of Jumper are presented in Figure G.2. The background of Jumper diverse, and the agent is always in the center of the screen (zoom to see the outlines clearly). In Table 6, we break down descriptions of the sampled images from each cluster and interpretations of the mean image for each cluster in the Jumper game.
Figure G.3 depicts various states from the Jumper game. C.3(a) and C.3(b) belong to the same episode and fall under Cluster 4, while C.3(c) and C.3(d) are from another episode, both categorized under Cluster 1. Notably, neither C.3(a) nor C.3(c) shows the presence carrots. This observation leads us suspect that the determination of these clusters is solely reliant on the radar and distance bar
21

---

Table 6: Cluster descriptions and mean image outlines for the Jumper game
Cluster
Description
Mean image outlines
The agent learns to jump up from the bottom left and move the left the top right.
The radar pointing and to the right, and the outline of the channel above and to the right..
The agent is touching the carrot the left upper left.
The radar is pointing to the upper left and is very close to the target.
The agent learns the skill of movement at the top of the scene.
The radar pointing to the left down, and the outline of the channels face the left down.
The agent is approaching the carrot the upper right.
The radar pointing the upper right, and the distance to the target is very close.
The agent touching the carrot below.
The radar is pointing down, and is very close to the target.
The agent approaching the carrot above or left.
The radar pointing up, and is very close the target.
The agent is touching the carrot on the right.
The radar pointing right, and is very close to the target.
The agent learns the skill of movement at the right bottom of the scene.
The radar pointing up or to the top left, and it is far from the target.
rather than the appearance of carrots. To test this hypothesis, we removed the carrot in C.3(c), which originally belonged to Cluster 4, and transformed it into Cf3(f). The result demonstrated that C.3(f) still belongs to Cluster 4, confirming our suspicion. However, this phenomenon might pose potential risks in practical applications. For example, in scenarios where sensor data and visual perceptions misalign, Al models might solely rely on sensor data for pecision-making e.g.nnng--g.--- an autonomous vehicle's sensors indicating an empty road while the occupants inside observe pedestrians crossing, yet the vehicle continues to accelerate.
NO00000 000000000
0000
(a)
(b) Figure G.3: Policy analysis examples in Jumper.
(c)
(d)
(e)
(1)
G.3 FruitBot
FruitBot is a bottom-to-top scrolling game where the agent moves left or right to collect fruits for points while avoiding negative scores upon touching non-fruit objects. The state examples and mean images from the clusters in the FDR space of FruitBot are presented in Figure G.4 and their descriptions in Table 7. FruitBot's mean images lack clarity due to the presence of diverse backgrounds, and the agent is constantly moving to the left and right at the bottom of the screen. However, we can still make out the outline of the wall and agent if we look carefully (zoom in to see the outlines clearly).
We examined a substantial number of video states and corresponding cluster information, and found that the factors determining clusters in FruitBot are the agent's position on the screen and its relative positioning to walls and gaps. This suggests that the agent has learned critical factors within the environment.
22

---

Mean
The rDR space
Cluster
Cleser
Cuslet
Cluster?
Cluster
Clsgpr 4
Cluster
Clöster
10.0
12.55
Claster
Clanter
Guster 8
Gluster
Figure G.4: State examples and mcan images from the FruitBot FDR space.
Table 7: Cluster descriptions and mean image outlines for the FruitBot game
Cluster
Description
Mean image outlines
The agent approaching the wall in the left area.
We the agent moving toward the gap on the wall that approaching on lower left.
The agent approaches the wall from the right area.
The agent moving toward the gap on the wall that approaching on the lower right.
The agent executes its policy from the wall from the left area.
The wall far away, and the agent moving in the lower left.
The agent approaches the wall from the right, but it is still some distance away.
The wall is far away, and the agent is moving in the lower right.
The agent is going through the gap in the middle and left, and insert the key the end of the scene.
The agent going through the final gap and in- serting the key.
The agent approaches the wall from the middle
We can identify the outline of the agent in the lower middle.
The agent going through the gap the right, and performs policy far from the wall in the right area.
The agent is crossing the gap in the bottom right.
The agent approaches is still some distance away.
from left, but
The agent is moving lower left, and the outline walls is in the middle of the screen.
H Hovering Examples
In figures H. and H.2, we present examples of our interactive visualization tool applied to Jumper and FruitBot. This tool is included in the supplementary material, allowing readers freely explore the semantie distribution of features and gain better understanding of the semantic clustering properties of DRL.
23

---

space
FOR space
speroo
space
(a) Overall perspective.
(b) First example point
(c) Second example point (d) Third example point
Figure H.1: Hover examples in the FDR space of Jumper. We observe sub-cluster in the FDR space example from the overall perspective (a) and the zoomed-in perspective (b), (c), and (d). The agent is standing on the edge of ledge. Although the scenarios (b), (c), and (d) are different, the proposed method effectively clusters semantically consistent features together in the FDR space.
The I0A soace
The FDR space
(a) Overall perspective.
First example point
(c) Second example point (d) Third example point
Figure H.2: Hover examples in the FDR space of Fruitbot. We observe sub-cluster in the FDR
space as an example from the overall perspective (a) and the zoomed-in perspective (b), (c), and (d). The agent is standing on the edge of ledge. Although the scenarios of (b), (c), and (d) are different, the proposed method effectively clusters semantically consistent features together the FDR space.
I Human Evaluation Details
I.1 Part Overview and Timeline Introduction
This section provides a detailed description of the evaluation process and presents the timeline for conducting the assessment. The evaluation aims to assess the clarity and interpretability of the semantic clusters in the FruitBot, Jumper, and Ninja games, with the objective of enhancing and quantifying the explainability of the system.
L1.1
Timeline
Fach participant will complete survey for two game environments (FruitBot, Jumper, Ninja). They follow the format as detailed:
Stage 1: Questionnaire minutes)
Participants are requested to complete a questionnaire that collects demographic information and gaming-related details. The questionnaire includes sections for gender, age group, education level, occupation, gaming experience, familiarity with evaluating game states, and preferred game genres.
Stage 2: Introduction to Game Environment (10 minutes)
During this stage, participants receive an introduction to the evaluation process. They are informed about the objectives of the assessment and the significance of evaluating the clarity and interpretability of the semantic clusters. They are also given a short description of the game environment (FruitBot, Jumper, Ninja), and shown short gameplay clip aid with the understanding of the game's objectives and features.
Stage Assessment (50 minutes)
Following the familiarization period, participants spend 50 minutes assessing the semantic

---

clusters in the game environment. They focus on evaluating the clarity and understandability the video clips within each semantic cluster. This is done online via a survey.
Total Evaluation Time: minutes.
1.2
Part 2: Questionnaire
This section of the evaluation plan presents the questionnaire that participants are required to complete. The questionnaire consists of the following sections:
I.2.1
Demographic Information
Age: Participants indicate their age.
Gender: Participants specify their gender Male, Female, Other.
Education Level: Participants indicate their highest level of education completed, including options such as High school and below, Bachelor's degree, Master's degree, and Doctorate and above.
Participants provide their current occupation, selecting from options such as Student, Employee, Self-employed, or Other.
Occupation:
1,2.2 Gaming-related Information
Participants indicate their level of gaming experience, choosing from options such as Beginner, Intermediate player, Advanced player, or Professional player.
Gaming Experience:
Participants indicate their frequency of gaming, choosing from options such as Daily, Several times week, Weekly, Monthly, or others.
Game Frequency:
Experience Evaluating Game States: Participants assess their experience in evaluating game states, selecting from options such as No experience, Some experience, Moderate experience, or Extensive experience.
Preferred Game Genres:
Participants specify their preferred game genres, including options such as Role-playing games, Shooting games, Strategy games, Puzzle games, or Other.
L.3 Part 3: Evaluation Questions for Clarity Assessment
In this section, a comprehensive set of questions is provided to assess the clarity and understandability of the semantic clusters. The questions capture participants' opinions and perceptions using Likert scale ranging from "Strongly Disagree" to "Strongly Agree'. The specific evaluation questions for the clarity assessment include:
The clips of each cluster consistently display the same skill being performed.
The clips of each cluster match the given skill description.
The two questions are asked each time the participant has been shown a semantic cluster.
L.4 Part 4: Evaluation Questions for Interpretability Assessment
This section outlines the question evaluating the interpretability the semantic clusters terms of their usefulness. The question is designed capture participants' opinions and perceptions using Likert scale ranging from "Strongly Disagree" "Strongly Agree." " Thee specific evaluation question for the interpretability assessment:
The identified skills aid in understanding the environment and the Al's decision-making process.
The above question is asked after the participants have seen all the semantic clusters.
25

---

1,5
Part 5: Personnel and Coordination
This section outlines the personnel and coordination aspects of the evaluation. includes information about evaluator recruitment and compensation. Specifically:
1.5.1
Evaluator Recruitment
15 evaluators are recruited participate in the evaluation.
1.5.2
Evaluator Compensation
Each evaluator receives $15 in compensation for their valuable time and contribution to the evaluation process.
By implementing this comprehensive evaluation plan, we gather valuable insights into the clarity and interpretability of the semantic clusters in the FruitBot, Jumper, and Ninja games. The evaluation results provide essential guidance for further quantifying the improved interpretability of DRL models using our proposed method.
I.6 Part 6: Grouping Details
Evaluator 1: FruitBot, Jumper
Evaluator 2: FruitBot, Jumper
Evaluator 3: FruitBot, Jumper
Evaluator 4: FruitBot, Jumper
Evaluator 5: FruitBot, Jumper
Evaluator 6: Jumper, Ninja
Evaluator 7: Jumper, Ninja
Evaluator 8: Jumper, Ninja
Evaluator 9: Jumper, Ninja
Evaluator 10: Jumper, Ninja
Evaluator 11: Ninja, FruitBot
Evaluator 12: Ninja, FruitBot
Evaluator 13: Ninja, FruitBot
Evaluator 14: Ninja, FruitBot
Evaluator 15: Ninja, FruitBot
This grouping plan ensures that each evaluator evaluates two different games, and each game receives total of 10 evaluations. It allows for comprehensive evaluations of each game and ensures that evaluators have an opportunity to provide feedback on multiple games.
J Potential Societal Impacts
This paper advances the interpretability of DRL. through semantic clustering, with potential ap- plications in safety-critical domains such autonomous systems and robotics. While primarily contributing to the field of Machine Learning, we encourage responsible application to mitigate potential misuse and do not identify immediate societal or ethical risks requiring specific emphasis.
26

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and the introduction precisely enumerate three key contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
A dedicated paragraph ($5) details the limitations.
27

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer:
[Yes]
Justification: Appendix B states all assumptions and supplies full proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: $4 and App. A list environment names, seeds, architecture, (learning rates, loss weights), compute budget, and provide the code in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
28

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The supplementary zip file contains anonymized source code, configuration files, and scripts that fetch the public PROCGEN dataset.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines pattps://nips.. cc/ Sttle//ggiies//odeSuuuee cwtmmiissoooooooiooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: httpssccccpbllccgcceeeee. /eepl////giiiie/eoeeeeeeeee SCoeeu iobbmissiii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
29

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Training details (optimiser, batch-size, PPO clip, etc.) appear in App. A; evaluation follows the PROCGEN protocol exactly.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All learning curves show mean and error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split. initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еттог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: App. states that each run uses a single Tesla V 100 (32 GB) for h; total GPU-days are reported.
Guidelines:
The answer NA means that the paper does not include experiments.
30

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics //neur///uuru wateerrrps.oom httpp///pbb..c//tteeeeueu.eooeee
Answer: [Yes]
Justification: The work uses only synthetic game data, releases code under MIT license, and poses no identifiable privacy or safety risk.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix discusses the impacts of this study.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
functioning correctly, harms that could arise when the
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
31

---

Justification:
The released artifacts are small game agents that carry no foreseeable dual-use
risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite the PROCGEN benchmark [19] and note its MIT license in App. A.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersw thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new datasets models are released beyond trained weights already covered by the code-release answer.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
32

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our user study surveyed adult volunteers on Al models online without studying human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our user study surveyed adult volunteers online, posed minimal risk, and was exempt under our institution's 's IRB policy.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: No large language models are used the proposed methodology.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
cc/ /ctp////eneeeeeee////////LL Please refer to our LLM policy htttts/s//eerrppppcc//////eeeeee what should or should not be described.
for
33


Reviews:
[{"review_id": "k2QEew0vlt", "reviewer": "Reviewer_cNxc", "summary": "The work addresses the interpretability aspect of reinforcement learning, by developing a semantic clustering approach, which is trained end to end while learning a new policy. To do this the authors use a VQ-VAE inspired novel architecture quantaization to obtain low dimensional state embeddings. Through experiments on Procgen game environments, they demonstrate that DRL models naturally group states representing similar behaviors or \"skills\" together, such as \"jumping to higher platforms\" or \"approaching targets,\" without requiring external supervision.", "questions": "1) “cluster inputs based on their semantic similarity in the internal space”: what is the internal space? \n2) Why element wise addition of the feature vector? why not concatenate it? \n3) If the goal is to analyse the DRL policy, then why make it conditional by adding to the policy term? What if the policy degrades? \n4) The results show that sematic similar states are clusterd together. But what if the features which procgen is changing (background), is ignored by the feature generator already (before the clustering module)? Implying the added VQ-VAE is not necessary.", "limitations": "The limitation of the work, would be the set of environments the algorithm was tested on. It would be interesting to see it tested on partially observable environments like Minecraft etc.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 1, "originality": 4, "strengths_and_weaknesses": "Strengths: \nThe empirical validation demonstrates compelling evidence for the method's effectiveness across multiple dimensions. The authors conduct experiments on four different Procgen environments and show through statistical comparisons that their approach outperforms traditional dimensionality reduction methods like UMAP, PCA, and t-SNE on standard clustering metrics.\n\nWeakness: \n1) No Ground Truth: They never establish what \"correct\" semantic clustering should look like\n2) The technical approach exhibits several design limitations that restrict its applicability. The number of clusters must be manually predetermined (they use K=8) with no adaptive mechanism to determine the optimal number for different environments or tasks.\n3) A critical practical limitation is the continued reliance on manual interpretation despite claims of enhancing interpretability. The method requires humans to examine clusters and manually assign semantic meanings, with no automated mechanism for generating natural language descriptions or understanding what behaviors each cluster represents.\n\nImprovements/suggestions: \n1) Introduction needs a bit more detail of the method the authors are proposing. The contribution section states that the paper proposes a novel architecture that addresses limitations. It would be better to directly mention what this method is here and what is the novelty. \n2) Missing related work: \n\n* patil et. al: contrastive abstraction for RL \n    * Uses contrastive learning to structure the space and uses hopfield network to obtain related clusters to a state. \n* Jan Robine et.al: smaller world models for reinforcement learning\n    * Use VQ-VAE to build world models. Related work for VQ-VAE and RL.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns which were raised during the reviewing process (though not all).", "rebuttal": "# Rebuttal to Reviewer cNxc\n\nWe sincerely thank Reviewer cNxc for their feedback and for highlighting our thorough empirical validation as well as the superior performance of our end-to-end semantic clustering approach compared to UMAP, PCA, and t‑SNE. Below, we address each concern.\n\n## Weaknesses\n\n> 1. No Ground Truth: They never establish what \"correct\" semantic clustering should look like\n\nSince semantics are inherently subjective, there is no universal 'correct' clustering. As is standard in interpretability research, we rely on human evaluation (lines 203–219) rather than a fixed ground truth (Doshi‑Velez & Kim, *Toward a Rigorous Science of Interpretable ML*, 2017).  Our method achieves high inter-rater agreement on cluster semantics (Table 2), confirming consistent human interpretation of our clustering results.\n\n> 2. The technical approach exhibits several design limitations that restrict its applicability. The number of clusters must be manually predetermined (they use K=8) with no adaptive mechanism to determine the optimal number for different environments or tasks.\n\nOur clustering approach in this paper is consistent with the state-of-the-art works on this topic (e.g., Xie et al. (2016), Caron, et al. (2018)), which use fixed cluster numbers rather than adaptive mechanisms to determine the optimal number of clusters.  That being said, we agree that this is a limitation (lines 247–248) and will explore adaptive mechanisms (e.g., elbow method, silhouette-score optimization) in future work. \n\n>  3. A critical practical limitation is the continued reliance on manual interpretation despite claims of enhancing interpretability. The method requires humans to examine clusters and manually assign semantic meanings, with no automated mechanism for generating natural language descriptions or understanding what behaviors each cluster represents.\n\nWe would like to note that the baseline works (Mnih et al. (2015) or Zahavy et al. (2016)) also use manual rather than automated description generation. Nevertheless, we agree that the reliance on manual generation of semantic descriptions is a limitation (lines 251–252), and will explore methods for automated generation of natural language descriptions (e.g., using GPT-4V) in future work.\n\nHowever, we disagree that the manual generation of description runs counter to our claim of enhancing interpretability. In a sense, these are orthogonal issues. Automated description generation could speed up the experiment design process, but would likely have little impact on the responses provided by the participants (who would not know whether the descriptions are generated manually or automatically), as automatically generated descriptions would still need to be manually verified by the researchers before being presented to the participants, given the possibility of hallucinations by VLMs (Liu et al. (2025))\n\nWe also note that even with the continued reliance on manual description generation, our method drastically reduces annotation workload compared to baseline works. For example, we estimate that annotating a single environment via manual per‑state feature analysis and semantic grouping using the methods proposed by Mnih et al. (2015) or Zahavy et al. (2016) would require **hours**, whereas our annotators reviewed a handful of short video clips and generated cluster descriptions (like the ones in Table 1) in approximately **15  minutes**.\n\n  \n## Questions\n\n> 1. \"cluster inputs based on their semantic similarity in the internal space\": what is the internal space? \n\nThe model's latent feature space.\n\n> 2. Why element wise addition of the feature vector? why not concatenate it? \n\nUnlike concatenation, element-wise addition does not change the original RL feature dimensionality. This enables integration without modifying downstream layers (lines 7, 78).\n\n> 3. If the goal is to analyse the DRL policy, then why make it conditional by adding to the policy term?  \n\nAdding the discrete code to the policy head enables extension to downstream tasks (e.g., macro action selection in hierarchical learning) (see Fig. 1 caption, lines 140–142, 226–231). \n\n> What if the policy degrades?\n\nThe policy does not degrade (see sections C.1,  D.1 and lines 153–154).\n\n> 4. The results show that sematic similar states are clusterd together. But what if the features which procgen is changing (background), is ignored by the feature generator already (before the clustering module)? Implying the added VQ-VAE is not necessary.\n\nModified VQ‑VAE is necessary for both clustering and enhancing interpretability (lines 77–80), regardless of background.\n\n- Lines 118–122 and section B.2 show that the modified VQ‑VAE acts like online k‑means for clustering latent features. \n- Lines 126–138 and section C.2 show that when trained alongside the FDR loss, the modified VQ‑VAE loss sharpens cluster boundaries, directly boosting the interpretability of the DRL model.\n  \n## Improvements/suggestions\n\nThank you for your suggested improvements regarding the introduction and related work! We will incorporate them into the camera-ready version.\n\n## References\n- Xie, et al, Unsupervised Deep Embedded Clustering, ICML 2016\n- Caron, et al., Deep Clustering for Unsupervised Learning of Visual Features, ECCV 2018\n- Liu, et al., Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering, ICLR 2025\n- Mnih, et al., Human-level control through deep reinforcement learning. Nature 2015.\n- Zahavy, et al., Graying the black box: Understanding DQN, ICML 2016"}, {"review_id": "PwikPYW3s2", "reviewer": "Reviewer_P422", "summary": "The authors employ the following argument structure to motivate their paper:\n\n1. t-SNE has been used in DRL works to visualize the encodings of DRL agents\n2. t-SNE is unstable (sensitive to initialization and therefore has inconsistent output)\n3. t-SNE requires extensive manual cluster annotation\n4. It would be good to provide an alternative to t-SNE without the issues identified in 2 and 3\n\nTheir solution is to integrate (online) clustering into DRL training. Their solution is built on top of VQ-VAE.", "questions": "- I found the sentence on line 107 beginning with \"We use the Student's t-distribution...\" unclear. What do you mean that it captures nonlinear structures?\n- What norm is being used in Equation (3) on line 110?\n- On Line 115-116 the authors say that \"the original distance relationship between features is maintained in the low-dimensional space.\" Is it a well-known or easy result that the student's t kernel is an isometry? If not, there should be a citation or proof for this in the appendix. \n- On line 127 did you mean to say that the clusters themselves are made more compact?\n- What are the qualifications of the human evaluators? I would have appreciated if this information was present in the main body of the paper near line 204-205. \n- How is the number of clusters controlled? Is it possible that there are overwhelmingly many for a human to interpret, and if so, what is to be done?", "limitations": "The authors did not try to recover an inner product in their embedding space whose quantification of similarity corresponds to semantic similarity in the input space.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- My own prior experiences using t-SNE to visualize DRL agent encodings left me desiring an alternative, so I am glad to see this paper propose one. \n- I think the authors successfully addressed the issue of t-SNE being unstable \n- Section 4.1 gives some evidence that their method yields more compact and separated clusters than t-SNE does.\n- Section 4.1 gives evidence that their method is not unstable like t-SNE is. \n- The authors provide a human evaluation. I think these are important for an XAI paper to be convincing. \n- I appreciate that the authors developed a visualization tool for analyzing the embedding space of DRL models. This is a nice addition that they didn't necessarily need to add. I liked the policy analysis at the end of section 4. I could see myself using it in the future if it is integrated into some standard python RL library. \n\n## Weaknesses\n\n- Nitpick: I would prefer if the authors gave a definition of \"semantic clustering\" that does not use the word \"semantic\" in it (unless the word \"semantic\" is also given a formal non-self-referential definition).\n\n- I would have appreciated if the **Online Clustering** paragraph in Section 3.1 did more to convince readers that previous work requires \"extensive manual annotation.\" The key word for me there is \"extensive.\" That should be substantiated. I view this as a significant omission. \n\n- I would have appreciated an explanation of Equation (4) rather than just showing the equation and ending the paragraph.", "was_revised": true, "final_justification": "The authors have responded to my concerns, the most significant of which related to substantiating their claim that prior methods require extensive manual annotation while the proposed method does not. I do not view this concern as fully resolved as I would need very detailed comparisons between the proposed method and more than just one or two prior works for such a bold, general claim.", "rebuttal": "# Rebuttal to Reviewer P422\n\nWe sincerely thank Reviewer P422 for their thoughtful feedback and for highlighting our alternative to t‑SNE, the stability and compactness of our clusters, the human evaluation, and the visualization tool. Below, we address each concern.\n\n---\n\n## Weaknesses\n\n> 1. Nitpick: I would prefer if the authors gave a definition of \"semantic clustering\" that does not use the word \"semantic\" in it (unless the word \"semantic\" is also given a formal non-self-referential definition).\n\nWe will revise the definition (lines 19–20) to state that clusters group states eliciting similar agent behaviors under comparable environmental conditions (e.g., approaching a target, jumping to a higher platform in Procgen), avoiding self‑reference.\n\n> 2. I would have appreciated if the Online Clustering paragraph in Section 3.1 did more to convince readers that previous work requires \"extensive manual annotation.\" The key word for me there is \"extensive.\" That should be substantiated. I view this as a significant omission.\n\nWe will expand §3.1 to state more details. In our setting, annotators labeled clusters for one environment by reviewing a handful of short clips in **~15 minutes**, whereas manual per‑state feature analysis in earlier approaches without automated clustering (e.g., Mnih et al. [17], Zahavy et al. [32]) typically takes **hours**.\n\n>3. I would have appreciated an explanation of Equation (4) rather than just showing the equation and ending the paragraph.\n\nWe will add:  “Minimizing the Eq. (4) encourages the low‑dimensional mapping to preserve the pairwise neighborhood structure of the high‑dimensional features.”\n\n---\n\n## Questions\n\n> 1. I found the sentence on line 107 beginning with \"We use the Student’s t-distribution...\" unclear. What do you mean that it captures nonlinear structures?\n\nBy “nonlinear structures,” we refer to curved or folded manifolds in feature space. The Student’s t affinity preserves such manifolds after projection—unlike linear methods (e.g., PCA).\n\n> 2. What norm is being used in Equation (3) on line 110?\n\nL₂. We will clarify this in the final version.\n\n> 3. On lines 115–116 the authors say that \"the original distance relationship between features is maintained in the low-dimensional space.\" Is it a well-known or easy result that the Student’s t kernel is an isometry? If not, there should be a citation or proof for this in the appendix.\n\nWe do not claim an exact isometry. Appendix B.1 formally proves that—when using the same degrees-of-freedom parameter in both spaces—the Student’s t affinity in the 2D embedding equals the original affinity up to a positive scale factor.\n\n> 4. On line 127 did you mean to say that the clusters themselves are made more compact?\n\nYes. We will rephrase to “each cluster become more compact (smaller intra‑cluster distances)...” to improve clarity.\n\n> 5. What are the qualifications of the human evaluators? I would have appreciated if this information was present in the main body of the paper near lines 204–205.\n\nWe will add near lines 204–205: annotators were adults (18+), native or highly proficient English speakers, with basic video‑gaming experience and a brief training session.\n\n> 6. How is the number of clusters controlled? Is it possible that there are overwhelmingly many for a human to interpret, and if so, what is to be done?\n\nThe number of clusters $K$ is a fixed hyperparameter (see L247–251, Appendix D). Very large $K$ fragments episodes into tiny segments, hindering complete description (lines 499–500)."}, {"review_id": "5iVSKKQDGR", "reviewer": "Reviewer_AFfc", "summary": "This paper investigates methods for interpreting policies in deep reinforcement learning. In particular, they propose a method to cluster states used by the policy in a way that respects semantic differences in the policy behaviour. There are a few qualitative investigations in ProcGen, with the rationale that policies in procedural generation would face a generalization challenge where interpretability would be useful for understanding the learned policy.", "questions": "- Figure 1: Why is it that the VQ code is integrated into the feature, rather than the VQ embedding?\n- Section 3.1 (curse of dimensionality): Wouldn't the curse of dimensionality also apply to the problem of dimension reduction? I am not sure I understand this motivation if the VQ code needs to be learned.\n- Section 3.1 (distance relationships): Would the reduced features learned from preserving consistency across distance relationships also serve to represent relationships with respect to other distances, like bisimulation metrics?\n- Section 3.2 (FDR loss): It is not clear to me what the relationship is between p and q here, and why the loss takes the form that it does. By optimizing the FDR loss, are you ensuring that the similarities of induced by f and g \\circ f are similar?\n- Section 3.2 (vector magnitude in student T): I understand that using the same $\\alpha$ ensures the same distance relationship, but wouldn't the magnitudes of the vectors differ if f and g \\circ f are of different dimensions?\n- Section 3.2 (Modified VQ-VAE): without the third term, how does the encoder of the VQ-VAE learn to maintain the assigned embedding? I understand that the $e_k$ can cover the space as centroids of K means, but an important difference is that the underlying features are also changing as the encoder is learned.\n- Section 4 (Cluster separation and t-sne): I wonder what exactly the t-sne is capturing on top of the learned clusters of the embeddings. Does t-sne on the fused state feature essentially return the cluster index?\n- Section 4 (sensitivity to number of states/seed): While the overall structure of the clustering remains the same, I wonder if there is any way in which the identity of these clusters could be identified? For example, does the specific embedding (e_k) assigned to these clusters change?\n- Figure 3 and Table 1: The results presented here are a bit difficult to follow because the change in subsequent images is difficult to detect. For example, cluster 4 which is described as performing a jump has mostly images of the agent near the top of its jump rather than a scene closer to the ground at the beginning of the jump. I am also not sure if the mean images accurately depict the dynamic nature.\n- Table 2: While these results show the proposed method in a favourable light, they would be much stronger if the evaluation included clips from clusters of a baseline method for comparison.\n- Section 5: an important limitation that was left out is whether the method has any effect on performance. I understand that this not the primary concern of this paper, but training the policy with the proposed fused features could result in changes to performance.\n\n### Minor Comments\n- Equation 2: I suggest using another symbol for the \"control factor\", since $f$ is usually used to denote a function\n- Section 3.2 (Improved Clustering): the first paragraph feels out of place, and i think that it can be demonstrated in the experiments. It doesn't connect with the actual description of the method earlier in the section. THe second paragraph is interesting, but difficult to follow. Could the statement be made as a formal proposition or theorem?\n- Figure 2: the titles of the subfigures could be more informative or removed.\n- Figure 3: the mean images could benefit increased contrast to highlight the difference between the clusters.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The method appears to be generally applicable, allowing it to be added to a conventional RL algorithm without any change to its operation. (Although, I do have a question about whether it can change performance/learning, see below).\n\n- Qualitative analysis suggests that the proposed approach for clustering in drl is generally effective. The experiments seem to be well-designed, thought-out and provide compelling evidence for improvements over t-sne when it comes to interpreting the policy's behaviour semantically.\n\n### Weaknesses\n- The presentation of the experimental results could be improved (see below).\n\n- Unclear whether if there is an effect of the proposed feature extraction method on downstream performance. If so, then this is a weakness without an investigation to demonstrate that the performance impact is negligible.", "was_revised": true, "final_justification": "The authors have clarified the questions I raised in my rebuttal, including further justification of including the VQ-code and omitting a term from the VQ-VAE loss, as well as demonstrating that the performance impact is minimal.", "rebuttal": "# Rebuttal to Reviewer AFfc\n\nWe sincerely appreciate Reviewer AFfc for their thoughtful review and for recognizing the general applicability of our method, the quality of the qualitative analyses, and the usefulness of the visualization tool. We address each concern below.\n\n---\n\n## Weaknesses\n\n> 1. The presentation of the experimental results could be improved (see below).\n\nPlease refer to our response to the questions below.\n\n> 2. Unclear whether the proposed feature extraction affects downstream performance. If so, this is a weakness without an investigation showing negligible impact.\n\nSections C.1, D.1, and lines 153–154 report that our proposed module has minimal impact on policy performance with experimental evidence. \n\n---\n\n## Questions\n\n> 1. Figure 1: Why integrate the VQ code into the feature rather than the VQ embedding?\n\nAs stated in lines 140–142, integrating the VQ code into policy training aligns with the definition of a latent-conditioned policy, facilitating downstream task integration. Furthermore, broadcasting the 1-D VQ code and adding it to the state feature is simpler than integrating a multi-dimensional VQ embedding with mismatched dimensions.\n\n> 2. Section 3.1 (curse of dimensionality): Wouldn't the curse of dimensionality also apply to the problem of dimension reduction? I am not sure I understand this motivation if the VQ code needs to be learned.\n\nWe avoid clustering in the original high‑D feature space by first compressing features into a low‑D manifold with the FDR network (lines 84–96) and then learning VQ codes on that reduced manifold for clustering (lines 97–100, 118–122), thereby mitigating sparsity and sidestepping the curse of dimensionality.\n\n\n> 3. Section 3.1 (distance relationships): Would the reduced features learned from preserving consistency across distance relationships also serve to represent relationships with respect to other distances, like bisimulation metrics?\n\nFDR optimizes over pairwise similarities; in principle, replacing this affinity with one induced by a bisimulation metric would extend the objective. We will note this as a future direction in §5. \n\n> 4. Section 3.2 (FDR loss): It is not clear to me what the relationship is between p and q here, and why the loss takes the form that it does. By optimizing the FDR loss, are you ensuring that the similarities of induced by f and g \\circ f are similar?\n\n$p$ and $q$ are pairwise similarities in the high‑D and low‑D spaces, respectively (lines 110–112). Appendix B.1 derives that minimizing the FDR loss encourages that the low‑dimensional mapping preserves the original similarities.\n\n> 5. Section 3.2 (vector magnitude in student T): I understand that using the same α ensures the same distance relationship, but wouldn't the magnitudes of the vectors differ if f and g \\circ f are of different dimensions?\n\nMagnitudes can differ. However, Appendix B.1 formally proves that by fixing the Student’s t degrees-of-freedom in both spaces, all pairwise affinities become proportional—identical up to a positive scale factor—thereby preserving the relative ordering of distances.\n\n> 6. Section 3.2 (Modified VQ-VAE): without the third term, how does the encoder of the VQ-VAE learn to maintain the assigned embedding? I understand that the $e_k$ can cover the space as centroids of K means, but an important difference is that the underlying features are also changing as the encoder is learned.\n\nAppendix B.2 proves that our modified VQ‑VAE is equivalent to online k‑means. The update mechanism ensures centroids continually track the moving feature distribution, yielding stable, semantically coherent clusters despite feature drift. Our experiments demonstrate the effectiveness and stability of this clustering method (see Sec. 4.1). \n\n> 7. Section 4 (Cluster separation and t-sne): I wonder what exactly the t-sne is capturing on top of the learned clusters of the embeddings. Does t-sne on the fused state feature essentially return the cluster index?\n\nWe use t‑SNE only as a 2D visualization baseline. Cluster colors (cluster indexes) in the t‑SNE plots are derived from our method to compare spatial relations; t‑SNE does not “recover” indices but provides a layout for visual inspection (see Sec. 4.1; Fig. 2 caption).\n\n> 8. Section 4 (sensitivity to number of states/seed): While the overall structure of the clustering remains the same, I wonder if there is any way in which the identity of these clusters could be identified? For example, does the specific embedding (e_k) assigned to these clusters change?\n\nAfter training, the learned codebook is fixed at inference. Each cluster maintains a consistent identity tied to its learned codebook vector $e_k$.\n\n> 9. Figure 3 and Table 1: The results presented here are a bit difficult to follow because the change in subsequent images is difficult to detect. For example, cluster 4 which is described as performing a jump has mostly images of the agent near the top of its jump rather than a scene closer to the ground at the beginning of the jump. I am also not sure if the mean images accurately depict the dynamic nature.\n\nCluster 4’s description refers to the in-air phase (“After performing a high jump, the agent loses sight of the ledge below” in Table 1), so the mean image at apex accurately captures this behavior. Static mean images provide a statistical summary; to illustrate true motion, we include dynamic video clips for every cluster shown in the paper in the supplementary material (lines 191–192). Upon acceptance, we will make these video clips and our code publicly available to facilitate reader understanding.\n\n\n> 10. Table 2: While these results show the proposed method in a favourable light, they would be much stronger if the evaluation included clips from clusters of a baseline method for comparison.\n\nThank you for the suggestion. Evaluating interpretability baselines (Mnih et al. [17]; Zahavy et al. [32]) is very challenging: t-SNE’s stochasticity yields non-reproducible clusters, and prior methods required extensive manual annotation of only a few large clusters, leaving many smaller clusters unexplained (lines 36–38). \n\nAs is standard in interpretability research, we rely on human evaluation (lines 203–219) rather than a baseline comparison (Doshi‑Velez & Kim, *Toward a Rigorous Science of Interpretable ML*, 2017). Table 2’s high scores confirm our method’s interpretability effectiveness. Meanwhile, Appendix E provides a quantitative clustering-quality comparison against PCA, UMAP, t-SNE, etc., underscoring our clear advantage.\n\n> 11. Section 5: an important limitation that was left out is whether the method has any effect on performance. I understand that this not the primary concern of this paper, but training the policy with the proposed fused features could result in changes to performance.\n\nAs noted earlier, our method has minimal impact on policy performance (see sections C.1, D.1, and lines 153–154).\n\n---\n\n## Minor Revisions  \nWe are very grateful for the suggestions and will incorporate them to improve clarity of the paper."}, {"review_id": "CGsMigSHnH", "reviewer": "Reviewer_fcAv", "summary": "The paper proposes a plug-in “semantic clustering module” for deep-reinforcement-learning (DRL) agents that pairs a lightweight 2-D feature-dimensionality-reduction (FDR) network with an online vector-quantiser. Jointly trained with any standard DRL algorithm (shown with PPO on the Procgen benchmark), the module yields a stable, human-interpretable partition of the latent state space into discrete clusters without hurting task performance. These clusters expose the structure of learned policies, enable video-level explanations such as “approaching the right-wall then jumping to a higher ledge,” and let practitioners inspect or segment roll-outs by semantic “skills.” Experiments across CoinRun, Ninja, Jumper and FruitBot demonstrate markedly clearer, seed-independent clusters than t-SNE visualisations, high agreement in a 15-person user study (Likert ≈ 4.2/5), and no significant score drop versus baseline PPO. Overall, the work shows that DRL already forms meaningful semantic groupings internally and that making them explicit dramatically improves transparency while remaining computationally light.", "questions": "* Limited discussion on the benefits of adding vector codes to the state features\n\nThe authors state that “Training the VQ code $k$ with a latent-conditioned policy $\\pi(a∣s,k)$ supports the extension to downstream tasks, such as macro action selection in hierarchical learning.” However, this appears to be the only justification provided for incorporating the VQ code into the state features. Given that the paper does not include any experiments in a hierarchical learning setting, nor an ablation study comparing performance with and without the VQ-code, this design choice feels insufficiently motivated. Since the main goal of the paper is to cluster state features—rather than to influence the policy—further discussion is warranted. An ablation study examining the effect of excluding the VQ-code could significantly enhance the clarity and rigor of the work.\n\n* Inconsistency between Section 4.1 and Appendix C.2\n\nIn Section 4.1, the authors state:\n“The t-SNE visualization of PPO (Figure 2a), spreads features across the space without forming clear clusters, limiting its utility for clustering analysis and requiring detailed manual examination of certain areas, as in previous studies.”\n\nHowever, in Appendix C.2, the “stop-gradient” experiment shows:\n“The results, as shown in Figures C.2 and C.3, demonstrated that states within the same semantic cluster continued to exhibit similar semantic interpretations, even without the influence of the semantic clustering module.”\n\nThis comparison seems somewhat inconsistent. If I understand correctly, the policy and the state representations in vanilla PPO and the stop-gradient method should be identical, assuming all other components remain the same. Figure C.2 in the appendix shows a t-SNE visualization of the stop-gradient method's feature space with cluster labels—yet these clusters should be equivalent to those from vanilla PPO, as the policy is unaffected. In that case, the two quoted statements are not strictly contradictory, but they do feel awkward when read together. The apparent discrepancy seems largely due to the absence of color labeling in Figure 2, which may lead to a misleading interpretation.\n\nI recommend revising Section 4.1 to explicitly incorporate and contrast the stop-gradient experiment, clarifying how your method differs from it and why joint training offers benefits. Additionally, it would be helpful to include an *unlabeled* version of your method's t-SNE plot, to mirror the presentation of vanilla PPO more symmetrically. This would provide a fairer and clearer comparison of visual cluster separation and further support your narrative.\n\n\nFurthermore, if the main difference between your method and the stop-gradient baseline lies in sharper cluster boundaries, then this distinction deserves richer discussion why is this important. Your analysis operates on the clusters themselves, not directly on the FDR or t-SNE visualizations (which shows sharper bounds). More evidence is needed to support the claim that sharper boundaries improve semantic grouping. For instance, examples of ambiguous or misclustered states at the boundaries would be valuable.\n\nLastly, the Semantic Formation experiment in Appendix F is not fully convincing. The statistics are averaged over 50,000 steps; it would be more informative to show episode-level metrics or frame-wise agreement scores. A comparison of Semantic Formation between your method and the stop-gradient baseline could help demonstrate the added value of your design. If your method indeed shows higher consistency or interpretability on these metrics, that would substantially strengthen your claims.\n\n* Limited performance comparison and sensitivity analysis of the newly introduced hyper-parameters\n\nPerformance comparisons are limited to a small set of Procgen environments. Evaluation on additional domains—such as Atari or other standard DRL benchmarks—would improve the generalizability of your claims.\n\nIt is commendable that your method preserves performance without requiring changes to the original PPO hyper-parameters and also demonstrates robustness to random seeds—both of which are important properties in the reinforcement learning domain and provides study of the effect of K. However, the method introduces new hyper-parameters (e.g., weights for the semantic loss components), and their sensitivity is not thoroughly explored. Understanding how these additional parameters affect performance and interpretability is crucial for assessing the stability and general usability of the approach. A more detailed sensitivity analysis would greatly strengthen the paper's practical value and reproducibility.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n* Novel approach to the important task of interpretable reinforcement learning\n* Strong and versatile analysis of the proposed method\n* Rigorous human study (though limited for making stronger behavioral claims)\n* Useful visualization tool\n\n**Weaknesses** (see detailed explanations in the questions):\n\n* Limited discussion on the benefits of adding vector codes to the state features\n* Inconsistency between Section 4.1 and Appendix C.2\n* Limited performance comparison and analysis of the sensitivity to the newly introduced hyper-parameter\n* Additional weaknesses noted in the discussion: choice of *k*, lack of automated description generation, etc.\n\nIn essence, this is a strong paper with several weaknesses that, however, do not undermine the overall contribution.", "was_revised": true, "final_justification": "The authors have addressed my questions and concerns regarding their method and the presented ablations. The additional metrics on semantic focus and temporal coherence make their claims more compelling and better substantiated.\n\nOverall, I found the work interesting and potentially impactful. It could inspire a new line of research, especially when combined with automatic state description generation using modern vision-language models—a direction I find particularly intriguing.", "rebuttal": "# Rebuttal to Reviewer fcAv\n\nWe sincerely appreciate Reviewer fcAv for their encouraging evaluation and for highlighting our novel method, analyses, user study, and visualization tool. Below, we address each concern.\n\n---\n\n## Questions & Weaknesses\n\n> 1. Limited discussion on the benefits of adding vector codes to the state features\n\nOur experiments confirm that integrating the VQ code into policy training has negligible impact on model performance (see lines 153–154, 475–476; App. C.1, D), while lines 226–231 presents how the VQ code aids analysis of hierarchical policy structures and improves interpretability in downstream hierarchical learning.\n\n---\n\n> 2. Inconsistency between Section 4.1 and Appendix C.2  \n\n1) **Consistency Clarification**  \n   As stated the captions of Fig 2 and Fig C.2, cluster colors (cluster indices) in the t-SNE visualizations are derived from our proposed method and are used solely to facilitate comparison of spatial relationships. This note will also be added to the main text to prevent misinterpretation in the final version.\n\n2) **Discussion of Stop-Gradient Baseline**  \n   In Section 4.1, we will introduce a discussion about the stop-gradient experiment in App C.2, highlighting how our approach yields sharper and more coherent clusters.\n\n3) **Boundary-State Ambiguities**  \n   In the stop-gradient t‑SNE (Fig. C.2), overlapping cluster regions cause semantically similar states to be split across clusters, weakening semantic separation (see lines 473–475). We will include concrete examples of these boundary-state ambiguities in the final version.\n\n4) **Semantic Formation Comparison & New Metrics**  \n   As suggested, we added two experiments that confirmed our method both improves semantic focus and strengthens temporal coherence.\n\n   - **Experiment 1: Stop‑Grad Baseline**  \n     We extended Table 4 to include the Stop‑Grad baseline:\n\n     | Environment  | Model      | Cluster transition probability | Pixel distance mean (Std Dev.) |\n     |:------------:|:-----------|:------------------------------:|:------------------------------:|\n     | **FruitBot** | Trained       | 0.1081                         | 100.00 (71.33)                  |\n     |              | Stop‑Grad  | 0.1520                         | 93.21 (68.09)                   |\n     |              | Raw        | 0.2834                         | 77.10 (49.94)                   |\n     | **Jumper**   | Trained       | 0.2224                         | 110.29 (62.29)                  |\n     |              | Stop‑Grad  | 0.3015                         | 108.38 (61.22)                  |\n     |              | Raw        | 0.5829                         | 104.57 (58.31)                  |\n     | **Ninja**    | Trained       | 0.2680                         | 141.57 (67.88)                  |\n     |              | Stop‑Grad  | 0.2705                         | 132.46 (66.12)                  |\n     |              | Raw        | 0.2712                         | 87.61 (62.43)                   |\n\n     *Analysis:* Stop‑Grad preserves the policy‑induced semantic grouping but lacks fully distinct boundaries, resulting in a higher cluster transition probability and a lower intra‑cluster pixel distance compared to our method (Trained).\n\n   - **Experiment 2: Episode‑ and Frame‑wise Metrics**  \n     We tested two more metrics:\n     - **Episode Cluster Entropy (ECE):** entropy of the episode’s cluster distribution (lower → more focused semantic grouping).  \n     - **Temporal Cluster Agreement (TCA@k):** fraction of frame pairs at lag k sharing the same cluster (higher → smoother, more stable semantics).  \n\n     | Environment  | Model      | ECE    | TCA@3  | TCA@6 |\n     |:------------:|:-----------|:-------|:-------|:-------|\n     | **FruitBot** | Stop‑Grad  | 1.9068 | 0.7050 | 0.5309 |\n     |              | Ours       | 1.8886 | 0.7302 | 0.5502 |\n     | **Jumper**   | Stop‑Grad  | 0.6987 | 0.7040 | 0.6715 |\n     |              | Ours       | 0.5572 | 0.7911 | 0.7089 |\n     | **Ninja**    | Stop‑Grad  | 1.5505 | 0.5798 | 0.3923 |\n     |              | Ours       | 1.3086 | 0.6787 | 0.4985 |\n\n     *Analysis:* Across all three games, our method achieves lower ECE (indicating more concentrated clusters) and higher TCA@3/6 (indicating stronger temporal coherence) compared to Stop‑Grad. Note that Jumper episodes are shorter than those in the other two games, which presents fewer clusters in single episodes and therefore yields lower ECEs.\n\n   - We will include these results in Appendix F of the final version. \n\n---\n\n> 3. Limited performance comparison and sensitivity analysis of newly introduced hyper-parameters\n\nThe primary goal of this paper is to enhance DRL interpretability (lines 39–48). While not the focus, we include performance analysis throughout: 1) Appendix C provides ablation results; 2) lines 406–407 outline hyperparameter selection; 3) Appendix D examines how varying the number of VQ embeddings affects both performance and interpretability. \n\nDuring hyper-parameter tuning, we found performance is mainly influenced by $w_{FDR}$, $w_{VQ\\text{-}VAE}$, and $f_{control}$, and is robust to the number of VQ embeddings and the degrees of freedom in Eq. (3). We will report this finding in the final version. Furthermore, we plan to expand benchmarks (lines 252–253) and provide further performance analysis in future work. We also commit to releasing our code upon paper acceptance, enabling the community to replicate and extend our results."}]

Author rebuttal:

Meta-review: (a) This paper proposes a new semantic clustering module for DRL that integrates feature dimensionality reduction and online VQ-based clustering directly into the RL training loop. It is effectively doing online K-means (with VQ VAEs) within the policy loop. Unlike other interpretability works that used t-SNE or manually defined features, this approach is automatic and simple. In its current form, a discrete VQ is trained and element-wise added to the original feature vector, but this can be generalized to have entire discrete differentiable programs via LLMs. This line of work could be interesting from the perspective of learning interpretable policies end-to-end rather than some post or pre hoc feature based thinking. 

(b) It is simple, end-to-end and can plug into existing DRL training pipelines. It also creates a trainable clustering mechanism that co-evolves with policy learning, which other methods fail to address. Experimental results are sound and validated through procgen environments, cluster quality metrics and human studies (more should be done). It also enables dynamic episode segmentation into semantic skills or options, which could have insights into hierarchical policy structure & learning.

(c) While the method reveals clear clusters, interpretation still requires human labeling of behaviors, and the paper lacks an automated mechanism for generating descriptions. The number of clusters must be manually set and no adaptive strategy is explored. Experiments are limited to four Procgen environments. 

(d) To the best of my knowledge, no prior DRL interpretability work has embedded such clustering end-to-end. The most common baseline is t-SNE and it is interesting to carefully study and explore this line of work. Perhaps it has implications for end-to-end neuro symbolic learning. I am interested in seeing this work expand where the VQ is generalized to have more flexible structures (e.g. code gen via LLMs). 

(e)  
- Reviewers asked how correct "semantic" clustering is defined. But there is no universal gold standard. This is inherently emergent and subjective.
- Concerns were raised about fixed cluster # (K). Authors acknowledged this as a limitation
- Reviewers noted reliance on humans to assign semantics. Authors argued their method reduces workload drastically compared to prior state annotation and suggested to exploring automatic description generation. I think this can be independently studied and done on top of such methods with multi-modal LLMs.
- Multiple reviewers asked about loss design (FDR, VQ-VAE), element-wise addition, and policy conditioning. Authors provided detailed clarifications and theoretical proofs in appx, showing their modified VQ-VAE is equivalent to online k-means.

I believe all major questions were addressed during the rebuttal period.
Decision: Accept (poster)