## 5 Conclusions and Future Works  
This paper investigated the role of observation space divergence as a contributing factor to performance variability in deep reinforcement learning. Through a series of experiments on PPO agents across eight Atari environments, we demonstrated a strong link between the similarity of states explored by different agents, the functional similarity of their learned networks, and the consistency of their final performance. The key findings include the following:  
* We expanded the analysis to eight Atari games, confirming that environments with low variance in performance (e.g., Freeway, Enduro) exhibit highly similar state space exploration across agents. In contrast, high-variance environments (e.g., Boxing, KFM, Qbert) show significant divergence. * Cross-evaluation of actor and value networks confirmed that agents with divergent observation distributions learn functionally different networks that do not generalize well to each other's experiences. * A new experiment with identically initialized networks revealed that performance variability and observation space divergence persist even without different initial weights, highlighting that stochasticity in the agent-environment interaction is a primary source of this divergence. * Our analysis was shown to be robust to the choice of the number of clusters (K) used for state discretization, and the computational cost of the analysis pipeline was found to be modest. This work highlights the importance of examining not only the performance levels achieved but also how agents attain them. Building on this foundational analysis in PPO and Atari, future research should extend this investigation in several key directions:  
* **Temporal and Causal Analysis:** Study the evolution of observation space differences during the training process. Analyzing the temporal relationship between when divergence occurs and when agent performance begins to vary would help establish a more direct causal link and better explain the dynamic nature of performance fluctuations. * **Systematic Component Analysis:** Conduct a comparative analysis of how different DRL components influence exploration behavior and performance variability. This includes systematically varying hyperparameters, network architectures, and regularization techniques to understand their impact on the observation space distribution of trained agents. * **Broadening Algorithmic and Environmental Scope:** Apply this analysis pipeline to a broader range of algorithms, including other on-policy variants and prominent off-policy algorithms (e.g., SAC, DQN), to determine how mechanisms like replay buffers affect divergence. Furthermore, expanding the study to other domains, such as continuous control benchmarks or environments with sparse rewards, is crucial to test the generality of our findings. * **Developing Mitigation Strategies:** Based on the insights from a component-level analysis, develop and benchmark novel exploration or regularization methods specifically designed to reduce undesirable trajectory divergence, promote more consistent learning outcomes, and directly mitigate DRL performance variability. * **Theoretical Foundations:** Work towards proposing a theoretical framework that can interpret these diverse experimental results, providing a more formal understanding of the relationship between stochastic exploration, state space coverage, and performance instability in DRL. **Author Contributions:** Conceptualization, S.J. and A.L. ; methodology, S.J. ; software, S.J. ; validation, S.J. and A.L. ; formal analysis, S.J. ; investigation, S.J. ; resources, S.J. ; data curation, S.J. ; writing--original draft preparation, S.J. ; writing--review and editing, S.J. and A.L. ; visualization, S.J. ; supervision, A.L. ; project administration, A.L. ; funding acquisition, S.J. All authors have read and agreed to the published version of the manuscript. **Funding:** This research was supported by the research fund of Hanbat National University in 2023.