## 1 Introduction  
Deep reinforcement learning (DRL) has demonstrated an extraordinary ability to solve high-dimensional decision-making problems, from Atari video games [1] to the board game Go [2]. Among modern on-policy methods, Proximal Policy Optimization (PPO) [3] has emerged as a de facto baseline due to its empirical performance and implementation simplicity, and it remains the algorithm of choice in a wide range of recent studies. Yet, a persistent challenge plagues the field. Despite identical hyperparameters and training budgets, independent runs of the same DRL algorithm can yield markedly different returns [4], complicating the fair evaluation and deployment of new methods. Existing research has identified several key sources contributing to this performance variability. One line of inquiry has focused on the profound impact of implementation details. Foundational work by [PERSON] et al. [4] demonstrated that differences in code-bases, hyperparameter settings, and even random seeds could lead to drastically different outcomes. This was further emphasized by [PERSON] et al. [5], who showed that seemingly minor code-level decisions--such as the choice of activation function or the order of operations--can alter performance by orders of magnitude. Another perspective has examined the dynamics of the training process itself. [PERSON] et al. [6] provided evidence that much of this variability originates early in training, as a few \"outlier\" runs drift onto low-reward trajectories and never recover. This aligns with the work by [PERSON] et al. [7], which explored how entropy-aware initialization can foster more effective exploration from the outset, thereby preventing early stagnation. A third approach has delved into the internal mechanics of the learning agent. For instance, [PERSON] et al. [8] recently established a connection between performance instability in PPO and internal \"representation collapse,\" where the network learns insufficiently diverse features, leading to trust issues in policy updates. While these studies provide crucial insights into implementation, training dynamics, and internal representations, a complementary perspective remains less explored: the divergence in the observation space that each agent actually experiences. RL agents learn from the specific trajectories of states and rewards they encounter. If two agents, due to chance events early in training, begin to explore different regions of the state space, they are effectively training on different datasets. This can lead them to converge to substantially different--and unequally performing--policies. This paper investigates the hypothesis that this divergence in lived experience is a primary cause of performance variability. Unlike prior work focused on implementation choices or internal network states [8], we aim to directly quantify the differences in state visitation distributions among independently trained agents. And while studies like [6,7] identify that runs diverge, we characterize how they diverge in terms of the states they encounter and explicitly link this to the functional dissimilarity of the resulting policies. In this work, we perform a controlled empirical study of five PPO agents, each trained with an independent random seed, across eight Atari environments--Alien, Boxing, Breakout, Enduro, Freeway, KungFuMaster, Pong, and Qbert--using the Arcade Learning Environment [9] and Gymnasium [10]. Training five independent PPO agents per game yields a spectrum of outcomes, from highly successful to comparatively poor policies. We then ask the following: Do higher-performing agents encounter a broader or different set of states than lower-performing agents? And do such differences manifest in the learned representations of the actor (policy) and critic (value function) networks? To answer these questions, we employ a range of analysis techniques. First, we visualize and quantify each agent's observation distribution using dimensionality-reduction methods and statistical measures of dispersion and similarity.