## 3 Results
### Qualitative Analysis of Observation Space via t-SNE  
The t-SNE visualizations in Figures 2 and 3 provide a qualitative view of the explored state manifolds, color-coded by an agent. The degree of overlap and separation between agent distributions varies significantly across games, correlating strongly with performance variance.  
* **Low-Variance Environments (Freeway, Enduro)**: In Freeway, the agent observations are thoroughly intermingled, forming a dense, homogeneous cloud with ring-like structures. This indicates that all agents explore nearly identical state spaces, consistent with their minimal performance variance. Enduro shows a similar pattern of high overlap, with points from all agents mixed together within several large clusters.
* **High-Variance Environments (Boxing, Qbert, KFM, Breakout)**: These environments show clear signs of divergent exploration. In Boxing, agents form highly distinct clusters; for example, agent #1 (blue) and agent #3 (green) occupy almost completely separate regions. In Qbert, agents trace out unique, winding paths with little overlap, suggesting highly specialized policies. In KFM, the lowest-performing agent #5 (purple) forms tight, isolated clusters, while the higher-performing agents explore a much larger, albeit still differentiated, central region. Breakout shows less distinct clustering, but agents still carve out visibly different trajectories.
* **Moderate-Variance Environments (Alien, Pong)**: These games represent an intermediate case. Alien agents share a large central cluster but also have unique \"tendrils\" of exploration. Pong agents form a series of concentric rings, but some agents are more  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Task** & **\#1** & **\#2** & **\#3** & **\#4** & **\#5** \\ \hline Alien & 1394.33 (448.74) & 1135.67 (491.36) & 971.00 (509.41) & 903.33 (257.38) & 824.00 (255.94) \\ Boxing & 99.73 (0.81) & 99.10 (0.70) & 99.03 (1.14) & 93.47 (3.20) & 92.10 (4.55) \\ Breakout & 343.60 (90.93) & 312.03 (114.54) & 171.80 (174.42) & 106.20 (143.42) & 41.20 (19.39) \\ Enduro & 874.20 (170.43) & 862.67 (143.47) & 771.20 (198.16) & 711.13 (164.85) & 700.93 (203.18) \\ Freeway & 22.03 (1.62) & 21.37 (1.33) & 21.30 (1.51) & 21.27 (1.31) & 21.27 (1.34) \\ KFM & 37,453.33 (7052.74) & 34,853.33 (6008.42) & 28,180.00 (7171.30) & 22,820.00 (7527.79) & 3600.00 (0.00) \\ Pong & 21.00 (0.00) & 21.00 (0.00) & 20.80 (0.40) & 19.83 (0.37) & 19.33 (3.73) \\ Qbert & 4050.00 (0.00) & 4050.00 (0.00) & 4050.00 (0.00) & 2008.33 (1553.87) & 1116.67 (977.84) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Performance of trained agents (mean and standard deviation of rewards).  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Task** & **\#1** & **\#2** & **\#3** & **\#4** & **\#5** \\ \hline Alien & 856.57 (80.35) & 731.37 (155.80) & 711.17 (183.88) & 699.20 (124.09) & 746.60 (113.81) \\ Boxing & 286.17 (21.92) & 667.27 (46.83) & 949.33 (29.09) & 1163.43 (93.63) & 1111.53 (68.38) \\ Breakout & 17,244.43 (11,941.62) & 22,010.73 (9967.87) & 22,905.47 (9143.04) & 23,686.77 (8433.69) & 25,257.17 (6501.71) \\ Enduro & 11,198.40 (1819.96) & 11,753.43 (2229.88) & 9535.17 (2540.15) & 9534.53 (2057.03) & 10,422.03 (2539.81) \\ Freeway & 2044.43 (2.54) & 2044.07 (2.26) & 2044.20 (2.30) & 2044.37 (1.89) & 2044.23 (1.67) \\ KFM & 3482.80 (788.32) & 3866.47 (448.77) & 3037.83 (595.05) & 4139.97 (553.75) & 1099.23 (2.54) \\ Pong & 1684.03 (20.47) & 1679.57 (26.72) & 1688.63 (23.34) & 2707.00 (302.06) & 1794.80 (85.39 confined to inner rings while others explore the outer regions more, and some agents (e.g., #5, purple) have more diffuse distributions.  
Figure 3: _Cont._  
Figure 2: Qualitative t-SNE analysis of observation space representations from five distinct agents in four Atari games: (**a**) Alien, (**b**) Boxing, (**c**) Breakout, and (**d**) Enduro.