## 3 Results
### Analysis of Actor and Value Network Differences  
Cross-evaluation of the learned networks (Tables 6-8) shows that divergence in observation space corresponds to functional divergence in the actor and value networks. \begin{table}
\begin{tabular}{c c c c c c} \hline  & Breakout \#1 & Breakout \#2 & Breakout \#3 & Breakout \#4 & Breakout \#5 \\ \hline Breakout \#1 & 0.000 & 0.288 & 0.352 & 0.295 & 0.361 \\ Breakout \#2 & 0.288 & 0.000 & 0.418 & 0.332 & 0.434 \\ Breakout \#3 & 0.352 & 0.418 & 0.000 & 0.220 & 0.209 \\ Breakout \#4 & 0.295 & 0.332 & 0.220 & 0.000 & 0.213 \\ Breakout \#5 & 0.361 & 0.434 & 0.209 & 0.213 & 0.000 \\ \hline  & Enduro \#1 & Enduro \#2 & Enduro \#3 & Enduro \#4 & Enduro \#5 \\ \hline Enduro \#1 & 0.000 & 0.023 & 0.017 & 0.020 & 0.021 \\ Enduro \#2 & 0.023 & 0.000 & 0.023 & 0.023 & 0.015 \\ Enduro \#3 & 0.017 & 0.023 & 0.000 & 0.019 & 0.014 \\ Enduro \#4 & 0.020 & 0.023 & 0.019 & 0.000 & 0.019 \\ Enduro \#5 & 0.021 & 0.015 & 0.014 & 0.019 & 0.000 \\ \hline  & Freeway \#1 & Freeway \#2 & Freeway \#3 & Freeway \#4 & Freeway \#5 \\ \hline Freeway \#1 & 0.000 & 0.011 & 0.013 & 0.014 & 0.013 \\ Freeway \#2 & 0.011 & 0.000 & 0.012 & 0.011 & 0.011 \\ Freeway \#3 & 0.013 & 0.012 & 0.000 & 0.010 & 0.013 \\ Freeway \#4 & 0.014 & 0.011 & 0.010 & 0.000 & 0.012 \\ Freeway \#5 & 0.013 & 0.011 & 0.013 & 0.012 & 0.000 \\ \hline  & KFM \#1 & KFM \#2 & KFM \#3 & KFM \#4 & KFM \#5 \\ \hline KFM \#1 & 0.000 & 0.067 & 0.060 & 0.078 & 0.315 \\ KFM \#2 & 0.067 & 0.000 & 0.076 & 0.078 & 0.334 \\ KFM \#3 & 0.060 & 0.076 & 0.000 & 0.098 & 0.282 \\ KFM \#4 & 0.078 & 0.078 & 0.098 & 0.000 & 0.317 \\ KFM \#5 & 0.315 & 0.334 & 0.282 & 0.317 & 0.000 \\ \hline  & Pong \#1 & Pong \#2 & Pong \#3 & Pong \#4 & Pong \#5 \\ \hline Pong \#1 & 0.000 & 0.113 & 0.137 & 0.114 & 0.476 \\ Pong \#2 & 0.113 & 0.000 & 0.152 & 0.120 & 0.485 \\ Pong \#3 & 0.137 & 0.152 & 0.000 & 0.119 & 0.358 \\ Pong \#4 & 0.114 & 0.120 & 0.119 & 0.000 & 0.444 \\ Pong \#5 & 0.476 & 0.485 & 0.358 & 0.444 & 0.000 \\ \hline  & Qbert \#1 & Qbert \#2 & Qbert \#3 & Qbert \#4 & Qbert \#5 \\ \hline Qbert \#1 & 0.000 & 0.225 & 0.013 & 0.156 & 0.225 \\ Qbert \#2 & 0.225 & 0.000 & 0.225 & 0.272 & 0.255 \\ Qbert \#3 & 0.013 & 0.225 & 0.000 & 0.159 & 0.225 \\ Qbert \#4 & 0.156 & 0.272 & 0.159 & 0.000 & 0.255 \\ Qbert \#5 & 0.225 & 0.255 & 0.225 & 0.255 & 0.000 \\ \hline \end{tabular}
\end{table} 
Table 5: _Cont._* For Freeway and Enduro, the off-diagonal values in the TD-error, RMSE, and KLD tables are very low, often close to the diagonal (self-evaluation) values. This means any agent's networks can accurately predict values and replicate policies for states experienced by any other agent, indicating functional convergence. * For Boxing, Qbert, and KFM, the off-diagonal values are substantially high. The KLD values in Boxing are frequently above 5.0, and the RMSE values in Qbert can be in the hundreds. This signifies that the agents have learned profoundly different policies and value functions that are highly specialized to their own unique experiences and do not generalize to those of their peers. This provides strong evidence that when agents see different things, they learn to do other things, and their internal models of the world (value functions) become incompatible.