# Policy Similarity Measure for Two-Player Zero-Sum Games
## 2 Related Work  
PSRO [21] is a population-based framework that extends the double oracle [18]. It provides a unified game-theoretic approach by iteratively expanding a restricted game through reinforcement learning [32], significantly improving scalability to large and complex games in MARL. Building upon PSRO, several adaptations have emerged to enhance its computational efficiency and approximation performance. For example, Pipeline-PSRO [16] employs parallelization to accelerate the training of multiple best response oracles, while neural fictitious self-play [19] integrates neural approximations of the best response into traditional fictitious play. These advances provide scalable solutions but often fail to fully exploit the policy space, especially in non-transitive settings. A critical direction for advancing PSRO involves promoting population diversity. Policy diversity encourages the discovery of robust strategies, facilitating better exploration of the policy space and, in turn, mitigating exploitability [17; 22; 30]. Various diversity-enhancing mechanisms have been proposed, often incorporating diversity metrics into the best response computation. Behavioral diversity (BD) [23; 29; 33] quantifies differences in trajectories or state distributions, drawing on concepts such as maximum entropy [34] and skill discovery [33]. While BD provides insights into policy behaviors, its limitations become apparent when minor behavioral differences result in significant reward disparities [23; 25]. To address these challenges, response diversity (RD) [17; 24; 35] leverages empirical payoff matrices to capture the strategic effectiveness of policies against diverse opponents. Recent works, such as determinantal point processes [17] and effective diversity [36], have further refined RD-based methods by emphasizing geometric and payoff-driven diversity measures. The UDM [24], a unified diversity framework, consolidates prior diversity measures into a generalized approach. However, it may overestimate population diversity when faced with highly similar strategies that yield little genuine variation in payoffs. This susceptibility to repeated or near-duplicate policies ultimately undermines the objective of cultivating a truly diverse policy set. Moreover, an effective visualization method for assessing diversity differences among policies remains absent, further limiting the interpretability and evaluation of policy diversity. Table 1 presents a concise summary of these main approaches. As illustrated, the primary unresolved challenge is the inability to effectively filter out redundant policies and visualize policy diversity. In response, we introduce the PSM, which is specifically designed to measure redundancy while offering a visual method for assessing policy diversity. By leveraging Gaussian similarity to measure Euclidean distance and cosine similarity to assess angular alignment, the PSM captures both the magnitude and direction of payoff differences, enabling a more precise differentiation of policies that might otherwise appear indistinguishable. Integrated into the policy optimization process, this method reduces population redundancy and enhances robustness against exploitation. Furthermore, the PSM facilitates the construction of a visual similarity matrix, providing a more interpretable assessment of diversity differences among policies. \begin{table}
\begin{tabular}{l l c c c} \hline \hline
**Method** & **Diversity Mechanism** & **Diversity Type** & **Redundancy Measurement** & **Visualization** \\ \hline DPP-PSRO [17] & Determinantal point process & RD & Indirect measurement & \(\times\) \\ BD\&RD-PSRO [23] & Occupancy measure and & BD\&RD & Not supported & \(\times\) \\  & convex hull & Eigenvalue of kernel & & & \\ UDM-PSRO [24] & representation & RD & Indirect measurement & \(\times\) \\ PSD-PSRO [25] & KL divergence & BD & Not supported & \(\times\) \\  & Gaussian and cosine & RD & Direct measurement & ✓ \\  & similarity & & & \\ \hline \hline \end{tabular}
\end{table}