# Policy Similarity Measure for Two-Player Zero-Sum Games
## 1 Introduction  
Two-player zero-sum games are a prominent setting in multi-agent reinforcement learning (MARL) [1] due to their analytical tractability and symmetry, making them a central focus of theoretical and applied research. In this context, self-play has emerged as a promising approach to addressing the inherent complexities of MARL in competitive scenarios. By training an agent against copies or past versions of itself [2, 3], self-play offers a stable and tractable framework for learning. This framework has been successfully applied in various game domains, such as Go [4, 5, 6, 7], chess [6, 7], poker [8, 9], and video games [10, 11, 12], often surpassing human-level performance. Despite these successes, two-player zero-sum games often exhibit non-transitive policy spaces [13], where the superiority of one policy does not guarantee universal dominance [14, 15]. Consequently, each player must acquire a diverse set of winning strategies to ensure robust performance against potential exploitation [16, 17]. A key challenge in these non-transitive games is identifying a Nash equilibrium (NE), which ensures optimal worst-case performance across the policy space. Traditional self-play [18, 19, 20] often fails to converge to an NE in the presence of strong non-transitivity [16, 17, 21]. One prominent framework for approximating the NE is policy space response oracles (PSRO) [21]. PSRO integrates conventional game-theoretic equilibrium computation withpolicy learning and iteratively expands a restricted game constructed from a limited set of policies. Ideally, the restricted game is significantly smaller than the original game, enabling a more tractable equilibrium approximation. Improving the capacity of PSRO to approximate the NE remains an active area of research. A promising direction involves promoting policy diversity within the restricted population, which is often accomplished by adding diversity regularization to the best response oracle of PSRO [17, 22, 23, 24, 25]. Encouraging policy diversity is crucial for achieving high-performance learning in non-transitive games. Diversity prevents agents from repeatedly exploring the same policies, thus enabling more efficient exploration of the policy space [26, 27]. Furthermore, ensuring a diverse policy repertoire mitigates exploitability, as any dominant policy is likely to be defeated by an alternative, but also fosters the adaptation necessary to maintain robust performance over time [28]. Existing methods attempt to quantify diversity by analyzing the differences in agent behaviors and response patterns [23, 24, 29, 30, 31]. The unified diversity measure (UDM) [24] further unifies these methods, offering a more effective means of quantifying population diversity. While the UDM provides a comprehensive and unified method for measuring population diversity, it is susceptible to redundant policies, which can cause it to converge to a local optimum. In this work, we propose the policy similarity measure (PSM), which focuses on measuring policy similarity. The PSM uses Gaussian and cosine similarities to quantify the similarity between payoff vectors, generating a similarity matrix for the population. This similarity matrix visually represents the population's diversity, allowing for an intuitive understanding of the differences between policies. Moreover, the PSM can be integrated as a regularization term during policy optimization, effectively guiding the learning process toward more heterogeneous policy sets. In brief, the PSM enhances the diversity of the overall population while providing a visual measure for assessing diversity. Furthermore, we demonstrate how redundant policies undermine the effectiveness of the UDM as an accurate measure of population diversity. In summary, this work makes the following contributions:  
* We point out the shortcomings of the UDM in handling redundant policies.