# Policy Similarity Measure for Two-Player Zero-Sum Games
## 4 Methodology
### Policy Similarity Measure  
To address this limitation, we introduce the PSM, which quantifies policy redundancy by integrating two distinct metrics: the Gaussian and cosine similarities. The Gaussian similarity captures the pairwise distances between payoff vectors, utilizing a Gaussian function to quantify the degree of similarity between policies. Simultaneously, the cosine similarity measures the angular distance between payoff vectors, reflecting their directional alignment. By combining these two measures, we derive a unified similarity matrix that captures both the magnitude of differences and the alignment of policies. This consolidated metric provides a comprehensive evaluation of policy similarity, and the resulting similarity matrix offers a visually intuitive representation, facilitating a clear analysis of the relationships between different policies. Gaussian similarity is computed by applying the Gaussian function to measure the pairwise distances between the payoff vectors of different policies. Specifically, for each pair of payoff vectors \(v_{i}\) and \(v_{j}\), the Euclidean distance \(d_{ij}=\|v_{i}-v_{j}\|_{2}\) is calculated. The Gaussian similarity \(s_{ij}\) between the two vectors is defined as follows:  
\[s_{ij}^{\text{Gaussian}}=\exp\left(-\frac{d_{ij}^{2}}{2\sigma^{2}}\right), \tag{10}\]  
where \(\sigma\) is the bandwidth parameter controlling the decay rate of the similarity. The resulting Gaussian similarity matrix \(S\) is a symmetric matrix with elements \(S_{ij}=s_{ij}\), where \(S_{ii}=1\) for all \(i\), indicating perfect self-similarity. Simultaneously, we compute the cosine similarity between the same pair of payoff vectors. This measure captures the angular distance between the vectors, reflecting the directional alignment of the policies. Specifically, for each pair of payoff vectors \(v_{i}\) and \(v_{j}\), the cosine similarity \(s_{ij}^{\text{Cosine}}\) is defined as follows:  
\[s_{ij}^{\text{Cosine}}=\frac{v_{i}\cdot v_{j}}{\|v_{i}\|\|v_{j}\|}, \tag{11}\]where \(\cdot\) represents the dot product and \(\|\cdot\|\) denotes the Euclidean norm of the vectors. The resulting cosine similarity matrix \(S^{\text{Cosine}}\) is symmetric, with elements \(S^{\text{Cosine}}_{ij}=s^{\text{Cosine}}_{ij}\), where \(S^{\text{Cosine}}_{il}=1\) for all \(i\), indicating perfect self-similarity. To integrate these two similarity measures, we combine the similarity values at corresponding positions in the two matrices, thus capturing both the relative distances and the directional alignment of the vectors. The resulting unified similarity measure is expressed as follows:  
\[S_{ij}=S^{\text{Gaussian}}_{ij}\cdot S^{\text{Cosine}}_{ij}. \tag{12}\]  
This consolidated metric provides a more comprehensive evaluation of policy similarity, accounting for both the magnitude of the differences and the directional alignment of the policies. The similarity matrix obtained through the PSM provides a visually intuitive representation, facilitating a straightforward analysis of the relationships between different policies. We incorporate the PSM as a regularization term into the policy optimization process. Specifically, for a policy pool \(\Pi\) and a newly introduced policy \(\pi^{n}\), the regularization term \(\text{PSM}(\Pi\cup\{\pi^{n}\})\) is defined as follows:  
\[\text{PSM}(\Pi\cup\{\pi^{n}\})=\sum_{j=1}^{n-1}S_{nj}=\sum_{j=1}^{n-1}S^{ \text{Gaussian}}_{nj}\cdot S^{\text{Cosine}}_{nj}. \tag{13}\]  
This term penalizes similar policies, encouraging the learning process to focus on exploring a more diverse set of policies. By incorporating the combined similarity measure into the PSRO optimization objective, we refer to this approach as PSM-PSRO. For player \(i\), the best response oracle of PSM-PSRO is expressed as follows:  
\[\mathcal{O}(\sigma_{-i})=\max_{\pi_{i}^{\prime}}\sum_{k}\sigma_{-i}^{k}E_{\pi_ {i}^{\prime},\pi_{-i}^{k}}[R]-\alpha\cdot\text{PSM}(\Pi_{i}\cup\{\pi_{i}^{ \prime}\}), \tag{14}\]  
where \(\alpha\) is a hyperparameter that controls the weight of the regularization term.