Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a unifying diversity measure of three diversity measures (ED, PD, and EC), and uses the unification to explain properties of the existing metrics. UDM-PSRO is proposed which uses the new diversity metric as an oracle. It is compared to other baselines on some simple normal-form games.
strengths_and_weaknesses: Strengths: This paper is well-written and clear to follow. The maths and proofs seem sound to me. The key idea is interesting and important research.

Weaknesses: Exploitability experiments show similar performance to other pre-existing methods. Exploitability of extensive-form games was not evaluated.

I think this is a strong paper. More thorough evaluation on (real) extensive form games would make it stronger.

Comments:

Experiments Section (Figure 1): I believe all the experiments are made on normal-form games? Therefore it might be good to mention/cite double oracle as well as PSRO.
questions: Q1: Can this diversity measure be easily extended to n-player, general-sum, or non-symmetric games?
limitations: The usual limits on symmetric tow-player zero-sum should be states again in the conclusion please.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper offers a unified diversity measure for multi-agent reinforcement learning. The authors first show the existing diversity measure and then provide Unified Diversity Measure (UDM) from a geometric perspective to unify all existing diversity measures. After showing the relationship between UDM and existing diversity measures, the authors provide two algorithms, UDM Fictitious Play and UDM PSRO, to provide diversity policies. Experiments on AlphaStar and Blotto show the performance of the proposed method over baselines.
strengths_and_weaknesses: - Strengths
    - The unified framework reveals the similarity among various diversity measures, and provides a new view of the diversity of policies.
    - The overall writing flow is good

- Weaknesses
    - Lack of the background of the geometric intuition.
    - The notations are too much and confusing, which are unfriendly to the audiences outside the game theory community.

questions: - Add the background of the geometric intuition.
- Polish the notation and the writing of this paper for more general audiences, not only for the game theory community but also the reinforcement learning/machine learning community.
- How to generate strategy feature and diversity kernel when applying UDM in the new games.
- What if the number of agents is not two?
- Line 293, missing the experiments of AlphaGO. 
- If a game has NE, why do we need to explore the diversity, especially when we can get the whole payoff matrix.
- I am curious about the experiments on AlphaStar. AlphaStar is trained for about 14 days with 16 TPUs for each agent. How to get meta-payoffs for 888 RL meta-strategies?

- Minor
    - Line 203 the meaning of Hadamard product of a vector and a matrix
    - The meaning of $|| \cdot ||_{F}$ above line 206
    - What is $m^*_i$ in line 207
    - Line 186 “similar with” -> “similar to”

limitations: I didn't find the description of the limitation of the proposed methods. And the authors didn't discuss any potential negative societal impacts. The author can describe the limitation, for example, is the proposed method limited to the two-player zero-sum game.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a unified diversity measure (UDM) for MARL learning. By choosing different diversity kernels and the function $f$, UDM can recover different existing diversity metrics. The authors establish some convergence properties under UDM and then conduct several experiments (3 in the main text and 2 additional in the appendix) including both transitive and non-transitive games. The experimental results show that UDM outperforms baselines without explicit diversity objectives and is comparable to baselines with diversity objectives in terms of exploitability and population effectivity.
strengths_and_weaknesses: **Strengths**:

This study is closely related to the learning for MARL. With a more diverse population, we are expected to achieve a faster learning speed to the target objective (e.g., lower exploitability, higher population effectivity). However, existing diversity metrics may be motivated by different practical observations and no well-defined unified diversity metric for the learning of multi-agent systems. UDM serves this purpose well by unifying existing diversity metrics into a single function, which may help to design new learning algorithms.

I think this paper is well-written and easy to follow. Based on a diversity kernel matrix, the authors show how UDM correlates to and differs from existing diversities. Some benefits of UDM are also revealed -- UDM can not only recover ED and PD separately but also tackle some of the notorious problems (ignoring weak but useful strategies and cannot distinguish redundant strategies).

Besides, UDM also seems technical sound where the methods and proofs are intuitive.


**Weaknesses**:

Current implementations (the choice of the kernel function and $f$) and experiment results are less informative. In the experiment part, the authors focus on $K\langle x,y \rangle=(\langle x,y\rangle+1)^3$ and $f(x)=\frac{1}{1+\exp(-x)}-\frac{1}{2}$ and only RD metrics. UDM generalizes RD and BD, while only a small part of the generation is shown in the current version. From the experiments, in terms of exploitability (expl) and PE, the differences among UDM, EC-PSRO, and FEP-PSRO are indistinguishable in AlphaStar888 and Blotto. I noticed that the experiments on two extensive-form games also demonstrated the above results, and the expl and PE of FEP-PSRO seems to be slightly better. The experiments for UDM-FP and UDM-$\alpha$-PSRO are missing. More insights (empirical or theoretical) from other instantiations of UDM are expected, which can be achieved by an ablation study.
questions: The questions are mainly concerning the instantiation of UDM.

(1) Could you elaborate why the kernel function $K\langle x,y \rangle=(\langle x,y\rangle+1)^3$ and $f(x)=\frac{1}{1+\exp(-x)}-\frac{1}{2}$ is chosen? Or, is there any principle for choosing these two functions in UDM?

(2) Will UDM-FP and UDM-$\alpha$-PSRO perform better (in terms of expl and PE) than the baselines that used FP and $\alpha$-PSRO respectively?

(3) This work seems to focus on RD, while UDM can be readily applied to BD. FEP-PSRO unified BD and RD, and the experiments showed it is helpful. Can UDM incorporate RD and BD simultaneously and achieve better performance than only considering one of RD and BD?

(4) The convergence of UDM-FP is provided by showing it is a GWFP process. Is it possible to show a faster convergence speed with this diversity term?
limitations: I favor the motivation of this paper, and the proposed method can provide a better perspective on the population diversity in MARL (especially in open-ended learning for two-player zero-sum games). It would be very helpful to provide an ablation study so that future researchers can know how to instantiate UDM.
review_title: null

Meta-review: metareview: This paper provides a unifying framework for promoting diverse behaviors in multi-agent RL. The framework---the unified diversity measure--- is general enough to be able to capture several other recently proposed measures as special cases (associated with specific kernel functions). The paper then provides extensions two MARL algorithms (PSRO and Fictitious-play) which make use of UDM to promote diverse behaviors in MARL and show that they converge asymptotically to relevant equilibria and provide numerical examples. 

Reviewers were generally positive on the paper, finding it well written and proposing an interesting idea for promoting diversity in MARL that seemed intuitive.  
confidence: Certain
Decision: Accept