Title: Automatic Curriculum Generation for Reinforcement Learning in Zero-Sum Games
Authors: Jiayu Chen
Yunfei Li
Zelai Xu
Huazhong Yang
Jiaming Song
Yu Wang
Yi Wu
Venue: ICLR 2023
Keywords: multi-agent reinforcement learning
curriculum learning
zero-sum games

Abstract: Curriculum learning (CL), whose core idea is to train from easy to hard, is a popular technique to accelerate reinforcement learning (RL) training. It has also been a trend to automate the curriculum generation process. Automatic CL works primarily focus on goal-conditioned RL problems, where an explicit indicator of training progress, e.g., reward or success rate, can be used to prioritize the training tasks. However, such a requirement is no longer valid in zero-sum games: there are no goals for the agents, and the accumulative reward of the learning policy can constantly fluctuate throughout training. In this work, we present the first theoretical framework of automatic curriculum learning in the setting of zero-sum games and derive a surprisingly simple indicator of training progress, i.e., the Q-value variance, which can be directly approximated by computing the variance of value network ensembles. With such a progression metric, we further adopt a particle-based task sampler to generate initial environment configurations for training, which is particularly lightweight, computation-efficient, and naturally multi-modal. Combining these techniques with multi-agent PPO training, we obtain our final algorithm, Zero-sum Automatic Curriculum Learning (ZACL). We first evaluate ZACL in a 2D particle-world environment, where ZACL produces much stronger policies than popular RL methods for zero-sum games using the same amount of samples. Then we show in the challenging hide-and-seek environment that ZACL can lead to all four emergent phases using a single desktop computer, which is reported for the first time in the literature. The project website is at https://sites.google.com/view/zacl.

TL;DR: In this work, we present the first theoretical framework of automatic curriculum learning in the setting of zero-sum game and derive a surprisingly simple indicator of training progress, i.e., the policy variance

Review discussion:
The authors haven’t mentioned nor compared to Neural Auto-Curricula [1], which is also a method for automatic curriculum generation in zero-sum games. This is essential given the claim in the abstract that ZACL is the first method to tackle the problem.
In related work, the paper states that the method is faster than emergent behavior methods, this claim is not supported in the results.
In the paragraph after equation 2, the energy-based formulation should be approximate, not exact. Equation 8 the Q function needs +-.
In the experimental section, the comparison to baseline methods was only in the easy MPE environment, while we see only comparisons to MAPPO and ablations in HnS.
It would be useful to report the time of the training time when assuring using a single desktop machine as a major contribution (given the method needs 2.0 billion samples in figure 5).
[1] Feng, X., Slumbers, O., Wan, Z., Liu, B., McAleer, S., Wen, Y., ... & Yang, Y. (2021). Neural auto-curricula in two-player zero-sum games. Advances in Neural Information Processing Systems, 34, 3504-3517.
Overall the writing is clear. The paper should present the gist of the experiments in the main paper instead of deferring everything to the appendix. The theoretical derivation is correct and the presentation quality is good. The idea of using the value disagreement is not novel, the authors pointed that in the related work. The theoretical formulation for zero-sum games is somehow novel but not significant. No code submission to reproduce the results.
NO.
1. “ We haven’t mentioned nor compared to Neural Auto-Curricula ”
Thanks for your reminder, we will supplement the comparison experiment in the final version. It's important to clarify that we are the first one to evaluate and design the curriculum at the level of subgame, while Self-Play, PSRO or Neural Auto-Curricula design the curriculum at the policy level and train against moderate opponents.
2. “ The paper states that the method is faster than emergent behavior methods, this claim is not supported in the results. ”
In MPE, we fix the policies of the prey trained at 15M environment samples for all the baselines, then train the best response policy of the predators to measure the exploitability. We additionally try allowing 45M (3 times) samples for PSRO since PBT typically requires more samples. This result indicates that our algorithm has the highest sample efficiency.
3. “In the paragraph after equation 2, the energy-based formulation should be approximate, not exact. Equation 8 the Q function needs +-. ”
Thank you for pointing out our typos in the theoretical proof.
4. “ The comparison to baseline methods was only in the easy MPE environment, while we see only comparisons to MAPPO and ablations in HnS ”
We will supplement all experiments in the future. Thank you for your suggestions.
## Strengths
The results on hide and seek sound impressive, and the ones for Predator-Prey are evaluated against good baselines.
The proposal of using convergence time on HnS is an interesting idea, and a useful contribution on its own, as it could become a common tool for evaluating curriculum learning methods.
## Weaknesses
I don't find the argumentation behind concentrating on 0-sum (the rewards are not growing) convincing, as there is a lot of work that define progress in 0-sum games this way or another (apart from the cited PSRO-derived ones, eg. [1] claims size of a NE is a useful measure of progress).
Furthermore, the mathematical motivation behind the method nor the method itself doesn't use the assumption of the game being 0-sum in any way, which makes the 0-sum part unintuitive.
The mathematical analysis provides a motivation for a significantly different method:
1. One that uses Q functions, and not value functions. There is no explanation provided on why using V provides an accurate approximation to Q. I don't understand why authors couldn't use Q-function (with actions sampled from the policy) instead, following their motivation.
2. A Maximum entropy one.
3. One with a fixed entropy weight, which on its own is a relatively strong assumption in a Curriculum Learning setting, where we may want to decrease entropy penalty over the course of training as the policy gets better and better.
The comparison with MAPPO on HnS doesn't provide enough detail to be properly judged: it's not clear what machine is used in both cases: the phrasing of the paper makes me assume that for ZACL there is a single GPU, able to process 2B samples in a single batch, but that seems beyond the current hardware possibilities. It's not clear why MAPPO uses 9600 workers while ZACL 256 and why the batch sizes are significantly different between the methods.
Finally, it would be useful to test one of the other SOTA methods (eg. from the ones tested on Predator-Prey) on this task for a reliable comparison.
As authors correctly cite, a very similar method for optimizing the tasks based on maximizing a Q-value variance was proposed in 2020.
## Future work suggestion
One unaddressed issue of the maximizing-variance method is the variance coming from the environment. If there are tasks which inherently possess such variance (eg. there are two doors chosen randomly and if the agent chooses wrongly, it's not able to get to the goal), they will always be on top of the priority queue, even when with the optimal policy of the agent, wasting environment interaction on tasks where nothing else could be improved.
[1]: Czarnecki et al. [Real World Games Look Like Spinning Tops](https://arxiv.org/abs/2004.09468)
I find the work relatively clear (with the issues expanded in the previous section), and of high quality.
I find the novelty of the method itself limited, but I find the proposed task to be of high experimental novelty.
I find the first experiment reproducible, the second one: not so much (again, details in the previous section).
NO.
1. " There is a lot of work that define progress in 0-sum games "
There is a lot of work to define progress in 0-sum games at the policy level and they concentrate on training against moderate opponents.
But we use a completely different metric, which automatically generates a curriculum at the subgame level to accelerate convergence to Nash.
2. " The mathematical motivation behind the method nor the method itself doesn't use the assumption of the game being 0-sum "
The advantage of our curriculum method is that it is not limited to 0-sum games. As mentioned in the article, we can solve fully cooperative games and accelerate the training process.
3. “ The mathematical analysis provides a motivation for a significantly different method ” \
( 1 ) We use Q functions in the proof, but Q functions and V functions are equivalent in the algorithm implementation. In our implementation, we only need to use the relative value of difficulty as a metric for sampling. We can represent  the variance of Q functions by V functions and the current policy : $Var_{j=1\sim m}[Q^j_{\pm}(s_t, a_t)] = Var_{j=1\sim m}[r^t + \gamma E_{ s_{t+1} \sim \rho_{s} }[V^j_{\pm}(s_{t+1})]] = \gamma E_{s_{t+1} \sim \rho_{s}}[Var_{j=1\sim m}[V^j_{\pm}(s_{t+1})]]$ .
So we directly use the V functions that are available in the MAPPO algorithm.
( 2 ) We will use the max entropy version of the algorithm for comparison later. Thanks for your suggestion.
( 3 ) For the convenience of proof, we use λ = 1, which corresponds to a certain degree of entropy regularization, but the argument is still valid for other λ > 0.
4. “ 2B samples in a single batch seems beyond the current hardware possibilities ”
We use a single a100 GPU to train ZACL, and consume 2B samples in the whole training process.
But as mentioned in the appendix, the sample in a single batch is 57,600 chunks of 10 timesteps, which takes up all the GPU memory.
5. “ It's not clear why MAPPO uses 9600 workers while ZACL 256 and why the batch sizes are significantly different between the methods. ”
Due to the limitation of compute,  ZACL only uses 256 workers to collect data in a single machine.
However, MAPPO training on HnS requires a very large number of samples and consumes a long training time.
So, like OpenAI, we use the distributed system and run a lot of parallel training workers to speed up the training process. The results show that we verify that MAPPO cannot exceed the performance of ZACL even if using a larger batch size in the distributed system.
6. “ Future work suggestion ” \
Thanks for your advice.
I liked that this paper was able to replicate and improve on the results from the OpenAI hide and seek paper with much less compute.
However, there are a number of issues with this paper.
First, I don’t think the motivation is there. If you are already doing self play, why do you need a curriculum? Usually the idea is that you need to use a curriculum to scale up to hard tasks, but the point of the OpenAI paper was to show that self play is an automatic curriculum.
Self play isn’t guaranteed to converge to Nash in two-player zero-sum games, and MAPPO is a cooperative RL algorithm
I don’t understand why PSRO is exploitable by a best response. By design, upon convergence PSRO is not exploitable by a best response. How many iterations did you train it? Could you show approximate exploitability over time for all algorithms?
Also, comparisons to CFR-based methods such as ESCHER would help the evaluation.
I thought the paper was well-written. The novelty is fine: to my knowledge I don't know of any papers looking at curriculum in two-player zero-sum games. The algorithm seems reproducible.
NO.
1. “ Why do we need a curriculum ? ” \
As mentioned in OpenAI's paper, hiders and seekers generate multiple rounds of strategies through self-play and our main contribution is to reduce the number of samples that reach each stage by generating an automatic curriculum that varies from easy to difficult. Self-play or PSRO-style algorithms use an automatic curriculum at the level of policy and only train against moderate opponents.
However, our method is to automatically generate curriculum at the subgame level to accelerate the convergence to Nash.
2. “ Why is PSRO exploitable by a best response ? ”  \
PSRO can theoretically converge to Nash, but needs sufficient samples.
Therefore, we fix the policies of the prey trained at 15M environment samples for all the baselines, then train the best response policy of the predators to measure the exploitability. We additionally try allowing 45M (3 times) samples for PSRO since PBT typically requires more samples. The results show that our method is faster than baselines.
Thank you for the response, but I remain unconvinced so I will keep my score.

Meta-review:
role: PC
data: metareview:_summary,_strengths_and_weaknesses: The reviewers agreed that the paper investigates an important setting of curriculum learning in two-player zero-sum games, is generally well-written, and has an interesting experimental evaluation. However, the reviewers pointed out several weaknesses in the paper, and there was a consensus that the work is not yet ready for publication. The reviewers have provided detailed and constructive feedback to the authors. We hope the authors can incorporate this feedback when preparing future revisions of the paper.
justification_for_why_not_higher_score: There was a consensus among the reviewers that the work is not yet ready for publication.
justification_for_why_not_lower_score: N/A

Decision: Reject