Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a curriculum learning method “VACL” to solve the multi-agents RL problems. The main idea of this paper is to train RL agents on an easy-to-hard task sequence based on an auto-generated curriculum.
main_review: The main idea of this paper is similar to some other curriculum RL methods, such as GoalGAN, setter and solver (https://arxiv.org/abs/1909.12892 ) and AMIGo. Due to the results of the experiments, this method achieves a great improvement on all previous work in a hide-and-seek environment. Meanwhile, VACL also achieves good performance on other tasks. 
 
Originality: The main idea of this paper is similar to the GoalGAN and AMIGo. However, the curriculum generator is not as tricky as the mentioned methods to learn. The author uses a smooth function to represent the learning progress of a specific task instead, which achieves a higher sample efficiency. The essential part of this method is to use an incremental transform \T to simplify q(\phi). Due to the results, it’s obvious that this transform is available.
 
Clarity: I think this paper is written well. The expansion of the theoretical part is very detailed and logically fluent. The details of the experiments seem sufficient. 
 
However, here are some questions and suggestions for this paper: 
 
Questions:
1. In algorithm 1. \geteasy(n_{0}). How to define easy tasks for agents? Are these easy tasks picked by humans? What is the difference of two \geteasy used in \geteasy(n_0) and \geteasy(n_k+1)?
2. Section 3.4 is not sufficient enough. Considering that the main idea of VACL is to sample an easy-to-hard curriculum for agents to learn, and also the evaluation of the difficulty of tasks is also based on the V. Why not add the method mentioned in section 4.1 to section 3.4? It causes me so puzzled when I first read section 4.1.
3. Does the forgetting phenomenon occur in the experiments? If yes, how to solve it? I did not find any restrictions or guarantees if the agent forgets some tasks that have been resolved.
4. For empirical, I think V cannot fully express the value of learning on the current task. Maybe you can consider another metric to move tasks into Q_{act}, such as \delta V. Of course, this is just a possible optimization. 
5. How to ensure the upper and lower bounds of "f" used in eq.4?
 
Overall, the paper offers a possible solution for evaluating the difficulty of tasks. 

=== Post Rebuttal ===

The paper studies an interesting problem. Its claims are well supported from both theoretical and experimental perspectives.



limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a variational automatic curriculum learning framework for solving goal-conditioned cooperative multiagent reinforcement learning problems. In this framework, the learning objective is decomposed into two parts: policy learning on current task distribution, and curriculum update to a new task distribution. The curricula are trained based on two components, including task configuration and the number of agents. The experimental results demonstrate the proposed method can achieve better performance than a few baselines with light weight computation resources. 
main_review: In all, this paper studied an important method for solving cooperative multiple agent sequential decision-making tasks by using curricula learning. The authors tried to develop and analyze the solution method in a rigorous way. The authors also provide some good discussion on the connection to existing work. The method is tested over two environments and shown to outperform a few baselines. However, the motivation and some details related to the claimed contributions are not clearly explained, and the problems settings considered seems restrictive to homogenous agents.  

Specifically, it is a little unclear to me why Stein variational gradient descent is necessarily better than regular variational inference method. It might be better to provide some more background information. 

There are some technical details not clearly explained. For example, In section 2.2.1, “to update the particles, we consider an incremental transform T(\phi)=…”. Why it is necessary to introduce this transform? This seems to be one of the main ingredients for deriving the gradient update of task distribution. It might better if some design choices can be justified.

What is the number of particles used for performing Stein variational inference? How sensitivity is results to it?

Another limitation is that the paper only considered homogenous agents and learn a shared goal-conditioned policy. It might worth discussing how the approach can be generalized to heterogenous agents’ cases, since many realworld problems involve collaboration among different types of agents.
------ after rebuttal------
I think the author should explicit demonstrate that the method can work on problems with heterogenous agents. Given that the rebuttal has addressed most of my concerns, I would like to increase my score to 6.
limitations_and_societal_impact: The author has provided some discussion regarding the limitation of their work. Additional limitations I see for this work are discussed in the Main Review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a curriculum learning method for cooperative multi-agent reinforcement learning in sparse reward tasks. The authors derive a curriculum learning approach by forming a lower bound on the expected reward under a target distribution p(\phi). This lower bound is maximized by alternatingly optimizing the policy under a variational distribution q(\phi) and then adjusting the variational distribution based on the current performance of the agent. The authors investigate the theoretical behavior showing that the optimization of q(\phi) minimizes a form of "weighted" KL-Divergence to p(\phi). Further, they derive a gradient-based approach to optimize q(\phi) and propose an approximate implementation of their approach that performs empirically well in different multi-agent RL benchmarks.
main_review: Strengths: \
The paper presents novel and insightful content. I especially like the transition from the derived gradient in Eq (4) to the approximate implementation in Section 3.1. The proposed quantization nicely connects concepts such as success buffers to the lower bound on the agent performance. As such concepts are rarely connected to any theory, this is an exciting variation.

The experiments demonstrate good performance. In combination with the convincing theoretical motivation, this seems to be a promising approach for curriculum learning in (multi-agent) RL. Further, the ablations in the experimental section give a good impression of the inner workings of the method.

Weaknesses: \
The theoretical motivation for the curriculum does not seem to carry over to the discrete variables describing the number of agents. The introduced entity progression scheme seems detached from the theory. While I understand that the gradient-based operator may not be computationally advisable for large N, why did the authors not subsume the number of agents into the parameter \phi. While this may not be in line with the continuity assumption about \phi, it should be fairly straightforward to implement. And with a properly adjusted noise process for exploration, it could yield an implementation more similar to the theoretical motivation. Nonetheless, the overall method seems to work well.

I see another weakness in the performance comparison in the experimental section. Apart from issues with clarity (see next section), the comparison methods did not use a curriculum over the number of agents. While I understand that there may be no related method ideally suited (because of the lack of a curriculum over both goal-state and agent number), I think one ablation should be added to the experiments: VACL without the curriculum over the number of agents. This would give an additional insight into the importance of the curriculum over agents and also yield a fairer comparison to the other curriculum learning methods like GoalGAN, RCG and AMIGo. 

In the "SVGD-Principled Update" experiment, the (principled) gradient-based version from Eq (6) was not compared to the rejection sampling method used in Algorithm 1. Instead, only a version without the rejection step has been used. Why is there no such comparison in the hard-spread scenario? I think it would give a good intuition of whether the rejection sampling used in VACL performs similarly to the gradient-based method.

Clarity: \
The paper is well written, and the concepts were understandable. However, some parts of the paper could be more concise, especially w.r.t. experimental details:
 - The rejection sampling makes use of an "oracle" that can identify infeasible regions. However, the infeasible regions are not specified in the experimental section except for hard-spread, where (as already mentioned) rejection sampling is not evaluated. The authors should clearly state whether additional knowledge was gained through the aforementionend oracle in the other environments. This would allow readers to judge whether VACL had an advantage over other methods like GoalGAN due to the access to the oracle.
 - Was GoalGAN also given access to the initial samples obtained by GetEasy(n)? I think it should be because it explicitly supports such seeding by initial tasks. I am curious because, in line 270, the authors mention that "GoalGAN and AMIGo takes an extremely long time to train a good neural goal-generator."
 - In Algorithm 1: A reference to the implementation of the "Sample" function in the appendix would be good to guide the reader.
 - How were the continuous goals \phi defined in the individual experiments? Unfortunately, I could not find any details on this. Especially in the hide-and-seek environment, this would be interesting. Further, what seed tasks are generated by the GetEasy function in the environments?
 - The discussion of the "Choice of seed task" in Section 4.1.2 seems to have an error. The text states that "Exp.Act.w.eval" works better than "Exp.Act." while the plots indicate the opposite.

I think the aforementioned points are another weakness of the paper, as some of those make the obtained results a bit less convincing.

Related Work: \
I have two suggestions for improvement for the related work:
 - The reference to the work by Wöhlke et al. (citation [30] in the main paper) seems to be presented a bit misleading in the related work section. Is it maybe because of a mix-up in the citations? While the authors say that "[30] propose to meta-learn f^*(\phi) to more effectively expand the curriculum distribution", it instead seems that [30] use a different characterization of the "active set" as the authors proposed here. In [30], the active set is defined as those parts of the task space in which the value function exhibits the largest gradient w.r.t. the task variable.
 - There exists work [1,2] that proposes a similar curriculum scheme and links it to a probabilistic inference perspective. It also minimizes the KL-Divergence between q(\phi) and p(\phi). While I don't think that this existing work hinders the novelty and originality of the proposed approach due to the different methodology, I think it should be included in the related work section. In combination with the corrected description of reference [30], this would yield a better overview of methods based on value functions and/or probabilistic inference.

[1] Klink, Pascal, et al. "Self-Paced Deep Reinforcement Learning." Neural Information Processing Systems, 2020.

[2] Klink, Pascal, et al. "Self-paced Contextual Reinforcement Learning." Conference on Robot Learning, 2019.


UPDATE:

After reading the authors responses, I increased my score from 5 to 6.
limitations_and_societal_impact: The authors discuss the limitations and societal impact of their work in the conclusion.
review_title: null

Meta-review: This work introduces a new curriculum learning algorithm for cooperative multi-agent reinforcement learning. All reviewers appreciated the method's novelty and that the paper was well-written. All found the theoretical motivation convincing and insightful. There were some initial concerns about the experimental results raised by reviewer spUJ, but they found the author's response convincing and thus increased their score.
Decision: Accept (Poster)