Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper tackles the task of automatically inducing a curriculum for agents learning through reinforcement. Specifically, they use two agents — a setter agent that sets goals, and a solver agent that solves the goals provided by the setter.  While this has been explored before, the difficulty lies in training both agents simultaneously in a robust fashion. If the goals are too difficult, the solver will be unable to solve them and if they are too easy, the solver will be unable to improve. The authors propose a combination of different losses to help the setter balance its goal predictions — validity, feasibility and coverage. In addition, they train a judge model predict the reward that the solver agent would achieve on a goal proposed by the setter. Empirical results on two setups demonstrate the effectiveness of this approach in learning a good curriculum. 

Pros:
1. Clear writing, method is easy to understand. 
2. Novel objectives for a multi-agent training setup

Cons:
1. Empirical results do not contain any baselines or prior work comparisons (only ablations of the proposed model)

——
Updates:
Thanks to the authors for their response. I realize I was not very clear in my comment above. I wanted to point out that the authors could consider adding other (simpler) baselines such as Sukhbaatar et al. (2017) to make their empirical results more complete for the navigation tasks (even though these methods make certain assumptions, it would be interesting to see how much of an effect they have compared to the proposed method). If you are able to, I’m convinced the paper will be much stronger. Nevertheless, I think this is very interesting work!
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes an autocurricula scheme to train a goal-conditional agent in a dynamic and sparse-rewarding environment. The main idea is to train a setter model to sample goals for next-step training, where the setter can make the decision either based on the training history or the environmental observation (conditional case). The paper proposes three criteria which leads to three types of loss to train the setter model, i.e., goal validity (the goal should be achievable by some existing policy), goal feasibility (how probable the current policy can achieve the goal), and goal coverage (the sampled goals by the setter need to cover all possible goals). A judge model is needed to output the feasibility of a given goal. So the autocurricula scheme contains the solver (agent), the setter, and the judge, each having its own combination of loss and they are trained together. Given a desired goal distribution, the paper proposes to additionally train a discriminator whose optimization objective is Wasserstein loss. In experiments, they evaluate the proposed method on three types of tasks in two environments, i.e., 3D color finding and grid world alchemy. The goals in the two environments are similar in that they all aim to achieve some color or color pairs. The difference lies in that the first one finds colors while the second pick up colors. Each environment can be changed between episodes by changing the colors of objects in the scenes. Experimental results show that different combinations of the three types of losses can bring improvements in some scenarios. Making setter and judge conditioned on environment observation can further improve the success rate. Given a desired distribution of goals, the learning becomes more efficient. The paper compares this method with Goal GAN as a baseline and outperforms it on the three tasks.

This paper is an early exploration of the effectiveness of autocurricula on goal-conditional tasks. The experiments to some extent verify the effectiveness of the proposed method. The high-level idea of setter-solver learning can be possibly generalized to other tasks. However, it is not convincing that the detailed techniques proposed in this paper can be easily generalized to more complicated environments and tasks. The writing of this paper is not clear and enough and the whole paper is difficult to understand. The experimental comparison is insufficient since only one baseline of curriculum (goal GAN) is compared (some of the methods mentioned in Section 2 can be easily applied though some of them were not specifically designed for goal-conditional tasks). 

Detailed comments:

1) There are too many details and hyperparameters involved when applied to specific tasks as shown in this paper. Considering the two environments in this paper is easier than most game environments studied in other papers (many of them also uses some types of the curriculum), the experimental result is not very strong. The proposed scheme is possible to be too simple for a slightly more complicated environment or task. 

2) The success of training relies on interactions between the three to four types of losses, but they all have the same weights in the combined loss. Is there any special reason for not setting different weights for different losses?

3) It is not clear how the inverse transform of the setter model (i.e., S^{-1} in L_val) is achieved.

4) There are three random quantities involved in the loss terms, i.e., the noise \xi applied to the sampled goals, the latent input z, the feasibility f. It is not clear whether the randomness on them will dominate the curriculum or not. 

5) The setter model takes z and f as inputs, where z is a vector but f is a scalar. Will it result in a setter model whose output does not change too much when changing f? 

6) How does the proposed method compare to simple goal selection by uncertainty, hardness, or curiosity? How does it compare to hindsight experience reply methods?

7) It is helpful to visualize the generated curriculum, i.e., the trajectory of selected goals during training in the 2D/3D grid.

8) Computing L_judge requires to test whether the agent can finally achieve the sampled goals. Is it too computational expensive during training? 

---------------------------

Update after rebuttal:

Thanks for the detailed reply from the authors! They answer most of my questions. I think this paper has a moderate contribution and thus will keep my "weakly accept" rating.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper studied the problem of reinforcement learning under the sparse or dynamic rewarding environment. 
The authors propose a promising automated curricula generation scheme, which considers goal validity, goal 
feasibility, and goal coverage to construct useful curricula for the underlying agents. Rewards and loss functions 
are proposed individually for the solver, judge, and setter. Empirical studies demonstrate the capability of the 
proposed model in generating task curricula across several complex goals. In general, I believe the studied 
problem is interesting, and the proposed model is promising.  However, I am not familiar with the curricula 
generation in the reinforcement learning setting. All I can say is the approach is intuitively appealing, the text is 
well written and easy to follow, even for an outsider. A minor concern is about the experiments. Most of the 
experiments are presented in the metric of Reward (% of best). It would be helpful if the authors can conduct 
some illustrative case studies (better in different scenarios) to show some specific tasks, the generated task-oriented 
curriculum, and provide some intuitive discussion to show the readers why the generated curriculum is beneficial 
for the agent. 
review_title: Official Blind Review #1

Meta-review: The authors introduce a method to automatically generate a learning curriculum (of goals) in a sparse reward RL setting, examining several criteria for goal setting to induce a useful curriculum.  The reviewers agreed that this was an exciting research direction but also had concerns about baseline comparisons, clarity of some technical points, hyperparameter tuning (and the effect on the strength of empirical results), and computational tractability.  After discussion, the reviewers felt most of these points were sufficiently addressed.  Thus, I recommend acceptance at this time.
Decision: Accept (Poster)