Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: 
This paper proposes a framework to progressively generate a sequence of tasks as curricula for hard-exploration tasks. It demonstrates better performance on MiniGrid and robot manipulation tasks compared with exploration baselines and curriculum goal generation baselines.

Pros
+ The paper is well-written and easy to follow. It provides enough implementation details.
+ This paper tackles a hard problem that incorporates exploration, policy learning, and curriculum task generation. The adversarial training solution is clean. It proposes a new way to learn the similarity score by encoding the MDP.

Questions & Concerns:
- The optimization problem on page 4.
 1. it's not an equation, it's an optimization problem. There are many references in the paper using "Eq. 1", "Eq. 2" for the optimization problem.
 2. There is nothing related to KKT condition for (2). KKT condition is some first-order necessary conditions for the saddle-point to be optimal. Form (2) in the paper doesn't use KKT. It's just a Lagrangian form.
 3. $\beta$ is usually called the Lagrangian multiplier rather than the KKT multiplier.
 4. Since $\beta$ is the dual variable, the optimization problem for $\beta$ is $min_{\beta} \text{ }L$ with $\beta$ lower bounded by *zero*. So that this problem is the upper bound of the primal problem. It shouldn't *explode* when you optimizing $\beta$ since you can just choose $\beta$ to be small enough to get close to the primal problem. 
 5. I understand the author may use some empirical way (an optimization problem with regularization) as they showed in their appendix, rather than primal/dual optimization. It's better to make this part rigorous.
- Can you explain why $V_1$ and $V_2$ only take $w$ as input? Both $V_1$ and $V_2$ are some total expected returns (of rewards or similarity), so they are also related to the policy, not only the tasks. In the original deep RL framework, a value network should take states as input. In your setting, it should take the states, or trajectory (embedding) as input. It's not reasonable to have a value estimation that doesn't take the data samples as input. Do you assume $V_1$ and $V_2$ can always fit the returns generated by an arbitrary policy? 
- In the algorithm, do you have any inner loops to train the discriminator and the RL algorithm? By meaning "while not converge", do you mean the convergence for all the losses as well as the RL performance? Will they work in a coherent way?
- In Fig 5, it seems APT-Gen without task progress can also work to some extent. This is confusing. Why the policy is not stuck on easy scenarios?
- (Minor concern) POET is an important baseline but the results are not included. 

I'm willing to change my score if some concerns are addressed.

========post rebuttal review=========

After reading the response from the authors and the new version of this paper, I decided to increase my score to 6.
review_title: Review for paper "Adaptive Procedural Task Generation for Hard-Exploration Problems"

final_score: rating: 7: Good paper, accept
review_content: This paper presents a procedurally content generation approach (APT-Gen) that generates a sequence of tasks for an agent to solve. These tasks are automatically generated in a way that helps an RL agent to learn hard-exploration problems. A main innovation of the approach is a task generator system that is both rewarded for generating tasks the agents can solve but also a sequence of tasks that are getting increasingly more similar to the target task.

I find the paper interesting, the results convincing and don't have too much to critique. My main critique would be to add what the limits of the current approach are. What are the hard-exploration problems it fails on and how much can the target task be outside the predefined task space before performance degrades? A more systematic study of these limitations would further add to the paper and highlight potential future directions for improvements. 

Minor comments:
- Based on how many independent runs are the results based on that are shown in Figure 3, 4, 5?  What do the shaded areas show? One standard deviation?
- A related paper that does adaptively and procedurally generate levels, instead of tasks, is "Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation " Justesen et al., 2018
review_title: Official Blind Review #2

final_score: rating: 4: Ok but not good enough - rejection
review_content: In the paper "Adaptive Procedural Task Generation for Hard-Exploration Problems", the authors introduce a new method to automatically generate a curriculum for reinforcement learning algorithms for hard-exploration problems.
The approach uses a generative network to generate variants of the agent's environment with varying difficulty and then a discriminator to automatically access the resemblance of the generated environment with the target environment. The generating process then uses this resemblance metric and the observed task return to progressively drive the learning process toward environments that can be solved by the agent and that become closer and closer to the target environment. 

The paper is well structure and clearly written. The illustrations, in particular, Figure 1 are particularly helping to understand the overall process of the algorithm. 


There are, however, a few elements that can be improved in the paper. 

First, in several places, the paper is quite vague on the claim and differences with other algorithms. For instance, "In contrast, we propose a general framework for highly configurable task spaces with parameters of much higher dimensions.". What is the exact scale of the difference? I think the paper will be more convincing with precise examples and the exact numbers of the dimensions. 

Related to this aspect, the task description provided in the paper is not sufficient to fully understand what are the degrees of freedom given to the generative network to generate different environments. In particular, what is changing between the different instances? This aspect is too central for the paper that it cannot be relegated to the appendix. 

After reading the appendix, we can then find the parameters of the environment are all discrete, with few possible values. This is a big deal actually because even if the number of parameters is larger in this paper, other papers referred here are using continuous parameter space, which is by definition infinitely larger. Thus the claims made in the paper need to be tuned down a little bit to underline this difference, and it could be important to discuss how the proposed approach to scale to continuous parameter spaces.  

Similarly, the justification of the baseline selection is not sufficient in the paper to understand the motivations of the authors and why these algorithms are used in this study, and also why some approaches are not included. For instance, the POET seems to follow a similar approach, yet working on continuous parameter spaces. According to the authors, the propose methods will work better because it is not trying to cover all the space of possible environment, but this remains to be demonstrated in the experimental section. 

Overall the method proposed in this paper shows some promising results, but the claims concerning the existing algorithm in the literature should be more clearly supported and demonstrated in the experimental section.


review_title: This paper shows some promising results, but the claims concerning the existing algorithm in the literature should be more clearly supported and demonstrated in the experimental section.

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
This paper tackles the problem of facilitating RL agents' learning in sparse reward, hard-exploration problems. The authors approached this challenge by generating a curriculum of tasks needed to finish the originally assigned task. Though using other auxiliary tasks to assist RL training has been heatedly discussed, their method has its own highlights and novelty. Pros and Cons are listed as follows:

[+] The overall framework is illustrated with clear explanations. The formulation of adversarial learning over the unrolling trajectories $\tau$ is the key for connecting the task generator and discriminator in this case and is interesting. The implicit formulation of task generation provided the prerequisite for solving out-of-scope tasks (when only $\mathcal{S}$ and $\mathcal{A}$ are shared).

[+] The experiments are well-designed with illustrative figures and analyses. Through the analysis, we see the advantage of having such curriculums do benefit agents' learning compared to previous methods. The ablative study also gave an intuitive explanation of the balance between having more difficult tasks in the curriculum and agents' performance on the original task.

[-] Although the general idea of generating easier tasks to facilitate agents' learning in complex tasks is intuitive, how the authors  justify their method has the same property is not fully justified in my mind. As far as I'm understanding, the authors used a hyperparameter $\delta$ to avoid making task generator $G$ moves too fast towards the target task, however, as was mentioned in Appendix B.2, the selection of the KKT multiplier is not critical, which causes some confusion on how the step-by-step difficulty enhancement was achieved. This result discussed is also having me question what if the task and reward scale is completely different, then would the effect of alternating $\beta$ still be small?

[-] The second problem is perhaps with the starting states in this training process. As the generator and discriminator was trained to adaptively fit the current RL agent's performance, the initiation of task generation should be important as a completely different setting of environment even with same state space and action space should be harmful for RL training in my mind(since there might be conflicts in $D_g$ and $D_{target}$ ). Then in this case, for example in a simpler scenario,  will this be a problem that affects RL training? Do we need to add constraints to the task generator at the beginning? This further leads to the question of whether the performance boost was provided by sampling on two similar environments (or even same as $G$ moves fast to the assigned task) and leading to more trajectories for RL training, or was the performance boost provided by the curriculum mentioned.

Because of the aforementioned questions, a borderline decision was made for now, please check the questions in the [-] to clear any misunderstandings.

=====================================================================================================

The authors addressed my concerns in the discussion period and I therefore created my score to 6.
review_title: This paper proposes a training framework to faciliate reinforcement learning in hard-exploration problems. The authors adopted ideas in curriculum learning and used a task generator to adaptively generate tasks that faciliate RL agent's learning. An adversarial learning frame work was used to train both the task generator and a discriminator which estimates differences between generated tasks and real target tasks. Their method show superior performance in several RL training scenarios.

Meta-review: I thank the authors for their submission and very active participation in the author response period. The paper is well written [R3,R4], tackles a hard problem [R4] in a novel way [R4] with interesting and convincing results [R2]. R3 noted that an empirical comparison to POET would be appropriate. However, in my view the authors addressed these concerns in a satisfactory manner. It seems that R3 has not updated their assessment nor confirmed their current score based on the author response. I am therefore discounting the only review voting for rejection and am siding with R1, R2 and R4. Thus, I recommend acceptance.
Decision: Accept (Poster)