Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes ZONE, a computational framework of curriculum task generation for reinforcement learning based on a concept from developmental psychology, i.e., zone of proximal development (ZPD). The target of the framework is to measure and schedule difficulty and progression of a student model. The approach is composed of two techniques, REJECT and GRAD. REJECT is to replace too easy or too hard tasks generated by teacher with proper tasks. GRAD is to maximize the norm of student model’s gradient, which reflects student progression. To evaluate the effectiveness, the authors conduct experiments on two baselines and their corresponding settings.
strength_and_weaknesses: Strengths:
1.	The paper is well written and easy to understand.
2.	The motivation of the method is intuitive and backed by psychology.
Weaknesses:
1.	The transition from psychological concepts to computational metrics seems not that natural, which causes some problems. For instance, the difficulty needs manual design based on different RL models, i.e., a fixed range for Goal GAN and a changeable range for PAIRED. Besides, the GRAD technique cannot be applied when the teacher is not trained with rewards.
2.	The number of baselines is a bit small, which degrades its universality and generality.
3.	The analysis of the experiment is not convincing. Figure 5a shows that ZONE generates more complex tasks, instead of better tasks matching the learning situation of a student. The more obstacle-filled or hallways-filled tasks are not necessarily better. Figure 5c shows that the gradient norm of student model without GRAD is larger than that with GRAD in the later stage of training, which seems weird and strange.

clarity,_quality,_novelty_and_reproducibility: The paper is good in clarity and novelty, and seems easy to reproducibility. However, the experiment part is not sufficient and there is still room for improvement in experimental analysis. 
summary_of_the_review: To summarize, the paper is good except for the experiments part. The motivation is interesting, based on developmental psychology. The method is well described and easy to understand. However, the experimental results are not so ideal and the analysis is not so convincing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to improve a known application of curriculum generation with two new techniques. One is to select tasks according to the difficulties to the student. The other one is to reward the teacher according to the student’s gradient norm such that the teacher can generate tasks which enhance the learner more. The authors add these two techniques to existing algorithms, PAIRED and Goal GAN and experimentally test the performances with these two techniques.

strength_and_weaknesses: The paper develops an analytical objective for task generation in Equation (1)-(5), clearly defining the problem. Also, the two techniques are well described, and the proposed algorithm is easy to follow and easy to apply. The experiments are well designed, including comparisons with two representative baselines and analysis of generated tasks.

However, there are still some improvements that could be made to the paper:

(1) This paper does not bring in enough novelty. The problem is controlling the difficulty of tasks in the zone of proximal development is already considered in the two baselines. There is no new approach to control the difficulty proposed. The first proposed technique, REJECT, is based on the developed methodologies of the baselines and is already developed in the algorithm POET, proposed by Wang et al. (2019). POET already proposes to reject or delete new environments that are too hard or too easy for the agent.

(2) The motivation of the approach needs more clarification. Neither of the two proposed techniques is proven to maximize the derived objective. Meanwhile, the paper is inspired by generating tasks in the zone of proximal development, which is not too hard nor too easy for the student. However, maximizing the student's gradient norm is unrelated to this main idea of generating ZPD tasks. 

(3) More description of the setting is needed. It is unclear whether the paper focuses on a generalization objective to maximize expected returns among a given environment distribution or what.

(4) Modelling the probability in the objective in section 3.3 on page 5 is unclear and technically incorrect. The paper claims that the first technique, REJECT, models the probability $p(r_{curr}|c_{curr},\theta_{curr})$ but how? Furthermore, this probability is decided by the dynamics of the MDP and cannot be influenced by the teacher. This modelling claim is technically incorrect.

(5) The test on the gradient maximization idea is limited. Some ablation studies to test the gradient maximization idea alone would be interesting, for example, whether gradient maximization is better than regret maximization proposed by PAIRED and difficulty control by Goal GAN.

(6) Some descriptions of experiments are missing. For example, how to set the gradient norm and returns scale when applying the gradient maximization technique? what contexts are all algorithms tested on for tasks (a)-(g) in Figure 2 on page 7? How the rejection rate is computed in Figure 5 on page 9? 

(7) I have a few questions about the experimental results:

(a) We see improvements in PAIRED-based algorithms. But PAIRED generates tasks which maximize the current regret. So why are there so many tasks given by the PAIRED rejected? Does PAIRED generate many negative regret tasks?

(b) It is shown in Table 1 on page 9 that PAIRED+REJECT+GRAD gives hard tasks from the beginning. Does it break the increasing difficulty set by the curriculum learning? Is it good?

(c) REJECT does not improve Goal GAN much. Can we say PAIRED is not controlling the difficulty of tasks well as claimed, but Goal GAN already fulfills the job?

(d) Why do PAIRED-based algorithms seem to have a larger variance in the acceptance rate?

clarity,_quality,_novelty_and_reproducibility: The paper lacks enough novelty and needs more clarification on the motivation. Meanwhile, more clarifications on the setting, modelling and experiments are needed.
summary_of_the_review: This paper is suggested to be rejected because (1) the algorithm does not propose new measurements on if a task is in the zone of proximal development and does not develop novel techniques to solve the problem, and (2) the paper needs more clarification.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors study to problem of finding and using the right balance in curriculum learning applied to reinforcement learning (RL). More specifically, at each training iteration of a student network, the proposed method aims to generate an appropriate, i.e. not too easy, not too hard, task for the student to learn. The goal is to achieve faster convergence and better performance. The approach is applied on top of two existing methods and tested on two benchmarks.
strength_and_weaknesses: Strengths:
- The paper is well written.

Weaknesses:
- Experimental results are mixed (sometimes better, sometimes worse). Thus, the benefit of the proposed method is not clearly shown.
- The title should be clearly tied to reinforcement learning. The title makes it look like the idea is more generic, while being tested only on RL. If the authors prefer to keep the title as is, they should demonstrate applicability in other areas, e.g. classification.
- There are many competing curriculum learning approaches for RL, see [A], missed by the authors in their discussion.

[A] Soviany et al., "Curriculum learning: A survey", International Journal of Computer Vision, 2022.

clarity,_quality,_novelty_and_reproducibility: The paper is clear, but I am not confident about the novelty of the proposed method.
summary_of_the_review: Due to the mixed results and the missed related works, I am not confident about accepting the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies curriculum learning in reinforcement learning. To achieve this, the paper formulates a teacher-student framework, where the teacher is responsible to generate the tasks for the student to learn. The key idea is two-folded. First, the student should not learn from the tasks that are outside of the difficulty scope. Second, the teacher should be rewarded for generating tasks that cause larger parameters update of the policy/value networks of the student. To implement this idea, the paper proposes REJECT, which simply rejects the tasks that are too easy or too hard based on the rollout reward, and GRAD, which makes the amount of the parameters update as one of the rewards for the teacher. Experiments show that the two strategies can improve efficiency.
strength_and_weaknesses: Strengths:

1. The proposed ideas are simple and intuitive.
2. The idea of using the amount of student model updates to guide the teacher is interesting. While there are some similar ideas in curriculum learning, it is nice to see it is effective in RL.

Weaknesses:

1. It is unclear to me why REJECT helps. It simply drops the tasks that are too difficult or easy. However, it still requires generating data with the student model. So it is still expensive. I am also curious whether such dropped samples are counted in the learning curve.
2. The improvement over the baselines is not significant. It is only comparable with goal GAN.
3. While the focus of the paper is curriculum learning, the paper should consider baselines that use intrinsic rewards in the experiment. The selected environments are mainly hard-exploration environments, so intrinsic reward methods (such as count-based intrinsic reward) could also perform well in these environments. So I would like to see a comparison.
4. Various exploration methods (such as intrinsic rewards) are not discussed in the related work. They try to tackle similar challenges in different ways.
clarity,_quality,_novelty_and_reproducibility: Yes
summary_of_the_review: This paper presents some interesting ideas. However, I have concerns about the REJECT. Particularly, it is unclear to me whether the dropped samples are counted when drawing the learning curve. I would like to ask the authors to clarify. If not counted, the comparison could be unfair. The experiments also need improvement.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors proposed a framework for curriculum task generation in reinforcement learning called Zone, where a “teacher” network generates tasks to enable “student” learning. The framework adds two techniques to existing algorithms. The REJECT technique removes tasks based on the learner's difficulty, and the GRAD technique rewards the teacher according to the student’s gradient norm such that the teacher can generate tasks that enhance the learner more.

Reviewers appreciated the work for being easy to understand and intuitive. However, they pointed out the issues of weak experimental results and the lack of clarity on the novelty and the motivation of the contribution.

I met with the reviewers to discuss these issues. Based on the discussion, we agreed that the paper has the following significant deficiencies that remain unresolved:
- The lack of performance improvement compared to goal-gan as well as the lack of comparisons with highly relevant baselines such as amigo indicates further empirical investigation and room for improvement.
- Although the work focuses on difficulty control, GRAD as a mechanism for accelerating learning is proposed as a significant contribution. The connection between them is unclear. 
- Unification is mentioned as a novelty and contribution, but the scope of unification remains specific to a narrow class of mechanisms. 
- These last two points indicate that the main message about the motivation and contribution of the work could not be fully appreciated, which can be addressed by focusing on these parts for the next version of the paper.

Addressing these deficiencies requires a significant revision of the current draft. Still, we believe that a resubmission that carefully addresses these concerns will make this paper much stronger and above a borderline paper.

justification_for_why_not_higher_score: The paper had the following significant deficiencies: lack of clarity in terms of contribution and motivation as well as clear performance advantage and relevant baselines.
Decision: Reject