Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an automatic curriculum learning framework for multi-objective curricula (MOC). The key insight is to use a multitask hyper-net module on top of multiple curricula generating neural networks. The hyper-net module resolves potential conflicting gradients between the lower level modules and parameterizes them accordingly. Each of these atomic module represents a single way of introducing curricula in RL: (i) subgoal sequence, (ii) initial states and (iii) reward function. Additionally, the external memory of the hyper-net is supposed to generate an “abstract“ curricula that is different from the predefined curricula above. The parameters of the abstract curriculum are added to the value function.


strengths_and_weaknesses: # Strengths
- The idea to learn individual curricula modules from scratch is interesting. It seems clever to learn distinct curricula and how to combine them at the same time to achieve good overall performance.
- Evaluations: MOC has been tested against state-of-the-art automatic curriculum learning approaches and seems to outperform them. Moreover, it is much appreciated that the authors perform a detailed ablation study.

# Weaknesses
- The paper is at parts hard to understand. For instance, it is not clear what the pros and cons of read and write-only memory are (discussion at the end of the introduction) at this part of the paper. The authors describe how their approach compares to previous work, but leave the reader with no intuition why restricting memory access is important at all. Another example is the last sentence of the automatic curriculum learning section, where it is not clear what the “history“ is.
- What exactly is this paper doing for the first time? Is it the introduction of the abstract curriculum module, or the combination of different types of curricula and their parallel tuning? The empirical evaluations are definitely also a contribution. A paragraph (that goes beyond the current contributions paragraph) saying explicitly how this paper differs from others would be helpful.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Despite that the paper is at parts not clearly written, the techniques introduced and the evaluations seem convincing. Also, this paper tackles an interesting and important research direction with potentially high impact.
issues: The paper seems to be written in a rush and needs to be revised to improve clarity and flow of reading. Moreover, the contributions need to be explicitly formulated.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a learning architecture that guides the learning process of an agent by simultaneously: 1) selecting an initial state from the agent starts the next episode; 2) selecting the agent's goal; and 3) selecting the reward function. Experiments on simulated robot environments (reaching, pick and place, and stacking) show some improvements relative to a few select baseline approaches. 
strengths_and_weaknesses: 
Strengths:

The approach is novel and the idea of learning how to initialize an episode to guide the agent's learning is creative. 

Ablation study is informative

Weaknesses:

There are a few clarity issues with the paper's writing and presentations -- see detailed comments under 'issues'

There needs to be better justification for why these particular baselines make sense -- see questions in detailed comments


robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The recommendation is based on the strengths and weaknesses outlined in the review, as well as the list of issues. The recommendation is subject to change depending on the authors' responses. 
issues: 

Questions for authors:

For the inputs to the Hyper-RNN, can you elaborate more on why you only consider the final state of the last episode?

Question on the 3 environments: these problems appear difficult mostly because the environmental reward function is sparse, in other words, it is engineered very poorly. Have you considered settings where some thought was put into the reward function to make the task easier to learn? Also, when each new episode is generated, what is randomized (e.g., positions of objects) and what remains the constant (e.g., the goal)?

In Figure 2, are the episodes likely to have more or less the same number of time steps? is it possible that some approaches result in episodes that are much shorter or longer? it may be good to plot the performance in terms of number of time steps as opposed to number of episodes as ultimately it is the interaction with the environment that we want to minimize

What were the fundamental limitations that prevent the baseline approaches from learning to complete the task all the time? theoretically, they should be able to do so given enough data but it looks like they've converged before hitting high success rates. Also, are the baselines essentially learning from scratch without a curricula? Why not have a simply DNN baseline that just tries to learn from scratch

In the limitations, cam you elaborate on what you mean by "elongates the training efficiency"?


Comments on parts of the paper:

In the last sentence of the 1st paragraph of introduction, more broadly, curriculum learning can also be used to modify the world itself, e.g., add new entities to the world, change the behaviors of other agents, etc., see:

Narvekar, Sanmit, et al. "Source task creation for curriculum learning." Proceedings of the 2016 international conference on autonomous agents & multiagent systems. 2016.

Somewhere in the introduction, you should elaborate what you mean by output a curriculum? how is curriculum represented? Sometimes a curriculum is a sequence of tasks, sometimes a graph of tasks, sometimes it is just an ordering of data points. 

A running illustrative example would really help is space permits

In preliminaries, the RL section should include the notion of goals since goals are used in this work

In the second paragraph of Section 2, "the output of D is a curriculum such as an initial state.", that's actually false. An episode with some particular initial state is part of a bigger curriculum, yes, but it is not in itself the curriculum. The curriculum is instead, the sequence of generated episodes, each with different initial state, goal, etc. 

In Figure 3, shouldn't all methods do the same at the very beginning with no training?
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes an architecture and learning pipeline to generate multi-objective curricula.

The basic ideas are:
1. Using a hypernet to generate the weights of models that further generate each manually designed curricula. This includes learnable initial states, subgoals, and shaped reward functions.
2. To enable futher flexibility, the authors propose to use an augmented memory to support the learning agent, which can be written by the hypernetwork. So this is a general form of "curricula" besides the aforementioned manually designed curricula.
3. Frame ACL as a bilevel optimization problem and optimize hypernet and $Q$ and $\pi$ jointly.
strengths_and_weaknesses: Strengths:
1. I think this paper proposes a novel problem (multi-objective curricula), a general perspective (bilevel optimization for ACL), and a novel architecture (hypernet with write-only memory as a general curricula).
2. The main ideas are straightforward and reasonable.
3. Experiment results are convincing. Particularly, it has been shown that the combination of hypernet and augmented memory is necessary.

Weakness:
I find this work very easy to follow and no significant weakness from my opinion.



robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think there are several novel points this paper brings:
1. A bilevel optimization (BO) view of ACL, though not completely novel, but to my knowledge this is the first paper explicitly using BO to train ACL.
2. Multi-objective curriculum via hypernetwork is an interesting proposal, which automatically connects all curricula.
3. The augmented memory is a novel form of "curricula" that is very flexible.
issues: 1. I do not see an obvious reason why using $\gamma$ for goal and not for the discount factor.
2. Line 145, For the shaping reward, why learning $\mu$ and not fixing it to the discount factor $\lambda$ (in your notation). If it is not $\lambda$, then in theory it cannot guarantee optimality anymore.
3. From the presentation perspective, I think the authors can explicitly list their contributions at the end of introduction using bullet points, which can potentially help others to quickly get the novelties proposed in this work.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a method to generate manually designed curricula as well as automatically generated abstract curricula for policy learning in the reinforcement learning setting. These curricula influence the data collection of the agent and thus can gradually help the agent to solve the task. By this, the agent's sample efficiency improves and it is able to solve the task with qualitative policies. The proposed curricula are coordinated via a multi-task hyper-net which parameterizes the curriculum base networks. The method is benchmarked against other automatic curriculum learning approaches on six robotic manipulation tasks. 
strengths_and_weaknesses: The paper is well written such that the reader can easily follow the intuition and the ideas. It shows a thorough experiments part including a thorough ablation study. 

Even though some related work was mentioned in the main text, I was missing an explicitly related work part while reading. In my opinion, summarizing the related work in a chapter and explicitly mentioning the differences between the proposed method to those is useful for the reader to be able to have an idea of the novelty. There are also a couple of other works which I could find on the internet which consider automatically adjusting their curriculum in the context of reinforcement learning and might be worth a discussion:
- https://proceedings.neurips.cc/paper/2020/file/68a9750337a418a86fe06c1991a1d64c-Paper.pdf
- http://proceedings.mlr.press/v100/klink20a/klink20a.pdf
- https://proceedings.mlr.press/v164/celik22a.html
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I like the proposed idea in this paper and I believe it is well written such that the reader can easily understand the purpose of the proposed method. The experiments show the advantages gained through the method. I would have appreciated an explicit related work part. However, I am willing to accept this paper. 
issues: My main concerns were already described in the 'Strengths and Weaknesses' section. I would be happy if the issues would be considered during the rebuttal.
review_title: null

Meta-review: metareview: 
# Strengths
- The proposed approach is novel and the idea of combining individual curricula focused on different aspects such as initial states or reward functions is interesting and promising
- The evaluation of the approach was well done and included comparisons to SOTA methods and detailed ablations

# Weaknesses
- The paper requires some additional clarifications as pointed out by the reviewers
- An dedicated related work section would further improve clarity
- The reviewers highlighted some additional related work that should be addressed in the paper.

# Post-Rebuttal Update
The additional related work section and the added clarifications definitely improve the presentation of the work and help to position the paper amongst prior work.
While additional baselines and more complex experiments/scenarios, ideally in real-world settings, would further improve the paper, I do believe the paper should be published.
I encourage the reviewer to polish the text once more. There are a few spelling/grammar mistakes, e.g., in the experiment section.
Furthermore, the paper should be shortened back to 8 pages for the camera-ready version.

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)