Learning Multi-Objective Curricula for Robotic Policy Learning
Authors: Jikun Kang
Miao Liu
Abhinav Gupta
Christopher Pal
Xue Liu
Jie Fu
Venue: CoRL 2022 Poster (2022)
Keywords: ACL
Hyper-net
Multi-objective Curricula

Abstract: Various automatic curriculum learning (ACL) methods have been proposed to improve the sample efficiency and final performance of robots' policies learning. They are designed to control how a robotic agent collects data, which is inspired by how humans gradually adapt their learning processes to their capabilities. In this paper, we propose a unified automatic curriculum learning framework to create multi-objective but coherent curricula that are generated by a set of parametric curriculum modules. Each curriculum module is instantiated as a neural network and is responsible for generating a particular curriculum. In order to coordinate those potentially conflicting modules in unified parameter space, we propose a multi-task hyper-net learning framework that uses a single hyper-net to parameterize all those curriculum modules. We evaluate our method on a series of robotic manipulation tasks and demonstrate its superiority over other state-of-the-art ACL methods in terms of sample efficiency and final performance.

Learning
Multi-Objective Curricula for Robotic Policy Learning
Jikun Kang McGill University
Miao Liu IBM Research Miao Liu1@ibm com
Abhinav Gupta Université de Montréal Mila Québec Al Institute abhinavgenyu, edu
Mila Québec AI Institute
j jikun. kang@mail waangmmail.mcgill.com
Christopher Pal Polytechnique Montréal Mila Quéhec AI Institute christopher, htt.stopphrr.plllpllymt.........
Xue Liu
Jie Fu"
McGill University
Beijing Academy of Al fuj merjie@bai.ccom .ac........ cn
Mila Québec AI Institute xueliu@cs www..lc.eele....
Various automatic curriculum learning (ACL) methods have been pro- posed improve the sample efficiency and final performance of robots' policies learning. They are designed to control how robotic agent collects data, which is inspired by how humans gradually adapt their learning processes to their capabili- ties. In this paper, propose a unified automatic curriculum learning framework to create multi multi-objective but coherent curricula that are generated by set of para- metric curriculum modules. Each curriculum module instantiated as a neural network and is responsible for generating a particular curriculum. In order to coordinate those potentially conflicting modules a unified parameter space, we propose multi-task hyper-net learning framework that uses a single hyper-net parameterize all those curriculum modules. We evaluate robotic manipulation tasks and demonstrate its superiority over other state-of-the- art ACL methods in terms of sample efficiency and final performance. Our code is available at https: h//ggttttbbbee// com
Abstract:
method
series of
Keywords: ACL, Hyper-net, Multi-objective Curricula
Introduction
The concept that humans frequently organize their learning into curriculum of interdependent processes according to their capabilities was first introduced to machine learning in [1]. Over time, curriculum learning has become more widely used in machine learning to control the stream of examples provided to training algorithms [2], to adapt model capacity [3], and to organize exploration [4]. Automatic curriculum learning (ACL)) for deep reinforcement learning (DRL. [5] has recently emerged promising tool to learn how adapt a robot's learning tasks to its capabilities during training. ACL can be applied to robotics' policies learning in various ways, including adapting initial states [6]. shaping reward functions [7]. generating goals [8]. More broadly, curriculum learning can also be used to modify the environment itself. For example, it can add new entities to the world or change the behaviors of other agents [9].
Oftentimes, only single ACI. paradigm (e.g., generating subgoals) considered. It remains an open question whether diffferent paradigms are complementary each other and if yes, how to combine them in more effective manner similar to how the "rainbow" approach of [10] has greatly improved DRL performance in Atari games. Multi-task learning is notoriously difficult, and Yu et al. [11] hypothesize that the optimization difficulties might be due to the gradients from different tasks
Cooresponding Author
6th Conference Robot Learning (CoRL. 2022), Auckland, New Zealand.

---

conflicting with each other thus hurting the learning process. In this work, we propose a multi-task bilevel learning framework for more effective multi-objective curricula robotics' policy learning. Concretely, inspired by neural modular systems [12] and multi-task RL [13], we utilize set of neural modules and train each of them to output sequence of tasks with different desired goals, rewards, initial states, etc. To coordinate potentially conflicting gradients from modules in a unified parameter space, we use single hyper-net [14] to parameterize neural modules so that these modules generate a diverse and cooperative set of curricula. Multi-task learning provides a natural curriculum for the hyper-net iself since learning easier curriculum modules can be beneficial for learning more difficult curriculum modules with parameters generated by the hyper-net.
Furthermore, existing ACL methods usually rely on manually-designed paradigms of which the target and mechanism have be clearly defined and it is therefore challenging create a very diverse set of curriculum paradigms. Consider goal-based ACL for example, where the algorithm is tasked with learning how to rank goals to form the curriculum [15]. Many of these curriculum paradigms are based on simple intuitions that are inspired by learning in humans, but they usually take too simple forms (e.g-, generating subgoals) to apply to neural models. Instead, we propose to augment the hand-designed curricula introduced above with an abstract curriculum of which paradigm is learned from scratch. More concretely, we take the idea from the hyper-net with a non-parametrin memory module, which is also directly connected to the DRL agent. The hyper-net can write entries to and update items in the memory, through which the DRL agent can interact with the environment under the guidance of the abstract curriculum maintained in the memory. The write-only permission given to the hyper-net over the memory is distinct from the common use of memory modules in meta-DRL literature, where the memories are both readable and writable. We point out that the hyper- net is instantiated as a recurrent neural network [17] which has its internal memory mechanism and thus a write-only extra memory module is enough. Another key perspective is that such a write-only memory module suffices to capture the essence of many curriculum paradigms. For instance, the subgoal-based curriculum can take the form of a sequence of coordinates in game which can be casily generated as hyper-net and stored in the memory module. The contributions of this work are as follows:
meta-DRL. [16] and equip
1. We introduce multi-objective curricula learning approach for improving the sample efficiency of solving challenging deep reinforcement learning tasks, which is an important problem that has not been adequately addressed before.
2. We further propose unified automatic curriculum learning framework to create multi- objective but coherent curricula that are generated by a set of parametric curriculum modules. Each curriculum module is instantiated as a neural network and is responsible for generating a particular curriculum. To coordinate those potentially conflicting modules in unified parameter space, we propose a multi-task hyper-net learning framework that uses a single hyper-net parameterize all those curriculum modules.
2 Related Work
Curriculum learning. Automatic curriculum learning (ACL) for deep reinforcement learning (DRL) [18, 19, 20, 21] has recently emerged as a promising tool to learn how to adapt an agent's s learning tasks based on its capacity during training. ACL [22] can be applied to DRL in variety of ways, including adapting initial states [6, 23], shaping reward functions [24, 25], or generating goals
[8, 15, 26, 27]. In closely related work [28], a series of related environments of increasing difficulty have been ereated to form curricula. There are other works related to curriculum reinforcement learning (CRL).
Multi-task and neural modules. Learning with multiple objectives is shown to be beneficial in DRL tasks [29, 30, 31, 32]. Sharing parameters across tasks [33, 34, 35) usually results in conflicting gradients from different tasks. One way to mitigate this is to explicitly model the similarity between gradients obtained from different tasks [11, 36, 37, 38, 39, 40, 41]. On the other hand, researchers propose to utilize different modules for different tasks, thus reducing the interference of gradients

---

from different tasks [42, 43, 44, 45, 46, 47, 48]. Most of these methods rely on pre-defined modules that make them not flexible in practice. One exception is [12], which utilizes soft combinations of neural modules for multi-task robotics manipulation. However, there is still redundancy in the modules in [12], and those modules cannot be modified during inference. Instead, we use a hyper-net dynamically update complementary modules on the fly conditional on the environments.
Memory-augmented meta DRL. Our approach is also related to episodic memory based meta DRL [49, 16, 50, 51]. Different from memory-augmennt meta DRL methods, the DRL agent in our case not allowed to modify the memory. This design prevents the agent from writing its information into the memory module, which can interfere with the generated curriculum. Note that is straightforward augment the DRL agent with both readable and writable neural memory just like [16, 49], which is different from read-only memory module designed for ACL..
Dynamic neural networks. Dynamic neural networks [52] can change their structures parameters based on different environments. Dynamic filter networks [53] and hyper-nets [14] can both generate parameters.
Our proposed framework generalizes previous work by unifying the aforementioned key concepts with focus automatically learning multi-objective curricula from scratch for DRL-based robot learning.
3 Preliminaries
Reinforcement learning (RL) is used to train agent policy with the goal of maximizing the (discounted) cumulative rewards through trial and error. A basic RL setting is modelled as a Markov decision process (MDP) with the following elements: S is the set of environment states. In this paper, goal g E 9 corresponds to a set of states S9 C S. The goal is considered to be achieved when the agent function, where 8(s*+1 s² a²) maps states time & R is the immediate reward after transition from to s' x(: (x) is the policy function parameterized by @n. and π ( s; Ox) denotes the probability of choosing action given an observation 8.
any state 8 E 59; A is the set of actions; is the state transition probability
paaeeactionn pair at time-step
probability distribution
Automatic curriculum learning (ACL) a learning paradigm where an agent is trained iteratively following a curriculum to ease learning and exploration in multi-task problem. Since it is not feasible manually design curriculum for each task, recent work has proposed create implicit curriculum directly from the task objective. Concretely, it aims to maximize metric P computed over a set of target tasks T Trarget after some episodes t'. Following the notation in [5]. the objective set to: maxp Sr~Trarge Pf dT, where D H - Tearget a task selection function. The input H. can be consist of any information about past interactions. For example, in our experiments, the history consists of the last state of the episode. and the output of D a sequence of generated episodes, which contain different desired goals, initial states, rewards, etc.
Hyper-networks were proposed in [14] where one network (hyper-net) is used to generate the weights of another network. All the parameters of both networks are trained end-to-end using backpropagation. We follow the notation in [54] and suppose that we aim to model a target function y : X' x I + R, where E X is independent of the task and / E I depends on the task. A base neural network fo(:; (I; f2))) can seen as a composite function, where fo X' - and fn I +,, Conditioned on the task information T, the small hyper-net (T; 02) generates the parameters 9 of base-net ft. Note that 0, is never updated using loss gradients directly.
Learning Multi-Objective Curricula
We use single hyper-net dynamically parameterize all the curriculum modules time and modify the memory module shared with the DRL agent. We call this framework a Multi Objeet-Oeecctive
3

---

Curricula (MOC). This novel design encourages different curriculum modules to merge and exchange information through the shared hyper-net.
Following the design of hyper-networks with recurrence [14], this hyper-net is instantiated as a recurrent neural network (RNN), which we refer as the Hyper-RNN, denoted (I; 0n), in the rest of this paper emphasize its dynamic nature. Additionally, the Hyper-RNN can be viewed as a configurator for other modules as suggested in [55]. Our motivation for the adoption of an RNN design is its capability for producing distinct set of curricula for every episode, which strikes better trade-off between the number of model parameters and its expressiveness. On the other hand, each manually designed curriculum module is also instantiated as an RNN, which is referred as Base-RNN fa 06) parameterized by 0g (I; 0.). Each Base-R -N-N is responsible for producing specific curriculum, e-g., series of sub-goals.
Algorithm 1: Multi-Ohjective Curricula
Curricula
Generation
Hyper-RNN Jn
Memory
Deep Reinforcement Learning (MOC-DRL).
for Episode e' in to Touter do
Base-RNN
Base-RNN
Base-RNN in
2
Sample new environment from the distribution of environments;
Cabs
Subgoal Sequences
Initial States
Reward Function
Hyper-RNN generates parameters for each curriculum module;
for Base- RNN in 3 do
Fpont
5
-Generate a curriculum;
Curricats
Action
Hyper-RNN updates the abstract
Kavironments
Distribution
Observation
@ (()d))
DRI Agent
curriculum in the memory;
Traiining step in to Timner do
8
DRL agent reads memory;
Figure Illustration of MOC-DRL with two loops. Curricula generation corresponds to the outer-level loop. The DRL agent interacts with the environment in the inner-level loop.
Train DRL agent following curricula;
Update Hyper-RNe RNN based on outer-level objective;
The architecture of MOC-DRL is depicted in Fig. 1, and its corresponding pseudo-code is given in Alg. 1. We formulate the training procedure as a bilevel optimization problem [56] where we minimize an outer-level objective that depends on the solution of the inner-level tasks.
In our case, the outer-level optimization comes from the curriculum generation loop where each step an episode denoted as t'. On the other hand, the inner-level optimization involves common DRL agent training loop on the interactions between the environment and the DRL. agent, where each time-step at this level is denoted as t. We defer the discussion on the details to Sec. 4.3.
Inputs, 1, the Hyper-RNN, Shs consist of: (1) the final state of the last episode, and (2) role identifier for each curriculum module (e.g., for initial states generation) represented one-hot encoding. Ideally, expect each Base-RNN to have its own particular role, which is specific to each curriculum. When generating the parameters for each Base-RNN, we additionally feed the role identifier representation to the Hyper-RNN.
Outputs of the Hyper-RNN at episode include: (1) parameters of for each Base-RNN, and (2) the abstract curriculum, hy maintained in the memory module. Here hy corresponds to the hidden states of the Hyper-RNN such that 10 h fa(1%;;,) (,))
4.1 Manually Designed Curricula
In Sec. 4.1, we describe the details of generating manually designed curricula while the process of updating the abstract curriculum is described in Sec. 4.2. We describe how to train them in Sec. 4.3.
In this work, we use three curriculum modules responsible for generating pre-defined curricula [5]: initial state generator, sub-goal state generator, and reward shaping generator. Our approach can be easily extended to include other forms of curricula (e.g.. selecting environments from discrete set [57]) by adding another curriculum generator to the shared hyper-net. These Basc-RNNs

---

simultaneously output the actual curricula for the DRL agent in a synergistic manner. It should be noted that these Base-RNNs are not directly updated by loss gradients, as their pseudo-parameters are generated by the Hyper-RNN.
Generating subgoal state g as curriculum Cgost with Base-RNN f.. As one popular choice in ACL for DRL, the subgoals can be selected from discrete sets [8] or continuous goal space [15]. A suitable subgoal state g can ease the learning procedures by guiding the agent on how to achieve subgoals step by step and ultimately solving the final task.
To incorporate the subgoal state in the overall computation graph, in this paper, we adopt the idea from universal value functions [58] and modify the action-value function, Q(=: dg), combine the generated subgoal state with other information Q == Q(s", a', g"; Q,) Q(s', a', Ggont; Qg), where sf is the state, a' is the action, and g" is the generated subgoal state. The loss is defined as Tgoal = Ba rr , at Hous [[Q(s' sissss,e,e, a', Cgoat; 0g) - 1) where ý is the one-step look-ahead:
ý= + A [[(("" ((ss",c a', Cyoati Qg) - log(x(a'*1) |s*t1, dx))),
(1)
Houf is the replay buffer and À the discount factor.
Generating initial state So as curriculum Cinit with Base-RNN Jh. Intuitively, if the starting state So for the agent is close to the end-goal state, the training would become easier, which forms natural curriculum for training tasks whose difficulty depends initial state and the end-goal state. This method has been shown effective in control tasks with sparse rewards [6, 59]. To simplify implementation, even though we only need a single initial state So which is independent of time, we still use a Base-RNN,Ji, Fo, to output it.
proper distance between the
The loss for this module is: Jnnit = Est a ) [[Q(s', a', Ciniti Qq) - =)) where ÿ is defined in Eqn. 1.
Generating potential-based shaping function as curriculum Crew with Base-RNN J Motivated by the success of using reward shaping for scaling RL. methods to handle complex domains [60], we introduce reward shaping as the third manually selected curriculum. The reward shaping function can take the form of: f' (s*, at, 1++) μ ff f1( - f (s*), where μ hyper-parameter and f. is base-RNN that maps the current state with reward. In this paper, add the shaping reward J' at, 1 to the original environment reward r. We further normalize the shaping reward between O and to deal with wide ranges.
Following the optimal policy invariant theorem [60], we modify the look-ahead function: 7 r' + If (s', a², s+11 + +AE (o a', Crew! Qg) log(a |stt1 (n))). Thus the loss is defined Treword = " a
aatt Hoor [[Q(s", al, Crews 23) T7
4.2 Abstract Curriculum with Memory Mechanism
Although the aforementioned hand-designed curricula are generic enough to be applied in any environmemt/task, it is still limited by the number of such predefined curricula. It is reasonable to conjecture that there exist other curriculum paradigms, which might be difficult to hand-design based on human intuition. As result, instead of solely asking the hyper-net to generate suman-engineeree curricula, we equip the hyper-nets with an external memory, in which the hyper-nets could read and update the memory's entries. The Hyper-RNN together with the memory can also be seen as an instantiation of shared workspaces [61] for different curricula modules, in which those modules exchange information.
By design, the content in the memory can serve as abstract curricula for the DRL agent, which is generated and adapted according to the task distribution and the agent's 's dynamic capacity during training. Even though there is no constraint on how exactly the hyper-net learns use the memory, observe that (see Sec. 5.2): 1) The hyper-net can receive reliable training signals from the manually designed curriculum learning objectives?; ; 2) Using the memory module alone would result in unstable
"To some extent, tasking the hyper-net to train manually designed curriculum modules can be seen as curriculum itself for training the abstract curriculum memory module.
5

---

training; 3) Utilizing both the memory and manual curricula achieves the best performance and stable training. Thus, training this memory module with other manually designed curriculum modules contributes to the shaping of the content that can be stored in the memory and is beneficial for overall performance.
Specifically, external memory is updated by the Hyper-RNN. To capture the latent curriculum information, we design neural memory mechanism similar to [62]. The form of memory is defined as matrix M. At each episode t', the Hyper-RNN emits two vectors m and m as [m² m 1 = [a, tanh]² (Wi, h where W the weight matrix of Hyper-RNN to transform its internal state hi, and [!-] denotes matrix transpose. Note that Wa are part of the Hyper-LSTM parameters b,.
The Hyper-RNN writes the abstract curriculum into the memory, and the DRL, agent can read the abstract curriculum information freely.
Reading. The DRL agent can read the abstract curriculum Cobs from the memory M. The read operation is defined as: ch = a M where a" E I represents an attention distribution over the set of entries of memory M Each scalar element a" in attention distribution a" can be calculated as: a sof tmax (cos cosineoeiiiiiiccooc (MM -1)) where we choose cosine(::-) as the align fuunction, M represents the k-th row memory vector, and m E R is add vector emitted by RNN.
The Hyper-RNN can write and update abstract curriculum in the memory module. The o m where m I corresponds to the extent to which the current contents in the memory should be deleted.
Updating.
write operation is performed as: M -M1-'(1- (1-a" +
Equipped with the above memory mechanism, the DRL. learning algorithm can read the mem- ory and utilize the retrieved information for policy learning. We incorporate the abstract cur- riculum into the value function by Q(s", a², g'', cabsi Øg). Similar to manually designed curric- ula, minimize the Bellman
and define the loss function for the abstract curriculum as: E as a0t1 Saha)Hown [(Q(s*, at, ca 0g) - =, where is defined in Eqn. 1.
Jabstract
4.3 Bilevel Training of Hyper-RNN
After introducing the manually designed curricula in Sec. 4.1 and the abstract curriculum in Sec. 4.2, here we describe how we update the Hyper-RNN's parameters 6%. the parameters associated with the DRL agent dy and dx. Since the Hyper-RNN's objective to serve the DRL agent, we naturally formulate this task bilevel problem [56] of optimizing the parameters associated with multi- objective curricula generation by nesting one inner-level loop in an outer-level training loop.
Outer-level training of Hyper-RNN. Specifically, the inner-level loop for the DRL agent learning and the outer-level loop for training Hyper-RNN with hyper-gradients. The outer-level loss is defined as :Touter Tsnstial + Tyoal + Treward + Jabs-
Since the manually designed curricula and abstract curricula all defined in terms of Q-function, for the implementation simplicity, we combine them together Jouter Egt,a' a Httt [[Q(s', a', Cgoal: Crew, Cinit, Cabsi ( ij) ") Following the formulation and implementation in [56], we obtain 0% argmin (0n: Touter (argmin (ø; J (0,, o)))).
Inner-level training of DRL agent. The parameters associated with the inner-level training, &y and on. can be updated based on any RL algorithm. In this paper, we use Proximal Policy Optimization (PPO) [63] which is popular policy gradient algorithm that learns stochastic policy.
5 Experiments
We evaluate and analyze our proposed MOC DRL on the CausalWorld [64], as this environment enables us to easily design and test different types of curricula in fine-grained manner. This environment also provides wrapper makes the environment execute actions on the real robot, which

---

can be used in sim2real experiments. It should be noted that we do not utilize any causal elements of the environment. It straightforward to apply our method to other DRI. environments without major modification. Moreover, the training and evaluation task distributions are handled by Causal World. Take task "Pushing" as an example: for each outer loop, we use CausalWorld to generate a task with randomly sampled new goal shapes from goal shape family
Mean episode reward
Reaching
Picking
Stacking
1000
-10
500
-10
-15
Episodes
1c7
Episodes
1e7
Episodes
Reaching
Picking
Stacking
Fractional Success
1.0
1.0
0.5
0.5
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1,0
1e7
Episades
1e7
Episades
1e7
MOC
GoalGAN
ALP GMM
Comparisons with state-of-the-ar ACL algorithms. Each learning curve is computed in three runs with different random seeds.
Figure 2:
5.1 Comparing MOC with state-of-the-art ACL methods
We compare our proposed approach with the other state-of-the-ard ACL methods: (1) GoalGAN [26]. which generative adversarial neural network (GAN) propose tasks for the agent to finish; (2) ALP-GMM [5], which models the agent absolute learning progress with Gaussian mixture models. None of these baselines utilize multiple curricula.
Fig. shows that MOC outperforms other ACL approaches in terms of mean episode reward, fractional success, and sample efficiency. Especially, MOC increases fractional success by up to 56.2% in all of three tasks, which illustrates the effectiveness of combining multiple curricula in synergistic manner.
5.2 Ablation Study
Our proposed MOC framework consists of three key parts: the Hyper-RNN trained with hyper- gradients, multi-objective curriculum modules, and the abstract memory module. To get better insight into MOC, conduct an in-depth ablation study probing these components. We first describe the MOC variants used in this section for comparison as follows: (1) MOC Base MOC has the Hyper-RNN RNN and the memory module but does not have the Base-RNNs for manually designed curricula. (2) MOC MOC has the Hyper- to generate three curriculum modules but does not have the memory module. (3) MOC Memory : MOC has Base-RNNs but does not have memory and Hyper-RNN components. It independently generates manually designed curricula. (4) MOCMemoryy Goult: MOC with Hyper-RNN and one Base-RNN, but without the memory module. It only generates the subgoal curriculum as pilot experiments show that it is consistently better than the other two manually designed curricula and is easier to analyze its behavior by visualizing the state visitation.
Ablations of Hyper-RNN. By comparing MOCM with MOC, Memory- -Hyperr as shown in Fig. 3, we can observe that letting a Hyper-RNN generate the parameters of different curriculum modules indeed helps in improving the sample efficiency and final performance. The advantage is more obvious in the harder tasks pick and place and stacking. The poor performance of
7

---

Mean episode reward
Reaching
Picking
-
1000
-10
500
-15
-20
0.2
0.4
0.6
0.8
1.0
0.4
0.6
1.0
Episodes
1e7
Episodes
Mean rpisode reward
Pick and Place
Stacking
10
-10
-10
0.2
0.4
0.6
0,8
1.0
0.2
0.4
0.6
0.8
1.0
Episodes
1e7
Episodes
1e7
MOCrummrr"
MOCMamory
MOC
VanillaDRL
MOCManary
MOCsase
Figure 3: Comparison of algorithms with and without memory component all four tasks. Each learning curve is obtained by three independent runs with different random seeds.
MOC Memory. yper- may be caused by the potential conflicts among the designed curricula. For example, without coordination between the initial state curriculum and the goal curriculum, the initial state generator may set an initial state close to the goal state, which is easy to achieve by agent but too trivial to provide useful training information to achieve the final goal. In sharp contrast, the Hyper-RNN can solve the potential conflicts from different curricula. All the curriculum modules are dynamically generated by the same hyper-net, and there exists an implicit information sharing between the initial state and the goal state curriculum generator. We put extra experimental results in Appendix Sec. A.3, Fig.
Ablations of the memory module. We aim provide empirical justification for the use of the memory module and its associated abstract curriculum. By comparing MOC with MOCMemore Memory as shown in Fig. 3, we can see that the memory module is crucial for MOC to improve sample efficiency and final performance. Noticeably, in pick and place and stacking, we see that MOC gains a significant improvement due to the incorporation of the abstract curriculum. We expect that the abstract curriculum could provide the agent with an extra implicit curriculum that is complementary to the manually designed curricula. We also find that is better for the Hyper-RNN to learn the abstract curriculum while generating other manually designed curricula. Learning multiple manually designed curricula provides natural curriculum for the Hyper-RNN RNN itself since learning easier curriculum modules can be beneficial for learning of more difficult curriculum modules with parameters generated by the Hyper-RNN.
Ablations of individual curricula. We now investigate how gradually adding more curricula affects the training of DRL agents. By comparing MOC MOCMemory .Goat and MOC MOCor as shown in Fig. 3, we observe that training an agent with single curriculum receives less environmental rewards as compared to the ones based on multiple curricula. This suggests that the set of generated curricula indeed helps the agent to reach intermediate states that are aligned with each other and also guides agent to the final goal state.
Limitations
This paper presents a multi-objective curricula learning approach for solving challenging deep robotics tasks. However, it may interesting to improve the sample efficiency of policy learning. Our method utilizes bi-level optimization, which elongates the training efficiency in terms of interactions between robots and environments. We argue that this limitation could be largely alleviated by replacing the model-free policy learning method with the model-based policy learning method.

---

References
[1] O. G. Selfridge, R. S. Sutton, and A. G. Barto. Training and tracking in robotics. In Ijcai, pages 670-672, 1985.
[2] Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In ICML, volume 382 of ACM International Conference Proceeding Series, pages 41-48. ACM, 2009.
[3] K. A. Krueger and Dayan. Flexible shaping: How learning in small steps helps. Cognition. 1100))380-394, 2009.
[4] J. Schmidhuber. Curious model-building control systems. In Proc. international joint conference on neural networks, pages 1458-1463, 1991.
[5] R. Portelas, C. Colas, I.. Weng, K. Hofmann, and P.-Y. Oudeyer. Automatic curriculum learning for deep rl: A short survey, arXiv preprint arXiv:2003. 046644 2020.
[6] C. Florensa, D. Held, M. Wulfmeier, M. Zhang, and P. Abbeel. Reverse curriculum generation for reinforcement learning. arXiv preprint arXiv: T.Xiv:7707.050000 2017.
[7] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and Munos. Unifying count-based exploration and intrinsic motivation. In NIPS, pages 1471-1479, 2016.
[8] N. Lair, C. Colas, R. Portelas, J.-M. Dussoux, P. Dominey, and P.-Y. Oudeyer. Language grounding through social interactions and curiosity-driven multi-goal learning. arXiv preprint arXiv: 0.0011.031199 2019.
[9] S. Narvekar, J. Sinapov, M. Leonetti, and P. Stone. Source task creation for curriculum learning. In AAMAS, pages 566-574. ACM, 2016.
[10] M. Hessel, J. Modayil, H. van Hasselt. T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot,
M. G. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning.
In AAAI, pages 3215-3222. AAAI Press, 2018.
[11] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn. Gradient surgery for multi-task learning, arXiv preprint arXhv:2001.06788 2020.
[12] R. Yang, H. Xu, Y. Wu, and X. Wang. Multi-task reinforcement learning with soft modulariza- tion. arXiv preprint arXiv:2003.13661, 2020.
[13] T. Wang, H. Dong, V. Lesser, and C. Zhang. Multi-agent reinforcement learning with emergent roles, arXiv preprint arXiv:2003.08033 2020.
[14] D. Ha, A. M. Dai, and Q V. Le. Hypernetworks. In ICLR (Poster). OpenReview.net, 2017.
[15] Sukhbaatar, Z. Lin, I. Kostrikov, G. Synnaeve, A. Szlam, and Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint arXiv: (xXiv:/703.0540) 2017.
[16] C. Blundell, B. Uria, A. Pritzel, Li, A. Ruderman, Z. Leibo, J. Rae, D. Wierstra, and D. Hassabis. Model-free episodic control. arXiv preprint arXiv: 1rX6vv1606...0 060666666600400 2016.
[17] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and
Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1406, /406 2014.
[18] S. Narvekar, J. Sinapov, and P. Stone. Autonomous task sequencing for customized curriculum design in reinforcement learning. In IJCAI, pages
2017.
[19] M. Svetlik, M. Leonetti, J. Sinapov, R. Shah, N. Walker, and P. Stone. Automatic curriculum graph generation for reinforcement learning agents. In AAAI, pages 2590-2590 AAAI Press, 2017.

---

[20] S. Narvekar and P. Stone. Learning curriculum policies for reinforcement learning. arXiv preprint arXiv: 18/2.00285, 2018.
[21] A. Campero, R. Raileanu, H. Küttler, B. Tenenbaum, T. Rocktäschel, and E. Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. arXiv preprint vrXi:2206.11122 2020.
[22] A. Graves, M. G. Bellemare, J. Menick, Munos, and K. Kavukcuoglu. Automated curriculum learning for neural networks. In international conference machine learning, pages 1311- 1320. PMLR, 2017.
[23] T. Salimans and R. Chen. Learning montezuma's revenge from a single demonstration. arXiv preprint arXiv:1812.033811 2018.
[24] D. Pathak, P. Agrawal, A. Efros, and Darrell. Curiosity-driven exploration by self- supervised prediction. In International Conference on Machine Learning, pages 2778-2787. PMLR, 2017.
[25] P. Shyam, W. Jaškowski, and F. Gomez. Model-based active exploration. In International Conference on Machine Learning, pages 5779-5788. PMLR, 2019.
[26] C. Florensa, D. Held, X. Geng, and P. Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pages 1515-1528, 2018.
[27] Q. Long, Z. Zhou, A. Gupta, Fang, Wu, and Wang. Evolutionary population curriculum for scaling multi-agent reinforcement learning. arXiv preprint arXhv:2003. 2020.
[28] R. Portelas, C. Romac, K. Hofmann, and P.-Y. Oudeyer. Meta automatic curriculum learning. arXiv preprint arXiv:2011.08463 2020.
[29] A. Wilson, A. Fern, S. Ray, and P. Tadepalli. Multi-task reinforcement learning: a hierarchical bayesian approach. In Proceedings of the 24th international conference on Machine learning, pages 1015-1022, 2007.
[30] I.. Pinto and A. Gupta. Learning to push by grasping: Using multiple tasks for effective learning. In 2017 IEEE international conference on robotics and automation (ICRA), pages 2161-2168. IEEE, 2017.
[31] M. Riedmiller, R. Hafner, T. Lampe, M. Neunert, J. Degrave, T. Wiele, V. Mnih, N. Heess, and J. T. Springenberg. Learning by playing solving sparse reward tasks from scratch. In International Conference on Machine Learning, pages 4344-4353, PMLR, 2018.
[32] K. Hausman, J. T. Springenberg, Z. Wang, N. Heess, and M. Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learning Representations, 2018.
[33] E. Parisotto, J. L. Ba, and R. Salakhutdinov. Actor-mimie: Deep multitask and transfer reinforcement learning, arXiv preprint arXiv:1511.06344 2015.
[34] A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell. Policy distillation. arXiv preprint 15 2015.
[35] Y. W. Teh, V. Bapst, W. M. Czarnecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and
R. Pascanu. Distral: Robust multitask reinforcement learning. arXiv preprint 170.0:1707.04155. 2017.
[36] Y. Zhang and D.-Y. Yeung. A regularization approach learning task relationships in multitask learning, ACM Transactions Knowledge Discovery from Data (TKDD), 8(3):1331, 2014.
10

---

[37] Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning, pages 794-803. PMLR, 2018.
[38] A. Kendall, Y. Gal, and R. Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 7482-7491, 2018.
[39] X. Lin, H. S. Baweja, G. Kantor, and D. Held. Adaptive auxiliary task weighting for reinforce- ment learning, Advances in neural information processing systems, 32, 2019.
[40] O. Sener and V. Koltun. Multi-task learning as multi-objective optimization. arXiv preprint 1 1810044600 2018.
[41] Y. Du, W. M. Czarnecki, S. M. Jayakumar, M. Farajtabar, R. Pascanu, and B. Lakshminarayanan. Adapting auxiliary losses using gradient similarity. arXiv preprint arXiv: 2018.
[42] S. P. Singh. Transfer of learning by composing solutions of elemental sequential tasks. Machine Learning, 8(3):323-339, 1992.
[43] Andreas, D. Klein, and Levine. Modular multitask reinforcement learning with policy sketches. International Conference on Machine Learning, pages 166-175 175-175. PMLR, 2017.
[44] A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671 2016.
[45] A. H. Qureshi, J. J. Johnson, Y. Qin, T. Henderson, B. Boots, and M. C. Yip. Composing task-agnostic policies with deep reinforcement learning, arXiv preprint 9rXiv:1905.1063) 2019.
[46] X. B. Peng, M. Chang, G. Zhang, P. Abbeel, and S. Levine. Mep: Learning composable hier- archical control with multiplicative compositional policies. arXiv preprint 1905 1905009808. 2019.
[47] T. Haarnoja, V. Pong, A. Zhou, M. Dalal, P. Abbeel, and S. Levine. Composable deep reinforce- ment learning for robotic manipulation. In 2018 JEEE International Conference Roboties and Automation (ICRA), pages 6244-6251. IEEE, 2018.
[48] H. Sahni, Kumar, F. Tejani, and C. Isbell. Learning to compose skills. arXiv preprint 17Xiv:1711.11288, 2017.
[49] M. Lengyel and P. Dayan. Hippocampal contributions to control: the third way. Advances in neural information processing systems, 20:889- 896, 2007.
[50] O Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra. Matching networks for one shot learning. arXiv preprint arXiv: 1606 04080, 2016.
[51] A. Pritzel, B. Uria, S. Srinivasan.
Badia, O. Vinyals, Hassabis. D. Wierstra, and C. Blundell. Neural episodic control. In International Conference on Machine Learning, pages 1827-2836. PMLR, 2017.
[52] Y. Han, G. Huang, S. Song, L. Yang, H. Wang, and Y. Wang. Dynamic neural networks: A survey, arXiv preprint arXiv:2102.0490 04900, 2021.
[53] X. Jia, B. De Brabandere, T. Tuytelaars, and L. Van Gool. Dynamic filter networks. In NIPS, 2016.
[54] T. Galanti and L. Wolf. On the modularity of hypemetworks.. arXiv preprint arXiv:2002.10006. 2020.

---

[55] Y. LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. OpenReview.net, 2022.
[56] E. Grefenstette, B. Amos, D. Yarats, M. Htut, A. Molchanov, F. Meier, D. Kiela, K. Cho, and
S. Chintala. Generalized inner loop meta-learning arXiv preprint arXiv:1910.01777, 2019.
[57] Matiisen, A. Oliver, T. Cohen, and Schulman. Teacher-student curriculum learning. IEEE transactions on neural networks and learning systems, 31(0):3732-3740, 2019.
[58] T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In JCML, volume 37 of JMLR Workshop and Conference Proceedings, pages 1312-1320. JogR.org, 2015.
[59] B. Ivanovic, J. Harrison, A. Sharma, M. Chen, and M. Pavone. Bare: Backward reachability curriculum for robotic reinforcement learning. In ICRA, pages 15-21. IEEE, 2019.
[60] A. Y. Ng. D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, pages 278-287. Morgan Kaufmann, 1999.
[61] A. Goyal, A. R. Didolkar, A. Lamb, K. Badola, N. R. Ke, N. Rahaman, J. Binas, C. Blundell, M. Mozer, and Bengio. Coordination among neural modules through a shared global workspace. In ICLR. OpenReview.net, 2022.
[62] S. Sukhbaatar, A. Szlam, J. Weston, and R. Fergus. End-to-end memory networks. arXiv preprint arXiv:1503.08895 2015.
[63] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, hts/1707.0660 2017.
[64] O. Ahmed, F. Träuble, A. Goyal, A. Neitz, M. Wuthrich, Y. Bengio, B. Schölkopf, and S. Bauer. Causalworld: A robotic manipulation benchmark for causal structure and transfer learning. In ICLR. OeenReview.net, 2021.
[65] O. Chang, L.. Flokas, and H. Lipson. Principled weight initialization for hypernetworks. In International Conference on Learning Representations, 2019.
12


Reviews:
[{"review_id": "znRhfneM8rM", "reviewer": "Reviewer_9eoj", "summary": "This paper proposes an architecture and learning pipeline to generate multi-objective curricula.\n\nThe basic ideas are:\n1. Using a hypernet to generate the weights of models that further generate each manually designed curricula. This includes learnable initial states, subgoals, and shaped reward functions.\n2. To enable futher flexibility, the authors propose to use an augmented memory to support the learning agent, which can be written by the hypernetwork. So this is a general form of \"curricula\" besides the aforementioned manually designed curricula.\n3. Frame ACL as a bilevel optimization problem and optimize hypernet and $Q$ and $\\pi$ jointly.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. I think this paper proposes a novel problem (multi-objective curricula), a general perspective (bilevel optimization for ACL), and a novel architecture (hypernet with write-only memory as a general curricula).\n2. The main ideas are straightforward and reasonable.\n3. Experiment results are convincing. Particularly, it has been shown that the combination of hypernet and augmented memory is necessary.\n\nWeakness:\nI find this work very easy to follow and no significant weakness from my opinion.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think there are several novel points this paper brings:\n1. A bilevel optimization (BO) view of ACL, though not completely novel, but to my knowledge this is the first paper explicitly using BO to train ACL.\n2. Multi-objective curriculum via hypernetwork is an interesting proposal, which automatically connects all curricula.\n3. The augmented memory is a novel form of \"curricula\" that is very flexible.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "1. I do not see an obvious reason why using $\\gamma$ for goal and not for the discount factor.\n2. Line 145, For the shaping reward, why learning $\\mu$ and not fixing it to the discount factor $\\lambda$ (in your notation). If it is not $\\lambda$, then in theory it cannot guarantee optimality anymore.\n3. From the presentation perspective, I think the authors can explicitly list their contributions at the end of introduction using bullet points, which can potentially help others to quickly get the novelties proposed in this work."}}, {"review_id": "d4l1SM7ZRDG", "reviewer": "Reviewer_HfsD", "summary": "The paper proposes a learning architecture that guides the learning process of an agent by simultaneously: 1) selecting an initial state from the agent starts the next episode; 2) selecting the agent's goal; and 3) selecting the reward function. Experiments on simulated robot environments (reaching, pick and place, and stacking) show some improvements relative to a few select baseline approaches. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n\nThe approach is novel and the idea of learning how to initialize an episode to guide the agent's learning is creative. \n\nAblation study is informative\n\nWeaknesses:\n\nThere are a few clarity issues with the paper's writing and presentations -- see detailed comments under 'issues'\n\nThere needs to be better justification for why these particular baselines make sense -- see questions in detailed comments\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The recommendation is based on the strengths and weaknesses outlined in the review, as well as the list of issues. The recommendation is subject to change depending on the authors' responses. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "\n\nQuestions for authors:\n\nFor the inputs to the Hyper-RNN, can you elaborate more on why you only consider the final state of the last episode?\n\nQuestion on the 3 environments: these problems appear difficult mostly because the environmental reward function is sparse, in other words, it is engineered very poorly. Have you considered settings where some thought was put into the reward function to make the task easier to learn? Also, when each new episode is generated, what is randomized (e.g., positions of objects) and what remains the constant (e.g., the goal)?\n\nIn Figure 2, are the episodes likely to have more or less the same number of time steps? is it possible that some approaches result in episodes that are much shorter or longer? it may be good to plot the performance in terms of number of time steps as opposed to number of episodes as ultimately it is the interaction with the environment that we want to minimize\n\nWhat were the fundamental limitations that prevent the baseline approaches from learning to complete the task all the time? theoretically, they should be able to do so given enough data but it looks like they've converged before hitting high success rates. Also, are the baselines essentially learning from scratch without a curricula? Why not have a simply DNN baseline that just tries to learn from scratch\n\nIn the limitations, cam you elaborate on what you mean by \"elongates the training efficiency\"?\n\n\nComments on parts of the paper:\n\nIn the last sentence of the 1st paragraph of introduction, more broadly, curriculum learning can also be used to modify the world itself, e.g., add new entities to the world, change the behaviors of other agents, etc., see:\n\nNarvekar, Sanmit, et al. \"Source task creation for curriculum learning.\" Proceedings of the 2016 international conference on autonomous agents & multiagent systems. 2016.\n\nSomewhere in the introduction, you should elaborate what you mean by output a curriculum? how is curriculum represented? Sometimes a curriculum is a sequence of tasks, sometimes a graph of tasks, sometimes it is just an ordering of data points. \n\nA running illustrative example would really help is space permits\n\nIn preliminaries, the RL section should include the notion of goals since goals are used in this work\n\nIn the second paragraph of Section 2, \"the output of D is a curriculum such as an initial state.\", that's actually false. An episode with some particular initial state is part of a bigger curriculum, yes, but it is not in itself the curriculum. The curriculum is instead, the sequence of generated episodes, each with different initial state, goal, etc. \n\nIn Figure 3, shouldn't all methods do the same at the very beginning with no training? \n\n\n"}}, {"review_id": "3KGYXbcI45S", "reviewer": "Reviewer_oFwa", "summary": "This paper proposes a method to generate manually designed curricula as well as automatically generated abstract curricula for policy learning in the reinforcement learning setting. These curricula influence the data collection of the agent and thus can gradually help the agent to solve the task. By this, the agent's sample efficiency improves and it is able to solve the task with qualitative policies. The proposed curricula are coordinated via a multi-task hyper-net which parameterizes the curriculum base networks. The method is benchmarked against other automatic curriculum learning approaches on six robotic manipulation tasks. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written such that the reader can easily follow the intuition and the ideas. It shows a thorough experiments part including a thorough ablation study. \n\nEven though some related work was mentioned in the main text, I was missing an explicitly related work part while reading. In my opinion, summarizing the related work in a chapter and explicitly mentioning the differences between the proposed method to those is useful for the reader to be able to have an idea of the novelty. There are also a couple of other works which I could find on the internet which consider automatically adjusting their curriculum in the context of reinforcement learning and might be worth a discussion:\n- https://proceedings.neurips.cc/paper/2020/file/68a9750337a418a86fe06c1991a1d64c-Paper.pdf\n- http://proceedings.mlr.press/v100/klink20a/klink20a.pdf\n- https://proceedings.mlr.press/v164/celik22a.html", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I like the proposed idea in this paper and I believe it is well written such that the reader can easily understand the purpose of the proposed method. The experiments show the advantages gained through the method. I would have appreciated an explicit related work part. However, I am willing to accept this paper. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "My main concerns were already described in the 'Strengths and Weaknesses' section. I would be happy if the issues would be considered during the rebuttal.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}, {"review_id": "0SdqzJFZelw", "reviewer": "Reviewer_g3te", "summary": "This paper presents an automatic curriculum learning framework for multi-objective curricula (MOC). The key insight is to use a multitask hyper-net module on top of multiple curricula generating neural networks. The hyper-net module resolves potential conflicting gradients between the lower level modules and parameterizes them accordingly. Each of these atomic module represents a single way of introducing curricula in RL: (i) subgoal sequence, (ii) initial states and (iii) reward function. Additionally, the external memory of the hyper-net is supposed to generate an “abstract“ curricula that is different from the predefined curricula above. The parameters of the abstract curriculum are added to the value function.\n\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The idea to learn individual curricula modules from scratch is interesting. It seems clever to learn distinct curricula and how to combine them at the same time to achieve good overall performance.\n- Evaluations: MOC has been tested against state-of-the-art automatic curriculum learning approaches and seems to outperform them. Moreover, it is much appreciated that the authors perform a detailed ablation study.\n\n# Weaknesses\n- The paper is at parts hard to understand. For instance, it is not clear what the pros and cons of read and write-only memory are (discussion at the end of the introduction) at this part of the paper. The authors describe how their approach compares to previous work, but leave the reader with no intuition why restricting memory access is important at all. Another example is the last sentence of the automatic curriculum learning section, where it is not clear what the “history“ is.\n- What exactly is this paper doing for the first time? Is it the introduction of the abstract curriculum module, or the combination of different types of curricula and their parallel tuning? The empirical evaluations are definitely also a contribution. A paragraph (that goes beyond the current contributions paragraph) saying explicitly how this paper differs from others would be helpful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Despite that the paper is at parts not clearly written, the techniques introduced and the evaluations seem convincing. Also, this paper tackles an interesting and important research direction with potentially high impact.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "The paper seems to be written in a rush and needs to be revised to improve clarity and flow of reading. Moreover, the contributions need to be explicitly formulated. ", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)