Reverse Forward Curriculum Learning for Extreme Sample and Demo Efficiency
Authors: Stone Tao
Arth Shukla
Tse-kai Chan
Hao Su
Venue: ICLR 2024 poster (2024)
Keywords: reinforcement learning
learning from demonstrations
curriculum learning
reverse curriculum learning
robot learning
robotics
Primary area: reinforcement learning
TL;DR: Combining a novel reverse curriculum approach with a forward curriculum to accelerate RL and massively reduce the demonstrations needed to solve complex tasks from sparse rewards.

Abstract: Reinforcement learning (RL) presents a promising framework to learn policies through environment interaction, but often requires an infeasible amount of interaction data to solve complex tasks from sparse rewards. One direction includes augmenting RL with offline data demonstrating desired tasks, but past work often require a lot of high-quality demonstration data that is difficult to obtain, especially for domains such as robotics. Our approach consists of a reverse curriculum followed by a forward curriculum. Unique to our approach compared to past work is the ability to efficiently leverage more than one demonstration via a per-demonstration reverse curriculum generated via state resets. The result of our reverse curriculum is an initial policy that performs well on a narrow initial state distribution and helps overcome difficult exploration problems. A forward curriculum is then used to accelerate the training of the initial policy to perform well on the full initial state distribution of the task and improve demonstration and sample efficiency. We show how the combination of a reverse curriculum and forward curriculum in our method, RFCL, enables significant improvements in demonstration and sample efficiency compared against various state-of-the-art learning-from-demonstration baselines, even solving previously unsolvable tasks that require high precision and control. Website with code and visualizations are here: https://reverseforward-cl.github.io/

Published as a conference paper at ICLR 2024
REVERSE FORWARD CURRICULUM LEARNING FOR EXTREME SAMPLE AND DEMO EFFICIENCY
Stone Tao & Arth Shukla & Tse-kai Chan & Hao Su
University of California, San Diego (stao, arshukla, tsc003,, hacsu)@ucsd.ed
ABSTRACT
Reinforcement learning (RL) presents a promising framework to learn policies through environment interaction, but often requires an infeasible amount of inter- action data to solve complex tasks from sparse rewards. One direction includes augmenting RL with offline data demonstrating desired tasks, but past work of- ten require lot of high-quality demonstration data that is difficult to obtain, es- pecially for domains such as robotics. Our approach consists of reverse cur- riculum followed by forward curriculum. Unique to our approach compared past work is the ability to efficiently leverage more than one demonstration via per-demonstration reverse curriculum generated via state resets. The result of our reverse curriculum is an initial policy that performs well on narrow initial state distribution and helps overcome difficult exploration problems. A forward curriculum is then used to accelerate the training of the initial policy to perform well on the full initial state distribution of the task and improve demonstration and sample efficiency. We show how the combination of reverse curriculum and forward curriculum in our method, RFCI., enables significant improvements in demonstration and sample efficiency compared against various state-of-the-art learning-froo seriing-from-eeee comm-mmmonstration baselines, even solving previously unsolvable tasks that require high precision and control. Website with code and visualizations are here: httpss//eeersefoowwadd...gg.ggge
INTRODUCTION
Adroit (1M, demos)
100
MetaWorld (1M, 5 demos)
The RL paradigm enables trial and error ap- proach to learn behaviors from rewards. How- ever, RL from scratch is sample-ineffciennt, es- pecially in high-dimensional tasks with complex dynamics and/or long horizons. Even when train- ing in fast simulators or highly GPU parallelized simulators (Makoviychuk et al., 2021]; Freeman al.] 2021) lack of priors means infeasible amounts of exploration and human programmed priors such as shaped reward functions or en- vironment simplifications are required, none of which are remotely scalable.
Sauctaso Nale
Nomalzed
95.2
80
40
62.3
RE,7D
RLPD
RFCL (Oun)
ManiSkal12 (2M, 5 demos)
Pfu gharggrr (10M, 10 demos)
Saccess Rate
80
60
Learning from demonstrations has become pop- ular paradigm for learning complex control skills without relying on engineered dense reward sig- nals, complex motion planning, or infeasible amounts of compute to brute force learn solu- tions. However, as often is the case in robotics, demonstrations are infeasible to collect at scale, making it difficult to scale learning from demon- stration methods. The classical approach to lever-
REPD
RECL (Oun)
RLPD
RTCL (Oun)
Figure 1: Results over 3 environment suites and the hardest task given fixed compute budgets ranging from 1M to 10M samples and few demonstrations. Our RFCL method drastically outperform recent ap- proaches like JSRL and RLPD which are included in the baselines.
aging demonstrations through behavior cloning (Atkeson & Schaal 1997). with more recent meth- ods performing behavior cloning via sequence modelling (Chen et al. 2021a Zheng et al. 2002)... A

---

Published as a conference paper at ICLR 2024
recent class of methods in offline RL learn value functions from demonstrations and finetune online (Kostrikov et al. 2022; Nakamoto et al. 2023). allowing them to learn from more demonstrations including suboptimal ones.
However, many of these methods require many demonstrations even with finetuning to solve hard tasks, which raises critical issue in domains such robotics since robotics demonstration data is scarce. Even human demonstrations, while flexible to collect, are not only difficult to collect scale but also exhibit undesirable properties like non-Markovianess and sub-optimality making them complicated to learn from (Mandlekar et al. 202T). Furthermore, these methods still cannot overcome the problem of exploration when comes to sparse reward tasks, where achieving the reward in online training is infrequent, or when the task has a very wide state distribution. Such difficult tasks include long-horizon tasks (Zakka et al. 2023), highly randomized and high-precisin robot manipulation tasks (Gu et al. 2023). or complex dynamics such as dexterous hand tasks (Chen Rajeswaran al. 2018).
al. 2023
The main motivating question then for our work is how can we effectively and practically overcome the exploration problem with access to only a few demonstrations? To address these challenges we leverage state reset from demonstrations and apply reverse and forward curriculums to online RL for sample and demonstration -ffficient learning. We show how our per-demonstration reverse curriculum can overcome the exploration problems that prevent prior works from ever solving some tasks by first training an initial weak policy capable of only solving a task from a narrow initial state distribution. We then demonstrate the sample-efficiency improvements forward curriculum brings when finetuning the reverse curriculum-trained policy to gradually learn to solve the task under the full initial state distribution.
Our primary contributions consist of our novel reverse forward curriculum learning (RFCL) algo- rithm and several key methods that accelerate RFCL, making the most demonstration and sample- efficient model-free RL algorithm. We rigorously evaluate RFCL against several state-of-the-an baselines across 21 fully-observable manipulation tasks from 3 benchmarks: Adroit, ManiSkil12, and MetaWorld (Rajeswaran et al. 2018 Gu et al.) 2023 Yu et al.] 2019). Critically, we show that RFCL is the only method that can solve every main task from just 5 demonstrations or less with strong sample-efficiency in addition to fast wall-time results. Through ablations, we show how reverse and forward curriculums enable sample and demonstratiin-eefiiccee learning.
2 RELATED WORK
Offline collected demonstrations provide an avenue for acceler- ating policy learning without excessive online interactions. The classical behavior cloning approach one method that tries to clone the exact behavior of the agent that generated the demonstrations via supervised learning (Atkeson & Schaal, 1997) and recently via sequence modelling (Chen al.) 2021a). Offline RL is another approach to imitate demonstrations without any online interactions. However, both BC and offline RL tend to be incapable of achieving high success rates/returns due being limited to the quality and diversity of the demonstrations. Some offline RL methods address this via online finetuning (Kostrikov et al. 2022 Nakamoto et al. 2023).
Learning from Demonstrations:
Finally, there are online RL solutions that incorporate demonstration data into their objective (Ra- jeswaran et al. 2018 Ball et al. 2023). However, when demonstration data extremely limited, prior approaches cannot solve complex tasks that are horg-oorioon, precise, and/or highly random- ized due to exploration difficulty.
Curriculum Learning: Prior work has investigated how to incorporate curriculum learning ap- proaches to gradually increase the difficulty of a task to facilitate training. Typically, these currieu- lums are hand-designed in order to solve complex long-horizon problems with sparse rewards and/or little guidance from other sources of data like demonstrations (Li ct al. 2020).
Alternatively, reverse curriculums have been leveraged by initializing the agent near states that are easier to achieve meaningful rewards from, and the curriculum progresses by initializing from grad- ually more difficult states. Reverse curriculums can be generated via reversible dynamics (Florensal jet al. 2017) or a model of backward reachability (Ivanovic et al. 2019). Some methods rollout a separate policy or demonstrator and initialize the learning agent the end of the rollout, with some
2

---

Published as a conference paper at ICLR 2024
rolling out random number of steps (Popov et al. 2017). 7), and recently JSRL (Uchendu et al. 2023) rolling out fewer steps over time to form a reverse curriculum.
State Reset: A number of approaches leverage state reset in order to overcome the problems of ex- ploration by initializing the agent directly in more often difficult-to-racee states instead of relying on sampled actions to reach there. Some methods leverage hardcoded states previously encountered
states during interaction (Florensa et al.] 2017] Ecoffet et al.] 2019 Chen et al. 2023) Other meth- ods use demonstrations as their source of initial states to initialize to, with some using uniform
curriculum over these initial states (Nair et al., 2018 Peng et al. 2018| Hosu & Rebedea 2016),
hand-defing curriculum (Zhu al. 2018), or reverse curriculum (Resnick al. 2018 Salimans & Chen 2018).
Our method is similar to some prior work in that we leverage a reverse curriculum and reset to states in demonstrations. Unique to our approach, however, is the ability to efficiently leverage more than one demonstration via a per-demonstration reverse curriculum, different to |Resnick et al. (2018) which only uses one demonstration and tackles tasks with small initial state distrbutions, and different to [air et al. (2018) which uses multiple demonstrations but applies a uniform curriculum and struggles with more difficult tasks due to exploration inefficiency. Furthermore, we propose novel approach of coupling reverse curriculum with a forward curriculum, which enables greater ktte-of-thee-e baselines.
demo efficiency and drastically improves sample efficiency compared Moreover, we introduce key methods that further accelerate learning in our curriculum designs not done by past work. These contributions enable our approach to even solve some previously unsolved environments from sparse rewards that even human designed dense rewards are insufficient for.
3 PRELIMINARIES
3.1 PROBLEM SETTING
We consider the standard Markov Decision Process (MDP) which can be described tuple M = (S, A, R, T, p. 7) where S is the continuous state space, A is the continuous action space, : S x A + IR is the scalar reward function, T S x A + S is the environment dynamics function, P is the initial state distribution, and y E [0, 1] is the discount factor. The goal is to learn a policy To S A parameterized by 9 that maximizes the expected discounted return, namely solving max, E., 1h== T'T,] where T3 is the reward at timestep t. One goal in RL is sample efficiency, learning No that maximizes the discounted return with few environment interactions as possible.
In this work, we consider complex MDPs with sparse reward functions where +1 is given for being in success state and 0 otherwise. This setting is motivated by how many environments in embodied AI research such as robotics are high-dimensional with difficult dynamics. Moreover, dense reward functions are non-trivial construct and may be sub-optimal (Singh et al. 2019] Amodei et al. 2016). While sparse reward functions are desirable as they reflect directly our desired goal (c.g. high success rates), they are more difficult to learn from due to exploration. A common approach is to leverage pre-collected demonstration data to aid in overcoming the problems of exploration sample-efficient learning under sparse rewards. We define dataset of demonstrations D (To: .....,, T to be set of N demonstrations where т, (st,0- Qi,0, ...; -i, O,,T:-1, $i.7; is a trajectory of length T; composed of a sequence of states and actions. practice, the observations used by the policy ma may be different from the actual environment state but for simplicity in this paper state also refers to observation.
3.2 CURRICUL.UM LEARNING
In RL., curriculum learning is an approach where during training, the agent presented with gradu- ally harder tasks, each leveraging skills from the previous easier tasks in order to accelerate learning, mimicing how humans learn (Narvekar et al., 2020). In this paper, the MDP M is changed through- out training by modifying the initial state distribution P to facilitate a curriculum. Define Smn to be the original initial state distribution of the MDP we wish to maximize discounted return. We can then define curriculum as sequence of initial state distributions ...): each of the elements are also referred to as stages of the curriculum. Each curriculum also comes with a criterion for whether to advance the next stage of the curriculum from p; to Pi+1-
3

---

Published as a conference paper at ICLR 2024
A key novelty of our work is leveraging both reverse curriculums and forward curriculums. The reverse curriculum initial state distributions are denoted with p and the forward curiculum reset states in the demonstrations similar to Nair et al.
with p². In the reverse curriculum, (0188. Different to prior work we auto generate per-demonstration reverse curriculum around these demonstration states in order accelerate learning under sparse rewards. The reverse cur- riculum enables sample-efficient learning of solving the MDP from the narrow set of initial states in the demonstrations {s detailed in Sec. 4.1 In the forward curriculum, we adopt similar approach to Prioritized Level Replay (PLR) (Jiang et al. (02T) which assumes priors about the parameterization of the state. Different to the reverse curriculum, we reset to states sampled from Sinit instead of states in demonstrations. The curriculum here is constructed in a way that prioritizes initial states that are at the edge of the ability of the policy mg. By sampling these initial states more frequently, the training focuses on learning solve from initial states just within reach of the agent, and as the agent improves, the curriculum advances to provide more difficult initial states, enabling the agent to solve the majority of initial states in Smitt This is detailed in Sec. 4.2] and results in our demonstration ablations Sec. 5.2 demonstrate how the forward curriculum coupled with the reverse curriculum enables our algorithm to fewer demonstrations than just using reverse curriculum and significantly fewer demonstrations than prior work.
4 REVERSE FORWARD CURRICULUM LEARNING (RFCL)
The goal of our method is to pro- vide a practical and flexible algorithm that accelerates learning in complex environments with a limited number of demonstrations without shaped re- wards. visual overview the two stage training process of RFCL is in Fig. 2 In stage 1, reverse cur- riculum is generated via state resets to states in demonstrations, learning policy to initially solve narrow initial state distribution. The reverse curriculum and state resets help over- come difficult exploration problems by initializing the agent near success states and gradually moving it fur- ther away. The forward curriculum then generalizes the initial policy to a larger initial state distribution without using more demonstration data by gradually sampling more difficult initial states to train on over time, enabling additional demonstration and sample efficiency. In both stages, we use the off-policy algorithm Soft Actor Critic (Haarnoja et al. 2018) with a Q-ensemble (Chen al. 2021b), see hyperparameters Appendix
Reverse Curriculum
Forward
Curriculum
Figure 2: A simplfied view of the reverse and forward curriculum. The blue arrows represent the given demonstration trajectories (2 this example), starting from an initial state and moving towards the goal marked by gold star. The area covered by dashed green lines represent the distribution of initial states from which the pol- icy can achieve high return. The area shaded in red represents the most frequently sampled initial states during each stage curricu- lum. From left to right represents the progression the trained policies ability over the course of the curriculum training.
4.1 STAGE REVERSE CURRICULUM
In this stage, we start with randomly initialized actor and critic networks and train using standard RL with SAC. We further aggressively oversample separate offline buffer consisting of all the given demonstration data, optimal and sub-optimal, as done in Hansen et al. (2023); Ball et al. (2023) during online RI. by sampling 50% of data from the online buffer and the rest from the offline buffer during optimization. We further adopt per-domonstration reverse curriculum that improves sample- efficiency compared to prior work that use other kinds of reverse curriculums or demonstration state resets. Finally, we introduce few key methods that further accelerate reverse curriculum learning.
Per-demonstration Reverse Curriculum Construction: We observe that not all demonstrations similar due to different initial states and are often multi-modal, is the case of human or motion- motion-planned planned demonstrations. As a result, a curriculum for each demonstration is necessary as opposed curriculum constructed from all demonstrations as done in prior work. A per-demo
4

---

Published as a conference paper at ICLR 2024
curriculum ensures noisy information arising from the multi-modality of demonstrations do not impact the reverse curriculum of each demonstration as much.
For each successful demonstration Tj assign start step f.. For each new episode during training, we first sample a demonstration т; from our dataset D with probability t,/T, sampling demonstra- tions progressing slower more frequently. Then, we sample discrete offset value k ~ K and reset the environment to the state si.e-k from T;, the state at the + & timestep of demonstration TG. We K.
make the design choice of using geometric distribution
Initializing all 4; Tú, the horizon of тi, at the start of training, we train mostly on states that success states these demonstrations are successful. As result, it is highly likely achieve positive rewards during training and thus easy to learn. We also define a small reverse step size constituting the distance between stages in the curriculum. Thus, the reverse curriculum for each demonstration T; is defined as PT: .........000.. PG) where P samples st,++ with probability px (k:).
We now define the stage transition criterion. During training and at curriculum stage pi, whenever we sample st, (when h = 0), we are sampling an initial state that is at the frontier of the agent's s abilities. the last mn times the agent achieved success at the end of episodes initialized at state si,ts then we progress the curriculum of demonstration т; to the next stage Pi-5- This ensures the agent only begins to tackle the task from the slightly more difficult demonstration state 8,,-- provided it is already succesful enough 8t,,, forming the curriculum.
Once all demonstration curriculums are at stage Pu. the reverse curriculum is considered complete, as now the agent can achieve high return all demonstration initial states 8,.. We visualize how the reverse curriculum progresses in Fig. [ and via videos on the project page, The red area demonstrates how the sampled initial states slowly reverse their way back to the first states of the two demonstrations, training the policy in reverse to perform well on the narrow initial state distribution visualized by the green region. An empirical investigation via simple pointmaze is done well in Sec. 5.2 We benchmark alternative curriculum choices in addition and show the sample efficiency gains of the per-demonstration reverse curriculum in Table
Dynamic Episode Timelimits:
Since we are often resetting to states not from the true initial state distribution Smi but from various points in demonstrations, the time it takes to achieve success from these states as quickly as possible varies significantly. Observe that if we sample a state near a success state that yields positive reward, we need far less environment interactions to then get the positive reward compared to sampling state farther away. This motivates the use of dynamic timelimit depending on which state is sampled, Suppose we sample state 81,6, then a simple choice set the episode timelimit to 1 + (T; - f)Φ-1 -1 ó is hyperparameter for the ratio of demonstration length to episode horizon. We fix the value of á for each environment suite and generally for better -imple-ffficieny ø can be set larger if the source demonstration is fairly slow (e.g. human demon- strations) and smaller if the source demonstration is fast or optimal (e.g. scripted policies). We show an ablation on dynamic epsiode timelimits on stage training and demonstrate the improved sample efficiency in Table I
4.2 STAGE 2: FORWARD CURRICULUM
In this stage, we continue the training of the actor and critics from stage 1 without resetting any networks. We instead reset the online buffer and initialize a new offline buffer. Like stage I, we continue aggressively sample from separate offline buffer. Different this time is that we fill the offline buffer with the online buffer of the stage 1 training, with the motivation being to ensure the value functions do not suddenly have to learn from completely unseen data and avoid unlearning phenomenon encountered by past work investigating finetuning of pretrained policies (Peng et al.) 2019 Kostrikov et al., 2022). Moreover, our choice of RI. algorithm is capable of learning from sub-optimal data it is helpful to add in the data that was collected from stage training instead of just using successful demonstrations. Crucial to this stage is the forward curriculum described below, which enables more demonstration--efiieee learning.
Forward Curriculum Construction: Following the reverse curriculum stage, we will have trained policy no that achieves a high return the MDP given the initial state is one of the initial states of a demonstration $;.0- It is critical to have the reverse curriculum as only applying a forward curriculum cannot work well in sparse reward settings due exploration difficulty. If there do not
5

---

Published as a conference paper at ICLR 2024
exist initial states So Smitt where is easy obtain any reward, the problem becomes very difficult and sample-inefficient, in addition to making it difficult to automatically generate a useful forward curriculum without relying on prior heuristics. Even when learning from demonstrations, if there are few demonstrations still remains difficult to get the sparse reward due to the exploration problem and lack of state coverage in the few demonstrations, especially in complex tasks.
Thanks to a few demonstrations and reverse curriculum learning, the problem of 'no easy initial states' is alleviated, we know that we can least perform well when starting from the initial states in the demonstrations. We only make a weak assumption that the initial state in the demonstrations we reverse-solvee
are close to states in Smit. which is often the case in collected demonstrations. Then, using a forward curriculum, with RL we initially prioritize training on initial states where there is the most learning potential. These are often close to the initial states of demonstrations because the policy can still get some nonzero reward if the initial state is similar enough to one seen during reverse curriculum learning. Eventually, the forward curriculum will train the policy on a sufficient number of initial states in Sinit and learn policy that achieves high return initial states sampled from Sma.. not just the demonstration initial states.
In order to maximize the outcome of each environment interaction, following PLR (Jiang et al.) 202I] we prioritize resetting to initial states that are at the edge of the agent's s abilities before reset- ting states that are too difficult to solve from states that are already easy to solve from. As PLR was originally designed for PPO, an on-policy algorithm, we make simpler adaptation of PLR that works well and robustly for off- aff-policy SAC and our environments.
First, we uniformally sample a set of n initial states Again = {si from Smit- As policy x after reverse curriculum learning has high return on initial states 51,0 we add these to Amiin Our method uses a simpler score function to PLR that assigns three scores with higher scores leading to higher priority, and it does not have a separate seen vs unseen set of initial states. Define q to be the fraction of episodes out of the last & episodes that receive nonzero return starting from a sampled initial state Siiii If q is 0, then assign a score of 2 to ii,init- If 0 < q < for a threshold a score of 3. If 2 assign a score of 1. In order of decreasing priority, we sample initial states that sometimes receive return, then states that receive no return, then states that consistently receive return. Following PLR, we adopt rank-based prioritization scheme which enables prioritization to be invariant to score scale, and we scale the importance of rank via a temperature value 3, resulting in the prioritiie distribution Ps. We also adopt the same staleness aware prioritization Pc.
w 1, assign
(s AAain S) rank(SS))
c- C2
Po (Ss,iiin C, = Co.eo--C;
(I)
Σ; rank(S,) 1/8: -1/)
where S; are the scores assigned to initial state 8t.init, e is the total number of episodes rolled out in training, and C; the episode count at which initial state iiiinit was last sampled. As q is based on whether episode received nonzero return or not, our score function is independent of scale of return of the environment under a optimal policy, making a more generalizable scoring scheme. We follow the same intuition PLR in that staleness prioritization ensures initial states that have not been sampled in a while get re-sampled in order to update their scores and ensure scores do not drift too far off their true value. The forward curriculum follows stages pb. .f..... advancing each time a new score assigned. The linear combination of the score and staleness prioritzation distributions form the initial state distribution at each stage
Po (Sc,init) = Ps(Si,ini |Arrin S) + (ii,iiiit C,
For all experiments we use the same hyperparameters that construct the forward curriculum. We visually showcase how the forward curriculum progresses over the course of training in Fig. 2ig.21m via toy example. While the initial policy starts off successful only on a narrow initial state distribution represented by the narrow green region, via the forward curriculum the policy prioritizes starting from initial states at the edge of what is capable of and expands the green region until the majority of initial states are covered.

---

Published as a conference paper at ICLR 2024
5 RESULTS
Through our experiments we aim to answer the following questions (1) How sample and demonstra- tion efficient is RFCL compared to other model-free baselines? (2) What methods accelerate reverse curriculum learning? (3) How does the curriculums in RFCL enable success from significantly less demonstrations and robustness to the demonstration source?
For our experiments, we rigorously evaluate and compare our algorithm against several baselines across robot environment suites for total of 21 fully-observable environments with sparse re- wards. The 3 environments suites are Meta World (Yu al.) 2019), Adroit (Rajeswaran et al.) 2018), and ManiSkill2 (Gu et al. 2023). see [. details on each task. We use an absolute sparse reward where +1 rewarded only on success states and 0 is given otherwise. The environments consist of range of easy to challenging tasks and we benchmark our algorithm on the low demon- stration regime, showcasing how RFCL is far more sample/demo efficient than prior approaches. We benchmark against 3 model-free baselines that also leverage demonstrations:
(1) RLPD (Ball et al., 2023) is well-tuned variation of SAC + demonstration data that at the time of writing achieves the ofate-of-the-art results on Adroit; (2) Jump Start RL (ISRL) (Uchendu et al. 2023) like RFCL. lever- ages reverse curriculums, although they generate the curriculum with a offline trained guide policy. JSRL code is currently not available we can only report on Adroit environments with numbers from their paper: (3) DAPG (Rajeswaran et al., 2018) uses a demo-augmented policy gradient for fifline-to-onlineg RL method. For each training run, select
RL. (4) Cal-QLin is state-of-the-ar a random seed. The seed also determines which uniformly sampled demonstrations are used for training. All baselines use the same seeds and thus the same demonstration data.
5.1
MAIN BENCHMARK RESULTS
Adroit (sparse)
MetaWorld (sparse)
ManiSkill2 (sparse)
100
100
NomaltedSran
80
Success Datte X
80
60
Sucoess Fatte
600
60
0
0
20
40
60
80
300
20
00
80
1000
50
1000
350
200
Environment steps Ox10%)
Enviicon ment steps (x110%)
Envinemment steps (x10")
JL
DAPS
RLFD
Cal-OL
RFCL tours)
Figure 3: Mcan success rate of algorithms for each environment suite across all tasks after 1M interaction steps with 5 demonstrations. Results are averaged within environment. Shaded areas represent 95% CIs over 5 seeds. The result show RFCL is significantly more performant and sample efficient compared to haselines. Note that RLPD and ISRI. can normally achieve decent results but require many more demonstrations for harder tasks.
Pick Place (5 demos)
Stick Pull demos)
PoginsertionSide (5 dames)
FlugCharger (10 demos)
100
80
Suctess Bate
60
60
50
60
40
4.0
40-
20
0
60
100
30
100
150
200
200
Enviitommment Staps DXL0ff
Ervironmmnt staps (x 1x
Ervir Environment Staps (x(10))
Environment stops (x330)
DAPG
RLFD
Cal QL
RFCL (ours)
Figure 4: On the harder tasks, RFCL significantly outperforms baselines. PlugCharger from ManiSkil12 is not of the most difficult environments, but RFCL can solve it
in the suite of tasks we test on usually as it given enough compute and demonstrations. Note PegInsertionSide and PlugCharger have high variance due to one seed failing for each, see Fig. 9 foor the same graph with 30 seeds.
The main results are summarized in Fig. 3 showing how RFCL outperforms all baselines that leverage demonstrations across all benchmarks and tasks. There are significant improvements in ManiSkill2 and MetaWorld and minor improvements in Adroit. RFCL is the only method that is capable of achieving nonzero success We argue that the critical reason why prior methods are unable to solve these tasks is because of the exploration bottleneck, which is more easily overcome via our method's state reset reverse currieu-
every environment within reasonable compute budget.
7

---

Published as a conference paper at ICLR 2024
PickCube
PickCabe
PirkCube
PickOube
PrrcCube demal
100
100
1.50
200
50
100
150
250
50
100
150
200
50
100
150
200
1.00
150
200
StackCube (1 demo)
StackCube demo)
StackCube demo)
StackCube (10 demo!
StackCube 120 demo)
100
SUCC5S
50
100
150
250
50
100
150
200
50
100
150
200
50
100
150
200
50
1:0
150
200
PeginsertiorSide (1 demo)
PeginserticnSide (3 demo)
PeginsertionSioo iii (5 demo)
PoginsertionS de (10 deme)
PeginsertionS4e (20 demot
300
100
1.50 1x
200
50
300
150
220
100
250 (x227)
200
Enveronment (6107)
Drviirom
Envirenment stops (x:2
Enveronment
Envrenment (x127)
RLFD
Revenu Cumcuiuu Only
Forward Cumiutum
RTCL (eura)
Figure 5: Mean success rate of algorithms on ManiSkill2 tasks after 2M interaction steps given varying amount of demonstrations to train on. Error bars represent 95% CIs over 5 seeds. Vertical gray lines indicate the average number of samples until the reverse curriculum completes. Reverse curriculum only RFCL. but instcad of a forward curriculum in stage 2 we sample uniformally from the initial state distribution. Forward curriculum only skips stage training entirely.
lum. Fig- 4 shows selection of the most difficult tasks and demonstrate that RFCL significantly better if the task is complex and difficult to explore. To visually see why these are more difficult see Appendix B.2 For disambiguation of each environment's 's results, see Appendix A.
5.2 ABLATIONS
For all ablations, we benchmark the ManiSkil12 environments as it is the most difficult set of environments due to high initial state randomization and precise manipulation requirements.
# of Demonstrations: RFCL provides a flexible approach leveraging wider range of demon- stration dataset sizes and still being able to perform well as shown in Fig. 5 We observe that when there are very few demonstrations, the forward curriculum important for improving performance. We see significant improvements on success rate for PickCube for demonstration, for Stackcube for 5 demonstrations or less, and for PegInsertionSide for 10 demonstrations or less. The same conclusions can be drawn from the same demonstration ablation on Metaworld which we show in Appendix A.3] We can attribute this effect to the narrowness of the distribution of initial states the reverse curriculum trained policy performs well on. When given very few demonstrations covering small set of initial states, the policy successful on a narrow distribution initial states. Without a forward curriculum, we sampling initial states uniformly from Smit meaning large majority of initial states are usually completely out of distribution and lead to return and mostly useless explo- ration. With forward curriculum, initial states that lead to nonzero return are prioritized, leading to sample-efficieng gains. Critically, we observe that RFCL still can solve difficult tasks that have high initial state randomization like PegInsertionSide from just few demonstrations whereas the strongest baseline fails completely and can only solve PickCube when given 20x the demonstrations RFCL needs solve PickCube.
Reverse Curriculum
Dynamic Timelimit
Steps (x (x10*) > 100.0
Demonstration Source: Adroit, Meta- world, and ManiSkill2 demonstrations are tele-operated, generated via scripts, and generated via motion planning re- spectively. For all these different kinds of demonstrations, RFCL can still solve the task showing that it is robust to demonstration source. This is exempli- fied by how human and motion planned demonstrations are often sub-optimal (with respect to sparse reward maximization objective), exhibiting multiple modes in actions,
Uniform
Global
41.5 + 10.8
Per Demo
X
33.5 + 3.0 24.1 + 5.9
Per Demo
Table 1: Ablations on the reverse curriculum stage comparing
number of environment steps until the agent can achieve ~000% success rate from the initial states (st.n) 5 demonstrations in the ManiSkil12 suite. Results shown are 95% CIs over 5 seeds. Uniform choice failed to reach 100% within 1M interactions.
8

---

Published as a conference paper at ICLR 2024
and possessing Man-ovanviinn characteristics (Mandlekar et al. 2021 The robustness can be at- tributed to the fact that we are learning from a purely sparse reward objective via online interaction, and not perfectly imitating past demonstrations. The demonstrations primarily provide means of overcoming exploration bottlenecks instead of leading to suboptima during training.
For stage 1 of training, alternative curriculums utilizing demonstration states for state resets are uniform and global. In a uniform "curriculum" " we benchmark the uniform state reset method used in Nair et al. (2018) where they uniformly sample a random demonstration and state in the demonstration. In a global curriculum, instead of assigning each demonstration a start step value t,, all demonstrations are assigned the start step value T; and we progress the curricu- lum of all demonstrations at the same time if the mean success rate of the past episodes exceeds a threshold. We perform a hyperparameter sweep over u and reported the best results. We further ablate on the use of a dynamic timelimit. From the results in Table we find that the uniform state reset approach when given just 5 demonstrations cannot get any reasonable results in 1M interac- tions. The per-demonstration reverse curriculum and dynamic timelimit both significantly improve the sample efficiency of the first training stage and learn a weak policy that can solve from {8,,)-
Reverse Curriculum:
Impact of Reverse and Forward Curriculums: The toy experiment on continuous state/action space pointmaze environment in Fig. 6
EF EF
EF
None
EA
Forward
demonstrates the agent's performance throughout training. The agent can re- set any square not covered by the demonstration and the default initial state distribution is heavily biased to the squares far away from the goal, order
Only
Reverse Forward
Reverse
Forward
making exploration difficult. to succeed, a policy would need to ex- plore around states along the demon- stration as the state space is continuous. Notably, under the reverse forward cur- riculum the agent quickly learns to first perform well around states along the demonstration, after which it relies on the forward curriculum to quickly learn to solve from states all around the maze in ever-growing frontier. Comparatively, without curriculum under the same time budget the agent is unable to improve due to no initial state prioritization and difficult exploration. The forward curriculum works eventually thanks to prioritizing initial states closer to the demonstration and goal, but is less sample efficient compared to reverse forward due to exploration difficulty.
Figure 6: Heatmap of agent's success rate al each maze cell over the course of training. comparing three kinds of training: None (no curriculum / normal RL). forward curriculum only, and method applying reverse and forward curriculums. Blue arrow is the demonstration provided. Red dot is the goal.
6 CONCLUSION
In this work, we introduce the RFCL algorithm that can solve complex tasks with significantly fewer demonstrations than before, including previously never-before solved sparse reward manipulation tasks. We show how the reverse curriculum trains an initial policy that solves the task from a narrow initial state distribution, and how the forward curriculum generalizes the policy to the full initial state distribution, enabling extreme demonstration and sample efficiency as shown in Fig. 3 and 5 We further motivate why the combination of a reverse and forward curriculum works, and introduce key methods that accelerate the reverse curriculum compared to alternatives.
A limitation of RFCL is the use of state reset and is thus generally restricted to training in simulation, relying on sim2real to be deployed in the real-world for robotics tasks. Despite this limitation, we argue sim2real is orthogonal research that can augment our research, and point to past work such as Chen et al. (2023) that succesfully perform sim2real on dexterous tasks and use state reset.
We hope our work provides a strong starting point for how far the capabilities of robot learning from demonstration methods can go if fully leverage the advantages of simulation vs real world training. With the demo efficiency of RFCL,, one can prioritize scaling up the diversity of tasks as opposed to the quantity of demonstrations. All code is open sourced on Github and are excited with how far the community can push when leveraging more properties of simulation.

---

Published as a conference paper at ICLR 2024
7 REPRODUCIBILITY STATEMENT
All experiments on the RFCL method can be reproduced with given runnable scripts + images uploaded htttss//giiuub.com///ooeee22/2/0
docker
AUTHOR CONTRIBUTIONS
Stone Tao proposed the initial research ideas and experiments, addition writing the majority of code for training and evaluation. Arth and Tse-kai contributed to research discussions around various experiments and significantly to baselines. Hao Su provided advising and guidance students on the research and proposed the toy experiment. All authors edited the manuscript.
ACKNOWLEDGEMENTS
Thank you to members of the Hao Su Lab for providing valuable feedback on the project. ST is supported in part by the NSF Graduate Research Fellowship Program grant under grant No. DGE- 2038238.
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. Concrete problems Al safety. CoRR, abs/1606.06565, 2016. URL /attp::/axxxi.. org/
abs/1606 06565)
Christopher G. Atkeson and Stefan Schaal. Robot learning from demonstration. In Douglas H. Fisher (ed.), Proceedings of the Fourteenth International Conference on Machine Learning (ICML 1997), Nashville, Tennessee, USA, July 8- 1997, pp. 12 20. Morgan Kaufmann, 1997.
Philip Ball, Laura M. Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforce- ment learning with offline data. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 1577-1594. PMLR, 2023. URL https: h/tps:///proceedngg..
mlr. wT.ppess//222/bbat
html
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Rein- forcement learning via sequence modeling. In Mare' Ranzato, Alina Beygelz- imer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Informa- tion Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 15084 15097. 2021a URL https:://proceed// 774896642a0cao 027011222022311112
newrips. wtt.cccpppppe wwtpppppe///2/2111 /2as//hhh2hhh////////// wwstt-bbbttttt..t........ html
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q- learning: Learning fast without model. In 9th International Conference on Learning Repre- sentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021b. URL. https:://ppeneeiae
net htte///orummidd//// 2m00td
Yuanpei Chen, Chen Wang, Li Fei-Fei, and C. Karen Liu. Sequential dexterity: Chaining dexterous policies for long-horizon manipulation. In Conference on Robot Learning, 2023.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Go-explore: new approach for explaaxpnon problems. CoRR, abs/1901.1009g, 2019. URL. http:
//a 7/arxiv. htt.v/.rgaab..... 901 10995
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In 1st Annual Conference on Robot Lear- ing, CoRL. 2017, Mountain View California, USA, November 13-15, 2017, Proceedings, vol-
10

---

Published as a conference paper at ICLR 2024
78 of Proceedings of Machine Learning Research, pp. 482 495. PMLR, 2017. URL html
.ml.. httt::/.pooooeen........eeeeeaaa
C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax A differentiable physics engine for large scale rigid body simu- lation. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neu- ral Information Processing Systems Track
Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL https:
neurips wwt.rips.ccppaperr////e wctpp/pppppppppp..c.c////ppppp 1/hash/
html
Jiayuan Gu, Fanbo Xiang. Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskill2: A unified benchmark for generalizable manipulation skills. In The Eleventh Inter- national Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https:
net / -idiidd__yyyyyD
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1856-1865, PMLR, 2018. URL http::ttt::/ceeeeeeeeeee mlr. prwss/ v80 haarno 181..b... html
Nicklas Hansen, Yixin Lin, Hao Su, Xiaolong Wang, Vikash Kumar, and Aravind Rajeswaran. Modem: Accelerating visual model-based reinforcement learning with demonstrations. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
net http://wwwweee/ddeeieeee
JdInc3gJVEJ
Jonel-Alexandru Hosu and Traian Rebedea. Playing atari games with deep reinforcement learning and human checkpoint replay. CoRR, (as/1607.05077 2016. URL /att:://axxv..rrr
abs/ moss/1607. 05077
Boris Ivanovic, James Harrison, Apoorva Sharma, Mo Chen, and Marco Pavone. Barc: Back- ward reachability curriculum for robotic reinforcement learning. In International Conference on Robotics and Automation, ICRA 2019, Montreal, QC. Canada, May 20-24, 2019, pp. 15- 21. IEEE, 2019. doi: 10. 10.110//CR..2019..0
URL httph:h////c wtt.:ei..or....00/1011111..11
ICRA. 2019 001198779422 794206
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 1021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 4940 4950. PMLR, 2021. URL Fiang21b.. html
mlr. haa..prees/vv39
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In The Tenth International Conference Learning Representations, ICLR 2022, Vir- tual Event, April 25-29, 2022. OpenReview.net, 2022. URL /ettpss:/cppereviiii .ne// Forum?i 20.0-68mmm w8n.6nnn2222nn
Richard Li, Allan Jabri, Trevor Darrell, and Pulkit Agrawal. Towards practical multi-object ma- nipulation using relational reinforcement learning. In 2020 IEEE International Conference on Robotics and Automation, ICRA 2020, Paris, France, May 31 August 31, 2020, pp. 4058-4088 IEEE, 2020, doi: 10. 000p//CRA4944.2000...911 TCRA40945. 2020. 9897468
URL /tll:ttss///cc.org////// http///ti.oogg//o//o/oo
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin. Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance gpu-based physics simulation for robot learning, 2021.
11

---

Published as a conference paper at ICLR 2024
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkamni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In Aleksandra Faust, David Hsu, and Gerhard Neumann (eds.), Conference on Robot Learning, 8-11 November 2021, I, London, UK, volume 164 of Proceedings of Machine Learning Research, pp. 1678-1690) PMLR, 2021. URL https:
//proceedings. mir.
mlr wwwpspeesssa..
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brishane, Australia, May 21-25, 2018, pp. 6292 6299. IEEE, 2018. doi: L109/ICRA. 8463162
UtLhttts::/ /ooi. org/1C.
Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline RL pre-training for efficient online fine- tuning. CoRR, abs/2303.05479, 2023. doi: 10448550/aXxi.2233.047777 URL https://doi.
455/100.8550/aa10 (118550/arxiv..o www.xxivv.33..... 05479
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. J. Mach.
Learn. Res., 21:181:1-181:50, 2020. URL http: / jmlr.. ortpppppessvv2/20-----e html
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: example- guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph.
37(4):143, 2018. doi: 10.1145///975173220311. URL httppps///eii..o//111.110
3197517. 3201311
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. CoRR, abs/1910.00177, 2019. URL
httt:://arxv..org/// wwwpg///b///9...ooo 00177
Ivaylo Popov, Nicolas Heess, Timothy P. Lillicrap, Roland Hafner, Gabriel Barth-Maron, Matej Vecerík, Thomas Lampe, Yuval Tassa, Tom Erez, and Martin A. Riedmiller. Data-efficient deep reinforcement learning for dexterous manipulation. CoRR, abs/1704.03073, 2017. |
7/arxiv.org 1atp//170.007777
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforce- ment Learning and Demonstrations. Proceedings of Robotics: Science and Systems (RSS), 2018.
Cinjon Resnick, Roberta Raileanu, Sanyam Kapoor, Alex Peysakhovich, Kyunghyun Cho, and Joan Bruna. Backplay: "man muss immer umkehren". CoRR, abs/1807.06919 2018. URL http:
// //axxiv.orr/bba/100...
Tim Salimans and Richard Chen. Learning montezuma's 's revenge from single demonstration.
CoRR, 0abs/8812.0332 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, Abs/1707.00344 2017. URL
www.xiiv..og/.a... abs/
1707.06347
Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Karen Liu, Dana Kulic, and Jeffrey Ich- nowski (eds.), Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pp. 1332-1344. 1.1.11444....... PMLR, 2022. httpst stttp::/prroeeeinggs mlr wwwpsppssssppp..
html
Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. Roboties: Science and Systems, 2019.
12

---

Published as a conference paper at ICLR 2024
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, Sergey Levine, and Karol Hausman. Jump-star reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engel- hardt, Sivan Sahato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICMI. 2023, 23-29 July 2023, Honolul,, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 34556-34583, PMLR, 2023. URL /procee//prceeiinggg mmrr. press/v202/ www.a//22cccnnnu222aa.2 .tml
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019. URL http::tt///// http:///wwwwww..aa////////.
10897
Kevin Zakka, Laura M. Smith, Nimrod Gileadi, Taylor A. Howell, Xue Bin Peng, Sumeet Singh, Yuval Tassa, Pete Florence, Andy Zeng, and Pieter Abbeel. Robopianist: A benchmark for high- dimensional robot control. CoRR, abs/2304.04150, 2023. doi: 10.48550VaxXx.223040415550 URL
https://doi.oo org 48550 wtt.://3w..vv......... 04150
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In Kamalika Chaud- huri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), Inter- national Conference Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 27042-27059, PMLR, 2022. URL
mir.
www.s/mww...rp....mmmmm.mmmmee.e wwwppressssssee 62/ zheng22c.ccc https///vee///eegeeeeeee html
Yuke Zhu, Ziyu Wang, Josh Merel, Andrei A. Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvu- nakool, János Kramár, Raia Hadsell, Nando de Freitas, and Nicolas Heess. Reinforcement and imitation learning for diverse visuomotor skills. In Hadas Kress-Gazit, Siddhartha S. Srinivasa, Tom Howard, and Nikolay Atanasov (eds.), Robotics: Science and Systems XIV, Carnegie Mellon University, Pirtsburgh. Pemnsylvania, USA, June 26-30, 2018, 2018 doi: N0.15507/RSS.200 2018. XIV.009. URL http:/ www..tocppprccceenegg.g.o.....o. org rew.////ssso.. p/0 p09. html
13

---

Published as a conference paper at ICLR 2024
A FULL ENVIRONMENT RESULTS
We show the training curves of our method compared to the primary baseline, RLPD, on all envi- romments here without averaging across environments in Appendix A.1
Because we are working with an extremely small number of demonstrations relative to the total number of demonstrations usually used with the tasks we tackle, there can be a high amount of variance in the result curves caused by the choice of demonstrations, not the algorithm itself. To disambiguate the source of variance, in Appendix [ we show results ManiSkill2 on one of the harder environments PegInsertion where we have 2 sets of demonstrations and for each set we run 5 training runs. The 2 sets of demonstrations are the same ones sampled for the main results shown in Figure
Finally we show some additional experiments with a demo ablation on Meta World in Appendix A.3 and compare against baseline that uses state reset by [air et al. (2018) on one of their tasks Appendix A.4 (their open soureed code does not use state reset so we can only test on their environment instead of benchmarking their algorithm on the more common robotics benchmarks like Metaworld).
A.1 RESULTS WITH RANDOM DEMONSTRATIONS
All algorithms given 5 randomly sampled demonstrations. We further mark the average step at which the reverse curriculum stage completes training with a vertical, dashed gray line. The bolded lines represent the mean values, and the shaded areas represent 95% CIs over seeds. Note that each seed uses the same demonstrations. Figures 7 and g show results on each environment in MetaWorld and Adroit respectively. The demo ablation curve in Fig. [ in the main paper shows results for ManiSkil12. We further explored just how variable demonstations are by running 30 seeds on the PegInsertion task with each seed using different sets of demonstrations in Fig. The figures show that we are the only method to succeed on all environments, significantly beating baselines. Furthermore, empirically our method is generally quite robust choice of demonstrations as shown in in Fig. / Only I seed out of 30 failed to get above 80% success rate after 6M interactions. Note that the robustness may vary between environments, and our results the main figures do include the seeds where nearly no success was obtained that occured. We believe there could be interesting future work in investigating what kind of demonstrations are easier to learn from using RFCL.
pepitanttaide
pushwal
106
Flandd mosert
20
80
100
20
Figure 7: Metaworld Results with random demonstrations. Results averaged over seeds with shaded area representing 95% Cls. Every single seed succesfully finished the reverse curriculum and the success rate trended upwards.
14

---

Published as a conference paper at ICLR 2024
dnorbinary-v
pen-binary--0
130
20
50
50
120
20
50
50
200
23
60
20
120
Enxenomentt OIIE D4309)
Dmu mo0ment strps 1-300)
Ensirmmment strpp (4300))
Figure 8: Adroit Results with random demonstrations. Results averaged
seeds with shaded repre-
senting 95% Cls. Every single seed succesfully finished the reverse curriculum the success rate (not shown) reached near 100%. The normalized return of RFCL plateaus just below the state-of the-----thhttr art method RLPD for the two easier environments, but significantly outperforms RLPD and Cal-QL on the harder Adroit Relocate environment.
102
RFCL SMinE
Figure 9: ManiSkil12 Peglnsertion results with random demonstrations. Results averaged over 30 seeds with shaded area representing 95% Cls. The results appear much better than the main figures due to the noise caused by hard-to to--to-llee Ie-eeear-fooo demonstrations being mostly cancelled out. Here, only out of the 30 seeds failed (the seed from the main results that failed.
105
M mv Wil yna
N
M
"Gooc"
Figure 10: ManiSkil12 PegInsertion results with fixed demonstration sets. "Bad" Demos correspond to runs using the only set of demonstrations that led no success in the main results in Fig. 4 "Good" Demos is randomly chosen to he of the other set of demonstrations used in the main results. Results averaged over seeds with shaded area representing 95% Cls.
A.2 RESULTS WITH FIXED DEMONSTRATIONS
We pick 2 of the sets of demonstrations used for training shown in Figures and 4 one of which was succesful and solved PeglnsertionSide rather fast, and the other demonstration set being the one where there demonstrations and show the results Fig. [O| We observe that the failed set demonstrations is indeed harder to learn from, although not impossible as 2/5 seeds can solve the task using that set of
success after training for 2M interactions. We then run seeds both sets of
15

---

Published as a conference paper at ICLR 2024
demonstrations while 3/5 seeds still has around success after 2M interactions. For the successful set of demonstrations, all seeds solved the task relatively quickly.
A.3 METAWORLD DEMO ABLATION
Fig. is another set of demo ablations on the Metaworld suite comparing RFCL with reverse cur-
riculum only and shows that when there are fewer demonstrations, the forward curriculum enables
more sample efficiency. The conclusions drawn from the main demo ablation on ManiSkill2 in Fig.
5 can also be drawn here.
Metawordd (1 demo)
Metawortd demos]
100
Success Rate
60
40
20
40
6.
80
1.00
20
40
50
80
100
Enviromment ste.0 (x:30F)
Environment (x10
Figure 11: Metaworld Results with varying number of demonstrations and comparing RFCL Reverse
riculum only. Results averaged over seeds with shaded area representing 95% Cls.
A.4 COMPARISON AGAINST PREVIOUS STATE RESET METHODS
Fig. 12 compares RFCL method with the state reset method proposed by Nair al. (2018)
the difficult Stack task, which tasks the agent to stack blocks into a tower. We label their method Nair al (2018). Our approach vastly outperforms their method by being able to solve the task in less than 10 million samples with just 20 demonstrations whereas Nair et al. (2018) requires more than 100 million samples using 100 demonstrations just get a 39% success
and has success even after 50 million samples (see their paper for their exact success rate curves). We note that while RFCL performs better, this comparison are different between papers (hence why this figure is in the appendix). For a more fair comparison we benchmarked the state reset strategy Nair al. (2018) uses Table T while keeping the rest of RFCL the same, which still shows that our per-demo reverse curriculum far more sample efficient compared to alternatives
little unfair as the core RL algorithms used
16

---

Published as a conference paper at ICLR 2024
Stack Cubes
100
50
Succes Rate
W
M
Environment steps (x100))
Figure 12: Stack 3 results of RFCL when given 20 random demonstrations. Results averaged over 5 seeds with shaded area representing 95% Cls.
B TASK DETAILS AND VISUALIZATIONS
TASK DETAILS
Below is the observation dimensions, action dimensions, and task horizons of all benchmarked tasks. For ManiSkil12 and Metaworld, we use the more robust and difficult metric of measuring success the end of episodes opposed allowing success and then failure (e.g. stopping an episode upon success being reached). Following recent past work on Adroit environments, we measure normalized score which is equal to the percent of timesteps in an episode where the agent is in success state. For all tasks we do not use any action repeat.
Task Adroit Door Adroit Pen Adroit Relocate
Observation Dimensions
Action Dimensions
Horizon
39
28
200
45
24
100
30
200
We consider the same 3 Adroit tasks and task horizons as our baselines JSRI. and RLPD. The controller used to control the Adroit hand is the default one in the benchmark, which controls each finger joint the hand position with delta actions.
Table 2: Adroit Tasks:
Task
Observation Dimensions
Action Dimensions
Horizon 200
All Tasks
39
We consider the most difficult 15 Metaworld tasks following the categorization by Seo al. 2022 The controller used control the robot end-effector the default one in the henchmark, which controls the position and gripper position of the end-effietor with delta actions.
Table 3: Metaworld Tasks:
B.2 TASK INITIAL START STATES AND CATEGORIZING DIFFICULTY
In Figures 13 to 18 we show visually what randomly sampled start states from the true start state distribution Siiit look like. We further note which of the environments have large amounts of ran- domization, which environments do not, and where the randomization comes from, in addition to briefly detailing where the complexity the task comes from that makes it difficult to solve. Our conclusion with regards to which environments
difficult in MetaWorld is corroborated by the categories used by Hansen et al. (2023 Seo et al. 2022).
Notably observe that environments that complex and have large amount of initial state randomization tend to need more compute or demonstrations to solve while those with less ran- domization get by with much less demonstrations. For example, Adroit Door has little to
17

---

Published as a conference paper at ICLR 2024
Task
Observation Dimensions
Action Dimensions
Horizon
PickCube StackCube
51
100
55
7
100
PegInsertionSide
50
100
PlugCharger
53
100
Table 4: Maniskill2 Tasks: We consider 4 tasks of various difficulty, with PickCube being the easiest (and solv- able by baselines when given enough demonstrations) and Siineettooniid/Pluhhagge being the most diffi- cult. PeglnsertionSide and PlugCharger have never been solved before under sparse rewards and few demon- strations. The controller used to control the robot end-effector is called pd-ce-delta -eseeddlta-ppse,, in the benchmark, which controls the pose and gripper position of the end-effector with delta actions.
pose, one of the defaults
randomization and easily solved by RFCL and the RLPD baseline with just demonstration. On the other end, ManiSkil12 PegInsertionSide and PlugCharger are complex and have lot of random- ization and can only be reliably solved by our method RFCL when given enough demonstrations and/or more compute. The randomization magnitude can stem from higher dimensions (e.g. more objects have randomized shape and positions) and/or stem from a wider range of random sampling.
Figure 13: ManiSkil12: PeglnsertionSide. High randomization: peg length, peg width, peg pose, block pose, hole in block position, robot position. Complex: Small clearance between peg and holc, requires learning grasp and insert precisely of different shapes from highly randomized poses
Figure 14: ManiSkill2: PlugCharger. High randomization: charger pose, receptacle pose, robot position. Complex: Very small clearance (much smaller than any other environment benchmarked in this paper) between the charger head and receptacle space, requires learning to grasp and insert very precisely
Figure 15: Adroit: AdroitHandRelocat. High randomization: ball position, goal position. Complex: Difficult dynamics with regards grasping ball with high nigh-dimenssonal robot
Figure 16: Adroit: AdroitHanddDor. Low randomization: door position. Simple: While robot hand high- dimensional, the door barely changes position
18

---

Published as a conference paper at ICLR 2024
Figure 17: MetaWorld: Pick Place. High randomization: block position, goal position. Complex: Highly randomized positions of goal and object, and object is small leaving little room for error.
Figure 18: MetaWorld: Assembly. Low randomization: cylinder position. Simple: Only the cylinder position randomized only a small range, other objects are initialized in the same place.
B.3 TASK REVERSE CURRICULUM
Below are samples of various states of the automatically generated per-demonstration reverse cur- riculum over the course of training for some of the more difficult tasks. Note that during the reverse curriculum stage of training, the reverse step size is small and the distance between adjacent stages of the curriculum are smaller than shown here.
Figure 19: ManiSkill2: PegInsertionSide
Figure 20: Adroit: Relocate
B.4 TASK SOLVES
For videos of example solves of these environments produced by our method RFCL, see our website https://reverrefoovaddd.gggiggii github.io/
B.5 BASELINE SELECTION METHODOLOGGY
As proposing learning crom-demoostrattons model-free RL algorithm that is more efficient in terms of samples and demonstrations, we select state-of-the-art model-free RL baselines that can demo data with code.
We select RLPD (Ball et al. 2023) as its the current state of the art method on the Adroit suite of tasks for learning from demos. It is the best baseline have well. As the RLPD paper shows out-perform standard SAC + adding demo data to the online buffer, we do not include SAC baseline.
19

---

Published as a conference paper at ICLR 2024
We also select Jump Start RL (Uchendu et al. 2023 Although it does not achieve state of the art results on Adroit, the concept reverse curiculum shares similarities with our proposed RFCL method.
We further select DAPG (Rajeswaran et al. 2018 and Cal-QL NNkammto et al. 2023) as they once achieved state-of-the art results on some tasks for learning from demos algorithms.
There are many on-policy algorithms including the popular PPO (Schulman et al. 201 baseline that could be benchmarked. However, it is well known that off- off-policy algorithms like SAC much more sample efficient than on-policy algorithms, and it would be unfair to compare against on-policy methods they would do poorly. The recent Sequential Dexterity (Chen al.) 2023) method in theory would be good to compare against as they are one of the few papers with code that use state resets, however they use PPO and rely on human engineering split a task into subtasks which is infeasible to do on the 20+ tasks we test on.
There are also model-based methods like MoDem (Hansen et al. 2023) that are very sample efficient thanks to the off off-policy setup and model-based approach. We do not compare against MoDem directly as they use world models, but we believe would interesting future work investigate reverse forward curriculums with world models for even better sample efficiency.
Finally, while we cite a number of past methods that use state resets, we currently are unable to benchmark any of them as the majority do not have code and/or have results tasks that are not open-sourced. We qualitatively comment on key differences in the related works section Sec. 2 and benchmark the uniform state reset strategy employed by Nair et al. (2018) and show that strategy performs worse compared to our per-demo reverse curriculum state reset approach in We further show results of RFCL on task that Nair et al. (2018) tests in Appendix|A.A A.4
20

---

Published as a conference paper at ICLR 2024
C HYPERPARAMETERS
Hyperparameter
Value
RL Hyperparameters (Soft-cctor-Critic)
Discount factor (7)
0.99 (Adroit and MetaWorld tasks)
0.99 (MS2 PegInsertionSide and Plugcharger)
0.9 (All other MS2 tasks)
Update to Data Ratio
Actor Update Frequency
20
Seed Steps
5,000
Replay Buffer Capacity
1,000,000
Batch Size
256
Number of Critics
10
Number of Sampled Critics
2
Polyak Averaging Coefficient (T)
0.005
Initial Temperature
1.0
Learnable Temperature
True
Total Interactions/ / Samples
1,000,000
Offline buffer to online buffer sample ratio
50:50
Networks and Optimization
Network Shape of Actor and Critic (MLP)
[256. 256, 256]
Layer Norm
Applied after every layer of critic only Rectified Linear Unit (RcLU)
Activation
Actor Learning Rate
3e-4
Critic Learning Rate
3e-4
Actor and Critic Optimizers
Adam
Environment and Data
Number of demonstrations
5/25 human teleoperated demonstrations for Adroit [1-20]/1000 motion planned demonstrations for ManiSkil12 5/00 scripted demonstrations for MetaWorld Sparse (+1 on success, otherwise)
out of total demonstrations available
Reward Function
Action Repeat
Episode Horizon
100 (ManiSkil12 and Adroit Pen) 200 (Adroit and Meta World Tasks) State
Observation Type
Reverse Curriculum
Reverse Step Size (6)
for Adroit and MetaWorld for ManiSkill 2
Por-trajectory Curriculum Buffer Size (m)
Demonstration Horizon to Episode Horizon Ratio (Φ)
Forward Curriculum
Forward Curriculum Score Threshold (w)
0.75
Forward Curriculum Buffer Size (k)
Forward Curriculum Score Temperature (8)
0.1
Forward Curriculum Ranking Strategy
Rank
Forward Curriculum Staleness Coefficient
Forward Curriculum Number Seeds/LLeees (n)
1000
Table 5: RFCL sample-efficient variation of hyperparameters. These are the used to generate all figures
results. Highlighted in blue indicates hyperparameters introduced by this paper, which are the automatic construction of reverse and forward curriculums.
The non-highlighted hyperparameters are standard ones used in Soft-Actor- Critic with Q-ensemble or Prioritized Level Replay (PLR). Furthermore, while a discount of 0.99 could be used for all tasks, we opt to use 0.9 for some ManiSkil12 tasks as it seems neither method nor any of the baselines can solve the tasks with a discount of 0.99 easily in 1M interactions when given significant amounts of demonstration data. Moreover, while it is strange only I Adroit task (Pen) has smaller horizon of 100, recent past work (Uchendu et al. 2023 Ball et al, 2023) use this specific horizon so we copy them for fair comparison.
21

---

Published as a conference paper at ICLR 2024
Finally, for just the PlugCharger environment, make a few modifications to make it work rela- tively more efficiently. We use a discount of 0.95, batch size of 512 to improve sample efficiency little. Importantly, we rescale the action space to be 125% the magnitude of the actions in the set of demonstrations used during training as the demonstration actions of PlugCharger on average are very small in magnitude, suggesting the full action space of the environment is unrealistic and pose great difficulty to exploration. This is fairly reasonable method to make the problem easier without relying on heavy heuristics, just introducing some bias about the magnitude of actions. We observe this rescaling not only accelerates learning but produces more reasonable policies that take smaller actions and are likely easier transfer to real. We only use the action rescaling for PlugCharger tests, leaving all other environments they were, although we believe this would likely accelerate learning for other environments requiring high precision like PegInsertionSide.
D WALL TIME EFFICIENT RESULTS
We note that while it standard in the field of robot learning and reinforcement learning to compare performance against the number of environment samples, for practical purposes it is also important to have fast wall-time performance. This is made more critical if onc's direction of research is towards sim2real of which then environment sample time is generally fast. With fast environment sample times, it is more wall-time optimal to increase the number of environment samples and lower the relative number of gradient updates, which is done with our wall time efficient set of hyperparameters. The only changes are to use a much smaller update to data ratio of 0.5 and increase actor update frequency to 1, essentially for e-g. every 32 environment steps update the critic and actor 16 times each. Moreover, the fast hyperparameters run 8 parallel environments with each taking 4 steps faster if 32 parallel environments are used.
time, after which then 16 gradient updates are performed. Training can go even
We did not heavily tune these hyperparameters since they were already quite fast, but we note that depending on environment speed, it may be optimal to tune the update to data ratio accordingly.
Time in Ta reported in GPU minutes representing the time until the running average success rate of the last 5 evaluations reaches 90%. Experiments all ran on RTX 2080 GPU. This is a very loose evaluation as it includes the non-trivial amount of time spent running test evaluation in addition to actual training, but it's clear that simply tuning few hyperparameters can easily improve training time.
Task
Wall Time Efficient Hyperparams Sample Efficient Hyperparams
Metaworld Soccer Metaworld Stick Pull Metaworld Pick Place
47min
143min
38min
104min
39min
180min
Table 6: Training time of select environments using different hyperparameters.
22


Reviews:
[{"review_id": "GZbE45kW8v", "reviewer": "Reviewer_UCiq", "summary": "This paper deals with curriculum learning in cases where there are only a small number of demonstrations available. The proposed method specifically design the curriculum generation as two stages: one along the demonstration paths, another explore around demonstration and eventually cover the entire space. The experiments are performed on several learning from demonstration (LfD) benchmarks.", "questions": "My questions are written in previous \"Weakness\" section. The authors can respond to the concerns/question written there.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- The presentation of this paper is good. \n- The experiment results are strong compared to multiple related baselines.", "weaknesses": "There are several issues/concerns on the method and experiments:\n\nMethod:\n\n- The most important issue is the design of curriculum. Basically, the author tried to separate the curriculum of reset states into two stages: in the first one the reset states are along the demonstrated states, and in the second one the reset states gradually move away from the demonstrated states. This does not make sense. Why not just combine the two stages into one, i.e. a curriculum that includes explorations of reset/initial states along the demonstration and also away from demonstrated states? The reset states need to cover the entire space in the end anyway. I don't see any reason for a two-stage design to make sense. Conceptually, reverse curriculum and \"forward curriculum\" are almost the same thing, with the latter has a difference of weighting on the exploration area.\n\n- What if there is no demonstration? Or intentionally forget about demonstration but just design curriculum that directly moves the reset/initial states away from goal? If this paper's assumption is that in the end, the initial/reset states should be able to cover the entire space anyway, then whether there is demonstrations provided should not matter: all starting states has to be explored sooner or later. I didn't see an ablation experiment to compare against the setting where no demonstration is available.\n\nExperiment:\n\n- As mentioned, it would be great to show the results under the setting where no demonstration is available, or the \"forward curriculum\"-only case. I understand the Maze task in Figure 6 tries to show this. But the experiment in Figure 6 is wrong: it does not show forward-curriculum-only is worse than reverse+forward curriculum. The reason it is worse is because forward-curriculum-only experiment mistakenly messed up with exploration (it should only explore in unexplored area, not around already explored demonstration area on blue lines). The explanation in the last two sentences of Section 5.2 is wrong too imo.\n\n- In Figure 5, do all experiments have the same reset/initial state distribution (cover all possible states) at the end of their curricula? If so, it's so hard to understand why the more demonstration there is, the faster it can be trained.\n\n- It seems that RFCL without \"forward curriculum\" works almost the same as RFCL with \"forward curriculum\" in most tasks (Fig 5). This again questions the necessity of having a second stage, as mentioned earlier. Without the two-stage setting, the novelty of this work is negatively impacted.\n\n- I'm not sure if the tasks selected in this paper are suitable for the proposed method:\n   - If the robot arms are moved by a positional controller, then the initial/reset position/states of the robot arm does not matter, because the positional controller should guide the robot arm/end effector converge to the goal position anyway regardless of initial states. Not much exploration is needed. So if this is the case, then the selected task cannot fully evaluate the potential of the proposed method.\n   - If the robot arms are not moved by a positional controller, then why not do so?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PLz4PMSYGu", "reviewer": "Reviewer_Rzng", "summary": "In this paper the authors show the benefits of first using a reverse curriculum on demonstrations (of simulated RL tasks with sparse rewards) followed by a forward curriculum that expands the set of initial states (from which the agent can achieve the task). In many different simulated RL tasks, it is shown that the method can achieve quite significant boosts in sample efficiency, and can sometimes succeed in tasks which other methods completely fail. Ablation studies show that the method is robust to the number of demonstrations and that the reverse-forward curriculum seems to be the best among various other choices.", "questions": "* Please do not use the word 'extreme', you used it several places throughout the paper.\n\n* \"In practice, the observations used by the policy πθ may be different from the actual environment state but for simplicity in this\npaper state also refers to observation.\"\n> So you don't consider noisy feedback or POMDPs? This should be mentioned clearly in the introduction. As the method shows significant improvement in learning curves, it is vital to indicate when/where we expect them to hold and where they would fail.\n\n* \"In both stages, we use the off-policy algorithm Soft Actor Critic (Haarnoja et al., 2018) with a Q-ensemble (Chen et al., 2021b), \"\n> What happens if you use another RL algorithm? Does it make a big difference? Which other methods, competitive to SAC, could you use?\n\n* Minor comment: \" As a result, a curriculum for each demonstration is necessary as opposed\nto a curriculum constructed from all demonstrations as done in prior work in order to ensure noisy\ninformation arising from the multi-modality of demonstrations do not impact the reverse curriculum\nof each demonstration as much.\" -> Too long sentence.\n\n* You use \\phi before you introduce it, and I didn't get what it's supposed to mean?\n\n* \"In this manner, initial states that sometimes receive\nreturn are prioritized the most, then initial states that receive no return, then initial states that are\nconsistently receiving return.\" -> Not a very clear sentence, do you want to use 'than' instead?\n\n* Would be nice to discuss why the methods compared against were chosen out of all the possible RL algorithms out there (maybe in an appendix?) Would the others be unsuitable for the task (e.g. on-policy, not suitable for demonstrations etc.)\n\n==== POST-REBUTTAL ====\n* My score remains the same, I think this paper should be accepted, as it has good results and detailed ablations. \n* Assumptions/limitations of the method (e.g. full observability, restricted to simulations, requires demonstrations) are mentioned throughout the paper and is not a deal-breaker. As mentioned in the rebuttal, just improving simulation efficiency is also a contribution.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper convincingly shows that their method and their particular choice of curriculum leads to significantly better performance in many different RL tasks with sparse reward structure. \n\n* Ablation studies are well done and cover significantly the possible variations.\n\n* Figures captions and plots are well done, as well as the visualizations in the website.", "weaknesses": "* The paper could benefit from an algorithmic summary. Algorithmic decisions not summarized succinctly by equations or in algorithmic form, but by verbose descriptions make the method look more 'alchemical' than it need be. e.q. \n\n\"Define q to be the fraction\nof episodes out of the last k episodes that receive nonzero return starting from a sampled initial state\nsi,init. If q is 0, then assign a score of 2 to si,init. If 0 < q < ω for a threshold 0 < ω < 1, assign\na score of 3. If q ≥ ω, assign a score of 1.\" \n\nThe actual numbers chosen detract from the concept of rejecting samples based on e.g. the expected return of exploration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iQwFZTX2n2", "reviewer": "Reviewer_gfCC", "summary": "In this study, a novel methodology is proposed, combining reverse and forward curricula. The authors suggest resetting from demonstrations to effectively perform exploration when access to a limited number of demonstrations is available. The reverse curriculum starts from positions backward from the goal point using demonstrations, while the forward curriculum is executed through score-based prioritizing, allowing the starting point to have an appropriate level of difficulty. As a result, it shows improved performance compared to baseline methods that leverage demonstrations.", "questions": "- In Figure 6, it seems that 'reverse' and 'forward' are only applicable to 'reverse-forward,' and not to 'none' and 'forward only,' which might be confusing.\n- Which algorithms among the compared baselines require the State Reset assumption?\n- If possible, could you please explain the reason for the initial high performance of RLPD in the stick pull task in Figure 4, followed by a drop?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method, combining reverse-forward curriculum approach is novel.\n\n- The learning curves presented in figures 3 and 4 demonstrate improved performance compared to the baselines, and the figure in 5 suggests that using both the reverse and forward curriculum with a limited number of demonstrations is beneficial.", "weaknesses": "- The assumption of resetting from demonstrations is a strong one and only applicable in simulation environments. While this study proposes an efficient way to leverage one or more demonstrations through the reverse curriculum, I still consider it to be a strong assumption.\n\n- The study may appear to be not significantly different from consecutively performing Jump-Start RL and PLR.\n\n- While it compares to various baselines, many of them seem to be algorithms that do not assume state resets or use curricula.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)