Title: TRL: DISCRIMINATIVE HINTS FOR SCALABLE RE-VERSE CURRICULUM LEARNING

Abstract: Deep reinforcement learning algorithms have proven successful in a variety of domains. However, tasks with sparse rewards remain challenging when the state space is large. Goal-oriented tasks are among the most typical problems in this domain, where a reward can only be received when the final goal is accomplished. In this work, we propose a potential solution to such problems with the introduction of an experience-based tendency reward mechanism, which provides the agent with additional hints based on a discriminative learning on past experiences during an automated reverse curriculum. This mechanism not only provides dense additional learning signals on what states lead to success, but also allows the agent to retain only this tendency reward instead of the whole histories of experience during multi-phase curriculum learning. We extensively study the advantages of our method on the standard sparse reward domains like Maze and Super Mario Bros and show that our method performs more efficiently and robustly than prior approaches in tasks with long time horizons and large state space. In addition, we demonstrate that using an optional keyframe scheme with very small quantity of key states, our approach can solve difficult robot manipulation challenges directly from perception and sparse rewards. Keyframes Keyframes Keyframes Keyframes Keyframes Keyframes

1. INTRODUCTION:

Reinforcement learning (RL) aims to learn the optimal policy of a certain task by maximizing the cumulative reward acquired from the environment. Recently, deep reinforcement learning has enjoyed great successes in many domains with short-term and dense reward feedback (Mnih et al., 2016 ) (e.g. Atari games, TORCS). However, many real world problems are inherently based on sparse, binary rewards, where the agent needs to travel through a large number of states before a success (or failure) signal can be received. For instance, in the grasping task of a robotic arm, the most accurate task reward would be a binary reward only if the agent successfully picked up the the object or not. This kind of goal-oriented tasks with sparse rewards are considered the most difficult challenges in reinforcement learning. That is, the environment only provides a final success signal when the agent has accomplished the whole task. The search space of these tasks may exponentially expand as state chain extends, which adds to the difficulty of reaching the final goal with conventional reinforcement learning algorithms.

There have been multiple lines of approaches for tackling such sparse reward problems, including the ideas based on intrinsic motivation (Stadie et al., 2015; Bellemare et al., 2016; Houthooft et al., 2016) , hierarchical reinforcement learning (Florensa et al., 2017a; Kulkarni et al., 2016) , curriculum learning (Karpathy et al., 2012; Sukhbaatar et al., 2017) and experience-based off-policy learning (Andrychowicz et al., 2017) . Recently, a particularly promising approach is the idea of reverse curriculum generation (Florensa et al., 2017b) . By reversing the traditional reinforcement learning process and gradually expanding the set of start states from the goal to the starting point, this method does achieve substantial progress in some simulation tasks. However, the newly sampled start set mixes old and new states, which leads to inefficiency in training, as the agent spends a large amount of time reviewing rather than learning new skills. This problem aggravates in large state space tasks where the expansion could be really slow, and the storing of history data is also impracticable due to the memory limitation.

To solve goal-oriented tasks and avoid the drawbacks mentioned above, we propose a Tendency Reinforcement Learning (TRL) architecture which shapes a reward function with former experience and use it to stabilize and accelerate training. To achieve this, a discriminative reward (the "tendency") is shaped to output a judgment on the success tendency of each state, and the reward function for the agent is hybrid, which combines the final goal reward with the tendency hints. We define a set of start states as a phase, and continue extending it from the final goal, until it reaches where the task starts. In each phase, the tendency reward influences the agent's training. After training each phase, the tendency reward is updated using collected experience. This mutual promotion contributes to a dramatic improvement in training efficiency and no longer requires keeping all history data.

In this paper, we introduce three novel techniques: First, we propose a hybrid reward design that combines the final reward with tendency hints. Secondly, with the introduction of a phase administrator, the curriculum for each phase is automated, which increases the efficiency. Lastly, we develop an optional keyframe scheme, that is, our framework is also compatible with additional keyframes with no strict accuracy requirement, as long as they are effective at reducing irrelevant search space.

The major contribution of this work is that we present a reliable tendency reinforcement learning method that is capable of training agents to solve large state space tasks with only the final reward, taking raw pixels as perception. In our experiments, we apply and test the model in diverse domains, including a maze game, Super Mario Bros and 3 simulated tasks (a robotic grasping task, a crane conveyance task and a pick-and-place challenge), where it proves more efficient and competitive than the prior works.

2. RELATED WORK:

Recently, great progress has been made in RL by approximating the policy and value functions through neural networks (Mnih et al., 2013; 2015; Schulman et al., 2015; Mnih et al., 2016) . However, these RL methods often suffer from extremely slow training progress or fail to learn in sparse reward tasks. There have been several lines of approaches to tackling such problems, namely curriculum learning, hierarchical RL, intrinsic motivation and demonstration-augmented RL, each with its own constraints and assumptions.

Curriculum learning works by deconstructing a difficult, long-horizon learning task into a sequence of easy to hard tasks (Karpathy et al., 2012) . While classic curriculum is often defined manually, recently in RL a number of work has shown promising results automated curriculum generation (Pinto et al., 2017; Sukhbaatar et al., 2017; Wulfe & Hershey, 2017; Florensa et al., 2017b) . Our work is closest to this line of work. In particular, reverse curriculum generation (Florensa et al., 2017a) has proposed a very similar curriculum generation through exploration backwards from the goal states, and has shown promising results in solving a variety of sparse reward problems. However, the critical difference of our approach from theirs is that we propose the use of the tendency reward trained discriminatively to summarize the past histories of successful and unsuccessful states, allowing our algorithm not to store the whole history of starting sets and further provide a dense shaping reward. Empirically our design choice is found to scale to larger state tasks with longer planning horizon better than this prior approach.

Hierarchical RL and intrinsic motivations are two other approaches for tackling the sparse reward problems. Hierarchical RL (Sutton et al., 1999; Kulkarni et al., 2016) learns to plan using a hierarchical of low-level policies, often directly provided to the high-level planner, to drastically reduce the search dimension of the problem, while intrinsic motivation often provides additional dense rewards in terms of novelty (Stadie et al., 2015; Houthooft et al., 2016; Bellemare et al., 2016) or some notion of empowerment (Mohamed & Rezende, 2015) to improve the exploration under no rewards. However, both approaches assume forward-based exploration to solve the sparse reward task, which becomes increasingly difficult as the dimensionality and the time horizon increase, unless strong structured prior is provided in HRL. Our method, with the similar assumptions as in reverse curriculum learning, allows much more efficient learning and exploration.

Lastly, there are a number of work that utilizes demonstrations to tackle such problems. Inverse reinforcement learning infers the underlying reward function from expert demonstrations (Ng et al., 2000; Abbeel & Ng, 2004; Ziebart et al., 2008) , and recent work has used such reward function as potential-based shaping reward to accelerate learning (Brys et al., 2015; Harutyunyan et al., 2015) . Another line of work uses demonstrations that consist of keyframes to do Keyframe-based Learning (Akgun et al., 2012) . These methods, however, make strong assumptions about the quality of demonstrations they receive. By contrast, our optional keyframe scheme only requires a small number of key states instead of several full demonstration trajectories, and is robust to noises. Specifically, for a challenging robotic manipulation task from perception, we only required 5 key states to solve the task. 

3. PRELIMINARIES AND ASSUMPTIONS:



3.1. PRELIMINARIES:

We revisit the conventional reinforcement learning setting where an agent interacts with the environment Env sequentially. At each time step t, the agent receives an observation O t and chooses an action a t from a set of action space A according to its policy π. Then the environment will give the agent the next state s t+1 and a reward r t . The goal of the agent is to maximize the expected return from each state s t through accumulating the reward by R t = ∞ k=0 γ k r t+k .

3.2. ASSUMPTIONS:

The tendency reinforcement learning model is based on three assumptions that follows (Florensa et al., 2017b) and hold true for a variety of practical learning problems. Assumption 1. We can arbitrarily reset the agent into any start state s ∈ P i .

Assumption 2. At least one state s g is provided such that s g ∈ S g . Assumption 3. The Markov Chain induced by taking uniformly sampled random actions has a communicating class 1 including all start states in P task and the given goal state s g . Assumption 1 can be satisfied in many robotic manipulation problems by storing the start states in P i with low-dimensional data (e.g. angle of joints and velocity of motors). Acquiring these data and sending commands to let robotic apply these settings are widely provided by standard robotic APIs, and such functions are similar in game engines. This assumption enables the agent to train from recommended start states, which proves to increase learning efficiency (Kakade & Langford, 2002) . Combining Assumption 1 with Assumption 2, we are able to reset the state to s g . Assumption 3 ensures that there exists a trajectory formed from a sequence of actions from any start state in P task to s g , and vice versa. This assumption can be satisfied by many robotic manipulation problems and games, as long as there are no major irreversibilities in the system. Symbols used in our paper are listed in table 1 for reference.

4. THE MODEL:

Naïve applications of conventional RL tend to fail in goal-oriented tasks with a large state space and sparse rewards. However, this kind of tasks are common in real world. TRL is able to tackle them by reducing aimless searching with the application of 1) a discriminative tendency reward shaping that utilizes the agent's former experience; 2) a phase administrator that assigns customized curriculum to the agent; 3) a keyframe scheme that shrinks search space and accelerates learning in large state space multistage tasks (optional).

Our key idea is to let the agent utilize previous experiences through a tendency classifier trained on history trajectories in a supervised manner (Sec. 4.1). We define each phase as a set of start states. In training, curriculum are automatically adjusted by a phase administrator (Sec. 4.2). In each phase, the agent learns from rewards produced by the tendency classifier, and trains the classifier with new experiences. The overview of training framework is shown in Fig. 1 and the corresponding algorithm is sketched in Alg. 1. 

Environment:



4.1. TENDENCY REWARD:

The tendency reward in our model is a binary classifier (e.g. SVM, NN) that takes an observation O as input and outputs a judgment on whether that state leads to the final success. We name it tendency reward mainly because it provides slight guiding hints to speed up the training process in each phase and those large state space tasks thus become trainable.

In training, each time when the agent steps onto a state, it would refer to the tendency reward for a judgment. If the state is evaluated as "favorable", the agent would receive a positive tendency hint as a reward. Otherwise, a negative reward would be issued. To prevent the agent from stalling at a favorable state for unlimited tendency rewards, we defined that such revisited states do not provide any reward, and applied a Gaussian kernel to detect them. The kernel is used with d most recent frames in a bid to detect repeated states, as is shown in Eq. (2):

EQUATION

We assume that s now is a revisited state if ρ is larger than 0.9. The parameter d is 20% of the max steps of each environment and σ = 1 & δ = 0 are constants in our experiments.

The binary tendency classifier trains on both success and failure trajectories and we use logistic loss as the classifier loss. The hybrid reward is defined in Eq. (3).

EQUATION

where λ is a scale factor (10 -3 in our experiments) based on the following hypothesis: a positive reward received from the environment should be more important than the output of T (•). Since the tendency hints are very subtle compared to the final reward, the agent's desire to achieve the final goal won't be interfered with.

4.2. PHASE ADMINISTRATOR:

Each time when the agent has made some progress in P current (being able to accomplish the task starting from 80% of start set), the x current for the generation of P current+1 will be adjusted by the phase administrator based on the agent's performance.

During the i th phase generation process, the agent start from P i and randomly take x i steps. The newly reached states would then be included into P i+1 if the agent's success rate on that state is between a lower-bound α and an upper-bound β, which guarantee the new states to a proper difficulty (α is 0.1 and β is 0.9 in our experiments).

In each phase generation, a reasonable x i is required. Large x i means less phases, while it might increase training iterations in P i+1 . Therefore, in order to strike a balance, the phase administrator adjusts x i according to the number of training iterations N i in P i . This adjustment is achieved with a sigmoid function shown in equation ( 4).

EQUATION

)

where N is the average of N 1 , N 2 , . . . , N i-1 .

The pseudo code of the phase administrator is sketched in Algorithm 2 in Appendix. B.

4.3. OPTIONAL KEYFRAME SCHEME FOR EXTREME LARGE STATE SPACE TASK:

In tasks with a vast state space, it is usually more difficult for P to reach any state s nearby P task . Those domains are difficult for both prior approaches and tendency RL. In those cases, we show that by providing just a few keyframes, our tendency RL algorithm can take advantage of them to shrink the search space. Ideally, the keyframes should be precise and indicate important states for the task, and in practice fine hand-engineering is often involved. However, for TRL, several rough key states would be sufficient. The agent still extends its phases with a regularly, while after each extension, it additionally checks whether the newly generated phase has covered some states nearby a certain key state (by a Gaussian kernel approximation function as is described in Eq. ( 5)). If so, the next phase would be directly sampled from that key state rather than generated form P current . In this way, the search space can decrease remarkably.

EQUATION

If J is larger than 30% of K, P current is believed to cover some states around s ki . Then the next phase is sampled nearby s ki rather than extended from P current . Pseudo code is shown in Alg. 3 in Appendix. B.

Note that our principle is distinct from LfD (learn from demonstration) methods in that the agent is not eager to follow the keyframes and doesn't get additional reward from them. In fact, these small number of keyframes are only used to shrink the search space, and our method also shows robustness to imperfect keyframe settings in experiment where we deliberately mix misleading states into the keyframes, and found that the influence is not significant (Sec. 5.3).

5. EXPERIMENT:

To evaluate our model, we design five experiments2 with different levels of difficulties (Fig. 2 ). In all of these experiments, we apply A3C (Mnih et al., 2016) as the RL solver. The details of environments and network are elaborated in Appendix. C and D. We first test the efficiency and validity of our model. After testifying the correctness of the overall model, we visualize the distribution of tendency hints and the starting sets of phases. Then we focus on the optional keyframe scheme, verify its compatibility to our method and prove our method's robustness to keyframes of different qualities. Finally, we apply our model to long-term manipulation challenges with large state spaces to show our advantages.

5.1. VALIDITY AND EFFICIENCY OF TRL:

The training of the 40 × 40 maze task takes around 35000 training steps (1 hour and 40 minutes) with 6 threads on GPU. Then the agent can reach the exit from the start point of the maze with a 9 × 9 observation around it. To investigate the influence of tendency reward on training, we conduct an experiment comparing the training effect under three conditions: using tendency reward without history phases, using history phases without tendency reward and using neither tendency reward nor history phases. This is not only an ablation experiment but also a comparison with prior work, since the second condition is basically the reverse curriculum generation learning method (Florensa et al., 2017b) . The result shows that our model performs well (Fig. 3 ). We also show the improvement when using phase administrator which let the curriculum automated (Fig. 4 ). 

5.2. INSPECTING THE LEARNING RESULTS:

The agent takes nearly 40000 training steps (around 21 hours) with 8 threads on GPU to learn how to complete the level 1-1 of Super Mario Bros. The policy learned by the agent is shown in Appendix. E.1.2. In order to find out the influence of the tendency reward, we traverse the whole states of some parts of the game and show both of the positive and negative tendency hints (Fig. 5 ). As we can see in these figures, our agent has acquired the ability to find enemies nearby and dodge or even kill them by jumping up to their heads. Besides, it also recognizes the danger of the pits and can choose the right time to jump over them. These figures show the guiding effect of the tendency reward, which does a great help for the agent when it is far away from the final goal. We also visualize parts of the phases in Mario, which effectively lead the agent to constantly learn new states farther from goal state (Fig. 5 ).

5.3. COMPATIBILITY AND ROBUSTNESS TO OPTIONAL KEYFRAMES:

We then enlarge the Maze to 100 × 100 also with a 9 × 9 observation. With 8 key states as the keyframes, the training takes around 50000 training steps (7 hours and 14 minutes). We also test our agent's robustness to imperfect keyframes, related results can be referred in Appendix. E.3.

We also compare our model with two LfD methods: data-efficient RL (Popov et al., 2017) and potential-based reward shaping (PBRS) from demonstration (Brys et al., 2015) . The former resets the agent randomly to someplace on an expert trajectory while the latter uses an artificial potential reward shaped based on demonstrations. We test them with both reasonable and unreasonable keyframes / demonstrations. As is shown in Fig. 6 , data-efficient RL is quite sensitive to the demonstration data and fluctuates constantly facing unreasonable demonstrations. PBRS from demonstration can achieve similar performance to TRL, but requires a lot more human efforts of hand-engineering its reward function (otherwise it just wouldn't work). Different from these methods, TRL does not need expert demonstration (10 labeled keyframes are sufficient) and requires no further human elaboration. Also, the training of TRL is the most stable of three all.

5.4. SUCCESS IN LARGE STATE SPACE PROBLEMS:

The following experiments mainly focus on illustrating the practicality of our model on solving difficult, real-world robotic manipulation problems with minimal assumptions. When confronted with long-term challenges with large state space, to avoid wasting time on unnecessary states, our keyframe scheme is applied to shrink the search space. In the three experiments, we add less than 15 keyframes to assist training (5 in robotic arm, 5 in conveyance challenge of the crane and 14 in pick-and-place task), and the results are quite satisfying.

5.4.1. GRASPING TASK:

The grasping task on robotic arm simulation is a typical large state space problem, where the gripper can move anywhere within the range of the arm. In our design, a camera is fastened to the forepart of the gripper and provides a 64 × 64 depth image as the agent's observation. We intuitively design 5 key states and train the whole task for 33360 iterations (4 hours 19 minutes) with 12 threads on GPU, and it turns out that our model can solve the task reliably.

To further investigate the relationship between training efficiency and scale of keyframes, we conduct the same experiment with 5, 8, 11, 13, 15, 18 key points respectively, as is shown in Appendix. E.4. The conveyance challenge is also a 3D manipulation task with a larger state space. The agent should pick up the ball, take it to correct basket, and drop it right into that basket to receive a positive reward. The camera is also fixed on the gripper that produces 64 × 64 RGB images as input. Our model is trained for 99240 iterations (5 hours 2 minutes) with 12 threads on GPU.

5.4.2. CONVEYANCE CHALLENGE:

Since this task involves multiple operations and has a large state space, we roughly add five key states to help the agent learn different skills, as is shown in Appendix. E.5.

With the intention of finding out the distribution of "favorable" and "adverse" states given by the tendency reward, we make the agent travel through the whole environment at an appropriate height under two conditions: 1) the gripper jaw is open with the target ball fixed on the center of plane; 2) the ball is grasped by the gripper and moves with it. Then we record the tendency reward at each position (Fig. 7 ).

As expected, areas close to the target basket are assigned the highest tendency hints, which is quite natural as these states are reached before of after a successful drop. On the other hand, locations above those incorrect baskets tend to incur negative tendency hints. More details can be found in Appendix. E.5.

5.4.3. RECURRENT PICK-AND-PLACE CHALLENGE:

Of all the three experiments in robotic manipulation, this challenge has the largest state space due to endless combinations of the agent's location and orientation and enormous possible positions of plates. This time, the agent observes through a panorama camera that provides 160 × 32 first-person point of view.

The whole task can be divided into several identical pick-and-place operations that resemble the conveyance challenge. We provide 14 intuitively designed keyframes to accelerate training (Appendix. E.6). However, these keyframes are not quite sufficient in such a huge state space, and the agent shows confusion from time to time.

Our phase administrator plays a significant role in phase arrangement, and actually makes up for the lack of expert data. Since the agent performs well within each pick-and-place operation, new phases are more concentrated on the intervals between operations.

6. DISCUSSION AND FUTURE WORK:

We develop a tendency reinforcement learning algorithm that resolves complicated goal-oriented tasks, and evaluate it with multiple experiments based on raw perceptions. Our method consists of two core components: a binary tendency classifier that provides dense hints on whether a state leads to success and a phase administrator that generates a reverse curriculum. In five distinct environments, we find that tendency RL method is efficient and stable enough to converge on a reliable policy within hours even facing intractable problems, and does not require history data which is impossible to acquire in large space domains.

One limitation of our current approach is that our model requires the environment to be capable of resetting to some arbitrary states, which is also a limitation of the reverse curriculum generation method (Florensa et al., 2017b) . A promising future direction is to additionally train reset controllers that automatically reset the system to the initial state following each episode, together with a corresponding forward controller (Han et al., 2015) .

In future work, our goal is to apply the tendency RL method to real world robotic manipulation tasks with perceptions. Furthermore, we plan to extend the binary tendency classifier into a predictor that outputs the expected success rate on current state, so that our model can also be applied to stochastic environments. Additionally, we intend to release a subset of our code and the trained models to facilitate further research and comparisons.

Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. 

A DEFINITIONS AND FORMULAE:

A.1 TERM DEFINITIONS

• Final goal: States that indicate the task is successfully finished (e.g. the exit in maze).

• Phase: A set of start states at a certain time period.

• Start state: A state where the agent starts for the final goal in a certain phase.

• Success state: The states leading to the final goal by current policy.

• Task start phase: States where the entire task begins, which is the final aim of phases extensions (e.g. the start point in maze).

• Keyframe: A small number of key states artificially designed to shrink the search space in large space tasks.

B OTHER ALGORITHMS 

C.4 CONVEYANCE CHALLENGE:

The fourth task is a conveyance challenge of a crane simulated by the physics engine MuJoCo. The agent can interact with a ball placed on a plane using a 3-finger gripper connected to the crane, with 8 actions available (6 for free movements and 2 for gripper manipulations). Surrounding the ball from a distance are some target baskets, among which one basket is randomly selected and colored green. The agent is able to observe through a camera fixed on the gripper that produces 64 × 64 RGB images, and only by dropping the ball into the green basket would the agent receive 1 point reward.

C.5 RECURRENT PICK-AND-PLACE:

The fifth task involves pick-and-place operations simulated in MuJoCo, which requires the agent to stack some plates into a neat pile. This time, the agent observes through a panorama camera that provides 160 × 32 first-person point of view. Fixed before the camera is a electromagnet capable of attracting one plate at a time, which is used for plate transfer. Available actions include turning left / right, stepping forward / backward, charging / releasing the electromagnet. The final goal is to gather and pile up all the plates into a straight stack, which is the only approach for the agent to obtain 1 point reward.

D NETWORK ARCHITECTURES D.1 MAZE:

The convolution neural network of the policy network has a first layer with 12 4 × 4 filters of stride 1, followed by a layer with 24 3 × 3 filters of stride 1, followed by a layer with 48 2 × 2 filters of stride 1. Then we add a LSTM layer with 384 units which followed by a fully connected layer with 96 hidden units.

D.2 SUPER MARIO BROS:

The convolution neural network of the policy network has a first layer with 48 4 × 4 filters of stride 1, followed by a layer with 48 3 × 3 filters of stride 1, followed by a layer with 96 2 × 2 filters of stride 1. Then we add a LSTM layer with 960 units which followed by a fully connected layer with 240 hidden units.

D.3 EXPERIMENTS WITH MUJOCO ENGINE:

The convolution neural network of the policy network has a first layer with 12 4×4 filters of stride 1, followed by a layer with 24 3 × 3 filters of stride 1, followed by a layer with 48 2 × 2 filters of stride 1, followed by a layer with 48 2 × 2 filters of stride 1. Then we add a LSTM layer with 1536 units which followed by a fully connected layer with 96 hidden units. Especially, in the pick-and-place challenge, there is an additional convolution layer with 96 2 × 2 filters of stride 1, and the LSTM layer contains 480 units. 

E SUPPLEMENTARY MATERIALS FOR EXPERIMENTS:



E.1.2 COMPARISON WITH CURIOSITY-DRIVEN METHOD:

In Super Mario Bros, we compare our model with another one trained based on curiosity-driven (Pathak et al., 2017) . We let the agents randomly move several steps at the beginning before run- ning the models, then compare the completion degree3 of two models. With the environment only providing a reward when Mario reaches the final goal, we find that our method obtains absolute advantage (Table 2 ). We also use the model trained in level 1-1 to run the level 1-2 and 1-3 of the game. Since the latter two levels are not trained and require new skills that the first level does not include (Fig. 9 ), both of the methods may suffer more failures. 

E.2 COMPARISON WITH MAXIMUM ENTROPY IRL:

We try the maximum entropy IRL (Ziebart et al., 2008) in the 100 × 100 maze task with more than two days' training and about 44G virtual memory used, but it still failed to form a feasible reward function. Since there are 10,000 states in the maze, the traditional IRL methods seem to suffer from the bottleneck of computing ability.

E.3 ROBUSTNESS TO IMPERFECT KEYFRAMES:

To test our model's robustness to imperfect keyframe settings, instead of arranging the keyframes onto the optimal trajectory, we set some of them onto a byway. We choose 8 special phases sampled from the misleading keyframes and show the policy the agent has learned in these phases. As is shown in Fig. 10 , in the sixth phase, the agent refuses to detour from the left side and finds a shortcut direct to the exit. A possible explanation is that our tendency reward only provide slight guiding hints which do not bother the agent's exploration for new policies. 

E.5 CONVEYANCE CHALLENGE DETAILS:

An interesting observation in conveyance challenge is, the areas with positive tendency hints are more concentrated and the boundary between "favorable" and "adverse" states is more explicit when the gripper jaw is open. This is because the states where the agent is not holding the ball are often related to grasping and dropping, which require more accuracy than just transferring the ball. This experiment also proves that our agent has superb robustness to imperfect keyframe settings. In our previous assumptions, the agent should rise to a proper height where all baskets were completely in view, before moving towards the correct basket (Fig. 12 ). However, it turns out that the agent learns to determine the target basket location at a lower height, where only portions of the baskets are in view. 

F OTHER DISCUSSIONS:

The regular reverse curriculum algorithm usually fails if there exists an irreversible process in the system. The irreversible process is defined as:

∃s, s ∈ S : (∃n > 0 : P (s n = s |s 0 = s) > 0) ∧ (∀n > 0 : P (s n = s|s 0 = s ) = 0) (6)

In such cases, the states s and s are not connected, and an agent starting from s will never reach s since that probability is 0. We define a absorbing state s a as a state that satisfies P (s a |s a ) = 1 ∧ ∀s ∈ S : s = s a -→ P (a|s) = 0 (7)

To be more generalized, we define a set S a of states to be a absorbing set if it satisfies

EQUATION

Consider a phase extension progress where P i+1 is generated from P i , if a large portion of states in P i+1 belong to some absorbing sets, it would be hard to for the new phase to include elements not in these absorbing sets. Therefore, the training is likely to be contained within these sets and no actual progress could be made since the phase extension makes no sense.

:

A communicating class is a maximal set of states C such that every pair of states in C communicates with each other. Two states communicate if there is a non-zero probability of reaching one from the other.

Demo videos of our experiments: https://sites.google.com/view/tendencyrl

The mean length they move divided by all length of current level

Figures and Tables:

Figure 1: The schematic illustration of TRL.

Figure 2: Five environments on which we evaluate our model

Figure 3: The training results of ablation experiments under three different conditions (mean to standard deviation with 5 random seeds).

Figure 4: The training results of 40 × 40 Maze task with 9 × 9 observations. Phase administrator proves to improve training efficiency.

Figure 5: The distribution of tendency hints and generated phases in parts of Super Mario Bros game. The purple background color represents positive hints while the blue ones indicate negative hints. The points in the figure indicate the starting points of Mario. Points with same color belong to a same phase.

Figure 6: Learning curves of our model, the data-efficient RL and PBRS from demonstration with keyframes / demonstrations of different qualities. Tendency RL is able to achieve a quite satisfactory performance without much human labor.

Figure 7: The tendency heat map at the height of 4cm, with different gripper poses in the conveyance task of a crane.

SUPER MARIO BROS DETAILS E.1.1 TRAINING RESULT The training result of Super Mario Bros is shown in Fig. 8.

Figure 8: The training result of the Super Mario Bros level 1-1. The red curve indicates the Mario's policy.

Figure 9: The cases where the agent needs new skills in level 1-2 and level 1-3.

Figure 10: The agent's policies in 8 special phases sampled from misleading key states. The purple points are collected by combining the testing results for 30 times. In the end the detouring keyframes are completely neglected by the agent.

Figure 11: Learning curve of the grasping task with different numbers of keyframes. The training efficiency is relatively higher when 13 keyframes are used. If the number of keyframes is too large or too small, training tends to become less efficient, but still stable.

Figure 12: Five key states for the conveyance challenge and skills required by these states.

Figure 13: Another tendency heat map of the conveyance task.

List of symbols

Table content: <table><tr><td colspan="2">Symbol Definition</td><td>Algorithm 1: TendencyRL</td></tr><tr><td colspan="2">Pi P task The task start phase. Set of start states in i th phase. K Number of states in each phase. O Observation acquired from environment.</td><td>input : Goal state s g , starting phase P 1 , task start phase P task . output: Trained policy π. Phase index i ← 1;</td></tr><tr><td>s s g</td><td>State that the agent arrives at. State where the agent reaches the goal.</td><td>if P 1 = ∅ then P 1 ← SampleN earby(s g , K);</td></tr><tr><td>r f π T (•) r(•) α β</td><td>Final goal reward(binary). Policy pursued by the agent. Function of tendency reward. Hybrid reward function combining T (•) and r f Lower bound of success rate in phase exten-sion. Upper bound of success rate in phase exten-</td><td>Train policy π beginning from P 1 with environmental reward; Collect trajectories &amp; train Tendency Reward T (•); while P i ∩ P task = ∅ do P i+1 ← P haseAdministrator(P i , N i );</td></tr><tr><td/><td>sion.</td><td>Train policy π beginning from P i+1</td></tr><tr><td>Ni</td><td>Number of training iterations in Pi.</td><td>with hybird Reward;</td></tr><tr><td colspan="2">xi xmax Max random walk steps in extension. Extension step size after the training of Pi. d Number of history frames in Gaussian kernel.</td><td>Update Tendency Reward; i ← i + 1; return π;</td></tr></table>

Current phase P i , number of training iterations in the i th phase N i . output: The next phase P i+1 . Calculate x i from N i with Equation (4); P i+1 ← ∅; while |P i+1 | < K do s old ← Sample (P i ); s new ← RandomW alk (s old , x i ); if α < SuccessRate (s new ) < β and s new / ∈ P i+1 then P i+1 ← P i+1 ∪ {s new };

Table content: <table><tr><td>B.1 PHASE ADMINISTRATOR</td></tr><tr><td>Algorithm 2: Phase Administrator</td></tr><tr><td>input :</td></tr></table>

Compare with the model trained based on curiosity-driven

Table content: <table><tr><td>Model</td><td>Completion degree in level 1-1(%)</td><td>Completion degree in level 1-2(%)</td><td>Completion degree in level 1-3(%)</td></tr><tr><td>ours</td><td>100</td><td>22</td><td>17</td></tr><tr><td>curiosity-driven</td><td>28</td><td>1</td><td>12</td></tr></table>

Peer review (iclr2018):
 This paper proposes a new method for reverse curriculum generation by gradually reseting the environment in phases and classifying states that tend to lead to success. It additionally proposes a mechanism for learning from human-provided "key states". The ideas in this paper are quite nice, but the paper has significant issues with regard to clarity and applicability to real-world problems: First, it is unclear is the proposed method requires access only high-dimensional observations (e.g. images) during training or if it additionally requires low-dimensional states (e.g. sufficient information to reset the environment). In most compelling problems settings where a low-dimensional representation that sufficiently explains the current state of the world is available during training, then it is also likely that one can write down a nicely shaped reward function using that state information during training, in which case, it makes sense to use such a reward function. This paper seems to require access to low-dimensional states, and specifically considers the sparse-reward setting, which seems contrived. Second, the paper states that the assumption "when resetting, the agent can be reset to any state" can be satisfied in problems such as real-world robotic manipulation. This is not correct. If the robot could autonomously reset to any state, then we would have largely solved robotic manipulation. Further, it is not always realistic to assume access to low-dimensional state information during training on a real robotic system (e.g. knowing the poses of all of the objects in the world). Third, the experiments section lacks crucial information needed to understand the experiments. What is the state, observation, and action space for each problem setting? What is the reward function for each problem setting? What reinforcement learning algorithm is used in combination with the curriculum and tendency rewards? Are the states and actions continuous or discrete? Without this information, it is difficult to judge the merit of the experimental setting. Fourth, the proposed method seems to lack motivation, making the proposed scheme seem a bit ad hoc. Could each of the components be motivated further through more discussion and/or ablative studies? Finally, the main text of the paper is substantially longer than the recommended page limit. It should be shortened by making the writing more concise. Beyond my feedback on clarity and significance, here are further pieces of feedback with regard to the technical content, experiments, and related work: I*m wondering -- can the reward shaping in Equation 2 be made to satisfy the property of not affecting the final policy? (see Ng et al. *09) If so, such a reward shaping would make the method even more appealing. How do the experiments in section 5.4 compare to prior methods and ablations? Without such a comparison, it is impossible to judge the performance of the proposed method and the level of difficulty of these tasks. At the very least, the paper should compare the performance of the proposed method to the performance a random policy. The paper is missing some highly relevant references. First, how does the proposed method compare to hindsight experience replay? [1] Second, learning from keyframes (rather than demonstrations) has been explored in the past [1]. It would be preferable to use the standard terminology of "keyframe". [1] Andrychowicz et al. Hindsight Experience Replay. 2017 [2] Akgun et al. Keyframe-based Learning from Demonstration. 2012 In summary, I think this paper has a number of promising ideas and experimental results, but given the significant issues in clarity and significance to real world problems, I don*t think that the current version of this paper is suitable for publication in ICLR. More minor feedback on clarity and correctness: - Abstract: "Deep RL algorithms have proven successful in a vast variety of domains" -- This is an overstatement. - The introduction should be more clear with regard to the assumptions. In particular, it would be helpful to see discussion of requiring human-provided keyframes. As is, it is unclear what is meant by "checkpoint scheme", which is not commonly used terminology. - "This kind of spare reward, goal-oriented tasks are considered the most difficult challenges" -- This is also an overstatement. Long-horizon tasks and high-dimensional observations are also very difficult. Also, the sentence is not grammatically correct. - "That is, environment" -> "That is, the environment" - In the last paragraph of the intro, it would be helpful to more clearly state what the experiments can accomplish. Can they handle raw pixel inputs? - "diverse domains" -> "diverse simulated domains" - "a robotic grasping task" -> "a simulated robotic grasping task" - There are a number of issues and errors in citations, e.g. missing the year, including the first name, incorrect reference - Assumption 1: .mathcal{P} has not yet been defined. - The last two paragraphs of section 3.2 are very difficult to understand without reading the method yet - "conventional RL solver tend" -> "conventional RL tend", also should mention sparse reward in this sentence. - Algorithm 1 and Figure 1 are not referenced in the text anywhere, and should be - The text in Figure 1 and Figure 3 is extremely small - The text in Figure 3 is extremely small