Scoring-Aggregating-Planning: Learning task-agnostic priors from interactions and sparse rewards for zero-shot generalization
Authors: Huazhe Xu
Boyuan Chen
Yang Gao
Trevor Darrell
Venue:  (2020)
Keywords: learning priors from exploration data
policy zero-shot generalization
reward shaping
model-based

Abstract: Humans can learn task-agnostic priors from interactive experience and utilize the priors for novel tasks without any finetuning. In this paper, we propose Scoring-Aggregating-Planning (SAP), a framework that can learn task-agnostic semantics and dynamics priors from arbitrary quality interactions as well as the corresponding sparse rewards and then plan on unseen tasks in zero-shot condition. The framework finds a neural score function for local regional state and action pairs that can be aggregated to approximate the quality of a full trajectory; moreover, a dynamics model that is learned with self-supervision can be incorporated for planning. Many of previous works that leverage interactive data for policy learning either need massive on-policy environmental interactions or assume access to expert data while we can achieve a similar goal with pure off-policy imperfect data. Instantiating our framework results in a generalizable policy to unseen tasks. Experiments demonstrate that the proposed method can outperform baseline methods on a wide range of applications including gridworld, robotics tasks and video games.

Under review as a conference paper at ICLR 2020
SCORIN--AGGREGATII--PAAANGG TASK- AGNOSTIC PRIORS FROM INTERACTIONS AND SPARSE REWARDS FOR ZERO-SHOT GENERALIZATION
LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Humans can learn task-agnostic priors from interactive experience and utilize the priors for novel tasks without any finetuning. this paper, we propose Scoring- perregating-Planniii (SAP), framework that can learn task-agnostic semantics and dynamics priors from arbitrary quality interactions with sparse reward and then plan on unseen tasks in zero-shot condition. The framework finds neural score function for local regional state and action pairs that can be aggregated approxi- mate the quality of full trajectory; moreover, dynamics model that is learned with self-supervision can be incorporated for planning. Many previous works that leverage interactive data for policy learning either need massive on-policy environmental interactions or assume access to expert data while we can achieve the similar goal with pure off-policy imperfect data. Instantiating our framework results in a generalizable policy to unseen tasks. Experiments demonstrate that the proposed method can outperform baseline methods on a wide range of applications including gridworld, robotics tasks and video games.
INTRODUCTION
While deep Reinforcement Learning (RL) methods have shown impressive performance on video games (Mnih et al., 2013) and robotics tasks (Schulman et al. 2015 Lillicrap et al.) 2015), they solve each problem kabula rasa. Hence, it will be hard for them generalize to new tasks without re-training even due to small changes. However, humans can quickly adapt their skills to a new task that requires similar priors e.g. physies, semantics, affordances to past experience. The priors can be learned from a spectrum of examples ranging from perfect demonstrative ones that accomplish certain tasks aimless exploration.
parameterized intelligent agent "Mario" which learns to move to the right in the upper level in Figure [ would fail transfer the priors from to the lower level in Figure Daure[andd and further play the game in the new level because change of configurations and background, e.g. different shapes of ladder, new fence. When inexperienced human player controlling the Mario to move it to the right in the upper level, it might take many trials for him/her to realize the falling to a pit and approaching the "koopa" Akoopa"(turtlae (turtle) from the left are harmful while standing on the top of the "koopa" ((trl())) is not. However, once learned, s/he can infer similar mechanisms in the lower level in Figure [ without additional trials because human have a variety of priors including the concept of object, similarity, semantics, affordance, etc (Gibson) 2014 Dubey et al., 2018). In this paper, we intend to teach machine agents to realize and utilize useful priors to generalize to new tasks without finetuning.
Toward addressing the generalization problem with learned priors, we follow the intuition that: (1) each trajectory in a video game or robotics task is composed of state-action pairs with object interactions (2) terminal rewards can be approximated by the aggregation of the scores for each state-action pairs. With those intuitions in mind, we summarize our proposed method in broad strokes. Given trajectory with terminal sparse reward, we first parameterize the score of action-local region pair with a convolutional neural network F and then aggregate the scores to approximate the final sparse reward. To further enable actionable agents to utilize the scores, neural dynamics model
Project page:

---

Under review as a conference paper at ICLR 2020
exploration
Stage
10
PPY 11
BES.
83
Terminal Reward
learning the priors
Action-state scores
Dynamics model
applying the priors
zero-shot generation World 2 Stage
Tenminal Rewerd
Figure I: Illustrative figure: An agent is learning priors from exploration data from World Stage 1 in Nintendo Super Mario Bros game. In this paper, the agent focuses on learning two types of priors: learning action-state preference score for local regions and dynamics model. The action-state scores on the middle left learns that approaching the "Koopa" from the left is undesirable while from the top desirable. On the middle right, dynamics model can be learned predict future state based on the current state and action. The agent can apply the priors to new task World 2 Stage to achieve reasonable policy with zero shot.
can be learned from the interaction data using self-supervision. We show that how an agent can take advantage the scoring function and the learned dynamics model with planning algorithms (Mayne et al. 2000). We adopt the sparse terminal reward setting because in most of the tasks, step-by-step rewards are hard to obtain while final evaluations for trajectories are relatively easy.
Readers may argue that learning dense score for every interaction step is reminiscent of Inverse Reinforcement Learning (Ng et al. 2000 Abbeel & Ng 2004). The distinctions between the proposed method and IRL are threefold: First, instead of learning reward function of state scoring function of a local state 8; and an action a, which is sufficiently rich in a physical environment generalize well. Second, with the scoring function in hand, model learned from passive data to obtain the actual policy in a model-based manner while IRL needs re-train agent that can be as data inefficient as model-free RL. However, IRL. have difficulty learning useful model because the expert demonstrations usually only cover small portion of the true dynamies. Third, we eliminate the assumption of expensive expert demonstrations with the cost of adding a relatively economical sparse reward in the end. This climination not only reduces the cost for data collection, but also includes more diverse data train a robust model.
learn
and experimentally
dynamics
The proposed scoring function, beyond being cost function for planning, can also be treated as an indicator of the existence objects, which affect the evaluation of trajectory. We empirically evaluate the scores for objects extracted in the context of human priors and hence find the potential of using our method as an unsupervised method for object discovery.
In this paper, we have three major contributions. First, we propose a framework that can learn task-agnostic priors that can generalize to novel tasks. Second, we incorporate a self-supervised learned dynamics model with scoring function learn useful policy. Third, we demonstrate the effectiveness the proposed method on a didactic grid- example, a well-known video game "Super Mario Bros" and robotics Blocked-Ress Reach environment and show method outperforms various baselines. Last but not least, we find that objects emerge from our method in an unsupervised manner, which could be useful to other visual tasks.
2

---

Under review as a conference paper at ICLR 2020
2 PRELIMINARIES
In this paper, we formulate each environment as an Markov Decision Process (MDP). We represent the MDP by tuple: (S,A, p, 7), where S is the state space and AA is the action space. MDP fully specified by state E S. A MDP evolves with an action a E A by a probability distribution p(s'ls, a). The MDP emits reward r(s, a) each step. Y (0, 1) the discount factor.
Reinforcement learning aims to learn conditional distribution over the action space given a state (-Is) that maximizes the discounted future rewards:
8
π
argmax
Σ
y (r(ss;a)))
(asls)) Pacep
The state transition probability p(s'|s, a) is treated unknown model-free RL problems. Model- based methods explicitly learn dynamics model MM-|s, a) that specifies the conditional distribution of the next state given the current state and action from environmental interactions.
With an environment model M, one can select an action to rollout with the model recurrently in order to maximize the discounted future reward. One method that approximately finds the optimal action is the Model Predictive Control (MPC) algorithm. It looks ahead for horizon of H steps and selects an action sequence that maximizes the discounted reward for the future steps:
argmax Σ r'r(s,,
where St+1 MMss, at). To simplify notation, here assume the environment deterministic and slightly abuse the notations such that M(5r, az) returns the next state instead of state distribution. We note that MPC can also use the ground truth environment dynamics p(-|s,
METHOD
3.1 PROBLEM FORMULATION
An intelligent agent should be able to learn priors from its past experiences and to generalize to related yet unseen tasks. To facilitate such goals, we formulate the problem follows:
1,2 ... N collected in A, p). Each т; trajectory and 7, = {(s,, a,)), 1,2 ... K₃- These trajectories are ramdom explorations//nteratioo the task by per step reward, which is the standard MDP setting, we propose to only evaluate the performance of each trajectory with terminal evaluation r(T) in E1 when task T is given.
The agent is first presented with bank of exploratory trajectories {Ti}.
a training environment E (S,A,))
with the environment. Instead of specifying
At test time, we would like the agent to perform the task T, using zero extra interaction with the new but related environment, represented by & A'p). We assume the test environment &g can have different state distribution and related but different dynamics from the experience environment E1 However, agent on task T by giving a single terminal reward r(T,)) per trajectory т; to its previous experiences. This reward is only used for evaluation and an agent can never utilize it while carrying off the task. The proposed formulation requires much less information and thus more realistic. In this paper, we focus locomotion tasks with object interactions, such as Super Mario running with other objects in presence, or Reacher robot acting with obstacles around.
assume the actions that agent could perform stay the same. We evaluate
3.2 THE SCORINGAAGGREGAATNG-P....
(SAP) FRAMEWORK
We propose the Berrin-Aggregating.. propose learn per step neural scoring function Fş that scores sub-region, subset of observation space that surrounds the agent. A local region score is simply sum over all the sub-regions that are in the local region. The local region scores can be aggregated along the trajectory to approximate
framework to solve this problem. As an overview, we
3

---

Under review as a conference paper at ICLR 2020
the terminal sparse reward. Meanwhile, dynamics model Ma is learned to approximate the true transition p(-|s, a) based on past experience Σg. After both the scoring function and the dynamics model are learned, perform Model Predictive Control algorithm to get the final policy the new environment & for the task T.
4
dynamics model
fret
&
agentt
sub-region
Subh Region Network
Ft
agent sur rounded by rogoregioss
A
action saladoor
S242
sub-region scores
abjective
local region score
sub-maiion scom
cvaluaaion
aggreasted
argmax over cimulaaive scores
Figure 2: An overview of SAP framework. (a) and (b) in the figure correspond to the Scoring, Aggregating part. (c) and (d) together describes the dynamics learning and planning. See Section 3.2 for details.
Scoring The per step sub-region scoring function can be described as Fg (S:),,), ar). Here 0 denotes parameters, W is a function that extracts local regions from states and is sub-region indicator in a metric space L. We note that the neural network's parameters are shared for every local region. A local region score then is > Fo (o( at). Intuitively, this function measures how well an action a performs the current task, based on a local region extracted from state s.
We presume local region can be extracted for scoring because in physical environment, an agent could only interact with the world within its sensing capabilities. From another perspective, can be seen decomposition of the sparse reward into the per-step rewards.
Specifically, on the metric space associated with the problem, we divide the local region around the agent into n sub-regions I. For example, in Cartesian coordinate system, we can divide each coordinate independently. For each I, we use a sub-region scoring network to produce a scoring table of size |A| x [,C. where [A] is the action dimension, and |C] denotes the number of possible relative positions around the agent. One entry selected from the table as the score for this sub-region / based the current action taken and the relative position of this sub-region to the agent.
Aggregating To approximate the terminal rewards and learn the scoring network, we aggregate the local region scores Diec Fo(((s)) ax) for each step into single aggregated score J, by aggregating function G:
(s;),
JT(T) = (
Fo Fo(Wi(S)), a:))
IEC
The aggregated score J are then fitted to the sparse terminal reward. In practice, G is chosen based on the form the final sparse terminal reward, e.g. max or sum function. the learning process, the Fo function is learned by back-propping errors between the terminal sparse reward and the predicted through the aggregation function. In this paper, we use &g loss that is: ming 1 (Jo(T) - r(7))²
Planning To solve the task, we propose to planning algorithms to find optimal action based on the learned scoring function and a learned dynamics model. As shown in the part (c) of Figure 2 we learn forward dynamics model Ms based on the exploratory data with supervised loss function. Specifically, we train a neural network that takes the action a,, state and output $,41, which is an estimate of 1 We use la loss the objective: minş (M a+) 8441
4

---

Under review as a conference paper at ICLR 2020
With the learned dynamics model and the scoring function, we solve optimization problem using the Model Predictive Control (MPC) algorithm to find the best trajectory for a task T in environment Eg. The objective of the optimization problem is to minimize -T)
Here r' is a H-step trajectory sampled based on MM starting from some state 8,, The actual algorithm can be stated as starting from the current state Si=i, we randomly select several action sequence up to length H. With the dynamics model, we can roll out the estimated states 9441.''. St, The learned scoring function and the aggregation function can give us an aggregated score for each of the action sequence. We select the action sequence that gives us the best aggregated score and execute the first action in the environment. And we repeat the procedure starting at the new state Sç.
4 RELATED WORK
Inverse Reinforcement Learning. The seminal work Ng et al. (2000) proposed inverse reinforce- ment learning (IRL). IRL aims to learn a reward function from a set of expert demonstrations. The original IRL is demonstrated on physical state representations, while recent work (Tucker et al., 2018 Rhinehart & Kitani 2017) has attempted to extend the work into visual states. Although IRL and SAP both learn functions from a set of off- off-policy data, they fundamentally study different problem - IRL learns a reward function, which can be used for a model-free RL algorithm, from expert demonstrations, while our method learns from exploratory data that is not necessarily related to any tasks. There are some works dealing with violation of the assumptions of IRL, such as inaccurate
perception of the state (Bogert & Doshi 2015; Wang et al. 2002 Bogert et al.] 2016 Choi & Kim 2011), or incomplete dynamics model (Syed & Schapire 2008 Bogert & Doshi 2015 Levine &
Abbeel 2014; Bagnell et al.) 2007, Ng 2004); however, IRL does not study the case when the dynamics model is purely learned and the demonstrations
suboptimal. Recent work (Xie et al., 2019) proposed to leverage failed demonstrations with model-free IRL. to perform grasping tasks; though sharing some intuition, our work is different because of the model-based nature.
Reward Shaping, Ng et al. 1999) studied the problem of reward shaping, i.e. how to change the form of the reward without affecting the optimal policy. The -agoriggggggggtiing part can also be thought as a novel form of reward shaping where reward functions are automatically learned. Most of the efforts in reward shaping require careful manual design (OpenAl 2018 Wu & Tian 2016).
A corpus of literature (Marthi 2007: Grzeš & Kudenko 2010; Marashi et al.) 2012) try to learn the reward shaping automatically. Marthi (2007) assumes that the state space can be abstracted, such that one can form an abstracted MDP, which can be solved exactly. Other automatic reward shaping methods, such as Gizze& & Kudenko (20T0): Marashi et al. (2012). try to build a graph on top of discretized states. However, the methods do not apply to the high-dimensional input such as image, while our SAP framework could. One recent work RUDDER (Arjona-Medde et al. 2018) utilizes an LSTM to decompose rewards into per-step rewards. This method is orthogonal and complementary to our framework.
RL with Sparse Rewards. When only sparse rewards are provided, RL agent suffers harder exploration problem. In the literature, there are mainly three categories of methods to deal with this problem for high dimensional tasks: (1) Unsupervised exploration strategies, such curiosity-drive exploration ( (Pathak et al. 2017] Burda et al. 201 Sa b|]b|pl (Schiidhuber 1991 Stadie et al., 2013 Achiam
& Sastry 2017). or count- based exploration (Tagg et al.) [017]] Strchl & Littman| 20088 Bellemare iet al.] [2166] Fu et al.) 2017 Ostrovski et al. 2017 Machado et al. 2018 Choshen et al. 2018), solve the sparse reward problem by more efficient explorations. (2) In condiiinit tasks, such as pushing an object to some random location, one can use Hindsight Experience Replay (Andrychowicz al., 2017) to learn from experiences with different goals. (3) More generally, one can define auxiliary tasks to learn a meaningful intermediate representations (Huang et al., 2209a Silver et al., 2018, Dosovitskiy & Koltun| 2016 Riedmiller et af.) 2018 Agarwal et al. 2019). Different from previous methods, approach this problem by learning scoring function for each timestep, based on the single terminal reward. This effectively converts the single terminal reward to a set of rich intermediate representations, on top of which
apply planning algorithms, such as MPC.
Model-Basee RL. In the planning part of our SAP framework, we train a dynamics model. i.e. under the umbrella of model-based algorithms (Sutton [191].. This idea has been widely studied in the area of robotics (Deisenroth et al., (2013] Deisenroth & Rasmussen, 201TT Morimoto & Atkeson, 2003 Deisenroth et al. 2011). The line of work uses a variety of methods to learn accurate dynamics
5

---

Under review as a conference paper at ICLR 2020
Exp trution
920
DARLA
Naive Human Phirrs
1200
Humna
SAP courst
22
1000
600
000
800
600
400
200
200
8
(a) Hidden Reward Gridworld Env
(b) Mario train world returns
(c) Mario test world returns
Figure 3: (a) The hidden reward gridworld example. The agent needs to learn the points of each object and collect as many points as possible a new world configuration. See Section [3.1 for more details. (b) & (c) The total returns of different methods on the Super Mario Bros. All the experiments are run with 50000 game steps on multiple episodes. Error bars are shown as 95% confidence interval except for the given interaction data. See Section 5.2. for details. Best view in color.
model ranging from Gaussian Process (Ko & Fox 2009), fime-varying linear models (Levine &
Koltun 2013 Lioutikov al. 2014) Xie et al. 2016), mixture of gaussian models (Khansari-Zadeh
& Billard (011) neural networks Hunt et al.) 1992 Tangkaratt et al. 2014 Kurutach et al. 2018
Chua et al. 2018: Luo et al.) 2018 Buckman et al., 2018). This paradigm has been applied to high
dimensional space, such simulated and real robotic applications (Watter et al. 2015 Finn et al.
2016b] Hafner et al., 2018), and Atari games (Kaiser et al., 2019, Weber et al.) 2017). Although
model-based RL has been extensively studied, none of the previous work has explored combining with learning the dense task-agnostic scores from sparse signals.
Zero-Shot Generalization and Priors Prior knowledge comes from previous experience including interaction with objects, etc. Recently, researchers have shown the importance of priors in playing video games (Dubey et al.) 2018). More works have also been done utilize visual priors such objects in many other domains such as robotics for generalization, etc. (Wang et al., 2019 Jang al., 2018 Devin al. 2018 Zhu et al. 2018, Du & Narasimhan 2019). Keramati et al. (2018); Li et al. (2019): Higgins et al. (2017) explicitly extended RL to handle object level learning. While our method does not explicitly model objects, we have shown that meaningful scores are learned for objects in our SAP framework, which explains why our method generalizes to new tasks without any finetuning. Other works (Sohn et al., 2018 Oh et al., 2017 try to learn compositional skills that can be transferred to new tasks, which is orthogonal and complementary to the proposed method.
EXPERIMENT
In this section, we would like to study how well our SAP framework performs compare other methods, and the roles of various components in the framework. We conduct experiments on three environments in different domains: didactic gridworld task, a video game "Super Mario Bros" (Kauten 2018) and robotics blocked reacher task (Huang et al. 2019b). Environment details, architectures, hyper-parameters are described thoroughly in Appendix.
5.1 DIDACTIC EXAMPLE: HIDDEN REWARD GRIDWORLD
In order investigate whether the proposed framework can learn meaningful scores and hence induce correct policy, we start with a simple didactic task Hidden Reward Gridworld where the environment matches the assumptions of our method. This environment will reveal what level can our method recover the per step scores. Figure P shows an illustrative example of the Hidden Reward Gridworld. In the grid world, there is an object at each location, where each type of object has some unknown number of points. The agent has already explored some trajectories, and only the sum of points is known by the end of each trajectory. It needs to leam the value of each object and collect much value as possible in a new environment with different object configurations. In our experiment, we an 8 x 8 grid, with 16 different types of objects, and each of them worth value of 0, 1 to 15

---

Under review as a conference paper at ICLR 2020
respectively. To make the task more challenging, instead of giving the identity of each object by object id, we generate a 16-dimensional noisy feature for each type object. The noisy feature representation of object mimics the output of perception network from visual inputs.
On this task, our method operates as follows. We use a two layer fully con- nected neural network predict per step score from the 16-dimensional feature. Those per step scores are aggregated by a sum opera- tor, and fitted to the training data. We opt to use the en- vironment dynamics model, because it can be perfectly learned by tabular function. As shown in Table 1 we find that neural network can fit the object value based on the features with an error 0.04 on a training environment and 0.34 on a new task even with the feature noise presence. To how well our method performs this task, we train two behavior cloning agents: an agent imitates the exploratory data (denoted as BC-random) and the other imitates the SAP behavior on the training environment (denoted as BC-SAP). As shown in Table | BC-random has far inferior performance. since it clones the exploration behavior, which does not maximize the value collected. BC-SAP performs as well SAP in the training environment but performs worse than SAP in the new environment. This shows that even allowed to clone the behavior of SAP, behavior cloning still do not generalize as well as our method in a new environment.
Table 1: returns. See See. 5.1 for analysis.
The average error for the learned object points and the total
World 1
World (new task)
error
returns
error
returns
SAP BC random BC - SAP
0.04
378.6
0.34
389.8
83.05
106.8
300.0
In this grid world environment, we have shown that SAP is capable of learning accurate scoring functions, even in the presence of object feature noises. We have also demonstrated that SAP would generalize better than alternative algorithms.
5.2 SAP ON SUPER MARIO BROS WITH SPARSE REWARDS
To evaluate our proposed algorithm in more realistic environment, we run SAP and few alternative methods in the Super Mario Bros environment. In this environment, we neither have a clear definition of the dense score for each step, nor the aggregation procedure. This environment also features high-dimensional visual observations, which is more challenging since we have a larger hypothesis space. The original game has 240 x 256 image input and discrete action space with 5 choices. We wrap the environment following Mnih et al. (2015) and what described in Appendix [A.2] finally, obtain a 84 x 84 size 4-frame stacked gray-scale stacked observation. The goal for an agent is to survive and go toward the right as far as possible. The environment returns how far the agent goes to the right at the end of a trajectory as the delayed terminal sparse reward.
We apply our SAP framework as follows. We first divide the local region around the agent into eight 12 by 12 pixel sub-regions based on relative position as illustrated in Figure. in the Appendix. Each sub-region is scored by CNN, which has final FC layer to output score matrix. The matrix has the shape dim(action) by dim(relative position), which are and 8 respectively. Then an action selector and sub-region selector jointly select row corresponding to the agent's action and the column corresponding to the relative position. The sum of all the sub-region scores forms the local region score. Then we minimize the & loss between the aggregated local region scores along the trajectory and the terminal reward. A dynamics model is also learned by training another CNN. The dynamics model takes a 30 by 30 size crop around the agent, the agent's location as well a one-hot action vector. Instead of outputting full generated image. we only predict the future location of the agent recursively. We avoid video predictive models because it suffers the blurry effect when predicting long term future (Lotter et al. 2016 Finn et al. 2016a). We plan with the learned scores and dynamics model with a standard MPC algorithm with random actions that looks ahead 10 steps.
5.2.1 COMPARISONS
We compare our method with the following methods:
7

---

Under review as a conference paper at ICLR 2020
Exploration Data Exploration Data is the data from which learn the scores, dynamics model and imitate. The data is collected from a suboptimal policy described in Appendix [A.2.5] The average reward on this dataset is baseline for all other methods. This is omitted in new tasks because only know the performance in the environment where the data is collected.
Behavioral Cloning Behavioral Cloning (BC) learns mapping from state to action the exploration data using supervised learning. We use cross-entropy loss for predicting the actions.
DARLA (Higgins et al., 2017) DARLA relies learning latent state representation that can be transferred from the training environments to the testing environment. It achieves this goal by obtaining a disentangled representation of the environment's generative factors before learning act. We use the latent representation as the observations for behavioral cloning agent.
Naive Human Priors Naive Human Priors method (NHP) incorporates model predictive control with predefined naive human priors, which is +1 score if the agent tries to move or jump toward the right and otherwise. NHP replaces the scoring aaggingggggrggtion step of our method by a manually defined prior. We note that it might be hard to design human priors in other tasks.
In Figure. BD the results show that the proposed SAP model outperforms all the baselines with large margin on the same level they are trained on. We believe that there are two major reasons for BC's unsatisfactory performance: 1. we only have access to the exploration data which suboptimal for the task. 2. When seeing rare events in the game (the exploration data hardly reach the ending part), it fails to generalize. SAP also outperforms DARLA on both training and generalization tasks. We believe it is because learning disentangled representations on the Mario games is hard, since observations can change dramatically in game settings. Comparing the SAP model with the NHP method, demonstrate the learned priors can empower agent with better understanding of the world thus stronger performance. We show qualitative results in the subsection 5.2.3 to validate that the learned priors contain meaningful scores that leads to better actions. SAP also outperforms all the baselines in an unseen level without any finetuning (Fig. which proves that it can generalize well.
5.2.2 ABLATIVE STUDIES
supomntion
4.42
Sayee leman frims
2000
577 hourd
390
Fake
3003
3003
300
3
3
(a) GT dynamics model on (b) GT dynamics model (c) GT dynamics model on (d) GT dynamics model & done on W2S1 (test)
W1S1 (train)
no done on WISI (train) W2S1 (lest)
Figure 4: Ablative results. (c) are SAP on WISI and W2S1 respectively with the groundtruth dynamics model. (b).(d) are similar to (c) but without the true "done" signal. Error bar is 95% confidence interval. We see that with perfect dynamics model the performance boosted for both model-based methods. However, even with a disabled "done" signal, SAP still works well while NHP performs significantly worse than before.
Towards understanding the effect of different components of SAP, we investigate the performance of an agent using learned scores on perfect dynamics model which shows the upper bound for improvement from better models. We further investigate how much the "done" signal from the perfect model help with the performance. Hence, we perform two main ablative experiments:
We apply the SAP method with a groundtruth dynamics. Note this setting is only feasible for video games and simulated environments; however, this gives an upper bound for the proposed method by continually improving the dynamics model. In Fig. 4b and Fig. 4 we find that with a perfect dynamics model, both NHP and SAP has performance boost on both the original task and novel task while SAP can still outperform baselines by a large margin. This suggests SAP can perform better if the dynamics model can be further improved.
Groundtruth Model:
8

---

Under review as a conference paper at ICLR 2020
We hypothesize that the "done" signal from the perfect model contributes to the superior performance because it naturally tells an MPC algorithm not to select a short trajectory reward at every step is positive (e.g. in the NHP). In Fig. 1p and Fig. 1 not provided, SAP still preserves the performance and outperforms all the baselines. However, we find NHP significantly dropped below the level of exploration data return which indicates the NHP heavily relies on the "done" signal.
No Done Signal:
see that if "done" signal
More ablation studies, such as planning horizon, visual representations in Appendix A.2.7
5.2.3 VISUALIZATION OF LEARNED SCORES AND ACTIONS
In this section, we qualitatively study the induced action by greedily maximizing one-step score. The actions can suboptimal and different from the policy described above because the greedy actions only consider one-step score but it still capture the behavior of policy.
We visualize the computed actions on World 5 Stage (Figure. 5 which is visually different from previous tasks. In this testing case, we see that the actions are reasonable, such avoiding obstacles and monsters by jumping over them, even in the face of previously unseen configurations and different backgrounds. However, the "Piranha Plants" are not recognized because all the prior scores are learned from WISI where never appears. More visualization of action maps is available in Appendix A.228
Figure Visualized greedy actions from the learned scores in new environment (W5S1).
Additionally, we visualize the prior scores for different representative local sub-regions in Appendix A.2.8 Fig- ure. 9 In this setting, we synthetically put an agent on different relative position method learns meaningful scores such as assigning low scores for walking towards "kooop""(urtlll and high score for jumping.
object. We find that our
Those qualitative studies further demonstrate that the SAP method can assign meaningful scores for different objects in unsupervised manner. It also produces good actions even in new environment.
5.3 SAP oN THE 3-D ROBOTICS TASK
(a)
(b)
(c)
(d)
Figure 6: Four variants the 3D robot reacher environments. See Section 5.2. for details.
In this section, we further study the SAP method to understand its property with higher dimensional observation space. We conduct experiments in a 3-D robotics environment, BlockedReacher-vvO In this environment, robot hand is initialized at the left side of a table and tries to reach the right. Between the robot hand and the goal, there are few blocks standing as obstacles. The task is moving the robot hand reach point on y 1.0 fast possible. To test the generalization capability, we create four different configurations of the obstacles, as shown in Figure Figure is the environment where collect exploration data from and Figure hp are the testing environments.
We apply the SAP framework as follows. The original observation is 25-dimensional continuous state and the action space is 3-dimensional continuous control. They are discretized into voxels and 8 discrete actions as described Appendix A.3.T In this environment, the local region is set 15 x 15 x 15 eube of voxels around robot hand end effector. We divide this cube into 27

---

Under review as a conference paper at ICLR 2020
Table 2: Evaluation of SAP and NHP on the 3D Reacher environment. Numbers are the avg. steps to
reach the goal. The lower the better. Numbers in the brackets are the 95% confidence interval.
Config +
Config 2
Config 3 4
Config 4 109.2[2.98]
SAP (w/ learned dynamics)
97.53[2.48]
86.53[1.88
113.3[3.03]
NHP (w/ learned dynamics)
124.2[4.51]
102.0[3.04]
160.7[7.89]
155.4[7.24]
SAP (w/ perfect dynamics)
97.60[2.60]
85.38[
112.2[3.07]
114.1[3.24]
NHP (w/ perfect dynamics)
125.3[4.25]
102.4[2.96]
153.4[5.30]
144.9[4.84]
x 5 x 5 sub-regions. The scoring function is a fully-connected neural network that takes a flattened voxel sub-region and outputs the score matrix with shape of 26 x 8. The scores for each step are aggregated by a sum operator along the trajectory. We also train a 3D convolutional neural net the dynamics model. The dynamics model takes in a 15 15 15 local region well an action, and outputs the next robot hand location. With the learned scores and the dynamics model, we plan using the MPC method with a horizon of steps.
We perform similar baselines as in the previous section that is detailed in Appendix A.3.4 In Table 2 we compare our method with the NHP method on the 3D robot reaching task. We found that our method needs significantly fewer steps than the NHP method, both in the training environment and testing ones. We find that SAP significantly moves faster to the right because learns a negative score for the obstacles. However, the NHP method, which has +1 positive for each 1 meter moved to the right, would be stuck by the obstacles for longer duration. We found that the learned dynamics model is relatively accurate in this domain, such that the performance using the learned dynamics is close to that of perfect dynamics. These experiments show that our method can be applied to robotics environment that can be hard for some algorithms due to the 3-D nature. Moreover, we demonstrate again that using partial states (local regions) with SAP generalize better than baselines.
CONCLUSION
We devise a novel learn generalizable priors from exploration and sparse rewards for novel tasks. We find the proposed method can capture the transferable priors and take advantage of the priors without any finetuning. Experimental results also show that following the SAP framework, designed algorithms outperform variety of baseline methods on both training and unseen testing tasks in different application domains.
Paaegggtinng-llaning (SAP) framework for designing algorithms that can
While this paper explores some applications under SAP framework, many compelling questions remain open. There is welcoming avenue for future work to improve each component of the framework. For example, for complex tasks, there might be priors beyond contact force and game dynamics with a much more complicated action space. Hence, how to extract relational priors from them to solve novel tasks is still yet to be explored, Dubey et al. 2018) thoroughly studied existing human priors in video game playing; however, it is still not clear how to use the correct priors for real-world applications an SAP framework (e.g. fire good when you want look but harmful while being too close).
There are many interactive samples the real world, but most of them are suboptimal. However, an evaluation of them can be given from a task-specific score human evaluation. Our method excels in this setting. In theory, it can be extended to the case a binary sparse reward is given by carefully choosing an aggregator such logic operators with sufficient samples. We leave those extensions for future works.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. Proceedings of the twenty-first international conference on Machine learning, pp. 1. ACM, 2004.
Joshua Achiam and Shankar Sastry, Surprise-basse baarise-baaedddd intrinsic motivation for deep reinforcement learning. arXiv preprint urXiv:/703.0172 2017.
Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. Learning to generalize from sparse and underspecified rewards. arXiv preprini 1902.0998, 2019.
Marcin Andrychowiez, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pp. 5048 50188-008.1000 2017.
Jose Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. arXiv preprint 1806.07877 2018.
JA Bagnell, Joel Chestnutt, David M Bradley, and Nathan D Ratliff. Boosting structured prediction for imitation learning. In Advances in Neural Information Processing Systems, pp. 1133-1660, 2007.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471-1479, 2016.
Kenneth Bogert and Prashant Doshi, Multi-robot inverse reinforcement learning under occlusion with state transition estimation. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pp. 1837-1838. International Foundation for Autonomous Agents and Multiagent Systems, 2015.
Kenneth Bogert, Jonathan Feng-Shun Lin, Prashant Doshi, and Dana Kulic. Expectation-maxiimzazio for inverse reinforcement learning with hidden data. In Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, pp. 1034-1042, International Foundation Autonomous Agents and Multiagent Systems, 2016.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample-eflicient reinforce- ment learning with stochastic ensemble value expansion. In Advances in Neural Information Processing Systems, pp. 8224-8234 8234, 2018.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale 2018a.
study of drrie learning. arXiv preprint
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.1 128994 2018b.
Jaedeug Choi and Kee-Eung Kim. Inverse reinforcement learning in partially observable environments. Journal of Machine Learning Research, 12(Mar):691-77 -300 2011.
Leshem Choshen, Lior Fox. and Yonatan Loewenstein. Dora the explorer: Directed outreaching reinforcement action-sclection. arXiv preprint arXiv: 2018.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using prohabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4765- 2018.
Mare Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach policy search. In Proceedings of the 28th International Conference on machine learning (ICML-I1), pp. 465-472, 2011.
Mare Peter Deisenroth, Carl Edward Rasmussen, and Dieter Fox. Learning control a low-cost manipulator using data-efficient reinforcement learning. Robotics: Science and Systems VII. pp. 57-64, 2011.
Marc Peter Deisenroth, Gerhard Neumann, Jan Peters, et al. A survey on policy search for robotics. Foundations and Trends in Robotics, 2(1 1111):111123 2013.
Coline Devin, Pieter Abbeel, Trevor Darrell, and Sergey Levine. Deep object-centric representations for generalizable robot learning. In 20/8 IEEE International Conference on Robotics and Automation (ICRA), pp. 7111-7118. IEEE, 2018.
11

---

Under review as a conference paper at ICLR 2020
Alexey Dosovitskiy and Vladlen Koltun. Learning act by predicting the future. arXiv preprint arXiv:1611.0177,, 2016.
Yilun Du Karthik Narasimhan. Task-agnostic dynamies priors for deep reinforcement learning. arXiv preprint arxiv: (XXx::99950008/0 2019.
Rachit Dubey, Pulkit Agrawal, Deepak Pathak, Thomas L Griffiths, and Alexci Efros. Investigating human priors for playing video games. arXiv preprint a 10217, 2018.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Advances in neural information processing systems, pp. 64-72, 2016a.
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoen- coders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 512- 519, IEEE, 2016b.
Justin Fu, John Co-Reyes, and Sergey Levine. Ex2: Exploration with exemplar models deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2p-227772587 2017.
James Gibson, The ecological approach to visual perception: classic edition. Psychology Press, 2014.
Marek Grzeš and Daniel Kudenko. Online learning of shaping rewards reinforcement learning. Neural Networks, 23(4))541-550, 2010.
Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lec, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint 1xXxv:811.00551, 2018.
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot transfer in reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1480-1490. JMLR. org, 2017.
Shiyu Huang, Hang Su, Jun Zhu, and Ting Chen. Combo-action: Training agent for fips game with auxiliary tasks. AAAI, 2019a
Zhiao Huang, Fangehen Liu, and Hao Su. Mapping state space using landmarks for universal goal reaching. arXiv preprint arXiv: / 1908.9008000 05451, 2019b.
K Jetal Hunt, D Sbarbaro. R Zbikowski, and Peter Gawthrop. Neural networks for control systems-a survey. Automatica, 13831100831112, 1992.
Eric Jang, Coline Devin, Vincent Vanhoucke, and Sergey Levine. Grasp2vec: Learning object representations from self-super grasping. arXiv preprint urXiv:181I.0610 2018
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy Н Campbell, Konrad Czechowski. Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-basee reinforcement leamning for atari. arXiv preprint 1rXiv:1903.0 00374,034444 2019.
Christian Kauten, Super Mario Bros for OpenAT Gym. GitHub, 2018. URL httphhtts//gthhhttet thtp...comtuub.commm com/ Kauten ja/ jawpga///aaaaaaemm suppr-uur-----ibbrroo
Ramtin Keramati, Jay Whang, Patrick Cho, and Emma Brunskill. Strategic object oriented reinforcement learning. arXiv preprint 806.0778 2018.
S Mohammad Khansari-7adch and Aude Billard. Learning stable nonlinear dynamical systems with gaussian mixture models. IEEE Transactions on Robotics, 27(5):943-99) 957-997722222 2011.
Jonathan Ko Dieter Fox. Gp-bayesfilter: Bayesian filtering using gaussian process prediction and observa- tion models. Autonomous Robots, 27(1):55-90, 2009.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprini 1802::1802.100300 2018
Sergey Levine and Pieter Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems, pp. 1071-1079, 2014.
Sergey Levine and Vladlen Koltun, Guided policy search. In International Conference on Machine Learning, pp. 1-9, 2013.
12

---

Under review as a conference paper at ICLR 2020
Minne Li. Pranay Nashikkar, and Jun Wang. Optimizing object-based perception and control by free-energy principle. CoRR, abs/1903.01385, 0.335 2019, URL http:://aaxxi..orggg//ooo/////// abw 01385
Timothy P Pillicrapp, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 1xXv:1109.00299 2015.
Rudolf Lioutikov, Alexandros Paraschos, Jan Peters, and Gerhard Neumann. Sample-based informationl- theoretic stochastic optimal control. In 2014 IEEE International Conference on Robotics and Automation (ICRA). pp. 3896-3902. IEEE, 2014.
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. arXiv preprint 1 1605.08/04, 2016.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model- deep reinforcement learning with theoretical guarantees. arXiv preprint a 1807.03858, 2018.
Marlos Machado, Mare G Bellemare, and Michael Bowling. Count-based exploration with successor representation. arXiv preprint arXiv:1807.11022 2018.
Maryam Marashi, Alireza Khalilian, and Mohammad Ebrahim Shiri. Automatic reward shaping in reinforce- ment learning using graph analysis. In 2012 2nd International e eConference on Computer and Knowledge Engineering (ICCKE), pp. 111-116. IEEE, 2012.
Bhaskara Marthi. Automatic shaping and decomposition of reward functions. In Proceedings the 24th International Conference on Machine learning, pp. 601-608. ACM, 2007.
David Q Mayne, James B Rawlings, Christopher V Rao, and Pierre OM Scokaert. Constrained model predictive control: Stability and optimality. Automatica, 36(6in):789-8) 2000.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 518(75400:529, 2015.
Jun Morimoto and Christopher G Atkeson. Minimax differential dynamic programming: An application to robust biped walking. In Advances in neural information processing systems, pp. 1563-1570, 2003. 2003.
Andrew Y Ng. Feature selection, vs. 2 regularization, and rotational invariance. In Proceedings of the first international conference on Machine learning, pp. 78. ACM, 2004.
Andrew Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application reward shaping. In ICML, volume 99, pp. 778-287, 1999.
Andrew Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. Icml, volume 1. pp. 2000.
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2661-2670. JMLR. org, 2017.
OPenAl. Openai five. https::/bbog.opeee openai wtw.pppnn...ooopppaaiiiiiieeeeee 2018.
Georg Ostrovski, Mare Bellemare, Aäron van den Oord, and Rémi Munos. Count-based exploration with neural density models. Proceedings of the 34th International Conference on Machine Learning-Void Vollue 70, pp. 2721-2730. JMLR. org, 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self- supervised prediction. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider. Josh Tobin, Maciek Chociej. Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802. 1802.09464, 2018.
Nicholas Rhinchart and Kris M Kitani. First-person activity forecasting with online inverse reinforcement learning. In Proceedings of the IEFE International Conference on Computer Vision, pp. 3696-3705, 2017.
13

---

Under review as a conference paper at ICLR 2020
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing-solving sparse reward tasks from scratch, arXiv preprint 1802 10567, 2018.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom model-building neural controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pp- 222-227, 1991.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. digeediinnsional continuous control using generalized advantage estimation. arXiv preprint 1556::5500.244444 2015.
David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, ct al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1 140-1144, 2018.
Sungryull Sohn, Junhyuk Oh, and Honglak Lee. Hierarchical reinforcement learning for zero-shot generalization with subtask dependencies. In Advances in Neural Information Processing Systems, pp. 7156-7166, 2018.
Bradly C Stadic, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint (rxXiv1507.000//1 2015.
Alexander L Strehl and Michael L Littman. An analysis of model-based interval estimation for markov decision processes, Journal Computer and System Sciences, 10011139913111 2008
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2 (4):160 16300116611100 1991.
Umar Syed and Robert E Schapire. A game-theoretic approach to apprenticeship learning. In Advances in neural information processing systems, pp. 1449 1456,,14622222 2008
Haoran Tang, Rein Houthooft. Davis Foote. Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. exploration: A study of count based exploration for deep reinforcement learning. In Advances in neural information processing systems, pp. 2753-2762 2017.
Voot Tangkaratt, Syogo Mori, Tingting Zhao, Jun Morimoto, and Masashi Sugiyama. Model-basee policy gradients with parameter- based exploration by least-squares conditional density estimation. Neural networks, 57:128-140, 2014.
Aaron Tucker, Adam Gleave, and Stuart Russell. Inverse reinforcement learning for video games. arXiv preprint Ttti:/880..0555 2018.
Dequan Wang, Coline Devin, Qi-Zhi Cai, Fisher Yu, Trevor Darrell. Deep object-centric policies for autonomous driving. 2019 International Conference on Robotics and Automation (ICRA), pp. 8853-8859.
IEEE, 2019.
Shaojun Wang, Ronald Rosenfeld, Yunxin Zhao, and Dale Schuurmans. The latent maximum entropy principle. In Proceedings IEEE International Symposium on Information Theory, pp. 131. IEEE, 2002.
Manuel Watter, Jost Springenberg, Joschka Boodecker, and Martin Riedmiller. Embed to control: locally lincar latent dynamics model for control from raw images. In Advances in neural information processing systems, pp. 2254,,2754,220 2015.
Theophane Weber, Sebastien Racaniere. David P Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende. Adria Puigdomenech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al. Imaginatiion-augmented agents for deep reinforcement lcarning. arXiv preprint arXiv: 1707.06203, 2017.
Yuxin Wu and Yuandong Tian. Training agent for first-person shooter game with actor-critic curriculum learning. 2016.
Chris Xie, Sachin Patil, Teodor Moldovan, Sergey Levine, and Pieter Abbecl. Model-based reinforcement learning with parametrized physical models and optimism-driver exploration. In 2016 IEEE international conference on robotics and automation (ICRA), pp. 604-511. IEEE, 2016.
Xu Xie, Changyang Li, Chi Zhang, Yixin Zhu, and Song-Chun Zhu. Learning virtual grasp with failed demonstrations via bayesian inverse reinforcement learning. IROS, 2019.
Guangxiang Zhu, Zhiao Huang, and Chongjic Zhang. Object-oriented dynamics predictor. In Advances Neural Information Processing Systems, pp. 9804 0014-9815,20 2018.
14

---

Under review as a conference paper at ICLR 2020
A EXPERIMENT SPECS
A.1 HIDDEN REWARD GRIDWORLD
A.1.1 ENVIRONMENT.
In Figure we visualize a sample the gridworld environment. Each entry correspond to a noised feature vector based the type of object in it. Each feature is length 16 vector whose entries are uniformly sampled from [0. 1]. Upon each feature, we add a small random noise from normal distribution with μ a 0.05. The outer-most entries correspond to padding objects whose rewards are O. The action space includes move toward four directions up. down, left, right. If an agent attempts take an action which leads outside of our grid, it will be ignored be the environment.
A.1.2 ARCHITECTURES FOR SCORE FUNCTION AND DYNAMICS MODEL..
We train two layer fully connected neural networks with 32 and 16 hidden units respectively and ReLU activation function to approximate the score for each grid.
In this environment, we do not have a learned dynamics model.
Hyperparameters. During training, we use an adam optimizer with learning rate 1e-3, B1 = 0.9, 82 0.999. The learning rate reduced le-4 after 30000 iterations. The batchsize is 128. We use horizon = 4 as our planning horizon.
A.2 SUPER MARIO BROS
A.2.1 ENVIRONMENT.
We wrap the original Super Mario environments with additional wrappers. We wrap the action space into discrete joypad actions, none, walk right, jump right, run right and hyper jump right. We follow (Burda et al.] (0188) to add sticky action wrapper that repeats the last action with probability of 20%. Besides this, we follow add the standard wrapper as in past work (Mnih et al. 2015).
A.2.2 ARCHITECTUREE FOR SCORE FUNCTION AND DYNAMICS MODEL.
For the score function, we train CNN taking each 12px by 12px sub-region input with conv layers and hidden fully connected layers. For each conv layer, we use filter of size 3 by 3 with stride 2 with number of output channels equals and 16 respectively. "Same padding" is used for each conv layer. The fully connected layers have 128 units. Relu functions are applied as activation except the last layer.
For the dynamics model, we train a neural network with the following inputs: 30 by 30 local observation around mario. b. current action along with 3 recent actions encoded in one-hot tensor. c. 3 most recent position shifts. d. one-hot encoding of the current planning step. Input a is encoded with 4 sequential conv layers with kernel size and stride Output channels are 8, 16, 32, 64 respectively. A global max pooling follows the conv layers. Input b, d are each encoded with 64 node fo layer. The encoded results are then concatenated and go through a 128 units hidden fe layer. This layer connects to two output heads, one predicting shift in location and one predicting "done" with sigmoid activation. Relu function is applied as activation for all intermediate layers.
A.2.3 HYPERPARAMETERS.
During training, we use an adam optimizer with learning rate 3c-4, 3, = 0.9, B2 = 0.999. The batchsize is 256 for score function training and 64 for dynamics model. We horizon 10 as planning horizon. We use a discount factor & = 0.95 and 128 environments in our MPC.
A.2.4 MORE ON TRAINING
In the scoring function training, each data point a tuple of a down sampled trajectory and calculated score. We down sample the trajectory in the exploration data by taking data from every
15

---

Under review as a conference paper at ICLR 2020
4
2
5
6
3
4
5
2
6
1
8
7
Figure 7: A visualization of the sub-regions in the Super Mario Bros game. In this game, there are in total 8 sub-regions.
16

---

Under review as a conference paper at ICLR 2020
Table Ablation Study for number of planning steps in MPC based methods. The averaged return is reported.
World Stage
World 2 Stage (new task)
plan 8
plan 10
plan 12
plan 8
plan 10
plan 12
SAP NHP
1341.5
1359.3
1333.5
724.2
790.1 587.9
682.6
1193.8
1041.3
1112.1
546.4
463.8
Table 4: Ablation Study of visual representation. The averaged return is reported. We omit the baselines that is weaker than NHP.
World 1 Stage
World 2 Stage (new task)
SAP subregion
1359.3
790.1
SAP localregion
1258.0
737.0
NHP
1041.3
587.9
two steps. Half the the trajectories ends with "done"(deeth) event and half are not. For those ends with "done", the score is the distance mario traveled by mario at the end. For the other trajectories, the score is the distance mario traveled by the end plus a mean future score. The mean future score of a trajectory is defined to be the average extra distance traveled by longer (in terms of distance) trajectories than our trajectory. We note that all the information are contained in the exploration data.
A.2.5 MORE DETAILS ON BASELINES
Behaviroal Cloning (BC). As super mario is a deterministic environment, we noticed pure behavior cloning trivially get stuck at a tube at the very beginning of level 1-1 and die at an early stage of 2-1. Thus select action using sampling from output logits instead of taking argmax.
Exploration Data We train a policy with only curiosity as rewards (Pathak et al. 2017). However, early stopped the training after 5e7 steps which is far from the convergence le9 steps. We further added an e-greedy noise when sampling demonstrations with c = 0.4 for 20000 episodes and 0.2 for 10000 episodes.
A.2.6 ADDITIONAL ABLATIONS
Ablation of Planning Steps In this section, we conduct additional ablative experiments to evaluate the effect of the planning horizon in MPC method. In Table. B little with different planning steps in a relatively small range and outperforms baselines constantly. In the main paper, choose horizon 10. We find that when plan steps are larger such 12, the performance does not improve monotonically. This might be due to the difficult to predict long range future with a learned dynamics model.
that method fluctuates
Ablation of visual representation In this section, we conduct experiments to evaluate the effect of the proposed visual representation local sub-regions. As a comparision, we include variant that takes in the whole local region input and output a score conditioned on actions. In Table 4 we see that the local subregions contribute to both the training performance and the zero-shot generalization performance. However, we also find that even without the subregions, SAP still outperforms our second strongest baseline. This is because the scoring and planning steps still empowers the agent the ability learn and generalize.
Model Dissection: To further understand each component of SAP, we ablate the scoring-aggregatting component and the planning component. The NHP method uses manually designed scoring function, with the original planning component. We further conducted the SAP-3 step and the Greedy method which only have 3 planning steps and no planning respectively. In the Table. 5 observe that without the scoring aggregating component or the planning component, the performance has a significantly drop. This shows that all components of SAP critical to the performance.
17

---

Under review as a conference paper at ICLR 2020
Table Model dissection comparisons. The averaged return is reported. 95% confidence interval is shown in the bracket.
Scoring
Aggregating
Planning
WISI(train)
W2S1(test)
SAP
yes yes yes
1359.3[68.5]
790.1[35.7]
NHP
manual
yes
1041.31149.0)
587.8[31.0]
SAP-3 steps
yes
yes
minimal
944.5[133.5]
444.3[14.7]
Greedy
yes
yes
302.0[27.9]
40.0[15.7]
Table 6: Additional baselines privileged BC and RL agent. The averaged return is reported.
World Stage
World 2 Stage (new task)
SAP
1359.3
790.1
Privileged BC
1241.4
432,4
RL curiosity
1183.0
347.1
A.2.7 ADDITIONAL BASELINES
In this section, we compare SAP with more baselines. The first baseline is privileged BC: We collected 8000 near-optimal trajectories (average score 1833.0) from the training environment. Then we train an imitation learning agent that mimics the near-optimal data. note that this baseline not a fair comparison because SAP only utilizes random exploratory data; present this baseline to test the generalization ability of an imitative agent that performs well on training environment. The second baseline is reinforcement learning agent that is trained with curiosity driven reward (Burdal jet al. 2018b) and the final sparse reward. We limit the training steps to 10M. This is also violating the setting have it interacts with the environment. We conduct this experiment test the generalization ability of a reinforcement learning agent. In Table we see that both baselines have large drop on the generalization task compared to SAP.
A.2.8 ADDITIONAL VISUALIZATION
In this section, we present additional visualization for qualitative study. In Figure. g we see that on few randomly sampled frames, even the greedy action be meaningful for most of the cases. We see the agent intend to jump over obstacles and avoid dangerous monsters.
show the scores of given state-action pair and find that the scores fulfill the human prior. For example, in Figure. - we synthetically put the mario agent in relative positions to "koopa" conditioned
In Figure.
the action "move right". The score is significantly lower when the agent's position is to the left of "koopa" compared to other position. In Figure. [b] it is the same setup as in
Figure. Gh but conditioned the action "jump". We find that across Figure. and Figure. dp the
Figure 8: More visualizations on the greedy action map on WISI, W2S 1251(mn)) (new task) and W5S (new
task). Note the actions be different from the policy from MPC.
18

---

Under review as a conference paper at ICLR 2020
left position score of Figure. p is smaller than that of Figure. which is consistent with human we substitute the object from "koopa" to the ground. We find the score similar for the top position which means there is
priors. In Figure. and Figure. d that on both Figure. gn and Figure. g not much difference between different actions.
(a) action: move right
(b) action: jump
(c) action: move right
(d) action: jump
Figure 9: Visualization of the learned score on pre-extracted objects. The grayscale area is the visualized score and the pink area is separator. Best viewed in color. In (a), we synthetically put the mario agent in 8 relative positions "koopa" conditioned on the action "move right". The score is significantly lower when the agent's 's position is to the left of "koopa" compared to other position. In (b), the same setup as in (a) conditioned on the action "jump". We find that across (a) and (b) the left position score of (b) is smaller than that of (a) which is consistent with human priors. In (c) and (d), we substitute the object from "koopa" the ground. We find that on both (c) and (d) the score are similar for the top position which means there is not much difference between different actions. Note this figure is only for visualizations and we even put the agent in positions that cannot be achieved in the actual game.
A.3 ROBOTICS BLOCCED REACH
A.3.1 ENVIRONMENT.
In the Blocked Reach environment, a 7-DoF robotics arm is manipulated for specific task, For more details, we refer the readers to PPlappert et al. (2018). We discretize the robot world into 200 x 200 x 200 voxel cube. For the action space, we discretize the actions into two choices for each dimension which moving 0.5 or -0.5. Hence, in total there are actions. We design four configurations for evaluating different methods as shown in Figure 6 For each configurations, there three objects are placed in the middle obstacles. The height of the objects in each configuration are (0.05, 0.1, 0.08), (0.1, 0.05, 0.08), (0.12, 1.12, 0.12), (0.07, 0.11, 0.12).
A.3.2 ARCHITECTURES FOR SCORE FUNCTION AND DYNAMICS MODEL.
For the score function, we train a hidden layer fully-connected neural networks with 128 units. We Relu functions as activation except for the last layer. Note that the input by 5 by voxels are flattened before put into the scoring neural network.
For the dynamics model, we train 3-D convolution neural network that takes in local region (voxels), action and last three position changes. The 15 by 15 by 15 local voxels are encoded using three 3d convolution with kernel size 3 and stride 2. Channels of these 3d conv layers are 16, 32, 64, respectively. A 64-umit FC layer is connected to the flattened features after convolution. The action is encoded with one-hot vector connected a 64-unit FC layer. The last three positions are also encoded with a 64-unit FC layer. The three encoded features are concatenated and go through 128-unit hidden FC layer and output predicted change in position. All intermediate layers use relu as activation.
A.3.3 HYPERPARAMETERS
During training, batchsize is 128 for score function training and 64 for dynamics model. We use horizon = 8 as our planning horizon.
adam optimizer with learning rate 3e-4, B1 0.9, 82 0.999. The
19

---

Under review as a conference paper at ICLR 2020
A.3.4 BASELINES
Our naive human prior baseline in the blocked robot environment 8-step MPC where score for each step is the component of the action vector at that step.
We omit the Behavioral cloning baselines, which imitates exploration data, consequence of two previous results.
20


Reviews:
[{"review_id": "r1lfJzG3qS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a framework (Scoring-Aggregating-Planning (SAP)) for learning task-agnostic priors that allow generalization to new tasks without finetuning. The motivation for this is very clear - humans can perform much better than machines in zero-shot conditions because humans have learned priors about objects, semantics, physics, etc. This is achieved by learning a scoring function based on the final reward and a self-supervised learned dynamics model.\n\nOverall, the paper is very clear and easy to follow.\nThe presented task is realistic and important, and the paper seems to address it in a reasonable approach. \nHowever, the evaluation seems lacking to me - the evaluation convinced me that SAP works, but I am not convinced that it works better than existing approaches (see below), and especially did not convince me that it is better in the zero-shot test environment.\nThe (anonymized) website contains nice videos that support the submission.\n\nQuestions for the authors:\n\n1. Page 3, 3rd paragraph of Section 3: the paper says that \"The proposed formulation requires much less information and thus more realistic and feasible\" - I agree that this is more realistic, but is it really more feasible? The requirement of much less information makes the proposed formulation much more sparse.\n\n2. A basic assumption in the SAP framework is that a local region score is a sum of all the sub-regions. As phrased in the paper: \"in the physical world, there is usually some level of rotational or transnational invariance\". I'm not sure that this assumption makes sense neither in the Mario case or in other tasks, e.g., robotics. Doesn't it matter if you have a \"turtle\" right in front of you (which means that the turtle is going to hit you), or below you (which means that you are going hit the turtle)?\n\n3. A question about the planning phase - page 5 says: \"We select the action sequence that gives us the best-aggregated score and execute the first action\". Do you select the entire sequence of actions in the new environment in advance? Can the agent observe the new state after every action, and decide on the next action based on the actual step that the action has reached, rather than on the state that was approximated in advance?\nIn other words - what happens if the first action in the new test environment yields an unexpected state, that was not predicted well by the dynamics model; does the agent continue on the initial planned trajectory (that ignores the \"surprise\"), or does it compute its next action based on the unexpected state?\n \n4. Experiments: in Gridworld and Mario - are there any stronger baselines in the literature, or reductions of known baselines to the zero-shot scenario? Are the chosen \"Human Priors\", BC-random and BC-SAP just strawmen? \nSince the main goal of this paper is the zero-shot task, what would convince me is a state-of-the-art model that does possibly *better than SAP on the training level*, but *worse than SAP in generalizing to the new level*. Additionally, are there other baselines that specifically address the zero-shot task in the literature?\n\nMinor (did not impact score):\nPage 2, 1st paragraph: \"... we show that how an intelligent agent\"...\nPage 3, 3rd paragraph: \"... in model-free RL problem\" - missing an \"a\" or \"problem*s*\"?\nPage 3, 3rd paragraph: \". Model based method ...\" - missing an \"a\" as well?\nPage 4, 1st paragraph:: \"... utilizing the to get the ...\"\nPage 4, last row: missing a dot after the loss equation, before the word \"In\".\nPage 7, Table 1: \"BC-random\" is called \"BC-data\" in the text. Aren't they the same thing?\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SkxYYEW4qr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I am not from this area and don't know much about reinforcement learning.\n\nThe paper discusses zero shot generalization (adaptation) into new environments. The authors propose an approach and then show results on Grid-World, Super Mario Bros, and 3D Robotics. \n\nIn the training environment E1 = (S, A, p) the algorithm sees a bank of exploratory trajectories \\tau_i = {(s_t, a_t)}_{t=1}^{T} but not rewards. The authors  then say that algorithm is tested on the test environment E2. They \" propose to only inform the new task per trajectory terminal evaluation r(τ ) in E1\" to give the training signal (where r is the reward).\n\nI am a bit confused by this setting. The model never sees any rewards for E1 but it does see rewards for E2? How is this zero shot?\n\nThe authors then propose their approach, I wish some of it had been described more rigorously with math (e.g. the loss etc.) so it was easier to understand for people not in the domain and familiar with some of the terminology. \n\nEmpirically the authors show results for 3 datasets and this seems thorough. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyghHQBr9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a method that aims to learn task-agnostic priors for zero-shot generalization. The main idea is to employ the following modeling approach on top of the model-based RL framework: a local convolution network is used to compute a score for each local state action pair, and then another network is used to aggregate all the scores. While the problem being studied is important and the experimental results seem positive, there are a few concerns.\n\nFirst, the baselines presented in the experiments are relatively weak. In Related Work, the authors discuss the differences between the proposed method and the related methods, but few of the related methods are used as baselines for comparison with the proposed method. Moreover, the experiments are quite insufficient in terms of ablating different components of the proposed methods.\n\nSecond, essentially the proposed method is trying to solve the zero-shot generalization by parameter initialization; a model is pretrained on related tasks and used as initializations for target tasks. The authors claim that it is different from prior work mainly because of the neural architecture that deals with sparse rewards via score aggregation. While the proposed architecture might be more suitable for solving tasks with sparse rewards, it is not intuitive whether it has something to do with learning zero-shot generalization. And apparently, the method will also rely on the similarity between the pretrained task and the target task, and such a scope constraint is not discussed in the paper. In other words, I'm not quite sure a better architecture is fundamental progress towards zero-shot RL.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an algorithm for zero-shot generalization in RL via learning a scoring a function from.

The reviewers had mixed feelings, and many were not from the area. A shared theme was doubts about the significance of the experimental setting, and also the generality of the approach.

As this is my field, I read the paper, and recommend rejection at this time. The proposed method is quite laborious and requires quite a bit of assumptions on the environments to work, as well as fine tuning parameters for each considered task (number of regions, etc). I also agree that the evaluation is not convincing -- stronger baselines need to be considered and the experiments to better address the zero-shot transfer aspect that the paper is motivated by. I encourage the authors to take the review feedback into account and submit a future version to another venue.
Decision: Reject