Scoring-Aggregating-Planning: Learning task-agnostic priors from interactions and sparse rewards for zero-shot generalization

1 INTRODUCTION . While deep Reinforcement Learning ( RL ) methods have shown impressive performance on video games ( Mnih et al. , 2015 ) and robotics tasks ( Schulman et al. , 2015 ; Lillicrap et al. , 2015 ) , they solve each problem tabula rasa . Hence , it will be hard for them to generalize to new tasks without re-training even due to small changes . However , humans can quickly adapt their skills to a new task that requires similar priors e.g . physics , semantics , affordances to past experience . The priors can be learned from a spectrum of examples ranging from perfect demonstrative ones that accomplish certain tasks to aimless exploration . A parameterized intelligent agent “ Mario ” which learns to move to the right in the upper level in Figure 1 would fail to transfer the priors from to the lower level in Figure 1 and further play the game in the new level because change of configurations and background , e.g . different shapes of ladder , new fence . When an inexperienced human player is controlling the Mario to move it to the right in the upper level , it might take many trials for him/her to realize the falling to a pit and approaching the “ koopa ” ( turtle ) from the left are harmful while standing on the top of the “ koopa ” ( turtle ) is not . However , once learned , s/he can infer similar mechanisms in the lower level in Figure 1 without additional trials because human have a variety of priors including the concept of object , similarity , semantics , affordance , etc ( Gibson , 2014 ; Dubey et al. , 2018 ) . In this paper , we intend to teach machine agents to realize and utilize useful priors to generalize to new tasks without finetuning . Toward addressing the generalization problem with learned priors , we follow the intuition that : ( 1 ) each trajectory in a video game or a robotics task is composed of state-action pairs with object interactions ( 2 ) terminal rewards can be approximated by the aggregation of the scores for each state-action pairs . With those intuitions in mind , we summarize our proposed method in broad strokes . Given a trajectory with a terminal sparse reward , we first parameterize the score of an action-local region pair with a convolutional neural network F and then aggregate the scores to approximate the final sparse reward . To further enable actionable agents to utilize the scores , a neural dynamics model 1Project page : https : //sites.google.com/view/sapnew/home . can be learned from the interaction data using self-supervision . We show that how an agent can take advantage of the scoring function and the learned dynamics model with planning algorithms ( Mayne et al. , 2000 ) . We adopt the sparse terminal reward setting because in most of the tasks , step-by-step rewards are hard to obtain while final evaluations for trajectories are relatively easy . Readers may argue that learning a dense score for every interaction step is reminiscent of Inverse Reinforcement Learning ( Ng et al. , 2000 ; Abbeel & Ng , 2004 ) . The distinctions between the proposed method and IRL are threefold : First , instead of learning a reward function of state s , we learn a scoring function of a local state sl and an action a , which is sufficiently rich in a physical environment and experimentally can generalize well . Second , with the scoring function in hand , we use a dynamics model learned from passive data to obtain the actual policy in a model-based manner while IRL needs to re-train an agent that can be as data inefficient as model-free RL . However , IRL can have difficulty learning a useful model because the expert demonstrations usually only cover a small portion of the true dynamics . Third , we eliminate the assumption of expensive expert demonstrations with the cost of adding a relatively economical sparse reward in the end . This elimination not only reduces the cost for data collection , but also includes more diverse data to train a robust model . The proposed scoring function , beyond being a cost function for planning , can also be treated as an indicator of the existence of objects , which affect the evaluation of a trajectory . We empirically evaluate the scores for objects extracted in the context of human priors and hence find the potential of using our method as an unsupervised method for object discovery . In this paper , we have three major contributions . First , we propose a framework that can learn task-agnostic priors that can generalize to novel tasks . Second , we incorporate a self-supervised learned dynamics model with scoring function to learn a useful policy . Third , we demonstrate the effectiveness of the proposed method on a didactic grid-world example , a well-known video game “ Super Mario Bros ” and a robotics Blocked-Reach environment and show our method outperforms various baselines . Last but not least , we find that objects emerge from our method in an unsupervised manner , which could be useful to other visual tasks . 2 PRELIMINARIES . In this paper , we formulate each environment as an Markov Decision Process ( MDP ) . We represent the MDP by a tuple : ( S , A , p , r , γ ) , where S is the state space and A is the action space . A MDP is fully specified by a state s ∈ S. A MDP evolves with an action a ∈ A by a probability distribution p ( s′|s , a ) . The MDP emits a reward r ( s , a ) each step . γ ∈ ( 0 , 1 ) is the discount factor . Reinforcement learning aims to learn a conditional distribution over the action space given a state π ( ·|s ) that maximizes the discounted future rewards : π∗ = argmax π E at∼π ( at|st ) st+1∼p ( st+1|st , at ) ∞∑ t=0 γtr ( st , at ) The state transition probability p ( s′|s , a ) is treated as unknown in model-free RL problems . Modelbased methods explicitly learn a dynamics modelM ( ·|s , a ) that specifies the conditional distribution of the next state given the current state s and action a from environmental interactions . With an environment modelM , one can select an action to rollout with the model recurrently in order to maximize the discounted future reward . One method that approximately finds the optimal action is the Model Predictive Control ( MPC ) algorithm . It looks ahead for a horizon of H steps and selects an action sequence that maximizes the discounted reward for the future H steps : argmax a0 , ··· , aH−1 H−1∑ t=0 γtr ( st , at ) where st+1 =M ( st , at ) . To simplify notation , here we assume the environment is deterministic and slightly abuse the notations such thatM ( st , at ) returns the next state instead of state distribution . We note that MPC can also use the ground truth environment dynamics p ( ·|s , a ) . 3 METHOD . 3.1 PROBLEM FORMULATION . An intelligent agent should be able to learn priors from its past experiences and to generalize to related yet unseen tasks . To facilitate such goals , we formulate the problem as follows : The agent is first presented with a bank of exploratory trajectories { τi } , i = 1 , 2 · · ·N collected in a training environment E1 = ( S , A , p ) . Each τi is a trajectory and τi = { ( st , at ) } , t = 1 , 2 · · ·Ki . These trajectories are ramdom explorations/interactions with the environment . Instead of specifying the task by per step reward , which is the standard MDP setting , we propose to only evaluate the performance of each trajectory with a terminal evaluation r ( τ ) in E1 when a task T is given . At test time , we would like the agent to perform the task T , using zero extra interaction with the new but related environment , represented by E2 = ( S ′ , A , p′ ) . We assume the test environment E2 can have different state distribution and related but different dynamics from the experience environment E1 . However , we assume the actions that an agent could perform stay the same . We evaluate an agent on task T by giving a single terminal reward r ( τi ) per trajectory τi to its previous experiences . This reward is only used for evaluation and an agent can never utilize it while carrying off the task . The proposed formulation requires much less information and thus more realistic . In this paper , we focus on locomotion tasks with object interactions , such as Super Mario running with other objects in presence , or a Reacher robot acting with obstacles around . 3.2 THE SCORING-AGGREGATING-PLANNING ( SAP ) FRAMEWORK . We propose the Scoring-Aggregating-Planning framework to solve this problem . As an overview , we propose to learn a per step neural scoring function Fθ that scores a sub-region , subset of observation space that surrounds the agent . A local region score is simply a sum over all the sub-regions that are in the local region . The local region scores can be aggregated along the trajectory to approximate the terminal sparse reward . Meanwhile , a dynamics modelMφ is learned to approximate the true transition p ( ·|s , a ) based on past experience E1 . After both the scoring function and the dynamics model are learned , we perform a Model Predictive Control algorithm to get the final policy on the new environment E2 for the task T . Scoring The per step sub-region scoring function can be described as Fθ ( Wl ( st ) , at ) . Here θ denotes parameters , W is a function that extracts local regions from states and l is a sub-region indicator in a metric space L. We note that the neural network ’ s parameters are shared for every local region . A local region score then is ∑ l∈L Fθ ( Wl ( st ) , at ) . Intuitively , this function measures how well an action at performs in the current task , based on a local region l extracted from state s. We presume a local region can be extracted for scoring because in a physical environment , an agent could only interact with the world within its sensing capabilities . From another perspective , it can be seen as a decomposition of the sparse reward into the per-step rewards . Specifically , on the metric space associated with the problem , we divide the local region around the agent into n sub-regions l. For example , in a Cartesian coordinate system , we can divide each coordinate independently . For each l , we use a sub-region scoring network to produce a scoring table of size |A| × |L| , where |A| is the action dimension , and |L| denotes the number of possible relative positions around the agent . One entry is selected from the table as the score for this sub-region l based on the current action taken and the relative position of this sub-region l to the agent . Aggregating To approximate the terminal rewards and learn the scoring network , we aggregate the local region scores ∑ l∈L Fθ ( W ( sl ) , at ) for each step into a single aggregated score J , by an aggregating function G : Jθ ( τ ) = G ( st , at ) ∈τ ( ∑ l∈L Fθ ( Wl ( st ) , at ) ) The aggregated score J are then fitted to the sparse terminal reward . In practice , G is chosen based on the form of the final sparse terminal reward , e.g . a max or a sum function . In the learning process , the Fθ function is learned by back-propping errors between the terminal sparse reward and the predicted J through the aggregation function . In this paper , we use ` 2 loss that is : minθ 12 ( Jθ ( τ ) − r ( τ ) ) 2 . Planning To solve the task , we propose to use planning algorithms to find optimal action based on the learned scoring function and a learned dynamics model . As shown in the part ( c ) of Figure 2 , we learn a forward dynamics modelMφ based on the exploratory data with a supervised loss function . Specifically , we train a neural network that takes in the action at , state st and output ŝt+1 , which is an estimate of st+1 . We use an ` 2 loss as the objective : minφ 12 ( Mφ ( st , at ) − st+1 ) 2 With the learned dynamics model and the scoring function , we solve an optimization problem using the Model Predictive Control ( MPC ) algorithm to find the best trajectory for a task T in environment E2 . The objective of the optimization problem is to minimize −Jθ ( τ ′ ) . Here τ ′ is a H-step trajectory sampled based onMφ starting from some state si . The actual algorithm can be stated as starting from the current state st=i , we randomly select several action sequence up to length H . With the dynamics model , we can roll out the estimated states ŝt+1 , · · · , ŝtH . The learned scoring function and the aggregation function can give us an aggregated score for each of the action sequence . We select the action sequence that gives us the best aggregated score and execute the first action in the environment . And we repeat the procedure starting at the new state si .

Summary: The paper describes a method that aims to learn task-agnostic priors for zero-shot generalization. The main idea is to employ the following modeling approach on top of the model-based RL framework: a local convolution network is used to compute a score for each local state action pair, and then another network is used to aggregate all the scores. While the problem being studied is important and the experimental results seem positive, there are a few concerns.