Scoring-Aggregating-Planning: Learning task-agnostic priors from interactions and sparse rewards for zero-shot generalization

1 INTRODUCTION . While deep Reinforcement Learning ( RL ) methods have shown impressive performance on video games ( Mnih et al. , 2015 ) and robotics tasks ( Schulman et al. , 2015 ; Lillicrap et al. , 2015 ) , they solve each problem tabula rasa . Hence , it will be hard for them to generalize to new tasks without re-training even due to small changes . However , humans can quickly adapt their skills to a new task that requires similar priors e.g . physics , semantics , affordances to past experience . The priors can be learned from a spectrum of examples ranging from perfect demonstrative ones that accomplish certain tasks to aimless exploration . A parameterized intelligent agent “ Mario ” which learns to move to the right in the upper level in Figure 1 would fail to transfer the priors from to the lower level in Figure 1 and further play the game in the new level because change of configurations and background , e.g . different shapes of ladder , new fence . When an inexperienced human player is controlling the Mario to move it to the right in the upper level , it might take many trials for him/her to realize the falling to a pit and approaching the “ koopa ” ( turtle ) from the left are harmful while standing on the top of the “ koopa ” ( turtle ) is not . However , once learned , s/he can infer similar mechanisms in the lower level in Figure 1 without additional trials because human have a variety of priors including the concept of object , similarity , semantics , affordance , etc ( Gibson , 2014 ; Dubey et al. , 2018 ) . In this paper , we intend to teach machine agents to realize and utilize useful priors to generalize to new tasks without finetuning . Toward addressing the generalization problem with learned priors , we follow the intuition that : ( 1 ) each trajectory in a video game or a robotics task is composed of state-action pairs with object interactions ( 2 ) terminal rewards can be approximated by the aggregation of the scores for each state-action pairs . With those intuitions in mind , we summarize our proposed method in broad strokes . Given a trajectory with a terminal sparse reward , we first parameterize the score of an action-local region pair with a convolutional neural network F and then aggregate the scores to approximate the final sparse reward . To further enable actionable agents to utilize the scores , a neural dynamics model 1Project page : https : //sites.google.com/view/sapnew/home . can be learned from the interaction data using self-supervision . We show that how an agent can take advantage of the scoring function and the learned dynamics model with planning algorithms ( Mayne et al. , 2000 ) . We adopt the sparse terminal reward setting because in most of the tasks , step-by-step rewards are hard to obtain while final evaluations for trajectories are relatively easy . Readers may argue that learning a dense score for every interaction step is reminiscent of Inverse Reinforcement Learning ( Ng et al. , 2000 ; Abbeel & Ng , 2004 ) . The distinctions between the proposed method and IRL are threefold : First , instead of learning a reward function of state s , we learn a scoring function of a local state sl and an action a , which is sufficiently rich in a physical environment and experimentally can generalize well . Second , with the scoring function in hand , we use a dynamics model learned from passive data to obtain the actual policy in a model-based manner while IRL needs to re-train an agent that can be as data inefficient as model-free RL . However , IRL can have difficulty learning a useful model because the expert demonstrations usually only cover a small portion of the true dynamics . Third , we eliminate the assumption of expensive expert demonstrations with the cost of adding a relatively economical sparse reward in the end . This elimination not only reduces the cost for data collection , but also includes more diverse data to train a robust model . The proposed scoring function , beyond being a cost function for planning , can also be treated as an indicator of the existence of objects , which affect the evaluation of a trajectory . We empirically evaluate the scores for objects extracted in the context of human priors and hence find the potential of using our method as an unsupervised method for object discovery . In this paper , we have three major contributions . First , we propose a framework that can learn task-agnostic priors that can generalize to novel tasks . Second , we incorporate a self-supervised learned dynamics model with scoring function to learn a useful policy . Third , we demonstrate the effectiveness of the proposed method on a didactic grid-world example , a well-known video game “ Super Mario Bros ” and a robotics Blocked-Reach environment and show our method outperforms various baselines . Last but not least , we find that objects emerge from our method in an unsupervised manner , which could be useful to other visual tasks . 2 PRELIMINARIES . In this paper , we formulate each environment as an Markov Decision Process ( MDP ) . We represent the MDP by a tuple : ( S , A , p , r , γ ) , where S is the state space and A is the action space . A MDP is fully specified by a state s ∈ S. A MDP evolves with an action a ∈ A by a probability distribution p ( s′|s , a ) . The MDP emits a reward r ( s , a ) each step . γ ∈ ( 0 , 1 ) is the discount factor . Reinforcement learning aims to learn a conditional distribution over the action space given a state π ( ·|s ) that maximizes the discounted future rewards : π∗ = argmax π E at∼π ( at|st ) st+1∼p ( st+1|st , at ) ∞∑ t=0 γtr ( st , at ) The state transition probability p ( s′|s , a ) is treated as unknown in model-free RL problems . Modelbased methods explicitly learn a dynamics modelM ( ·|s , a ) that specifies the conditional distribution of the next state given the current state s and action a from environmental interactions . With an environment modelM , one can select an action to rollout with the model recurrently in order to maximize the discounted future reward . One method that approximately finds the optimal action is the Model Predictive Control ( MPC ) algorithm . It looks ahead for a horizon of H steps and selects an action sequence that maximizes the discounted reward for the future H steps : argmax a0 , ··· , aH−1 H−1∑ t=0 γtr ( st , at ) where st+1 =M ( st , at ) . To simplify notation , here we assume the environment is deterministic and slightly abuse the notations such thatM ( st , at ) returns the next state instead of state distribution . We note that MPC can also use the ground truth environment dynamics p ( ·|s , a ) . 3 METHOD . 3.1 PROBLEM FORMULATION . An intelligent agent should be able to learn priors from its past experiences and to generalize to related yet unseen tasks . To facilitate such goals , we formulate the problem as follows : The agent is first presented with a bank of exploratory trajectories { τi } , i = 1 , 2 · · ·N collected in a training environment E1 = ( S , A , p ) . Each τi is a trajectory and τi = { ( st , at ) } , t = 1 , 2 · · ·Ki . These trajectories are ramdom explorations/interactions with the environment . Instead of specifying the task by per step reward , which is the standard MDP setting , we propose to only evaluate the performance of each trajectory with a terminal evaluation r ( τ ) in E1 when a task T is given . At test time , we would like the agent to perform the task T , using zero extra interaction with the new but related environment , represented by E2 = ( S ′ , A , p′ ) . We assume the test environment E2 can have different state distribution and related but different dynamics from the experience environment E1 . However , we assume the actions that an agent could perform stay the same . We evaluate an agent on task T by giving a single terminal reward r ( τi ) per trajectory τi to its previous experiences . This reward is only used for evaluation and an agent can never utilize it while carrying off the task . The proposed formulation requires much less information and thus more realistic . In this paper , we focus on locomotion tasks with object interactions , such as Super Mario running with other objects in presence , or a Reacher robot acting with obstacles around . 3.2 THE SCORING-AGGREGATING-PLANNING ( SAP ) FRAMEWORK . We propose the Scoring-Aggregating-Planning framework to solve this problem . As an overview , we propose to learn a per step neural scoring function Fθ that scores a sub-region , subset of observation space that surrounds the agent . A local region score is simply a sum over all the sub-regions that are in the local region . The local region scores can be aggregated along the trajectory to approximate the terminal sparse reward . Meanwhile , a dynamics modelMφ is learned to approximate the true transition p ( ·|s , a ) based on past experience E1 . After both the scoring function and the dynamics model are learned , we perform a Model Predictive Control algorithm to get the final policy on the new environment E2 for the task T . Scoring The per step sub-region scoring function can be described as Fθ ( Wl ( st ) , at ) . Here θ denotes parameters , W is a function that extracts local regions from states and l is a sub-region indicator in a metric space L. We note that the neural network ’ s parameters are shared for every local region . A local region score then is ∑ l∈L Fθ ( Wl ( st ) , at ) . Intuitively , this function measures how well an action at performs in the current task , based on a local region l extracted from state s. We presume a local region can be extracted for scoring because in a physical environment , an agent could only interact with the world within its sensing capabilities . From another perspective , it can be seen as a decomposition of the sparse reward into the per-step rewards . Specifically , on the metric space associated with the problem , we divide the local region around the agent into n sub-regions l. For example , in a Cartesian coordinate system , we can divide each coordinate independently . For each l , we use a sub-region scoring network to produce a scoring table of size |A| × |L| , where |A| is the action dimension , and |L| denotes the number of possible relative positions around the agent . One entry is selected from the table as the score for this sub-region l based on the current action taken and the relative position of this sub-region l to the agent . Aggregating To approximate the terminal rewards and learn the scoring network , we aggregate the local region scores ∑ l∈L Fθ ( W ( sl ) , at ) for each step into a single aggregated score J , by an aggregating function G : Jθ ( τ ) = G ( st , at ) ∈τ ( ∑ l∈L Fθ ( Wl ( st ) , at ) ) The aggregated score J are then fitted to the sparse terminal reward . In practice , G is chosen based on the form of the final sparse terminal reward , e.g . a max or a sum function . In the learning process , the Fθ function is learned by back-propping errors between the terminal sparse reward and the predicted J through the aggregation function . In this paper , we use ` 2 loss that is : minθ 12 ( Jθ ( τ ) − r ( τ ) ) 2 . Planning To solve the task , we propose to use planning algorithms to find optimal action based on the learned scoring function and a learned dynamics model . As shown in the part ( c ) of Figure 2 , we learn a forward dynamics modelMφ based on the exploratory data with a supervised loss function . Specifically , we train a neural network that takes in the action at , state st and output ŝt+1 , which is an estimate of st+1 . We use an ` 2 loss as the objective : minφ 12 ( Mφ ( st , at ) − st+1 ) 2 With the learned dynamics model and the scoring function , we solve an optimization problem using the Model Predictive Control ( MPC ) algorithm to find the best trajectory for a task T in environment E2 . The objective of the optimization problem is to minimize −Jθ ( τ ′ ) . Here τ ′ is a H-step trajectory sampled based onMφ starting from some state si . The actual algorithm can be stated as starting from the current state st=i , we randomly select several action sequence up to length H . With the dynamics model , we can roll out the estimated states ŝt+1 , · · · , ŝtH . The learned scoring function and the aggregation function can give us an aggregated score for each of the action sequence . We select the action sequence that gives us the best aggregated score and execute the first action in the environment . And we repeat the procedure starting at the new state si .

Summary: The paper proposes a framework (Scoring-Aggregating-Planning (SAP)) for learning task-agnostic priors that allow generalization to new tasks without finetuning. The motivation for this is very clear - humans can perform much better than machines in zero-shot conditions because humans have learned priors about objects, semantics, physics, etc. This is achieved by learning a scoring function based on the final reward and a self-supervised learned dynamics model.