Environment Probing Interaction Policies

1 INTRODUCTION . Over the last few years deep reinforcement learning ( RL ) has shown tremendous progress . From beating the world ’ s best at Go ( Silver et al. , 2016 ) to generating agile and dexterous motor policies ( Schulman et al. , 2015 ; Lillicrap et al. , 2015 ; Peng et al. , 2018 ) , deep RL has been used to solve a variety of sequential decision-making tasks . However , one key issue that deep RL faces is transfer : a policy trained in one environment is extremely dependent on the specific parameters of that environment ( Rajeswaran et al. , 2017 ) and fails to generalize to a new environment . Because of this , training agents that can generalize to novel environments has become an active area of research ( Rajeswaran et al. , 2017 ; Pinto et al. , 2017b ; Yu et al. , 2017 ; Peng et al. , 2017 ) . The most common approach for environment generalization is to train a single policy over a variety of different environments ( Rajeswaran et al. , 2017 ; Peng et al. , 2017 ) , thus allowing the policy network to learn invariances to the underlying environmental factors . However , a monolithic policy that works on multiple environments often ends up being conservative since it is unable to exploit the dynamics of a given environment . We argue that instead of learning a policy that is invariant to environment dynamics we need a policy that adapts to its environment . A popular approach for this is to first perform explicit system identification of the environment , and then use the estimated parameters to generate an environment specific policy ( Yu et al. , 2017 ) . However , there are three reasons why this approach might be infeasible : ( a ) it is based on the structural assumption that only a known set of parameters matter ; ( b ) estimating those parameters from few observations is an extremely challenging problem ; ( c ) these approaches require access to environment parameters during training time , which might be impractical . So is it possible to learn an environment dependent policy without estimating every environment parameter ? When humans are tasked to perform in a new environment , we do not explicitly know what parameters affect performance ( Braun et al. , 2009 ) . Instead , we probe the environment to gain an intuitive understanding of its behavior . The purpose of these initial interactions is not to complete the task immediately , but to extract information about the environment . This process facilitates learning in that environment . Inspired by this , we present a framework in which the agent first performs “ environment-probing ” interactions that extract information from an environment , then leverages this information to achieve the goal with a task-specific policy . This setup entails three challenges : 1 ) How should an agent interact with an environment ? 2 ) How can information be extracted from such interactions ? 3 ) How can the extracted information be used to learn a better policy ? One way to interact is by executing random actions in the environment . However , this may not expose the nuances of an environment . Another way is to use samples from the task-specific policy to infer the environment ( Yu et al. , 2017 ) . However , a good task-specific policy may not be optimal for understanding an environment . Instead , we separately optimize an “ Environment-Probing ” Interaction policy ( EPI-policy ) to extract environment information . To assign rewards to the EPIpolicy , our key insight is that the agent ’ s ability to predict transitions in an environment can be used as a metric of its understanding of that environment . If an “ environment-probing ” trajectory can help in predicting transitions , then it should be a good trajectory . More specifically , we use the “ environment-probing ” trajectory to generate an environment embedding vector . This compact embedding vector is passed to a prediction model that provides higher rewards to the trajectories that are useful in estimating environment-dependent transitions . Once we have learned the EPI-policy , we can use the generated environment embedding as an input to the task-specific policy . Since this task-specific policy is conditioned on the information from the environment , it can now produce actions dependent on the nuances of the specific environment . We evaluate the effectiveness of using EPI on Hopper and Striker from OpenAI Gym ( Brockman et al. , 2016 ) with randomized mass , damping and friction . In both of these environments , the performance of our policies is better than several commonly used techniques and is even comparable to an oracle policy that has full access to the environment information . These results , along with a detailed ablation analysis , demonstrates that implicitly encoding the environment using EPI-policies improves task performance in unseen testing environments . 2 RELATED WORK . Developing generalizable machine learning models is a longstanding challenge . One approach to transfer models across different input distributions is domain adaptation . Duan et al . ( 2012 ) along with several others ( Hoffman et al. , 2014 ; Kulis et al. , 2011 ; Long et al. , 2015 ) domain-adapt visual models . With the onset of deep learning , several works have investigated finetuning ( retraining ) pretrained networks in different target domains ( Yosinski et al. , 2014 ; Li et al. , 2016 ; Tzeng et al. , 2014 ) . In reinforcement learning , Gupta et al . ( 2017 ) learns different encoders for different environments that share a common feature space to enable transfer . Improving the generalizability of models can also be formulated as a meta-learning problem . Finn et al . ( 2017 ) and Yu et al . ( 2018 ) finetune the models in new environments . We argue that agents should have the capacity to generalize without updating their policy . Duan et al . ( 2016b ) and Wang et al . ( 2016 ) optimize the policy over multiple episodes and use RNN to implicitly model the underlying changing dynamics in the hidden states . Hausman et al . ( 2018 ) purposes to learn an embedding in a supervised manner by performing variational inference on task ids . These methods are based on model-free meta-RL , while Sæmundsson et al . ( 2018 ) and Clavera et al . ( 2018 ) investigate in model-based meta-RL . Sæmundsson et al . ( 2018 ) infers latent variables using Gaussian Process , while we use neural networks to create an embedding and experimented with more complex high-dimensional continuous control problems . Concurrent work Clavera et al . ( 2018 ) proposes to train a dynamics model that adapts online using either gradient based method ( Finn et al. , 2017 ) or recurrent based method ( Duan et al. , 2016b ) . The most important difference is that we have a separate policy to probe the environment instead of using the trajectories from the execution of the task . In our method , training prediction models is somewhat related to learning dynamics model in the model-based RL . However , the prediction model in our method does not to be very accurate since we are only taking the difference . Another stream of research that is relevant to our work is simulation-to-real transfer . Here the goal is to train a policy in a physics simulator and then transfer that policy to a real robot . The challenge here is to bridge the “ Reality Gap ” between the real world and the simulator . Rusu et al . ( 2016 ) attempt to do this via architectural novelty for visual domain adaptation . Sadeghi & Levine ( 2016 ) ; Tobin et al . ( 2017 ) ; Pinto et al . ( 2017a ) and James et al . ( 2017 ) perform domain randomization on the inputs during training to learn a transferable policy . These methods target the mismatch of observation space . However , we target the mismatch in the dynamics of the environment . Peng et al . ( 2017 ) proposes domain randomization of the environment parameters and learning an LSTM policy to implicitly learn the environment . We compare our method to this approach and show that a separate process that extracts the nuances of an environment is vital for better generalization . Rajeswaran et al . ( 2017 ) and Pinto et al . ( 2017b ) also target generalization to environments , however they do not incorporate environment information and hence are only able to learn a conservative policy . The traditional approach of system identification ( Milanese et al. , 2013 ; Gevers et al. , 2006 ; Bongard & Lipson , 2005 ; Armstrong , 1987 ) aims to explicitly estimate the varying environment parameters . Yu et al . ( 2017 ) propose the UP-OSI framework that can perform online system identification and takes the estimated environment parameters as additional input to the policy . This works well when the number of parameters to estimate is few . However , explicit identification of every environment parameter is a extremely challenging . Denil et al . ( 2016 ) trains an policy to gather information about physical properties through interactions in intuitive physics to answer questions , while we are trying do dealt with a more complex continuous control setting . A recent approach by Pathak et al . ( 2017 ; 2018 ) uses errors in prediction models as an intrinsic reward ( Chentanez et al. , 2005 ) to improve exploration in RL . The strategy is to take actions to reach the states that maximize the error in the learned prediction model . In contrast , our method selects actions that reduce the prediction error on the heldout prediction dataset of task policy trajectories . 3 BACKGROUND : REINFORCEMENT LEARNING . We use the framework of reinforcement learning to learn the interaction policy and the final task target policy . This section contains brief preliminaries for reinforcement learning . Note that a more comprehensive exposition can be found in Kaelbling et al . ( 1996 ) and Sutton & Barto ( 1998 ) . We model our problem as a continuous space Markov Decision Process represented as the tuple ( S , A , P , r , γ , S ) , where S is a set of continuous states of the agent , A is a set of continuous actions the agent can take at a given state , P : S × A × S → R is the transition probability function , r : S ×A → R is the reward function , γ is the discount factor , and S is the initial state distribution . The goal for an agent is to maximize the expected cumulative discounted reward ∑T−1 t=0 γ tr ( st , at ) by performing actions according to its policy πθ : S → A . Here , θ denotes the parameters for the policy π which takes action at given state st at timestep t. There are various techniques to optimize πθ . The one we use in this paper is a policy gradient approach , TRPO ( Schulman et al. , 2015 ) . 4 APPROACH . Our objective is to learn a generalizable policy that is trained on a set of training environments , but can succeed across unseen testing environments . To this end , instead of a single policy , we have two separate policies : ( a ) an Environment-Probing Interaction policy ( EPI-policy ) that efficiently interacts with an environment to collect environment-specific information , and ( b ) a task-specific policy that uses this additional environment information ( estimated via EPI-policy ) to achieve high rewards for a given task . In this section , we present our methodology for learning these two policies .

Summary: The submission presents a reinforcement learning method for exploring/probing the environment to determine an environment’s properties and exploit these during later tasks. The method relies on jointly learning an embedding that simplifies prediction of future states and a policy that maximises a curiosity/intrinsic motivation like reward to learn to explore areas where the prediction model underperforms. In particular, the reward is based on the difference between prediction based on the learned embedding and prediction based on a prior collected dataset, such that the reward optimises to collect data with a large difference between the prediction accuracy of both models. The subsequently frozen policy and embedding are then used in other domains in a system identification like manner with the embedding utilised as input for a final task policy. The method is evaluated on a striker and hopper environment with varying dynamics parameters and shown to outperform a broad set of baselines.