Assessing Generalization in Deep Reinforcement Learning

1 INTRODUCTION . Deep reinforcement learning ( RL ) has emerged as an important family of techniques that may support the development of intelligent systems that learn to accomplish goals in a variety of complex real-world environments ( Mnih et al. , 2015 ; Arulkumaran et al. , 2017 ) . A desirable characteristic of such intelligent systems is the ability to function in diverse environments , including ones that have never been encountered before . Yet , deep RL algorithms are commonly trained and evaluated on a fixed environment . The algorithms are evaluated in terms of their ability to optimize a policy in a complex environment , rather than their ability to learn a representation that generalizes to previously unseen circumstances . Indeed , their sensitivity to even subtle changes in the environment and the dangers of overfitting to a specific environment have been noted in the literature ( Rajeswaran et al. , 2017b ; Henderson et al. , 2018 ; Zhang et al. , 2018 ; Whiteson et al. , 2011 ) . Generalization is often regarded as an essential characteristic of advanced intelligent systems and a central issue in AI research ( Lake et al. , 2017 ; Marcus , 2018 ; Dietterich , 2017 ) . It refers to both interpolation to environments similar to those seen during training and extrapolation outside the training data distribution . The latter is particularly challenging but is crucial to the deployment of systems in the real world . Generalization in deep RL has been recognized as an important problem and is under active investigation ( Rajeswaran et al. , 2017a ; Pinto et al. , 2017 ; Kansky et al. , 2017 ; Yu et al. , 2017 ; Wang et al. , 2016 ; Duan et al. , 2016b ; Sung et al. , 2017 ; Clavera et al. , 2018 ; Sæmundsson et al. , 2018 ) . However , each work uses a different set of environments and experimental protocols . For example , Kansky et al . ( 2017 ) propose a graphical model architecture , evaluating on variations of the Atari game Breakout . Rajeswaran et al . ( 2017a ) propose training on a distribution of domains in risk-averse manner and evaluate on two continuous control tasks from MuJoCo ( Hopper and HalfCheetah ) . Duan et al . ( 2016b ) aim to learn a policy that automatically adapts to the environment dynamics and evaluate on bandits , tabular Markov decision processes , and maze navigation . Sæmundsson et al . ( 2018 ) combine learning a hierarchical latent model for the environment dynamics and model predictive control , evaluating on two continuous control tasks ( cart-pole swing-up and double-pendulum swing-up ) . What appears to be missing is a common testbed for evaluating generalization in deep RL : a clearly defined set of tasks , metrics , and baselines that can support concerted community-wide progress . In other words , research on generalization in deep RL has not yet adopted the ‘ common task framework ’ , a proven catalyst of progress ( Donoho , 2015 ) . Only by creating such testbeds and evaluating on them can we fairly compare and contrast the merits of different algorithms and accurately measure progress made on the problem . Our contribution is to establish a reproducible framework for investigating generalization in deep RL , with the hope that it will catalyze progress on this problem , and to present an empirical evaluation of generalization in deep RL algorithms as a baseline . We select a diverse but manageable set of environments , comprising classic control problems and MuJoCo locomotion tasks , built on top of OpenAI Gym for ease of adoption . Like Rajeswaran et al . ( 2017a ) and others , we focus on generalization to changes in the system dynamics , which is implemented by specifying degrees of freedom ( parameters ) along which the environment specifications can be varied . Significantly , we test generalization in two regimes : interpolation and extrapolation . Interpolation implies that agents should perform well in test environments where parameters are similar to those seen during training . Extrapolation requires agents to perform well in test environments where parameters are different from those seen during training . To provide the community with a set of clear baselines , we evaluate two deep RL algorithms on all environments and under different combinations of training and testing regimes . We chose one algorithm from each of the two major families : A2C from the actor-critic family and PPO from the policy gradient family . Using the same experimental protocol , we also evaluate two schemes for tackling generalization in deep RL : EPOpt , which learns a policy that is robust to environment changes by maximizing expected reward over the most difficult of a distribution of environment parameters , and RL2 , which learns a policy that can adapt to the environment at hand by taking into account the trajectory it sees . Because each scheme is constructed based on existing deep RL algorithms , our evaluation is of four algorithms : EPOpt-A2C , EPOpt-PPO , RL2-A2C , and RL2-PPO . We analyze the results and draw conclusions that can guide future work on generalization in deep RL . The experimental results confirm that extrapolation is more difficult than interpolation and show that the ‘ vanilla ’ deep RL algorithms ( A2C and PPO ) were able to interpolate fairly successfully . Somewhat surprisingly , they interpolate and extrapolate better than their EPOpt and RL2 variants , with the exception of EPOpt-PPO . RL2-A2C and RL2-PPO proved to be difficult to train and were unable to reach the level of performance of the other algorithms given the same amount of training resources . 2 RELATED WORK . Generalization in RL . There are two main approaches to generalization in RL : learning policies that are robust to environment variations , or learning policies that adapt to such variations . A popular approach to learn a robust policy is to maximize a risk-sensitive objective , such as the conditional value at risk ( Tamar et al. , 2015 ) , over a distribution of environments . Morimoto & Doya ( 2001 ) maximize the minimum reward over possible disturbances , proposing robust versions of the actorcritic and value gradient methods in a control theory framework . This maximin objective is utilized by others in the context where environment changes are modeled by uncertainties in the transition probability distribution function of a Markov decision process . Nilim & Ghaoui ( 2004 ) assume that the set of possible transition probability distribution functions are known , while Lim et al . ( 2013 ) and Roy et al . ( 2017 ) estimate it using sampled trajectories from the distribution of environments of interest . A recent representative of this approach applied to deep RL is the EPOpt algorithm ( Rajeswaran et al. , 2017a ) , which maximizes the conditional value at risk , i.e . expected reward over the subset of environments with lowest expected reward . EPOpt has the advantage that it can be used in conjunction with any RL algorithm . Adversarial training has also been proposed to learn a robust policy ; for MuJoCo locomotion tasks , Pinto et al . ( 2017 ) trains an adversary that tries to destabilize the agent during training . A robust policy may sacrifice performance on many environment variants in order to not fail on a few . Thus , an alternative , recently popular approach to generalization in RL is to learn a policy that can adapt to the environment at hand ( Yu et al. , 2017 ) . To do so , a number of algorithms learn an embedding for each environment variant using trajectories sampled from that environment , which is input into a policy . Then , at test time , the current trajectory can be used to compute an embedding for the current environment , enabling automatic adaptation of the policy . Duan et al . ( 2016b ) , Wang et al . ( 2016 ) , Sung et al . ( 2017 ) , and Mishra et al . ( 2018 ) , which differ mainly in the way embeddings are computed , consider model-free RL by letting the embedding be input into a policy and/or value function . Clavera et al . ( 2018 ) consider model-based RL , in which the embedding is input into a dynamics model and actions are selected using model predictive control . Under a similar setup , Sæmundsson et al . ( 2018 ) utilize probabilistic dynamics models and inference . This literature review has focused on RL algorithms for generalization that do not require updating the learned model or policy at test time , in keeping with our benchmark ’ s evaluation procedure . There has been work on generalization in RL that utilize such updates , primarily under the umbrellas of transfer learning , multi-task learning , and meta-learning . Taylor & Stone ( 2009 ) surveys transfer learning in RL where a fixed test environment is considered , with Rusu et al . ( 2016 ) being an example of recent work on that problem using deep networks . Ruder ( 2017 ) provides a survey of multi-task learning in general , which , different from our problem of interest , considers a fixed finite population of tasks . Finn et al . ( 2017 ) present a meta-learning formulation of generalization in RL and Al-Shedivat et al . ( 2018 ) extend it for continuous adaptation in non-stationary environments . Empirical methodology in deep RL . Shared open-source software infrastructure , which enables reproducible experiments , has been crucial to the success of deep RL . The deep RL research community uses simulation frameworks , including OpenAI Gym ( Brockman et al. , 2016 ) , the Arcade Learning Environment ( Bellemare et al. , 2013 ; Machado et al. , 2017 ) , DeepMind Lab ( Beattie et al. , 2016 ) , and VizDoom ( Kempka et al. , 2016 ) . The MuJoCo physics simulator ( Todorov et al. , 2012 ) has been influential in standardizing a number of continuous control tasks . For ease of adoption , our work builds on OpenAI Gym and MuJoCo tasks , allowing variations in the environment specifications in order to study generalization . OpenAI recently released a benchmark for transfer learning in RL ( Nichol et al. , 2018 ) , in which the goal is to train an agent to play new levels of a video game with fine-tuning at test time . In contrast , our benchmark does not allow fine-tuning and focuses on control tasks . Our work also follows in the footsteps of a number of empirical studies of reinforcement learning algorithms , which have primarily focused on the case where the agent is trained and tested on a fixed environment . Henderson et al . ( 2018 ) investigate reproducibility in deep RL , testing state-ofthe-art algorithms on four MuJoCo tasks : HalfCheetah , Hopper , Walker2d , and Swimmer . They show that results may be quite sensitive to hyperparameter settings , initialization , random seeds , and other implementation details , indicating that care must be taken not to overfit to a particular environment . The problem of overfitting in RL was recognized earlier by Whiteson et al . ( 2011 ) , who propose an evaluation methodology based on training and testing on multiple environments sampled from a distribution and experiment with three classic environments : MountainCar , Acrobot , and puddle world . Nair et al . ( 2015 ) evaluate generalization with respect to different starting points in Atari games . Duan et al . ( 2016a ) present a benchmark suite of continuous control tasks and conduct a systematic evaluation of reinforcement learning algorithms on those tasks . They consider generalization in terms of interpolation on a subset of their tasks . In contrast to these works , we address a greater variety of tasks , extrapolation as well as interpolation , and algorithms for learning deep RL agents that generalize .

Summary: This paper presents a new benchmark for studying generalization in deep RL along with a set of benchmark results. The benchmark consists of several standard RL tasks like Mountain Car along with several Mujoco continuous control tasks. Generalization is measured with respect to changes in environment parameters like force magnitude and pole length. Both interpolation and extrapolation are considered.