Deep Reinforcement Learning of Universal Policies with Diverse Environment Summaries

Deep reinforcement learning has enabled robots to complete complex tasks in simulation . However , the resulting policies do not transfer to real robots due to model errors in the simulator . One solution is to randomize the simulation environment , so that the resulting , trained policy achieves high performance in expectation over a variety of configurations that could represent the real-world . However , the distribution over simulator configurations must be carefully selected to represent the relevant dynamic modes of the system , as otherwise it can be unlikely to sample challenging configurations frequently enough . Moreover , the ideal distribution to improve the policy changes as the policy ( un ) learns to solve tasks in certain configurations . In this paper , we propose to use an inexpensive , kernel-based summarization method method that identifies configurations that lead to diverse behaviors . Since failure modes for the given task are naturally diverse , the policy trains on a mixture of representative and challenging configurations , which leads to more robust policies . In experiments , we show that the proposed method achieves the same performance as domain randomization in simple cases , but performs better when domain randomization does not lead to diverse dynamic modes . 1 INTRODUCTION . Reinforcement learning ( RL , Sutton & Barto ( 1998 ) ) is a powerful paradigm that has enabled impressive results in controlled environments such as games ( Mnih et al. , 2015 ) and simulated robotic systems ( Lillicrap et al. , 2015 ) . However , it has proven difficult to transfer these successes to physical systems , as the simulations that are used for training are only an approximation of the real world . Consequently , the trained policies may fail to complete the task on the physical system and expensive and impractical retraining in the physical world might be required . In this paper , we consider the problem of training policies that are robust with respect to these errors in the simulator . Specifically , we present an algorithm that adaptively identifies a representative and diverse set of simulator configurations , which is then used to train the policy , see Fig . 1 . Related work Training policies that are robust towards model errors in robotics has mostly been considered in model-based RL . There , Bayesian models have been used in order to compute policies that are robust towards the worst-case model ( Akametalu et al. , 2014 ; Berkenkamp et al. , 2017 ) . By estimating and improving the model online , these methods allow for safe policy improvement on the real system . However , these methods suffer from the curse of dimensionality . In high-dimensional state spaces , model-free RL has been the most successful ( Kober & Peters , 2014 ) . In particular , methods based on the natural policy gradient ( Kakade , 2002 ) , e.g. , proximal policy optimization ( PPO ) ( Schulman et al. , 2017 ) , and deterministic policy gradients ( DPG ) ( Silver et al. , 2014 ) together with neural network policies have been successful . These methods estimate the gradient of the performance based on trajectories induced by the current policy . However , the resulting policies are not robust towards model errors and may fail to work on the real system . One solution is to consider transfer learning ( Taylor & Stone , 2009 ) or multi-task learning and environments . Wulfmeier et al . ( 2017 ) uses data gathered on the real system in order to improve the policy on the simulator in parts of the state space that were visited by the real system . An alternative approach by Marco et al . ( 2017 ) uses a Bayesian approach to transfer knowledge about the optimal policy from simulation to a robot . Both approaches require experiments on the real system . A related Figure 1 . Illustration of the algorithm . In order to train robust and universal policies effectively , our algorithm maintains a representative set of environments for policy training . This set is diverse in statetrajectories and captures diverse dynamic modes of the underlying system . The summary is updated by evaluating random configurations , which leads to a data-efficient algorithm . approach is used by ( Rusu et al. , 2017 ) , which first trains a policy in simulation and then extended the neural network for additional training on the real system . Alternatively , one can render policies more robust towards model errors on the simulator directly , without experiments on the physical system . This idea is similar to multi-task learning , where the goal is to train policies that perform well across multiple tasks ( Devin et al. , 2017 ; Teh et al. , 2017 ) . Training a policy accross multiple simulators has first been used in computer vision , where color and lighting are randomized to make the resulting policy transfer better to real-world conditions ( Sadeghi & Levine , 2017 ; Tobin et al. , 2017 ) . Tan et al . ( 2018 ) use the same idea to randomize physical parameters of the robot . However , while lighting conditions affect the state observations directly and are designed to be diverse , sampling physical parameters uniformly does not necessarily lead to diverse state trajectories . We will show that this difference can cause randomization to perform poorly when difficult parameter combinations are only sampled infrequently . Instead of considering average performance , both Yu et al . ( 2017 ) and Rajeswaran et al . ( 2016 ) aim to optimize the worst-case performance across all configurations . To this end , they sample simulator configurations and only use the ones with the worst performance under the current policy for training . However , these approaches require a large number of simulations , most of which are not used to update the policy , and are thus computationally expensive . Another issue that can arise in these settings is catastrophic forgetting ( Goodfellow et al. , 2013 ) , where by training only on the worst-case simulator the policy unlearns how to perform in previously solved simulator configurations . Another approach by Pinto et al . ( 2017 ) considers adversarial training , where an adversary locally disrupts the simulation . However , local approaches typically do not lead to policies that perform well across all configurations . We avoid these problems by using a small but representative summary of simulator configurations instead . These kind of representative summaries have previously been considered for sensor placement ( Krause et al. , 2008 ) , where the goal is to cover a space with sensors in order to increase the amount of information gained while minimizing the cost of placing the sensors . Such representative summaries have also been used in sequential action selection ( Dey et al. , 2013 ) , path planning ( Dey et al. , 2012 ) , and information gathering tasks ( Choudhury et al. , 2017 ; Binney et al. , 2010 ) . These works exploit structure in the problem known as submodularity ( Nemhauser et al. , 1978 ) , which leads to tractable algorithms with provably close-to-optimal performance . Contribution In this paper , we use ideas from data summarization in order to obtain a representative summary of simulator configurations that capture the diverse and relevant dynamic modes of the underlying system . Our key contributions are , first , to propose a similarity metric that allows us to quantify differences in simulator configurations that are invariant to parameter transformations by encoding dynamic behavior that is physically meaningful . Specifically , we use a kernel that measures similarity of simulator configurations based on the trajectories induced by the current policy . Secondly , we propose a tractable algorithm based on streaming submodular optimization to efficiently and adaptively compute a summary of configurations based on the similarity metric . Together , these two contributions ensure that the resulting summary covers the space of statetrajectories induced by different simulator configurations well . Thus , by training on this diverse summary , the resulting policies perform well across all dynamic modes of the simulators . In cases where uniform sampling of configurations already leads to diverse dynamic modes , our method achieves the same performance and robustness as domain randomization . However , when this is not the case and domain knowledge would be needed to specify a suitable sampling distribution , training on diverse configurations leads to significantly improved performance and robustness . 2 PROBLEM STATEMENT AND BACKGROUND . We consider a simulator with state space S ⊆ Rd and actions in A . During simulation , the whole state space is observed . The simulator is parameterized by a set of physically meaningful parameters M. These parameters represent different simulator configurations by , for example , modify the task or changing physical properties of a robot , such as mass or friction . The goal is to train a single policy that performs well across all the possible configurations in M. Since this set represents all uncertainties about the real-world system , policies that perform well across all configurations are likely to perform well in the real-world too . The simulator parameters affect the transition dynamics , so that each choice of parameters m ∈M encodes a separate Markov decision process 〈S , A , Tm , c , γ〉 , where the stochastic transition dynamics Tm depend on the simulator parameters m. Performance for a fixed simulator configuration m ∈ M under a policy πθ is measured via the expected γ-discounted future rewards r ( · ) , which is given by Jθ ( m ) = Eat∼πθ ( st ) , st+1∼Tm ( st , at ) [ ∑∞ t=0 γ tr ( st , at ) ] . This objective can be maximized for a fixed model m by estimating the gradients with respect to the policy parameters θ from state trajectories obtained through simulated rollouts ( Kakade , 2002 ) . To achieve robustness across all configurationsM , ideally we would optimize the worst-case performance directly : maxθ minm∈M Jθ ( m ) . However , this problem is intractable and , even if it was , gradient-based updates based on the worst-case configuration may not converge for neural network policies . Consider the problem in Fig . 2a , where the goal is to reach a target behind an obstacle and the simulator is parameterized by the starting position at the bottom . If the initial worst-case starting position is left of the obstacle , the gradient update will push state trajectories away from the obstacle towards the right . In contrast , when the starting position is right of the obstacle , the policy update will push the trajectory to the left . These two opposing updates can lead to an oscillating cycle during the optimization with slow or no convergence to the optimal policy . The underlying problem is that the gradient is estimated based on trajectories that do not cover the state-space sufficiently , so that policy improvement to the left of the obstacle leads to worse performance on the right side and vice-versa . This kind of forgetting is well-known in multi-task reinforcement learning ( Goodfellow et al. , 2013 ) and was also hinted at in ( Rajeswaran et al. , 2016 ) . To avoid this issue in practice , one can instead replace the minimum by a finite sum over a small , but representative set of modelsMs ⊂M , max θ ∑ m∈Ms Jθ ( m ) , ( 1 ) which can be easily solved using standard batch policy gradient methods . The choice of Ms is critical , since it needs to capture all relevant dynamic modes inM that cover the state space in order to avoid training issues as in Fig . 2a . Domain randomization typically samples Ms uniformly at random fromM after every policy update . However , this only ensures coverage in the parameter space M. The resulting coverage in terms of state-trajectories depends on the parameterization ofM , transition dynamics , and the policy . For example , in the model in Fig . 2a uniform sampling in parameter space leads to trajectories that pass to the left of the obstacle more frequently than to the right . This leads to a training bias , where parameters on the right have worse performance , since the corresponding trajectories are underrepresented during training . Instead , the setMs should be selected to be diverse in state-trajectories , as shown by the blue lines in Fig . 2a . This ensures that also tasks that parameter configurations that are sampled infrequently under the uniform distribution achieve good performance . As a consequence , the resulting policies trained on diverse trajectories are more robust . In summary , we want to select a setMs that represents diverse dynamic modes and state-trajectories of the underlying system , without exploiting domain knowledge about how to parameterize M. Moreover , we want to compute this set efficiently , without requiring large amount wasteful rollouts that are not used for training as in ( Yu et al. , 2017 ; Rajeswaran et al. , 2016 ) . Submodular Optimization As stated previously , the parameters in Ms should represent diverse induced state-trajectories . This kind of diversity requirement has previously been considered , especially in form of the sensor placement problem ( Krause et al. , 2008 ) . There , the goal is to cover a space with sensors in order to maximize the amount of information gained . In general , this is a difficult combinatorial optimization problem , which requires evaluating every possible combination of sensors in order to find the best one . To obtain a tractable and probably close-to-optimal algorithm , Krause et al . ( 2008 ) exploit a diminishing returns property of the problem known as submodularity . Conceptually , submodularity states that adding a sensor to a small set of existing sensors always provides more information than when we already have a large set of sensors that cover the space . More precisely , a set function f that maps a subset Ms of the domain M to the reals is called submodular if , for any two summariesMs ⊂ U ⊂M and any additional new element m ∈ M , it holds that that f ( Ms ∪ { m } ) − f ( Ms ) ≥ f ( U ∪ { m } ) − f ( U ) ; that is , for the larger set U adding a new sensor m helps less than for the small set S. Another natural assumption is monotonicity , which means that larger summaries are always better , f ( U ) ≥ f ( Ms ) . Maximizing monotone submodular functions can be done efficiently using the greedy algorithm with an approximation ratio of at least 1− 1/e ≈ 0.68 relative to the optimal solution ( Nemhauser et al. , 1978 ) . The greedy algorithm starts with the empty set Ms = { } and greedily selects the element that maximally improves the function value within one step , until a constraint on the size ofMs is reached . Ms ←Ms ∪ argmax m∈M f ( Ms ∪ { m } ) . ( 2 ) Streaming Submodular Optimization The greedy algorithm in ( 2 ) iteratively constructs the set Ms by evaluating all possible elements m in M that could be added to the current set . This requirement is relaxed by streaming submodular optimization algorithms , which assume that the data comes in as a stream and one has to decide for each element whether to add it to the summary or disregard it . The STREAM-GREEDY algorithm by Krause & Gomes ( 2010 ) considers a conceptually straight-forward extension of the greedy algorithm ( 2 ) . For each new element m in the stream , they consider if there is a benefit of replacing one of the current elements m′ ∈ Ms in the current summary with m , ( m′ , m ) = argmax m′∈Ms , m∈Ms∪ { m } f ( Ms \ { m′ } ∪ { m } ) , Ms ←Ms \ { m′ } ∪ { m } , ( 3 ) This algorithm also enjoys constant-factor optimization guarantees ( Krause & Gomes , 2010 ) . The bounds can be further improved with a more complicated algorithm ( Badanidiyuru et al. , 2014 ) .

Summary: This paper addresses the problem of finding a policy that will perform well in a real environment when training in a simulator that may have errors.  It takes the now standard approach of trying to find a policy that performs well in an ensemble of simulated environments that are perturbations of the basic simulator.   The question is:  how can we construct an ensemble that represents the uncertainty about the real world well while being small enough for computational efficiency.  The idea is to construct a diverse set of samples that represents the whole space of important variations in the simulation;  the particular novelty here is to ensure that the sample set attains coverage over the *behaviors* of the simulator rather than the parameters of the simulation.  This problem is made difficult by the fact that there is no finite space of samples to choose from and the fact that we don't have a natural distance metric on the simulator behavior.