Learning Synthetic Environments and Reward Networks for Reinforcement Learning

We introduce Synthetic Environments ( SEs ) and Reward Networks ( RNs ) , represented by neural networks , as proxy environment models for training Reinforcement Learning ( RL ) agents . We show that an agent , after being trained exclusively on the SE , is able to solve the corresponding real environment . While an SE acts as a full proxy to a real environment by learning about its state dynamics and rewards , an RN is a partial proxy that learns to augment or replace rewards . We use bi-level optimization to evolve SEs and RNs : the inner loop trains the RL agent , and the outer loop trains the parameters of the SE / RN via an evolution strategy . We evaluate our proposed new concept on a broad range of RL algorithms and classic control environments . In a one-to-one comparison , learning an SE proxy requires more interactions with the real environment than training agents only on the real environment . However , once such an SE has been learned , we do not need any interactions with the real environment to train new agents . Moreover , the learned SE proxies allow us to train agents with fewer interactions while maintaining the original task performance . Our empirical results suggest that SEs achieve this result by learning informed representations that bias the agents towards relevant states . Moreover , we find that these proxies are robust against hyperparameter variation and can also transfer to unseen agents . 1 INTRODUCTION . Generating synthetic data addresses the question of what data is required to achieve a rich learning experience in machine learning . Next to increasing the amount of available data , synthetic data can enable higher training efficiency that opens up new applications for Neural Architecture Search ( Such et al. , 2020 ) , may improve algorithm analysis or facilitate custom datasets ( Jhang et al. , 2020 ) . In this paper , we consider learning neural synthetic data generators for Reinforcement Learning ( RL ) . We investigate the question of whether we can learn a synthetic Markov Decision Process of a real ( target ) environment which is capable of producing synthetic data to allow effective and more efficient agent training , that is , to achieve similar or higher performance more quickly compared to when training purely on the real environment . When learning to produce both states and rewards , we refer to these neural network proxies as synthetic environments ( SEs ) . Additionally , we investigate the same question for learning reward proxies that do not learn about the state dynamics and which we refer to as a Reward Networks ( RNs ) . We depict our procedure in Figure 1which resembles a bi-level optimization scheme consisting of an outer and inner loop . The inner loop trains the agent on an SE or RN . Since our method is agnostic to both domain and agent , we can interchangeably adopt standard RL algorithms in the inner loop . In the outer loop , we assess the agent ’ s performance by evaluating it on the real environment ; we then take the collected reward as a score to update the SE ’ s or RN ’ s neural parameters used in the inner loop . In this way , the SE/RN is gradually updated such that an agent being trained on it , scores higher on a real environment . For the outer loop we use Evolution Strategies ( Rechenberg , 1973 ; Salimans et al. , 2017 ) with a population of SE/RN parameters . After discussing related work ( Section 2 ) , we make the following contributions : • We introduce synthetic environments ( Section 3 ) , a novel concept that focuses on the environment instead of agent learning using a bi-level optimization scheme , which is guided purely by the agent performance . This concept goes beyond the usual learning of a onetime internal environment model inside an agent ( such as in Dyna ( Sutton , 1990 ) ) . • As a sub-problem of SEs , we investigate reward networks ( Section 4 ) , contrasting several types of potential-based reward shaping ( Ng et al. , 1999 ) variants . • We show that it is possible to learn SEs ( Section 5 ) and RNs ( Section 6 ) that , when used for agent training , yield agents that successfully solve the Gym tasks ( Brockman et al. , 2016 ) CartPole and Acrobot ( SEs ) , as well as Cliff Walking , CartPole , MountainCarContinuous , and HalfCheetah ( RNs ) . • We show that SEs and RNs are efficient and robust in training agents , require fewer training steps compared to training on the real environment and are able to train unseen agents • We report empirical evidence showing that SEs and RNs achieve their efficiency gains for training new agents through condensed and informed state and reward representations . Overall , we find it noteworthy that it is actually possible to learn such proxies , and we believe our research will improve their understanding . Since these learned proxies can train agents quickly and transfer to unseen agents , we believe this work might open up possible avenues of research in RL . Possible future applications include cheap-to-run environments for AutoML ( Hutter et al. , 2019 ) or for robotics when training on targets is expensive , as well as agent and task analysis , or efficient RL agent pre-training . Our PyTorch ( Paszke et al. , 2019 ) code and models are made available publicly.1 2 RELATED WORK . Synthetic Environments In the context of RL , learning synthetic environments is related to model-based RL ( MBRL ) where the dynamics model can be viewed as an SE . In MBRL , one jointly learns both a dynamics model and a policy as in Dyna ( Sutton , 1990 ) or an existing dynamics model is used with planning methods to learn a policy ( Silver et al. , 2017 ; Moerland et al. , 2020 ) . Our work does not involve planning , nor does it use supervised learning for the dynamics model and it does not mix synthetic and real data during policy learning . Instead , we use purely synthetic data to train agents similar to World Models ( Ha & Schmidhuber , 2018 ) but use the cumulative rewards from the real environment in a bi-level optimization for learning our model jointly with our agent . For a more extensive discussion on the differences between our work and model-based RL , we refer the reader to Appendix D. Analogous to learning SEs is Procedural Content Generation ( Togelius et al. , 2011 ) and Curriculum Learning that concerns automatically selecting ( Matiisen et al. , 2020 ) or generating the content of training environments ( Volz et al. , 2018 ; Shaker et al. , 2016 ; Wang et al. , 2019 ; Cobbe et al. , 2020 ) , with the closest work being Generative Playing Networks ( GPNs ) ( Bontrager & Togelius , 2020 ) . GPNs learn an environment generator that creates increasingly difficult SEs according to what a critic ’ s value function estimates as challenging . While GPNs are a method to generate environment curricula , our approach studies learning an SE for effective and efficient agent training by compressing the relevant information into a single model of the environment . We also use a more generally 1https : //github.com/automl/learning environments applicable objective that does not rely on actor-critic formulations but purely on the achieved cumulative reward . Less related areas are methods like domain randomization ( Tobin et al. , 2017 ) that generate environments regardless of agent performance or minimax methods that adversarially ( instead of cooperatively ) generate environments based on the difference between the reward of two competing agents that are trying to solve the generated environment ( Dennis et al. , 2020 ) . Related to our work , and inspiring it , are Generative Teaching Networks ( Such et al. , 2020 ) . While we similarly use a bi-level optimization to learn a synthetic data generator , our approach is different in central aspects : we use Evolution Strategies to avoid the need for explicitly computing Hessians , we do not use noise vectors as input to our SEs , and we target sequential decision-making problems instead of supervised learning . Reward Networks Reward shaping concerns the question of how to enhance the reward signal to allow agents to be trained more effectively or efficiently . Common learned reward shaping approaches are curiosity or count-based exploration ( Pathak et al. , 2017 ; Burda et al. , 2019 ; Singh et al. , 2010 ; Bellemare et al. , 2016 ; Tang et al. , 2017 ) . Others achieve reward shaping with prior knowledge through expert demonstrations ( Judah et al. , 2014 ; Brys et al. , 2015 ; Ibarz et al. , 2018 ) . In contrast to our work , these contributions all apply a single-level optimization . When using a bilevel optimization , the reward shaping function is usually learned in the outer loop while the policy using the learned rewards is optimized in the inner loop . Here , one way is to meta-learn the parameterization of reward functions ( Faust et al. , 2019 ; Hu et al. , 2020 ; Jaderberg et al. , 2019 ) . Another way is to learn a neural network that resembles the reward function . While learning full synthetic environments is entirely novel , there exists prior work on learning reward shaping networks . The most related works are ( Zheng et al. , 2018 ) for single tasks , ( Zou et al. , 2019 ) for entire task distributions , or ( Zheng et al. , 2020 ) that additionally take into account the entire lifetime of an agent to learn a “ statefulness across episodes ” -reward function . Despite the similarities , some noteworthy differences exist . Importantly , the approaches in ( Zheng et al. , 2018 ; Zou et al. , 2019 ) are not agent-agnostic , making it less straightforward to exchange agents as in our work . Moreover , the transferability of learned shaped rewards is studied only limitedly ( Zheng et al. , 2018 ) , not at all ( Zou et al. , 2019 ) or only for grid world-like environments ( Zheng et al. , 2020 ) . 3 LEARNING SYNTHETIC ENVIRONMENTS . Problem Statement Let ( S , A , P , R ) be a Markov Decision Process ( MDP ) with the set of states S , the set of actions A , the state transition probabilities P and the immediate rewards R when transitioning from state s ∈ S to the next state s′ ∈ S through action a ∈ A . The MDPs we consider are either human-designed environments Ereal or learned synthetic environments Esyn , ψ ( SE ) represented by a neural network with parameters ψ. Interfacing with the environments is identical in both cases , i.e . s′ , r = E ( s , a ) . The crucial difference is that for SEs , the state dynamics and rewards are learned . The main objective of an RL agent when acting on an MDP Ereal is to find a policy πθ parameterized by θ that maximizes the cumulative expected reward F ( θ ; Ereal ) . We consider the following bi-level optimization problem : find the parameters ψ∗ , such that the agent policy πθ parameterized with θ that results from training on Esyn , ψ∗ achieves the highest reward on a target environment Ereal . Formally that is : ψ∗ = arg max ψ F ( θ∗ ( ψ ) ; Ereal ) s.t . θ∗ ( ψ ) = arg max θ F ( θ ; Esyn , ψ ) . ( 1 ) We use standard RL algorithms for optimizing the agents on the SE in the inner loop . Although gradient-based optimization methods can be applied in the outer loop , we chose Natural Evolution Strategies ( Wierstra et al. , 2008 ) ( NES ) to allow the optimization to be independent of the choice of the agent in the inner loop and to avoid potentially expensive , unstable meta-gradients ( Metz et al. , 2019 ) . Additional advantages of NES are that it is better suited for long episodes , sparse or delayed rewards , and parallelization ( Salimans et al. , 2017 ) . Algorithm We now explain our method . The overall scheme is adopted from ( Salimans et al. , 2017 ) and depicted in Algorithm 1 . It consists of an Evolutionary Strategy in the outer loop to learn the SE and an inner loop which trains RL agents . The performances of the trained agents are then used in the outer loop to update the SE . We instantiate the population search distribution as a multivariate Gaussian with mean 0 and a fixed covariance σ2I . The main difference to ( Salimans et al. , 2017 ) is that , while they maintain a population over agent parameter vectors , our population consists of SE parameter vectors . Moreover , our approach involves two optimizations ( the agent and the SE parameters ) instead of one ( agent parameters ) . Algorithm 1 : Learning Synthetic Env . with NES 1 Input : initial SE parameters ψ , real environment Ereal , NES noise std . dev . σ , number of episodes ne , population size np , NES step size α 2 repeat 3 foreach member of pop . i = 1 , 2 , . . . , np do 4 ϵi ∼ N ( 0 , σ2I ) 5 ψi = ψ + ϵi 6 θi = TrainAgent ( θi , Esyn , ψi , ne ) 7 Fψ , i = EvaluateAgent ( θi , Ereal ) 8 Update SE : ψ ← ψ + α 1npσ ∑np i Fiϵi 9 until no steps Our algorithm first stochastically perturbs each population member i which results in ψi ( Line 5 ) . Then , a new randomly initialized agent is trained on the SE parameterized by ψi for ne episodes ( L6 ) . The trained agent with fixed parameters is then tested across 10 episodes on the real environment ( L7 ) , yielding the average cumulative reward which we use as a score Fψ , i . Finally , we update ψ with a stochastic gradient estimate based on all member scores ( L8 ) . We use a parallelized version of the algorithm and an early stopping heuristic to stop in fewer than ne episodes when progress plateaus ( more in Appendix A.1 ) .

Summary: This paper aims to learn proxy environments (synthetic environments or SEs) and reward functions (reward networks or RNs), parameterized as neural networks, such that these proxy models provide beneficial transitions to make it more sample-efficient to learn a policy for a fixed target environment (referred to as the real environment). SEs replace the observed state and rewards during training, while reward networks provide a synthetic reward that is a function of the real reward, current state, and next state. The proposed method formulates this problem as a bi-level optimization, where the inner loop consists of standard RL under the proxy model, and the outer loop consists of NES with the aim of optimizing either the performance on the true, target environment (SEs) or a the number of training steps needed to reach a certain return threshold on the real environment.