Do recent advancements in model-based deep reinforcement learning really improve data efficiency?

1 INTRODUCTION . Producing fully independent agents that learn optimal behavior and develop over time purely by trial and error interaction with the surrounding environment is one of the prominent dilemmas in the field of artificial intelligence . A mathematical framework that encapsulates the problem of these autonomous systems is reinforcement learning . Over the past few years , exceptional progress has been made in devising artificial agents that can learn and solve problems in a variety of domains using deep RL methods ( Mnih et al. , 2015 ; Schulman et al. , 2015 ; Silver et al. , 2016 ) . However , these algorithms are perceived as extremely data inefficient . They are thought to require an immense amount of non-optimal interaction with the real environment before they begin to operate acceptably well ( Irpan , 2018 ) . One of the most popular benchmarks for assessing overall performance and data complexity of deep RL algorithms is Atari Learning Environment ( Bellemare et al. , 2013 ; Machado et al. , 2018 ) . The state-of-the-art model-free approaches , at least in the way they were presented so far , need millions of frames to learn how to play these games acceptably well ( Schulman et al. , 2017 ; Hessel et al. , 2018 ) . It corresponds to days of play experience using the standard frame rate . However , human players can achieve the same within minutes ( Tsividis et al. , 2017 ) . A lot of work has been produced to circumvent these shortcomings . Most successful studies focus on model-based strategies inspired by the classical Dyna approach ( Sutton , 1991 ) and actionconditional prediction methods ( Oh et al. , 2015 ; Leibfried et al. , 2016 ) . Although some of them manage to drastically reduce the amount of data required by the standard algorithms , they do it by highly increasing both conceptual and computational complexity of the models . In this paper , we argue , and experimentally prove , that already existing model-free techniques can be much more data-efficient than it is assumed . We introduce simple change to the state-of-theart Rainbow DQN algorithm . In some environments like Pong or Hero , it can achieve the same results given only 5 % - 10 % of the data it is often presented to need . Furthermore , it results in the same data-efficiency as the state-of-the-art model-based approaches while being much more stable , simpler , and requiring much less computation . Following the introduction , section 2 gives a brief background behind reinforcement learning with the focus on Q-learning and its deep learning equivalents . Section 3 provides an overview of recent studies aimed at improving data efficiency using model-based approaches . Section 4 argues that model-free methods can be much more efficient than it tends to be presented and that existing model-based techniques only give an illusion of efficiency . Then , the description and analysis of experiments follow in sections 5 and 6 . Finally , section 7 concludes this study . 2 BACKGROUND . Reinforcement learning is a problem of learning a policy that maximises the reward signal for a given task . To define RL setting we need a set of possible environment states S , a set of available actions A , and relations between those . These relations are described by a transition function T : S × A → S that defines dynamics of transitions from one state to another , and a reward function R : S × A → R that defines the real-valued reward signal . Together , T and R constitute the model of the environment . The goal of reinforcement learning is to find a policy π : S → A that maximises the total cumulative reward over time . One of the most popular reinforcement learning algorithms is Q-learning ( Watkins & Dayan , 1992 ) . Q-learning decides on an optimal policy based on the state-action value function Q : S × A → R that maps state and action performed in that state to the expected total cumulative reward following the action . The algorithm chooses an action that maximizes Q , i.e . at = argmaxaQ ( st , a ) . Q is learned in the process of interacting with the environment . At every agent ’ s step , tuple ( st ∈ S , at ∈ A , rt ∈ R , st+1 ∈ S ) is obtained and immediately used to update the Q function . Because state-action combination is often too big or continuous to represent directly in a tabular manner , Q is commonly approximated using different supervised learning algorithms . However , using deep learning to approximate Q is not trivial because Q-learning breaks important assumptions required by neural networks . Namely , Q update is recursive and experience tuples are highly correlated when used sequentially . Recently introduced DQN ( Mnih et al. , 2015 ) bypassed this issue by introducing two concepts : target network and replay buffer . Target network is simply a fixed snapshot of the network that approximates Q value ( online network ) taken every τt steps . Instead of updating the online network towards itself , it is updated towards the target network . This approach maintains the logic of Qlearning while stopping the online network from diverging due to recursive updates . Replay buffer , on the other hand , guarantees a much higher level of independence between experience tuples . They are not used immediately , one after another anymore but stored in the replay buffer instead . Then , every τu steps , a single training step is performed , i.e . a mini-batch of randomly sampled experience from the replay buffer is used to update the online network . It reduces the correlation between experience samples by breaking their ordering . Rainbow DQN ( Hessel et al. , 2018 ) is a combination of several incremental improvements on top of DQN that increased both sample efficiency and the total performance of the algorithm achieving state-of-the-art results . It is an architecture that we use as an example that current model-free deep RL is not as inefficient as it is often stated . Throughout the paper hyperparameters from Hessel et al . ( 2018 ) are employed , unless stated otherwise . 3 MODEL-BASED REINFORCEMENT LEARNING . The most successful approach to improving data efficiency of deep RL is based on the premise of model-based techniques ( Sutton & Barto , 2018 ) . Having access to transition and reward mechanics of the environment would make it possible to construct an artificial simulation where the agent could be trained without performing often costly interactions with the real environment . However , in most scenarios , the agent is not given any prior information about the model of its environment . This issue is often overcome by learning the model instead . Oh et al . ( 2015 ) and Leibfried et al . ( 2016 ) have shown that it is possible with a very high level of accuracy . Ability to learn the model of the environment was subsequently leveraged to successfully improve different aspects of deep RL ( Racanière et al. , 2017 ; Oh et al. , 2017 ; Buesing et al. , 2018 ; Ha & Schmidhuber , 2018 ) . Azizzadenesheli et al . ( 2018 ) , Holland et al . ( 2018 ) , and Kaiser et al . ( 2019 ) , however , focused directly on employing the learned models to increase data efficiency of deep RL algorithms . Azizzadenesheli et al . ( 2018 ) proposed Generative Adversarial Tree Search ( GATS ) . Unlike in the standard approach to learning the environment dynamics , GATS creates two separate models : Generative Dynamics Model ( GDM ) based on a modified Pix2Pix ( Isola et al. , 2017 ) to learn the transition model T : S×A→ S ; and Reward Predictor ( RP ) , a simple 3-class classification architecture to learn the reward model R : S×A→ R. Both models learn from experience stored in DQN ’ s replay buffer and are then used for bounded Monte Carlo tree search as in ( Silver et al. , 2016 ) . GATS is evaluated primarily on the game Pong where it learns an optimal policy using around 42 % of the data required by using standalone model-free agent what is a tiny improvement compared to the methods described next . Holland et al . ( 2018 ) explored the performance of the model-based approach given either perfect model , model pretrained on expert data ( pretrained model ) , or model learned alongside the agent ’ s value function ( online model ) . Both non-perfect models followed standard architecture for the task ( Oh et al. , 2015 ; Leibfried et al. , 2016 ) . These models are then used to generate 100 samples of simulated experience for every interaction with the real environment . All three variations outperformed state-of-the-art Rainbow DQN in terms of data efficiency on 5 out of 6 games . Nevertheless , only the results of the online model are used for further discussion to ensure a fair comparison between the algorithms . Kaiser et al . ( 2019 ) introduced Simulated Policy Learning ( SimPLe ) . Similarly to the previous two architectures , it learns the model of the environment using a modified version of Oh et al . ( 2015 ) . It differs from previous approaches by employing PPO ( Schulman et al. , 2017 ) as its RL agent and by using the learned model much more exhaustively . It uses the model similarly to Holland et al . ( 2018 ) , however it provides at least 800k samples of artificial data after every 6.4k interactions . The approach is then evaluated on a range of 26 different Atari games . It provides results that highly outperform both Holland et al . ( 2018 ) and Azizzadenesheli et al . ( 2018 ) in terms of data efficiency achieving at least 2x improvement on over half of the games and more than 10x improvement on Freeway . To the best of our knowledge , SimPLe is the state of the art in terms of data-efficient deep reinforcement learning ; thus , it will be used as a primary baseline throughout the rest of the paper . 4 DATA EFFICIENCY OF STANDARD APPROACHES . We argue that DQN-like model-free methods are not as data inefficient as they are often portrayed . They are simply used in a very inefficient way . Let us define ratio r describing the number of training steps to the number of interactions with the environment . In the default setting τu = 4 . It means that the algorithm performs a single update of the network for every 4 interactions with the environment , i.e. , r = 1/4 . As explained in section 3 , both the online-model-based algorithm from Holland et al . ( 2018 ) and SimPLe from Kaiser et al . ( 2019 ) first learn the approximated model of the environment . Then , this approximation is used to provide simulated samples of experience alongside the real data . Nevertheless , these samples , in the best case , can only provide as much real signal to the agent as was provided in the original data . However , as a byproduct of the agent ’ s interactions with the learned model , the ratio r significantly increases . Holland et al . ( 2018 ) performs 100 simulated steps for each real step causing r = ( 1 + 100 ) ∗ ( 1/4 ) = 25.25 . SimPLe executes 800k simulated steps after every 6.4k interactions with the real environment . Thus , if SimPLe was using DQN as its model-free component ratio r would be even higher ( r = ( 800k + 6.4k ) /6.4k/4 = 126/4 = 31.5 ) . It seems unfair to allow model-based methods to perform more training steps for each gathered data point without letting model-free baselines to do the same . However , from the studies discussed above , only Holland et al . ( 2018 ) performed tests allowing DQN for extra updates1 . GATS was compared solely to the standard version of DQN and SimPLe to the standard version of PPO algorithm together with the Rainbow DQN that , as stated in the paper , was hypertuned for sample efficiency ( HRainbow ) . However , hyperparameters for HRainbow were not disclosed . We hypothesize , that the main reason behind improved data efficiency in the results is essentially increased r. 1Their results showed that indeed model-based approach with the online model does not overperform modelfree approach with extra updates . However , the study was mainly interested in thorough analysis , rather than improving the state of the art .

Summary: The paper presents a data-efficient version of the Rainbow DQN by Hessel et al. (2018) that matches the performance of a recent state of the art model-based method by Kaiser et al. (2019) on several Atari games. Particularly, the paper empirically shows that a simple hyper-parameter tuning, in this case increasing the ratio of number of training steps to the environment interactions as well as decreasing epsilon-decay period, can result in significant improvements in sample efficiency of the Rainbow DQN agent. They show that their method (which requires significantly less computation) can outperform the model-based variant on half of the games tested, while performing worse on the rest.