Measuring Progress in Deep Reinforcement Learning Sample Efficiency 

1 INTRODUCTION . Recent successes of deep reinforcement learning ( DRL ) in Go ( Silver et al. , 2016 ; 2017 ; 2018 ) and complex real-time strategy games ( Berner et al. , 2019 ; Vinyals et al. , 2019 ) indicate the vast potential for automating complex economically relevant tasks like assembling goods in non-standardized environments or physically assisting humans using DRL . However , sample efficiency is likely to remain an important bottleneck for the economic feasibility of real world applications of DRL . This is because samples of state transitions in the environment caused by an agent ’ s action are essential for training DRL agents but automated systems interacting with the real world are often fragile , slow or costly , which makes DRL training in the real world expensive both in terms of money and time ( Dulac-Arnold et al. , 2019 ) . As most widely used benchmarks in DRL still consist of computer games and simulations where samples can be obtained risk-free , fast and cheaply1 , progress in DRL does not necessarily translate to future real world applications unless it corresponds to progress in sample efficiency or simulation accuracy and transfer learning . This makes information about progress in sample efficiency an important input to researchers studying topics like the future of employment ( Frey & Osborne , 2017 ) , the potential for malicious uses of DRL ( Brundage et al. , 2018 ) and other potentially transformative impacts of AI systems in the future ( Gruetzemacher & Whittlestone , 2019 ) . 2 BACKGROUND AND RELATED WORK . 2.1 SAMPLE EFFICIENCY . While research on progress in the field of AI used to have a strong focus on benchmark performance metrics in AI ( Eckersley & Nasser , 2017 ) , there have been calls to pay more attention to the importance of other metrics : Martinez-Plumed et al . ( 2018 ) enumerate previously neglected dimensions of AI progress and explore the relationship between computing resources and final performance in RL . The wider implications of sample efficiency , more specifically , are explored by Tucker et al . ( 2020 ) . While their scope is broader than DRL , most of their points do apply to reinforcement learning as a 1Espeholt et al . ( 2019 ) reach up to 2.4 M FPS and 1 B frames per 25 $ on DM Lab ( Beattie et al. , 2016 ) . special case of AI technology . For example , as the authors point out , robotics is an especially important case of a data-limited domain , in which new applications are likely to be unlocked for many actors with improved sample efficiency . While some benchmark papers make explicit quantitative comparisons of sample efficiency ( or “ data efficiency ” ) to isolated previous methods ( Hessel et al. , 2018 ; Hafner et al. , 2019a ; Kaiser et al. , 2019 ; Srinivas et al. , 2020 ) , we are not aware of any more systematic investigation of sample efficiency in DRL . 2.2 COMPUTING POWER AND SCALING LAWS . While DRL and other applications of deep learning have produced many impressive results , this has gone hand in hand with an ever-increasing amount of computing power ( compute ) being devoted to model training : Compute usage for large AI projects has been in line with Moore ’ s law prior to 2012 ( Sastry et al. , 2019 ) , but saw a doubling time of 3.4-months in the 2012 to 2018 period , which is seven times faster than Moore ’ s law ( Amodei & Hernandez , 2018 ) . Compute usage would not increase if there was nothing to gain : For example , the performance of Transformer-based language models ( Vaswani et al. , 2017 ) follows a power law in the number of model parameters when differently sized models are trained on the same dataset indicating that in the current regime , Transformer language models become more sample efficient with more parameters . Data is still important , as there is another power law relating dataset size for models with the same amount of parameters to performance ( Kaplan et al. , 2020 ) . Things likely look similar for the ImageNet benchmark ( Deng et al. , 2009 ) where progress has long been accompanied by larger models using a fixed amount of data ( Sun et al. , 2017 ) while recent progress relies on additional data ( Touvron et al. , 2019 ) . 2.3 ALGORITHMIC PROGRESS AND INCREASED EFFICIENCY . As increased sample usage might make progress in DRL look faster , increased compute usage might do so for machine learning . Grace ( 2013 ) found that hardware improvements account for half of the progress for several problems in the vicinity of AI research like computer chess , computer go and physics simulations . More recently , Fichte et al . ( 2020 ) observed the same for progress in SAT solvers . In the realm of deep learning , Hernandez & Brown ( 2020 ) investigated improvements in algorithmic efficiency by holding final performance on various benchmarks constant and analyzing how the number of FLOPS needed to reach that level changed over time . They find that this amount of compute needed to reach the performance of AlexNet ( Krizhevsky et al. , 2012 ) on ImageNet went down with a halving time of 16 months over 6 years . Looking at DawnBench ( Coleman et al. , 2017 ) submissions , they also find a cost reduction in terms of dollars of 184x2 between October 2017 and September 2018 for reaching a fixed performance level on ImageNet . Another submission managed to reduce the training time by a factor of 808x from 10 days and 4 hours to 18 minutes in the same time interval while still cutting the cost by a factor of over nine . 3 METHODS . 3.1 WE MEASURE GROWTH IN SAMPLE EFFICIENCY BY HOLDING PERFORMANCE CONSTANT . As proposed by Dulac-Arnold et al . ( 2019 ) and similar to work by Hernandez & Brown ( 2020 ) on measuring algorithmic progress , we measure progress in sample efficiency by comparing the number of samples needed for systems to reach a fixed performance level over time and define the sample efficiency of an algorithm for a given task and performance level as 1/S , where S is the number of samples needed to train to the given performance level on the task : Sample Efficiency ( Task , Score , Algorithm ) = 1 Samples Algorithm needs for Score on Task Compared to the dual approach of comparing the performance of different algorithms for a fixed amount of samples , this approach has the advantage of being more comparable between different benchmarks and a lot easier to interpret : The marginal difficulty of gaining score at a given performance level could vary a lot between games , and it is unclear whether twice as much score corresponds to twice better performance in any meaningful sense ( Hernandez-Orallo , 2020 ) , while 2This might be 88x , as there is a second initial contribution which cost $ 1113 rather than $ 2323 . twice as many samples to reach the same level of performance can straightforwardly be interpreted as half the sample efficiency . 3.2 WE CONSIDER BOTH ATARI AND CONTINUOUS CONTROL . We look at sample efficiency in three areas : 1 ) for the median human-normalized score on 57 games in the Arcade Learning Environment ( ALE ) ( Bellemare et al. , 2013 ) using the no-op evaluation protocol and pixel observations ; 2 ) single task performance on continuous control tasks from the rllab-benchmark ( Duan et al. , 2016 ) using simulation states3 as input ; and 3 ) on tasks from the DeepMind Control Suite ( Tassa et al. , 2018 ) using pixel observations as input . We chose these tasks because of their popularity and the existence of some de facto standards on evaluation and reporting protocols , which yields more mutually comparable published results . The tasks ’ popularity also implies attention from the research community , which ensures that they are somewhat representative of overall progress in DRL . Another advantage is the tasks ’ moderate difficulty and the wide range of partial solutions between failing the task and solving it optimally . This allows for a meaningful comparison of a wider set of different algorithms as more algorithms reach the performance threshold , while there is still space for improvement using advanced techniques . 3.3 WE TREAT EACH PUBLICATION AS A MEASUREMENT OF THE STATE-OF-THE-ART .. For our main results on progress in sample efficiency , we treat every considered result as a measurement of the state-of-the-art at publication : For each publication , we plot the sample usage ( grey dot in figures 2 ( a ) and 3 ) and the state-of-the-art in sample efficiency at the time ( blue stars in figures 2 ( a ) and 3 ) such that results that improve on the state-of-the-art are represented by a grey dot with a blue star in the middle . Then we fit the blue stars . Compared to fitting a dataset only containing jumps in the SOTA ( grey dots with a blue star ) , this dampens the effect single outliers have on the fitted model and means that prolonged periods of stagnation are taken into account adequately . However , our way of fitting the SOTA is sensitive to the precise dates associated with results that did not improve on the SOTA . Another approach would be to graph the SOTA on a continuous-time axis and approximate the corresponding step function by an exponential . This solves the sensitivity to measurements that don ’ t affect the SOTA but comes with two drawbacks : it requires an arbitrary choice of end date that affects the estimated doubling time and leads to high sensitivity to isolated early measurements . Full results for the estimated doubling times for the different approaches can be found in appendix D. Compared to the between task/score variation , the effect of the approach is usually small , but we did observe some larger effects when there have only been few and early changes to the SOTA . In these cases , the models using only jumps predict fast doubling times despite long periods of stagnation , while both other models are more conservative . On the other hand , the continuous-time model yielded the fastest doubling times for most pixel-based MuJoCo tasks , as it puts a lot of weight on improvements relative to the early , weak D4PG ( Barth-Maron et al. , 2018 ) baseline . 3.4 OUR STUDY IS BASED ON PREVIOUSLY PUBLISHED TRAINING CURVES . Due to the difficulty and overhead associated with the independent replication of results in DRL ( Islam et al. , 2017 ; Henderson et al. , 2018 ) , we decided to base our study on training curves in published papers . To that extent , we conducted a systematic search for papers tackling the respective benchmarks and included all relevant4 identified papers which reported on the metrics under consideration in a sufficiently precise way . A more detailed description of the results we included and the search process we employed to identify them can be found in appendix B . While our approach implies multiple limitations of our study that are discussed in appendix C , such as a biased selection of results and the inability to tune hyperparameters , we believe that the limited methodology still offers substantial value in exploring trends in DRL sample efficiency . 3As the dynamics depend on the simulation state , training on the state is easier as seen in Yarats et al . ( 2019 ) . 4A single paper that reported results ( Doan et al . ( 2020 ) ) was excluded for lack of relevance .

Summary: The paper proposes to retrospectively benchmark the sample-efficiency on widely used simulated deep RL benchmarks such as Atari and DMControl across the years. The paper shows some interesting trends with respect to how both the algorithmic improvements as well as the use of increased number of frames have driven the progress in the SoTA scores reported on the Atari HNS benchmark. The paper also shows that there has been exponential progress in the sample-efficiency on both Atari and DMControl with nice log-linear plots. This is an interesting analysis, on the lines of OpenAI's papers on Scaling Laws, but done for Deep RL.