Measuring Progress in Deep Reinforcement Learning Sample Efficiency 

1 INTRODUCTION . Recent successes of deep reinforcement learning ( DRL ) in Go ( Silver et al. , 2016 ; 2017 ; 2018 ) and complex real-time strategy games ( Berner et al. , 2019 ; Vinyals et al. , 2019 ) indicate the vast potential for automating complex economically relevant tasks like assembling goods in non-standardized environments or physically assisting humans using DRL . However , sample efficiency is likely to remain an important bottleneck for the economic feasibility of real world applications of DRL . This is because samples of state transitions in the environment caused by an agent ’ s action are essential for training DRL agents but automated systems interacting with the real world are often fragile , slow or costly , which makes DRL training in the real world expensive both in terms of money and time ( Dulac-Arnold et al. , 2019 ) . As most widely used benchmarks in DRL still consist of computer games and simulations where samples can be obtained risk-free , fast and cheaply1 , progress in DRL does not necessarily translate to future real world applications unless it corresponds to progress in sample efficiency or simulation accuracy and transfer learning . This makes information about progress in sample efficiency an important input to researchers studying topics like the future of employment ( Frey & Osborne , 2017 ) , the potential for malicious uses of DRL ( Brundage et al. , 2018 ) and other potentially transformative impacts of AI systems in the future ( Gruetzemacher & Whittlestone , 2019 ) . 2 BACKGROUND AND RELATED WORK . 2.1 SAMPLE EFFICIENCY . While research on progress in the field of AI used to have a strong focus on benchmark performance metrics in AI ( Eckersley & Nasser , 2017 ) , there have been calls to pay more attention to the importance of other metrics : Martinez-Plumed et al . ( 2018 ) enumerate previously neglected dimensions of AI progress and explore the relationship between computing resources and final performance in RL . The wider implications of sample efficiency , more specifically , are explored by Tucker et al . ( 2020 ) . While their scope is broader than DRL , most of their points do apply to reinforcement learning as a 1Espeholt et al . ( 2019 ) reach up to 2.4 M FPS and 1 B frames per 25 $ on DM Lab ( Beattie et al. , 2016 ) . special case of AI technology . For example , as the authors point out , robotics is an especially important case of a data-limited domain , in which new applications are likely to be unlocked for many actors with improved sample efficiency . While some benchmark papers make explicit quantitative comparisons of sample efficiency ( or “ data efficiency ” ) to isolated previous methods ( Hessel et al. , 2018 ; Hafner et al. , 2019a ; Kaiser et al. , 2019 ; Srinivas et al. , 2020 ) , we are not aware of any more systematic investigation of sample efficiency in DRL . 2.2 COMPUTING POWER AND SCALING LAWS . While DRL and other applications of deep learning have produced many impressive results , this has gone hand in hand with an ever-increasing amount of computing power ( compute ) being devoted to model training : Compute usage for large AI projects has been in line with Moore ’ s law prior to 2012 ( Sastry et al. , 2019 ) , but saw a doubling time of 3.4-months in the 2012 to 2018 period , which is seven times faster than Moore ’ s law ( Amodei & Hernandez , 2018 ) . Compute usage would not increase if there was nothing to gain : For example , the performance of Transformer-based language models ( Vaswani et al. , 2017 ) follows a power law in the number of model parameters when differently sized models are trained on the same dataset indicating that in the current regime , Transformer language models become more sample efficient with more parameters . Data is still important , as there is another power law relating dataset size for models with the same amount of parameters to performance ( Kaplan et al. , 2020 ) . Things likely look similar for the ImageNet benchmark ( Deng et al. , 2009 ) where progress has long been accompanied by larger models using a fixed amount of data ( Sun et al. , 2017 ) while recent progress relies on additional data ( Touvron et al. , 2019 ) . 2.3 ALGORITHMIC PROGRESS AND INCREASED EFFICIENCY . As increased sample usage might make progress in DRL look faster , increased compute usage might do so for machine learning . Grace ( 2013 ) found that hardware improvements account for half of the progress for several problems in the vicinity of AI research like computer chess , computer go and physics simulations . More recently , Fichte et al . ( 2020 ) observed the same for progress in SAT solvers . In the realm of deep learning , Hernandez & Brown ( 2020 ) investigated improvements in algorithmic efficiency by holding final performance on various benchmarks constant and analyzing how the number of FLOPS needed to reach that level changed over time . They find that this amount of compute needed to reach the performance of AlexNet ( Krizhevsky et al. , 2012 ) on ImageNet went down with a halving time of 16 months over 6 years . Looking at DawnBench ( Coleman et al. , 2017 ) submissions , they also find a cost reduction in terms of dollars of 184x2 between October 2017 and September 2018 for reaching a fixed performance level on ImageNet . Another submission managed to reduce the training time by a factor of 808x from 10 days and 4 hours to 18 minutes in the same time interval while still cutting the cost by a factor of over nine . 3 METHODS . 3.1 WE MEASURE GROWTH IN SAMPLE EFFICIENCY BY HOLDING PERFORMANCE CONSTANT . As proposed by Dulac-Arnold et al . ( 2019 ) and similar to work by Hernandez & Brown ( 2020 ) on measuring algorithmic progress , we measure progress in sample efficiency by comparing the number of samples needed for systems to reach a fixed performance level over time and define the sample efficiency of an algorithm for a given task and performance level as 1/S , where S is the number of samples needed to train to the given performance level on the task : Sample Efficiency ( Task , Score , Algorithm ) = 1 Samples Algorithm needs for Score on Task Compared to the dual approach of comparing the performance of different algorithms for a fixed amount of samples , this approach has the advantage of being more comparable between different benchmarks and a lot easier to interpret : The marginal difficulty of gaining score at a given performance level could vary a lot between games , and it is unclear whether twice as much score corresponds to twice better performance in any meaningful sense ( Hernandez-Orallo , 2020 ) , while 2This might be 88x , as there is a second initial contribution which cost $ 1113 rather than $ 2323 . twice as many samples to reach the same level of performance can straightforwardly be interpreted as half the sample efficiency . 3.2 WE CONSIDER BOTH ATARI AND CONTINUOUS CONTROL . We look at sample efficiency in three areas : 1 ) for the median human-normalized score on 57 games in the Arcade Learning Environment ( ALE ) ( Bellemare et al. , 2013 ) using the no-op evaluation protocol and pixel observations ; 2 ) single task performance on continuous control tasks from the rllab-benchmark ( Duan et al. , 2016 ) using simulation states3 as input ; and 3 ) on tasks from the DeepMind Control Suite ( Tassa et al. , 2018 ) using pixel observations as input . We chose these tasks because of their popularity and the existence of some de facto standards on evaluation and reporting protocols , which yields more mutually comparable published results . The tasks ’ popularity also implies attention from the research community , which ensures that they are somewhat representative of overall progress in DRL . Another advantage is the tasks ’ moderate difficulty and the wide range of partial solutions between failing the task and solving it optimally . This allows for a meaningful comparison of a wider set of different algorithms as more algorithms reach the performance threshold , while there is still space for improvement using advanced techniques . 3.3 WE TREAT EACH PUBLICATION AS A MEASUREMENT OF THE STATE-OF-THE-ART .. For our main results on progress in sample efficiency , we treat every considered result as a measurement of the state-of-the-art at publication : For each publication , we plot the sample usage ( grey dot in figures 2 ( a ) and 3 ) and the state-of-the-art in sample efficiency at the time ( blue stars in figures 2 ( a ) and 3 ) such that results that improve on the state-of-the-art are represented by a grey dot with a blue star in the middle . Then we fit the blue stars . Compared to fitting a dataset only containing jumps in the SOTA ( grey dots with a blue star ) , this dampens the effect single outliers have on the fitted model and means that prolonged periods of stagnation are taken into account adequately . However , our way of fitting the SOTA is sensitive to the precise dates associated with results that did not improve on the SOTA . Another approach would be to graph the SOTA on a continuous-time axis and approximate the corresponding step function by an exponential . This solves the sensitivity to measurements that don ’ t affect the SOTA but comes with two drawbacks : it requires an arbitrary choice of end date that affects the estimated doubling time and leads to high sensitivity to isolated early measurements . Full results for the estimated doubling times for the different approaches can be found in appendix D. Compared to the between task/score variation , the effect of the approach is usually small , but we did observe some larger effects when there have only been few and early changes to the SOTA . In these cases , the models using only jumps predict fast doubling times despite long periods of stagnation , while both other models are more conservative . On the other hand , the continuous-time model yielded the fastest doubling times for most pixel-based MuJoCo tasks , as it puts a lot of weight on improvements relative to the early , weak D4PG ( Barth-Maron et al. , 2018 ) baseline . 3.4 OUR STUDY IS BASED ON PREVIOUSLY PUBLISHED TRAINING CURVES . Due to the difficulty and overhead associated with the independent replication of results in DRL ( Islam et al. , 2017 ; Henderson et al. , 2018 ) , we decided to base our study on training curves in published papers . To that extent , we conducted a systematic search for papers tackling the respective benchmarks and included all relevant4 identified papers which reported on the metrics under consideration in a sufficiently precise way . A more detailed description of the results we included and the search process we employed to identify them can be found in appendix B . While our approach implies multiple limitations of our study that are discussed in appendix C , such as a biased selection of results and the inability to tune hyperparameters , we believe that the limited methodology still offers substantial value in exploring trends in DRL sample efficiency . 3As the dynamics depend on the simulation state , training on the state is easier as seen in Yarats et al . ( 2019 ) . 4A single paper that reported results ( Doan et al . ( 2020 ) ) was excluded for lack of relevance .

Summary: The paper is trying to make extensive and systematic investigation in deep RL papers to measure the recent progress in the broad literatures. The authors look into published state-of-the-art results in Atari, state-based continuous control and pixel-based continuous control. Inferring from papers' published training curves, the authors estimated and found sample efficiency doubling time is 10-18 months for Atari, 5-24 months for continuous control and 4-9 months for pixel-based control tasks.