text: 67,209,1283444494050816002,885528008,William Fedus,"The interplay of RL algorithms with experience replay is poorly understood. We study this and uncover a relationship between n-step returns and replay capacity. ICML '20 paper: <LINK> Prajit R.*, @agarwl_ , Yoshua, @hugo_larochelle , Mark R., @wwdabney <LINK> We study two properties in experience replay: 1. Size of replay capacity 2. Oldest policy in the buffer Together, these jointly define a replay ratio: an experience learning to new data acquisition ratio. Since DQN, these have often not varied (1M, learn each 4-steps). <LINK> The performance of a Rainbow agent (Hessel et al., 2017) varies substantially with these factors: improving with more ""on-policy"" data and capacity. In the easiest Deep RL boost, we find a 29% median improvement in Atari games simply by increasing replay capacity from 1M -&gt; 10M <LINK> But the story is completely different with a DQN algorithm -- regardless of whether we control for replay ratio or the oldest policy -- there is no change. Why? <LINK> Through ablative and additive studies, we isolate n-step returns as the crucial factor. Improvements with larger replay capacity are found only when using n-step returns. No clear signal found with prioritized experience replay, optimizer, or distributional learning. <LINK> The importance of n-step returns even holds in the logical extreme: batch reinforcement learning.  This is non-intuitive. Uncorrected n-step returns -- mathematically incorrect -- yield gains in a regime where they are the most incorrect. <LINK> A bias-variance trade-off partially explains the importance of n-step returns (refer to paper for experimental details), but it's still not complete. The entanglement between data generation and RL algorithms is an important issue in the design of better agents! <LINK> This was jointly led by Prajit Ramachandran with a great collaboration across Mila, Brain, DeepMind including: @agarwl_ , Yoshua Bengio, @hugo_larochelle , Mark Rowland, @wwdabney  Paper: <LINK> Code: <LINK> ICML 2020: <LINK>",https://arxiv.org/abs/2007.06700,"Experience replay is central to off-policy algorithms in deep reinforcement learning (RL), but there remain significant gaps in our understanding. We therefore present a systematic and extensive analysis of experience replay in Q-learning methods, focusing on two fundamental properties: the replay capacity and the ratio of learning updates to experience collected (replay ratio). Our additive and ablative studies upend conventional wisdom around experience replay -- greater capacity is found to substantially increase the performance of certain algorithms, while leaving others unaffected. Counterintuitively we show that theoretically ungrounded, uncorrected n-step returns are uniquely beneficial while other techniques confer limited benefit for sifting through larger memory. Separately, by directly controlling the replay ratio we contextualize previous observations in the literature and empirically measure its importance across a variety of deep RL algorithms. Finally, we conclude by testing a set of hypotheses on the nature of these performance benefits. ",Revisiting Fundamentals of Experience Replay,8,"[""The interplay of RL algorithms with experience replay is poorly understood. We study this and uncover a relationship between n-step returns and replay capacity.\n\nICML '20 paper: <LINK>\n\nPrajit R.*, @agarwl_ , Yoshua, @hugo_larochelle , Mark R., @wwdabney <LINK>"", 'We study two properties in experience replay:\n1. Size of replay capacity\n2. Oldest policy in the buffer\n\nTogether, these jointly define a replay ratio: an experience learning to new data acquisition ratio. Since DQN, these have often not varied (1M, learn each 4-steps). https://t.co/0MOKpLQEHn', 'The performance of a Rainbow agent (Hessel et al., 2017) varies substantially with these factors: improving with more ""on-policy"" data and capacity.\n\nIn the easiest Deep RL boost, we find a 29% median improvement in Atari games simply by increasing replay capacity from 1M -&gt; 10M https://t.co/wNVfFttK0h', 'But the story is completely different with a DQN algorithm -- regardless of whether we control for replay ratio or the oldest policy -- there is no change.\n\nWhy? https://t.co/cdJgyz240o', 'Through ablative and additive studies, we isolate n-step returns as the crucial factor.\n\nImprovements with larger replay capacity are found only when using n-step returns. No clear signal found with prioritized experience replay, optimizer, or distributional learning. https://t.co/SlKT7Mnwqx', 'The importance of n-step returns even holds in the logical extreme:  batch reinforcement learning. \n\nThis is non-intuitive. Uncorrected n-step returns -- mathematically incorrect -- yield gains in a regime where they are the most incorrect. https://t.co/HubfHztWuW', ""A bias-variance trade-off partially explains the importance of n-step returns (refer to paper for experimental details), but it's still not complete.\n\nThe entanglement between data generation and RL algorithms is an important issue in the design of better agents! https://t.co/9lPfzcadKi"", 'This was jointly led by Prajit Ramachandran with a great collaboration across Mila, Brain, DeepMind including: @agarwl_ , Yoshua Bengio, @hugo_larochelle , Mark Rowland, @wwdabney \n\nPaper:  https://t.co/KnGLGapNGU\nCode:  https://t.co/xcg5ZI7Tt2\nICML 2020:  https://t.co/QSGtPz4Iwk']",20,07,2010