Title: Recurrent Off-Policy Deep Reinforcement Learning Doesn’t Have to be Slow
Primary area: reinforcement learning
Keywords: Reinforcement Learning, Memory, LSTM, Transformers, Atari, Procgen, VizDoom, Partially Observable

Abstract: Recurrent off-policy deep reinforcement learning models achieve state-of-the-art performance but are often sidelined due to their high computational demands. In response, we introduce RISE (Recurrent Integration via Simplified Encodings), a novel approach that can leverage recurrent networks in any image-based off-policy RL setting without significant computational overheads via using both learnable and non-learnable encoder layers. When integrating RISE into leading non-recurrent off-policy RL algorithms, we observe a 35.6\% human-normalized interquartile mean (IQM) performance improvement across the Atari benchmark. We analyze various implementation strategies to highlight the versatility and potential of our proposed framework. We make our code available in the supplementary material.

TL;DR: We introduce a method for allowing Reinforcement Learning agents to utilize sequence models such as LSTMs with far less overhead than previous methods.