Q-Value Weighted Regression: Reinforcement Learning with Limited Data

Sample efficiency and performance in the offline setting have emerged as among the main challenges of deep reinforcement learning . We introduce Q-Value Weighted Regression ( QWR ) , a simple RL algorithm that excels in these aspects . QWR is an extension of Advantage Weighted Regression ( AWR ) , an off-policy actor-critic algorithm that performs very well on continuous control tasks , also in the offline setting , but struggles on tasks with discrete actions and in sample efficiency . We perform a theoretical analysis of AWR that explains its shortcomings and use the insights to motivate QWR theoretically . We show experimentally that QWR matches state-of-the-art algorithms both on tasks with continuous and discrete actions . We study the main hyperparameters of QWR and find that it is stable in a wide range of their choices and on different tasks . In particular , QWR yields results on par with SAC on the MuJoCo suite and – with the same set of hyperparameters – yields results on par with a highly tuned Rainbow implementation on a set of Atari games . We also verify that QWR performs well in the offline RL setting , making it a compelling choice for reinforcement learning in domains with limited data . 1 INTRODUCTION . Deep reinforcement learning has been applied to a large number of challenging tasks , from games ( Silver et al. , 2017 ; OpenAI , 2018 ; Vinyals et al. , 2017 ) to robotic control ( Sadeghi & Levine , 2016 ; OpenAI et al. , 2018 ; Rusu et al. , 2016 ) . Since RL makes minimal assumptions on the underlying task , it holds the promise of automating a wide range of applications . However , its widespread adoption has been hampered by a number of challenges . Reinforcement learning algorithms can be substantially more complex to implement and tune than standard supervised learning methods and can have a fair number of hyper-parameters and be brittle with respect to their choices , and may require a large number of interactions with the environment . These issues are well-known and there has been significant progress in addressing them . The policy gradient algorithm REINFORCE ( Williams ( 1992 ) ) is simple to understand and implement , but is both brittle and requires on-policy data . Proximal Policy Optimization ( PPO , Schulman et al . ( 2017 ) ) is a more stable on-policy algorithm that has seen a number of successful applications despite requiring a large number of interactions with the environment . Soft Actor-Critic ( SAC , Haarnoja et al . ( 2018 ) ) is a much more sample-efficient off-policy algorithm , but it is defined only for continuous action spaces and does not work well in the offline setting , known as batch reinforcement learning , where all samples are provided from earlier interactions with the environment , and the agent can not collect more samples . Advantage Weighted Regression ( AWR , Peng et al . ( 2019 ) ) is a recent offpolicy actor-critic algorithm that works well in the offline setting and is built using only simple and convergent maximum likelihood loss functions , making it easier to tune and debug . It is competitive with SAC given enough time to train , but is less sample-efficient and has not been demonstrated to succeed in settings with discrete actions . We replace the value function critic of AWR with a Q-value function . Next , we add action sampling to the actor training loop . Finally , we introduce a custom backup to the Q-value training . The resulting algorithm , which we call Q-Value Weighted Regression ( QWR ) inherits the advantages of AWR but is more sample-efficient and works well with discrete actions and in visual domains , e.g. , on Atari games . To better understand QWR we perform a number of ablations , checking different number of samples in actor training , different advantage estimators , and aggregation functions . These choices affect the performance of QWR only to a limited extent and it remains stable with each of the choices across the tasks we experiment with . We run experiments with QWR on the MuJoCo environments and on a subset of the Atari Learning Environment . Since sample efficiency is our main concern , we focus on the difficult case when the number of interactions with the environment is limited – in most our experiments we limit it to 100K interactions . The experiments demonstrate that QWR is indeed more sample-efficient than AWR . On MuJoCo , it performs on par with Soft Actor-Critic ( SAC ) , the current state-of-the-art algorithm for continuous domains . On Atari , QWR performs on par with OTRainbow , a variant of Rainbow highly tuned for sample efficiency . Notably , we use the same set of hyperparameters ( except for the network architecture ) for both our final MuJoCo and Atari experiments . 2 Q-VALUE WEIGHTED REGRESSION . 2.1 ADVANTAGE WEIGHTED REGRESSION . Peng et al . ( 2019 ) recently proposed Advantage Weighted Regression ( AWR ) , an off-policy , actorcritic algorithm notable for its simplicity and stability , achieving competitive results across a range of continuous control tasks . It can be expressed as interleaving data collection and two regression tasks performed on the replay buffer , as shown in Algorithm 1 . Algorithm 1 Advantage Weighted Regression . 1 : θ ← random actor parameters 2 : φ← random critic parameters 3 : D ← ∅ 4 : for k in 0 .. n_iterations− 1 do 5 : add trajectories { τi } sampled by πθ to D 6 : for i in 0 .. n_critic_steps− 1 do 7 : sample ( s , a ) ∼ D 8 : φ← φ− αV∇φ [ ||Rs , aD − Vφ ( s ) ||2 ] 9 : end for 10 : for i in 0 .. n_actor_steps− 1 do 11 : sample ( s , a ) ∼ D 12 : θ ← θ + απ∇θ [ log πθ ( a|s ) exp ( 1β ( R s , a D − Vφ ( s ) ) ] 13 : end for 14 : end for AWR optimizes expected improvement of an actor policy π ( a|s ) over a sampling policy µ ( a|s ) by regression towards the well-performing actions in the collected experience . Improvement is achieved by weighting the actor loss by exponentiated advantage Aµ ( s , a ) of an action , skewing the regression towards the better-performing actions . The advantage is calculated based on the expected return Rs , aµ achieved by performing action a in state s and then following the sampling policy µ . To calculate the advantage , one first estimates the value , Vµ ( s ) , using a learned critic and then computes Aµ ( s , a ) = Rs , aµ − Vµ ( s ) . This results in the following formula for the actor : argmax π Es∼dµ ( s ) Ea∼µ ( ·|s ) [ log π ( a|s ) exp ( 1 β ( Rs , aµ − Vµ ( s ) ) ) ] , ( 1 ) where dµ ( s ) = ∑∞ t=1 γ t−1p ( st = s|µ ) denotes the unnormalized , discounted state visitation distribution of the policy µ , and β is a temperature hyperparameter . The critic is trained to estimate the future returns of the sampling policy µ : argmin V Es∼dµ ( s ) Ea∼µ ( ·|s ) [ ||Rs , aµ − V ( s ) ||2 ] . ( 2 ) To achieve off-policy learning , the actor and the critic are trained on data collected from a mixture of policies from different training iterations , stored in the replay buffer D . 2.2 ANALYSIS OF AWR WITH LIMITED DATA . While AWR achieves very good results after longer training , it is not very sample efficient , as noted in the future work section of ( Peng et al. , 2019 ) . To understand this problem , we analyze a single loop of actor training in AWR under a special assumption . The assumption we introduce , called state-determines-action , concerns the content of the replay buffer D of an off-policy RL algorithm . The replay buffer contains all state-action pairs that the algorithm has visited so far during its interactions with the environment . We say that a replay buffer D satisfies the state-determines-action assumption when for each state s in the buffer , there is a unique action that was taken from it , formally : for all ( s , a ) , ( s′ , a′ ) ∈ D : s = s′ =⇒ a = a′ . This assumption may seem very limiting and indeed – it is not true in many artificial experiments with RL algorithms , such as Atari games . Even a random policy starting from the same state could violate the assumption the second time it collects a trajectory . But note that this assumption is almost always satisfied in real-world experiments with high-dimensional state spaces as any amount of noise added to a high-dimensional space will make repeating the exact same state highly improbable . For example , consider a robot observing 32x32 pixel images . To repeat a state , each of the 1024 pixels would have to have exactly the same value , which is highly improbable even with a small amount of noise . This assumption also holds in cases with limited data . When only a few trajectories are collected in a large state space , it is unlikely a state will be repeated in the replay buffer , which makes the assumption relevant to the study of sample efficiency . How does AWR perform under the state-determines-action assumption ? In Theorem 1 , formulated and proven in Appendix A.2 , we show that the AWR update rule under this assumption will converge to a policy that assigns probability 1 to the actions already present in the replay buffer , thus cloning the previous behaviors . This is not the desired behavior , as an agent should consider various actions from each state , to ensure exploration . The state-determines-action assumption is the main motivating point behind QWR , whose theoretical properties are proven in Theorem 2 in Appendix A.3 . We now illustrate the importance of this assumption by creating a simple environment in which it holds with high probability . We verify experimentally that AWR fails on this simple environment , while QWR is capable of solving it . The environment , which we call BitFlip , is parameterized by an integer N . The state of the environment consists of N bits and a step counter . The action space consists of N actions . When an action i is chosen , the i-th bit is flipped and the step counter is incremented . A game of BitFlip starts in a random state with the step counter set to 0 , and proceeds for 5 steps . The initial state is randomized in such a way to always leave at least 5 bits set to 0 . At each step , the reward is +1 if a bit was flipped from 0 to 1 and the reward is −1 in the opposite case . Since BitFlip starts in one random state out of 2N , at large enough N it is highly unlikely that the starting state will ever be repeated in the replay buffer . As the initial policy is random and BitFlip maintains a step counter to prevent returning to a state , the same holds for subsequent states . BitFlip is a simple game with a very simple strategy , but the initial replay buffer will satisfy the statedetermines-action assumption with high probability . As we will see , this is enough to break AWR . We ran both AWR and QWR on BitFlip for different values of N , for 10 iterations per experiment . In each iteration we collected 1000 interactions with the environment and trained both the actor and the critic for 300 steps . All shared hyperparameters of AWR and QWR were set to the same values , and the backup operator in QWR was set to mean . We report the mean out of 10 episodes played by the trained agent . The results are shown in Figure 1 . As we can see , the performance of AWR starts deteriorating at a relatively small value of N = 8 , which corresponds to a state space with 5 ·28 = 1280 states , while QWR maintains high performance even at N = 30 , so around 5 · 109 states . Notice how the returns of AWR drop with N – at higher values : 20− 30 , the agent struggles to flip even a single zero bit . This problem with AWR and large state spaces motivates us to introduce QWR next .

Summary: This paper presents a Q-value weighted regression (QWR) on top of the advantage weighted regression (AWR) to improve the sample efficiency for offline RL settings. Through the analysis to the AWR, the authors claim that it performs poorly in scenarios with discrete actions, which motivates the development of QWR. Empirically, the authors show that QWR is comparable to SAC in continuous tasks and a variant of Rainbow in discrete tasks.