Q-Value Weighted Regression: Reinforcement Learning with Limited Data

Sample efficiency and performance in the offline setting have emerged as among the main challenges of deep reinforcement learning . We introduce Q-Value Weighted Regression ( QWR ) , a simple RL algorithm that excels in these aspects . QWR is an extension of Advantage Weighted Regression ( AWR ) , an off-policy actor-critic algorithm that performs very well on continuous control tasks , also in the offline setting , but struggles on tasks with discrete actions and in sample efficiency . We perform a theoretical analysis of AWR that explains its shortcomings and use the insights to motivate QWR theoretically . We show experimentally that QWR matches state-of-the-art algorithms both on tasks with continuous and discrete actions . We study the main hyperparameters of QWR and find that it is stable in a wide range of their choices and on different tasks . In particular , QWR yields results on par with SAC on the MuJoCo suite and – with the same set of hyperparameters – yields results on par with a highly tuned Rainbow implementation on a set of Atari games . We also verify that QWR performs well in the offline RL setting , making it a compelling choice for reinforcement learning in domains with limited data . 1 INTRODUCTION . Deep reinforcement learning has been applied to a large number of challenging tasks , from games ( Silver et al. , 2017 ; OpenAI , 2018 ; Vinyals et al. , 2017 ) to robotic control ( Sadeghi & Levine , 2016 ; OpenAI et al. , 2018 ; Rusu et al. , 2016 ) . Since RL makes minimal assumptions on the underlying task , it holds the promise of automating a wide range of applications . However , its widespread adoption has been hampered by a number of challenges . Reinforcement learning algorithms can be substantially more complex to implement and tune than standard supervised learning methods and can have a fair number of hyper-parameters and be brittle with respect to their choices , and may require a large number of interactions with the environment . These issues are well-known and there has been significant progress in addressing them . The policy gradient algorithm REINFORCE ( Williams ( 1992 ) ) is simple to understand and implement , but is both brittle and requires on-policy data . Proximal Policy Optimization ( PPO , Schulman et al . ( 2017 ) ) is a more stable on-policy algorithm that has seen a number of successful applications despite requiring a large number of interactions with the environment . Soft Actor-Critic ( SAC , Haarnoja et al . ( 2018 ) ) is a much more sample-efficient off-policy algorithm , but it is defined only for continuous action spaces and does not work well in the offline setting , known as batch reinforcement learning , where all samples are provided from earlier interactions with the environment , and the agent can not collect more samples . Advantage Weighted Regression ( AWR , Peng et al . ( 2019 ) ) is a recent offpolicy actor-critic algorithm that works well in the offline setting and is built using only simple and convergent maximum likelihood loss functions , making it easier to tune and debug . It is competitive with SAC given enough time to train , but is less sample-efficient and has not been demonstrated to succeed in settings with discrete actions . We replace the value function critic of AWR with a Q-value function . Next , we add action sampling to the actor training loop . Finally , we introduce a custom backup to the Q-value training . The resulting algorithm , which we call Q-Value Weighted Regression ( QWR ) inherits the advantages of AWR but is more sample-efficient and works well with discrete actions and in visual domains , e.g. , on Atari games . To better understand QWR we perform a number of ablations , checking different number of samples in actor training , different advantage estimators , and aggregation functions . These choices affect the performance of QWR only to a limited extent and it remains stable with each of the choices across the tasks we experiment with . We run experiments with QWR on the MuJoCo environments and on a subset of the Atari Learning Environment . Since sample efficiency is our main concern , we focus on the difficult case when the number of interactions with the environment is limited – in most our experiments we limit it to 100K interactions . The experiments demonstrate that QWR is indeed more sample-efficient than AWR . On MuJoCo , it performs on par with Soft Actor-Critic ( SAC ) , the current state-of-the-art algorithm for continuous domains . On Atari , QWR performs on par with OTRainbow , a variant of Rainbow highly tuned for sample efficiency . Notably , we use the same set of hyperparameters ( except for the network architecture ) for both our final MuJoCo and Atari experiments . 2 Q-VALUE WEIGHTED REGRESSION . 2.1 ADVANTAGE WEIGHTED REGRESSION . Peng et al . ( 2019 ) recently proposed Advantage Weighted Regression ( AWR ) , an off-policy , actorcritic algorithm notable for its simplicity and stability , achieving competitive results across a range of continuous control tasks . It can be expressed as interleaving data collection and two regression tasks performed on the replay buffer , as shown in Algorithm 1 . Algorithm 1 Advantage Weighted Regression . 1 : θ ← random actor parameters 2 : φ← random critic parameters 3 : D ← ∅ 4 : for k in 0 .. n_iterations− 1 do 5 : add trajectories { τi } sampled by πθ to D 6 : for i in 0 .. n_critic_steps− 1 do 7 : sample ( s , a ) ∼ D 8 : φ← φ− αV∇φ [ ||Rs , aD − Vφ ( s ) ||2 ] 9 : end for 10 : for i in 0 .. n_actor_steps− 1 do 11 : sample ( s , a ) ∼ D 12 : θ ← θ + απ∇θ [ log πθ ( a|s ) exp ( 1β ( R s , a D − Vφ ( s ) ) ] 13 : end for 14 : end for AWR optimizes expected improvement of an actor policy π ( a|s ) over a sampling policy µ ( a|s ) by regression towards the well-performing actions in the collected experience . Improvement is achieved by weighting the actor loss by exponentiated advantage Aµ ( s , a ) of an action , skewing the regression towards the better-performing actions . The advantage is calculated based on the expected return Rs , aµ achieved by performing action a in state s and then following the sampling policy µ . To calculate the advantage , one first estimates the value , Vµ ( s ) , using a learned critic and then computes Aµ ( s , a ) = Rs , aµ − Vµ ( s ) . This results in the following formula for the actor : argmax π Es∼dµ ( s ) Ea∼µ ( ·|s ) [ log π ( a|s ) exp ( 1 β ( Rs , aµ − Vµ ( s ) ) ) ] , ( 1 ) where dµ ( s ) = ∑∞ t=1 γ t−1p ( st = s|µ ) denotes the unnormalized , discounted state visitation distribution of the policy µ , and β is a temperature hyperparameter . The critic is trained to estimate the future returns of the sampling policy µ : argmin V Es∼dµ ( s ) Ea∼µ ( ·|s ) [ ||Rs , aµ − V ( s ) ||2 ] . ( 2 ) To achieve off-policy learning , the actor and the critic are trained on data collected from a mixture of policies from different training iterations , stored in the replay buffer D . 2.2 ANALYSIS OF AWR WITH LIMITED DATA . While AWR achieves very good results after longer training , it is not very sample efficient , as noted in the future work section of ( Peng et al. , 2019 ) . To understand this problem , we analyze a single loop of actor training in AWR under a special assumption . The assumption we introduce , called state-determines-action , concerns the content of the replay buffer D of an off-policy RL algorithm . The replay buffer contains all state-action pairs that the algorithm has visited so far during its interactions with the environment . We say that a replay buffer D satisfies the state-determines-action assumption when for each state s in the buffer , there is a unique action that was taken from it , formally : for all ( s , a ) , ( s′ , a′ ) ∈ D : s = s′ =⇒ a = a′ . This assumption may seem very limiting and indeed – it is not true in many artificial experiments with RL algorithms , such as Atari games . Even a random policy starting from the same state could violate the assumption the second time it collects a trajectory . But note that this assumption is almost always satisfied in real-world experiments with high-dimensional state spaces as any amount of noise added to a high-dimensional space will make repeating the exact same state highly improbable . For example , consider a robot observing 32x32 pixel images . To repeat a state , each of the 1024 pixels would have to have exactly the same value , which is highly improbable even with a small amount of noise . This assumption also holds in cases with limited data . When only a few trajectories are collected in a large state space , it is unlikely a state will be repeated in the replay buffer , which makes the assumption relevant to the study of sample efficiency . How does AWR perform under the state-determines-action assumption ? In Theorem 1 , formulated and proven in Appendix A.2 , we show that the AWR update rule under this assumption will converge to a policy that assigns probability 1 to the actions already present in the replay buffer , thus cloning the previous behaviors . This is not the desired behavior , as an agent should consider various actions from each state , to ensure exploration . The state-determines-action assumption is the main motivating point behind QWR , whose theoretical properties are proven in Theorem 2 in Appendix A.3 . We now illustrate the importance of this assumption by creating a simple environment in which it holds with high probability . We verify experimentally that AWR fails on this simple environment , while QWR is capable of solving it . The environment , which we call BitFlip , is parameterized by an integer N . The state of the environment consists of N bits and a step counter . The action space consists of N actions . When an action i is chosen , the i-th bit is flipped and the step counter is incremented . A game of BitFlip starts in a random state with the step counter set to 0 , and proceeds for 5 steps . The initial state is randomized in such a way to always leave at least 5 bits set to 0 . At each step , the reward is +1 if a bit was flipped from 0 to 1 and the reward is −1 in the opposite case . Since BitFlip starts in one random state out of 2N , at large enough N it is highly unlikely that the starting state will ever be repeated in the replay buffer . As the initial policy is random and BitFlip maintains a step counter to prevent returning to a state , the same holds for subsequent states . BitFlip is a simple game with a very simple strategy , but the initial replay buffer will satisfy the statedetermines-action assumption with high probability . As we will see , this is enough to break AWR . We ran both AWR and QWR on BitFlip for different values of N , for 10 iterations per experiment . In each iteration we collected 1000 interactions with the environment and trained both the actor and the critic for 300 steps . All shared hyperparameters of AWR and QWR were set to the same values , and the backup operator in QWR was set to mean . We report the mean out of 10 episodes played by the trained agent . The results are shown in Figure 1 . As we can see , the performance of AWR starts deteriorating at a relatively small value of N = 8 , which corresponds to a state space with 5 ·28 = 1280 states , while QWR maintains high performance even at N = 30 , so around 5 · 109 states . Notice how the returns of AWR drop with N – at higher values : 20− 30 , the agent struggles to flip even a single zero bit . This problem with AWR and large state spaces motivates us to introduce QWR next .

Summary: This paper focuses on offline policy learning with a limited dataset and proposes a sample efficient algorithm called Q-Value Weighted Regression. Based on the Advantage Weighted Regression algorithm, this algorithm calculates the advantage of the sampling policy \mu by estimated Q-value function. Experiment results show that the QWR algorithm has better performance than the AWR algorithm with limited data.