Deep Q-Learning with Low Switching Cost

1 INTRODUCTION . Reinforcement learning ( RL ) is often used for modeling real-world sequential decision-making problems such as medical domains , personalized recommendations , hardware placements , database optimization , etc . For these applications , oftentimes it is desirable to restrict the agent from adjusting its policy frequently . In medical domains , changing a policy requires a thorough approval process by experts . For large-scale software and hardware systems , changing a policy requires to redeploy the whole environment . Formally , we would like our RL algorithm admits a low switching cost . In this setting , it is required that the deployed policy that interacts with the environment can not change many times . In some real-world RL applications such as robotics , education , and dialogue system , changing the deployed policy frequently may cause high costs and risks . Gu et al . ( 2017 ) trained robotic manipulation by decoupling the training and experience collecting threads ; Mandel et al . ( 2014 ) applied RL to educational games by taking a data-driven methodology for comparing and validating policies offline , and run the strongest policy online ; Jaques et al . ( 2019 ) developed an off-policy batch RL algorithms for dialog system , which can effectively learn in an offline fashion , without using different policies to interact with the environment . All of these work avoid changing the deployed policy frequently as they try to train the policy offline effectively or validate the policy to determine whether to deploy it online . For RL problem with a low switching cost constraint , the central question is how to design a criterion to decide when to change the deployed policy . Ideally , we would like this criterion to have the following four properties : 1 . Low switching cost : This is the purpose of this criterion . An algorithm equipped with this policy switching criterion should have low switching cost . 2 . High Reward : Since the deployed policy determines the collected samples and the agent uses fewer deployed policies , the collected data may not be informative enough to learn the optimal policy with high reward . We need this criterion to deploy policies that can collect informative samples . 3 . Sample Efficiency : Since the agent only uses a few deployed policies , there may be more redundant samples , which will not be collected if the agent switches the policy frequently . We would like algorithms equipped with a criterion with similar sample efficiency as the case without the low switching cost constraint . 4 . Generality : We would like this criterion to be effective not only on a specific task but also broadly effective on a wide range of tasks . In this paper , we take a step toward this important problem . We focus on designing a principled policy switching criterion for deep Q-networks ( DQN ) learning algorithms , which have been widely used in applications . For example , Ahn & Park ( 2020 ) apply DQN to control balancing between different HVAC systems , Ao et al . ( 2019 ) propose a thermal process control method based on DQN , and Chen et al . ( 2018 ) try to apply it to online recommendation . Notably these applications all require low switching cost . Our paper conducts a systematic study on DQN with low switching cost . Our contributions are summarized below . Our Contributions • We conduct the first systematic empirical study on benchmark environments that require modern reinforcement learning algorithms . We test two naive policy switching criteria : 1 ) switching the policy after a fixed number of steps and 2 ) switching the policy after an increasing step with a fixed rate . We find that neither criterion is a generic solution because sometimes they either can not find the best policy or significantly decrease the sample efficiency . • Inspired by representation learning theory , we propose a new feature-based switching criterion that uses the feature distance between the deployed Q-network and the underlying learning Q-network . Through extensive experiments , we find our proposed criterion is a generic solution – it substantially decreases the switching cost while maintains a similar performance to the case without the low-switching-cost constraint . • Along the way , we also derive a deterministic Rainbow DQN ( Hessel et al. , 2018 ) , which may be of independent interest . Organization This paper is organized as follows . In Section 2 , we review related work . In Section 3 , we describe our problem setup and review necessary backgrounds . In Section 4 , we describe deterministic Rainbow DQN with the low switching cost constraint . In Section 5 , we introduce our feature-based policy switching criterion and its theoretical support . In Section 6 , we conduct experiments to evaluate different criteria . We conclude in Section 7 and leave experiment details to appendix . 2 RELATED WORK . Low switching cost algorithms were first studied in the bandit setting ( Auer et al. , 2002 ; CesaBianchi et al. , 2013 ) . Existing work on RL with low switching cost is mostly theoretical . To our knowledge , Bai et al . ( 2019 ) is the first work that studies this problem for the episodic finite-horizon tabular RL setting . Bai et al . ( 2019 ) gave a low-regret algorithm with an O ( H3SA log ( K ) ) local switching upper bound where S is the number of stats , A is the number of actions , H is the planning horizon and K is the number of episodes the agent plays . The upper bound was improved in Zhang et al . ( 2020b ; a ) . The only empirical study on RL with switching cost is Matsushima et al . ( 2020 ) , which proposed a concept of deployment efficiency and gave a model-based algorithm . During the training process , the algorithm fixes the number of deployments , trains a dynamics model ensemble , and updates the deployed policy alternately . After each deployment , the deployed policy collects transitions in the real environment to enhance the models , and then the models optimize the policy by providing imagined trajectories . In other words , they reduce the number of deployments by training on simulated environments . Our goal is different : we design a criterion to decide when to change the deployed policy , and this criterion could be employed by model-free algorithms . There is a line of work on offline RL ( also called Batch RL ) methods , where the policy does not interact with the environment directly and only learns from a fixed dataset ( Lange et al. , 2012 ; Levine et al. , 2020 ) . Some methods Interpolate offline and online methods , i.e. , semi-batch RL algorithms ( Singh et al. , 1995 ; Lange et al. , 2012 ) , which update the policy many times on a large batch of transitions . However , the switching cost is not their focus . 3 PRELIMINARIES . 3.1 MARKOV DECISION PROCESS . Throughout our paper , we consider the episodic Markov decision model ( S , A , H , P , r ) . In this model , S is the state space , A is the action space , H ∈ Z+ is the planning horizon . P is the transition operator where P ( x′|x , a ) denotes the transition probability of taking action a from state x to state x′ . r : S ×A → R is the reward function . A policy is a mapping from a state to an action , π : S → A . In this paper , we focus on deterministic policies as required by motivating applications . The dynamics of the episodic MDP can be view as the interaction of an agent with the environment periodically . We let K be the total number of episodes the agent plays . At the beginning of an episode k ∈ [ K ] , the agent chooses a policy πk . The initial xk1 ∈ S is sampled from a distribution , and the agent is then in step 1 . At each step h ∈ [ H ] in this episode , based on the current state xkh ∈ S the agent chooses the action akh = πk ( xkh ) . The environment will give the reward for the step r ( xkh , a k h ) and move the agent to the next state xh+1 ∼ P ( ·|xkh , akh ) . The episode automatically ends when the agent reaches the step H + 1 . Q-function is used to evaluate the long-term value for the action a and subsequent decisions . The Q-function of a policy π at time step h is defined as follows : Qπh ( x , a ) : = rh ( x , a ) + E [ H∑ i=h+1 r ( xi , π ( xi ) ) ∣∣∣∣∣xh = x , ah = a ] ( 1 ) The goal of this agent is to find a policy π∗ which maximizes the expected reward , π∗ = arg maxπ Eπ [ ∑H h=1 rh ] . Ideally , we want to use a few episodes ( K ) as possible to learn π∗ . 3.2 SWITCHING COST . The concept of switching cost is used to quantify the adaptability of RL algorithms . The switching cost is defined as the number of policy changes of deployed policies in the running of the algorithm in K episodes , namely : Nswitch : = K−1∑ k=1 I { πk 6= πk+1 } ( 2 ) The goal of this paper is to equip algorithm with a criterion that learns π∗ using a few episodes while at the same time has small Nswitch . 3.3 DEEP Q-LEARNING If we can estimate Q-function for each state and action pair well , there is no difficulty in finding π∗ . For example , we can always select a∗ = arg maxaQ ( x , a ) . However , it is not easy to learn Q value estimates for each state and action pair , especially when the state or the action space is large . In deep Q-learning ( DQN ) , Mnih et al . ( 2015 ) combine deep networks and reinforcement learning successfully by using a deep neural network to approximate Q ( x , a ) . Given the current state xh , the agent selects a action ah greedily based on the Q ( xh , a ) , then the state move to xh+1 and a reward rt+1 is obtained . The transition ( xt , at , rt+1 , xt+1 ) is saved to the replay memory buffer . At each time , a batch of transitions is sampled from this buffer , and the parameters of neural networks are optimized by using stochastic gradient descent to minimize the loss ( rh+1 + γh+1 max a′ qθ̄ ( xh+1 , a ′ ) − qθ ( xh , ah ) ) 2 ( 3 ) where γh+1 is the chosen discount of time step h + 1 , θ is the parameters of the online network , and θ̄ represents the parameters of the target network . The gradient of the loss is back-propagated only to update θ , while θ̄ is not optimized directly . DQN has been successful as it led to superhuman performance on several Atari games . Nevertheless , there are also several limitations of this algorithm and many extensions have been proposed . Rainbow ( Hessel et al. , 2018 ) combines six of these extensions and obtains an excellent and stable performance on many Atari games . In the Appendix A , we review these tricks . In this paper , we focus on Rainbow DQN with low switching cost .

Summary: In the RL context, this paper aims at designing a generic solution for reducing the number of policy switches during training (called switching cost) while maintaining the performance. This study is done in the context of deep reinforcement learning. A few generic baselines solutions are provided as well as a more complex solution that empirically outperforms the baselines.