Deep Q-Learning with Low Switching Cost

1 INTRODUCTION . Reinforcement learning ( RL ) is often used for modeling real-world sequential decision-making problems such as medical domains , personalized recommendations , hardware placements , database optimization , etc . For these applications , oftentimes it is desirable to restrict the agent from adjusting its policy frequently . In medical domains , changing a policy requires a thorough approval process by experts . For large-scale software and hardware systems , changing a policy requires to redeploy the whole environment . Formally , we would like our RL algorithm admits a low switching cost . In this setting , it is required that the deployed policy that interacts with the environment can not change many times . In some real-world RL applications such as robotics , education , and dialogue system , changing the deployed policy frequently may cause high costs and risks . Gu et al . ( 2017 ) trained robotic manipulation by decoupling the training and experience collecting threads ; Mandel et al . ( 2014 ) applied RL to educational games by taking a data-driven methodology for comparing and validating policies offline , and run the strongest policy online ; Jaques et al . ( 2019 ) developed an off-policy batch RL algorithms for dialog system , which can effectively learn in an offline fashion , without using different policies to interact with the environment . All of these work avoid changing the deployed policy frequently as they try to train the policy offline effectively or validate the policy to determine whether to deploy it online . For RL problem with a low switching cost constraint , the central question is how to design a criterion to decide when to change the deployed policy . Ideally , we would like this criterion to have the following four properties : 1 . Low switching cost : This is the purpose of this criterion . An algorithm equipped with this policy switching criterion should have low switching cost . 2 . High Reward : Since the deployed policy determines the collected samples and the agent uses fewer deployed policies , the collected data may not be informative enough to learn the optimal policy with high reward . We need this criterion to deploy policies that can collect informative samples . 3 . Sample Efficiency : Since the agent only uses a few deployed policies , there may be more redundant samples , which will not be collected if the agent switches the policy frequently . We would like algorithms equipped with a criterion with similar sample efficiency as the case without the low switching cost constraint . 4 . Generality : We would like this criterion to be effective not only on a specific task but also broadly effective on a wide range of tasks . In this paper , we take a step toward this important problem . We focus on designing a principled policy switching criterion for deep Q-networks ( DQN ) learning algorithms , which have been widely used in applications . For example , Ahn & Park ( 2020 ) apply DQN to control balancing between different HVAC systems , Ao et al . ( 2019 ) propose a thermal process control method based on DQN , and Chen et al . ( 2018 ) try to apply it to online recommendation . Notably these applications all require low switching cost . Our paper conducts a systematic study on DQN with low switching cost . Our contributions are summarized below . Our Contributions • We conduct the first systematic empirical study on benchmark environments that require modern reinforcement learning algorithms . We test two naive policy switching criteria : 1 ) switching the policy after a fixed number of steps and 2 ) switching the policy after an increasing step with a fixed rate . We find that neither criterion is a generic solution because sometimes they either can not find the best policy or significantly decrease the sample efficiency . • Inspired by representation learning theory , we propose a new feature-based switching criterion that uses the feature distance between the deployed Q-network and the underlying learning Q-network . Through extensive experiments , we find our proposed criterion is a generic solution – it substantially decreases the switching cost while maintains a similar performance to the case without the low-switching-cost constraint . • Along the way , we also derive a deterministic Rainbow DQN ( Hessel et al. , 2018 ) , which may be of independent interest . Organization This paper is organized as follows . In Section 2 , we review related work . In Section 3 , we describe our problem setup and review necessary backgrounds . In Section 4 , we describe deterministic Rainbow DQN with the low switching cost constraint . In Section 5 , we introduce our feature-based policy switching criterion and its theoretical support . In Section 6 , we conduct experiments to evaluate different criteria . We conclude in Section 7 and leave experiment details to appendix . 2 RELATED WORK . Low switching cost algorithms were first studied in the bandit setting ( Auer et al. , 2002 ; CesaBianchi et al. , 2013 ) . Existing work on RL with low switching cost is mostly theoretical . To our knowledge , Bai et al . ( 2019 ) is the first work that studies this problem for the episodic finite-horizon tabular RL setting . Bai et al . ( 2019 ) gave a low-regret algorithm with an O ( H3SA log ( K ) ) local switching upper bound where S is the number of stats , A is the number of actions , H is the planning horizon and K is the number of episodes the agent plays . The upper bound was improved in Zhang et al . ( 2020b ; a ) . The only empirical study on RL with switching cost is Matsushima et al . ( 2020 ) , which proposed a concept of deployment efficiency and gave a model-based algorithm . During the training process , the algorithm fixes the number of deployments , trains a dynamics model ensemble , and updates the deployed policy alternately . After each deployment , the deployed policy collects transitions in the real environment to enhance the models , and then the models optimize the policy by providing imagined trajectories . In other words , they reduce the number of deployments by training on simulated environments . Our goal is different : we design a criterion to decide when to change the deployed policy , and this criterion could be employed by model-free algorithms . There is a line of work on offline RL ( also called Batch RL ) methods , where the policy does not interact with the environment directly and only learns from a fixed dataset ( Lange et al. , 2012 ; Levine et al. , 2020 ) . Some methods Interpolate offline and online methods , i.e. , semi-batch RL algorithms ( Singh et al. , 1995 ; Lange et al. , 2012 ) , which update the policy many times on a large batch of transitions . However , the switching cost is not their focus . 3 PRELIMINARIES . 3.1 MARKOV DECISION PROCESS . Throughout our paper , we consider the episodic Markov decision model ( S , A , H , P , r ) . In this model , S is the state space , A is the action space , H ∈ Z+ is the planning horizon . P is the transition operator where P ( x′|x , a ) denotes the transition probability of taking action a from state x to state x′ . r : S ×A → R is the reward function . A policy is a mapping from a state to an action , π : S → A . In this paper , we focus on deterministic policies as required by motivating applications . The dynamics of the episodic MDP can be view as the interaction of an agent with the environment periodically . We let K be the total number of episodes the agent plays . At the beginning of an episode k ∈ [ K ] , the agent chooses a policy πk . The initial xk1 ∈ S is sampled from a distribution , and the agent is then in step 1 . At each step h ∈ [ H ] in this episode , based on the current state xkh ∈ S the agent chooses the action akh = πk ( xkh ) . The environment will give the reward for the step r ( xkh , a k h ) and move the agent to the next state xh+1 ∼ P ( ·|xkh , akh ) . The episode automatically ends when the agent reaches the step H + 1 . Q-function is used to evaluate the long-term value for the action a and subsequent decisions . The Q-function of a policy π at time step h is defined as follows : Qπh ( x , a ) : = rh ( x , a ) + E [ H∑ i=h+1 r ( xi , π ( xi ) ) ∣∣∣∣∣xh = x , ah = a ] ( 1 ) The goal of this agent is to find a policy π∗ which maximizes the expected reward , π∗ = arg maxπ Eπ [ ∑H h=1 rh ] . Ideally , we want to use a few episodes ( K ) as possible to learn π∗ . 3.2 SWITCHING COST . The concept of switching cost is used to quantify the adaptability of RL algorithms . The switching cost is defined as the number of policy changes of deployed policies in the running of the algorithm in K episodes , namely : Nswitch : = K−1∑ k=1 I { πk 6= πk+1 } ( 2 ) The goal of this paper is to equip algorithm with a criterion that learns π∗ using a few episodes while at the same time has small Nswitch . 3.3 DEEP Q-LEARNING If we can estimate Q-function for each state and action pair well , there is no difficulty in finding π∗ . For example , we can always select a∗ = arg maxaQ ( x , a ) . However , it is not easy to learn Q value estimates for each state and action pair , especially when the state or the action space is large . In deep Q-learning ( DQN ) , Mnih et al . ( 2015 ) combine deep networks and reinforcement learning successfully by using a deep neural network to approximate Q ( x , a ) . Given the current state xh , the agent selects a action ah greedily based on the Q ( xh , a ) , then the state move to xh+1 and a reward rt+1 is obtained . The transition ( xt , at , rt+1 , xt+1 ) is saved to the replay memory buffer . At each time , a batch of transitions is sampled from this buffer , and the parameters of neural networks are optimized by using stochastic gradient descent to minimize the loss ( rh+1 + γh+1 max a′ qθ̄ ( xh+1 , a ′ ) − qθ ( xh , ah ) ) 2 ( 3 ) where γh+1 is the chosen discount of time step h + 1 , θ is the parameters of the online network , and θ̄ represents the parameters of the target network . The gradient of the loss is back-propagated only to update θ , while θ̄ is not optimized directly . DQN has been successful as it led to superhuman performance on several Atari games . Nevertheless , there are also several limitations of this algorithm and many extensions have been proposed . Rainbow ( Hessel et al. , 2018 ) combines six of these extensions and obtains an excellent and stable performance on many Atari games . In the Appendix A , we review these tricks . In this paper , we focus on Rainbow DQN with low switching cost .

Summary: This paper studies RL with low switching cost under the deep RL setting. It points out several naive algorithms like switching after a certain number of steps and then propose a new heuristic. This heuristic learns a new policy offline using the experience replay the behavior collected and switches the behavior policy once the similarity of the feature embeddings of the current state by these two policies becomes large. The paper also makes an attempt to provide a theoretical justification for a better understanding of the heuristic. This method might outperform the naive algorithms by some margin, if any. It would be a more interesting manuscript if some stronger results could be provided from the perspective of any of theory, experiments, or applications.