CAN ALTQ LEARN FASTER: EXPERIMENTS AND THEORY

Differently from the popular Deep Q-Network ( DQN ) learning , Alternating Qlearning ( AltQ ) does not fully fit a target Q-function at each iteration , and is generally known to be unstable and inefficient . Limited applications of AltQ mostly rely on substantially altering the algorithm architecture in order to improve its performance . Although Adam appears to be a natural solution , its performance in AltQ has rarely been studied before . In this paper , we first provide a solid exploration on how well AltQ performs with Adam . We then take a further step to improve the implementation by adopting the technique of parameter restart . More specifically , the proposed algorithms are tested on a batch of Atari 2600 games and exhibit superior performance than the DQN learning method . The convergence rate of the slightly modified version of the proposed algorithms is characterized under the linear function approximation . To the best of our knowledge , this is the first theoretical study on the Adam-type algorithms in Q-learning . 1 INTRODUCTION . Q-learning ( Watkins & Dayan , 1992 ) is one of the most important model-free reinforcement learning ( RL ) problems , which has received considerable research attention in recent years ( Bertsekas & Tsitsiklis , 1996 ; Even-Dar & Mansour , 2003 ; Hasselt , 2010 ; Lu et al. , 2018 ; Achiam et al. , 2019 ) . When the state-action space is large or continuous , parametric approximation of the Q-function is often necessary . One remarkable success of parametric Q-learning in practice is its combination with deep learning , known as the Deep Q-Network ( DQN ) learning ( Mnih et al. , 2013 ; 2015 ) . It has been applied to various applications in computer games ( Bhatti et al. , 2016 ) , traffic control ( Arel et al. , 2010 ) , recommendation systems ( Zheng et al. , 2018 ; Zhao et al. , 2018 ) , chemistry research ( Zhou et al. , 2017 ) , etc . Its on-policy continuous variant ( Silver et al. , 2014 ) has also led to great achievements in robotics locomotion ( Lillicrap et al. , 2016 ) . The DQN algorithm is performed in a nested-loop manner , where the outer loop follows an one-step update of the Q-function ( via the empirical Bellman operator for Q-learning ) , and the inner loop takes a supervised learning process to fit the updated ( i.e. , target ) Q-function with a neural network . In practice , the inner loop takes a sufficiently large number of iterations under certain optimizer ( e.g . stochastic gradient descent ( SGD ) or Adam ) to fit the neural network well to the target Q-function . In contrast , a conventional Q-learning algorithm runs only one SGD step in each inner loop , in which case the overall Q-learning algorithm updates the Q-function and fits the target Q-function alternatively in each iteration . We refer to such a Q-learning algorithm with alternating updates as Alternating Q-learning ( AltQ ) . Although significantly simpler in the update rule , AltQ is well known to be unstable and have weak performance ( Mnih et al. , 2016 ) . This is in part due to the fact that the inner loop does not fit the target Q-function sufficiently well . To fix this issue , Mnih et al . ( 2016 ) proposed a new exploration strategy and asynchronous sampling schemes over parallel computing units ( rather than the simple replay sampling in DQN ) in order for the AltQ algorithm to achieve comparable or better performance than DQN . As another alternative , Knight & Lerner ( 2018 ) proposed a more involved natural gradient propagation for AltQ to improve the performance . All these schemes require more sophisticated designs or hardware support , which may place AltQ less advantageous compared to the popular DQN , even with their better performances . This motivates us to ask the following first question . • Q1 : Can we design a simple and easy variant of the AltQ algorithm , which uses as simple setup as DQN and does not introduce extra computational burden and heuristics , but still achieves better and more stable performance than DQN ? In this paper , we provide an affirmative answer by introducing novel lightweight designs to AltQ based on Adam . Although Adam appears to be a natural tool , its performance in AltQ has rarely been studied yet . Thus , we first provide a solid exploration on how well AltQ performs with Adam ( Kingma & Ba , 2014 ) , where the algorithm is referred to as AltQ-Adam . We then take a further step to improve the implementation of AltQ-Adam by adopting the technique of parameter restart ( i.e. , restart the initial setting of Adam parameters every a few iterations ) , and refer to the new algorithm as AltQ-AdamR . This is the first time that restart is applied for improving the performance of RL algorithms although restart has been used for conventional optimization before . In a batch of 23 Atari 2600 games , our experiments show that both AltQ-Adam and AltQ-AdamR outperform the baseline performance of DQN by 50 % on average . Furthermore , AltQ-AdamR effectively reduces the performance variance and achieves a much more stable learning process . In our experiments for the linear quadratic regulator ( LQR ) problems , AltQ-AdamR converges even faster than the model-based value iteration ( VI ) solution . This is a rather surprising result given that the model-based VI has been treated as the performance upper bound for the Q-learning ( including DQN ) algorithms with target update ( Lewis & Vrabie , 2009 ; Yang et al. , 2019 ) . Regarding the theoretical analysis of AltQ algorithms , their convergence guarantee has been extensively studied ( Melo et al. , 2008 ; Chen et al. , 2019b ) . More references are given in Section 1.1 . However , all the existing studies focus on the AltQ algorithms that take a simple SGD step . Such theory is not applicable to the proposed AltQ-Adam and AltQ-AdamR that implement the Adam-type update . Thus , the second intriguing question we address here is described as follows . • Q2 : Can we provide the convergence guarantee for AltQ-Adam and AltQ-AdamR or their slightly modified variants ( if these two algorithms do not always converge by nature ) ? It is well known in optimization that Adam does not always converge , and instead , a slightly modified variant AMSGrad proposed in Reddi et al . ( 2018 ) has been widely accepted as an alternative to justify the performance of Adam-type algorithms . Thus , our theoretical analysis here also focuses on such slightly modified variants AltQ-AMSGrad and AltQ-AMSGradR of the proposed algorithms . We show that under the linear function approximation ( which is the structure that the current tools for analysis of Q-learning can handle ) , both AltQ-AMSGrad and AltQ-AMSGradR converge to the global optimal solution under standard assumptions for Qlearning . To the best of our knowledge , this is the first non-asymptotic convergence guarantee on Q-learning that incorporates Adam-type update and momentum restart . Furthermore , a slight adaptation of our proof provides the convergence rate for the AMSGrad for conventional strongly convex optimization which has not been studied before and can be of independent interest . Notations We use ‖x‖ : = ‖x‖2 = √ xTx to denote the ` 2 norm of a vector x , and use ‖x‖∞ = max i |xi| to denote the infinity norm . When x , y are both vectors , x/y , xy , x2 , √ x are all calculated in the element-wise manner , which will be used in the update of Adam and AMSGrad . We denote [ n ] = 1 , 2 , . . . , n , and bxc ∈ Z as the largest integer such that bxc ≤ x < bxc+ 1 . 1.1 RELATED WORK . Empirical performance of AltQ : AltQ algorithms that strictly follow the alternating updates are rarely used in practice , particularly in comparison with the well-accepted DQN learning and its improved variants of dueling network structure ( Wang et al. , 2016 ) , double Q-learning ( Hasselt , 2010 ) and variance exploration and sampling schemes ( Schaul et al. , 2015 ) . Mnih et al . ( 2016 ) proposed the asynchronous one-step Q-learning that is conceptually close to AltQ with competitive performance against DQN . However , the algorithm still relies on a slowly moving target network like DQN , and the multi-thread learning also complicates the computational setup . Lu et al . ( 2018 ) studied the problem of value overestimation and proposed the non-delusional Q-learning algorithm that employed the so-called pre-conditioned Q-networks , which is also computationally complex . Knight & Lerner ( 2018 ) proposed a natural gradient propagation for AltQ to improve the performance , where the gradient implementation is complex . In this paper , we propose two simple and computationally efficient schemes to improve the performance of AltQ . Theoretical analysis of AltQ : Since proposed in Watkins & Dayan ( 1992 ) , Q-learning has aroused great interest in theoretic analysis . The line of theoretic research of AltQ that are most relevant to our study lies in the Q-learning with function approximation . A large number of works study Q-learning with linear function approximation such as Bertsekas & Tsitsiklis ( 1996 ) ; Devraj & Meyn ( 2017 ) ; Zou et al . ( 2019 ) ; Chen et al . ( 2019b ) ; Du et al . ( 2019 ) , to name a few . More recently , convergence of AltQ with neural network parameterization was given in Cai et al . ( 2019 ) , which exploits the linear structure of neural networks in the overparamterized regime for analysis . It is worth noting that all the existing analysis of AltQ with function approximation considers the simple SGD update , whereas our analysis in this paper focuses on the more involved Adam-type updates . Convergence analysis of Adam : Adam was proposed in Kingma & Ba ( 2014 ) and has achieved a great success in training deep neural networks . Kingma & Ba ( 2014 ) and Reddi et al . ( 2018 ) provided regret bounds under the online convex optimization framework for Adam and AMSGrad , respectively . However , Tran et al . ( 2019 ) pointed out errors in the proofs of the previous two papers and corrected them . Recently , convergence analysis of Adam and AMSGrad in nonconvex optimization was provided in Zou et al . ( 2018 ) ; Zhou et al . ( 2018 ) ; Chen et al . ( 2019a ) ; Phuong & Phong ( 2019 ) , in which the Adam-type algorithms were guaranteed to converge to a stationary point . To the best of our knowledge , our study is the first convergence analysis of the Adam-type of algorithms for Q-learning . 2 PRELIMINARIES . We consider a Markov decision process with a considerably large or continuous state space S ⊂ RM and action space A ⊂ RN with a non-negative bounded reward function R : S ×A → [ 0 , Rmax ] . We define U ( s ) ⊂ A as the admissible set of actions at state s , and π : S → A as a feasible stationary policy . We seek to solve a discrete-time sequential decision problem with γ ∈ ( 0 , 1 ) as follows : maximize π Jπ ( s0 ) = EP [ ∞∑ t=0 γtR ( st , π ( st ) ) ] , subject to st+1 ∼ P ( ·|st , at ) . ( 1 ) Let J ? ( s ) : = Jπ ? ( s ) be the optimal value function when applying the optimal policy π ? . The corresponding optimal Q-function can be defined as Q ? ( s , a ) : = R ( s , a ) + γEPJ ? ( s′ ) , ( 2 ) where s′ ∼ P ( ·|s , a ) and we use the same notation hereafter when no confusion arises . In other words , Q ? ( s , a ) is the reward of an agent who starts from state s and takes action a at the first step and then follows the optimal policy π ? thereafter .

Summary: This paper claims to propose a method to train q-based agents that use “alternating” Q-learning. However, the alternating approach given in the paper appears to be the normal Bellman update implemented in most versions of DQN. Furthermore, the citation given for AltQ (Mnih et al. 2016) makes no mention of the term “Alternating Q learning”.