A Convergent Variant of the Boltzmann Softmax Operator in Reinforcement Learning

1 INTRODUCTION . In sequential decision making problem , an agent learns to find an optimal policy that maximizes the expected discounted long-term reward , which can be modeled by a Markov decision process ( MDP ) . In an MDP , the optimal value function is the fixed point of the Bellman operator . Thus , the optimal value function can be computed by iterative updates from arbitrary initial value function , i.e. , value iteration ( Bellman ( 2013 ) ) . Littman & Szepesvári ( 1996 ) proposed the generalized MDP considering generalized action selection operators , according to which the value function is estimated . If the generalized action selection operator satisfies the non-expansion property , the uniqueness of the fixed-point of the generalized Bellman operator is guaranteed , thus ensuring the convergence of generalized value iteration algorithm . Examples of non-expansive operators include the max and the mean operators . Without full information about transition dynamics and reward function of the environment , in reinforcement learning , the agent aims to learn an optimal policy by interacting with the unknown environment from experience . Reinforcement learning has achieved groundbreaking success for many decision making problems , both in discrete and continuous domains , including robotics ( Kober et al . ( 2013 ) ) , game playing ( Mnih et al . ( 2015 ) ) , and many others . One of the most fundamental challenges is how to balance exploration and exploitation , where the exploration-exploitation dilemma occurs in action selection and value function optimization ( Asadi & Littman ( 2016 ) ) . For value function optimization , the agent estimates the value function according to the action selection operator and then updates the estimated value . A number of effective algorithms to address explorationexploitation dilemma have their root in alternating the action selection operator , which falls in the generalized MDP framework . One of the most important reinforcement learning algorithm , Q-learning , employs the the max operator for value function optimization . The max operator always greedily selects the action that gives the best value and updates current estimation according to the value . As the current estimation for value functions is not accurate , such greedy selection may lead to misbehavior , e.g. , the overesti- mation phenomenon ( Hasselt ( 2010 ) ) . In fact , the max operator is the extreme case of exploitation , lacking the ability to explore or consider other choices . On the other hand , the mean operator estimates the value by computing the average . Thus , the mean operator solely explores which fails to utilize current estimation . The Boltzmann softmax operator is a natural summary operator that has been widely applied ( CesaBianchi et al . ( 2017 ) ) . Specifically , it is an exponentially weighting scheme , where the weights are computed according to the current estimation and its parameter β . The parameter β trades off between exploration and exploitation . When β → ∞ , it behaves as the max operator which solely favors exploitation while it behaves as the mean operator when β → 0 which purely focuses on exploration . However , despite from the advantages , it is very challenging to apply the operator in value function optimization . First , the parameter β is difficult to choose ( Sutton et al . ( 1998 ) ) . Second , as shown in ( Littman & Szepesvári ( 1996 ) ; Asadi & Littman ( 2016 ) ) , the Boltzmann softmax operator is not a non-expansion , which may lead to multiple fixed-points and thus the value function of this policy is not well-defined . The non-expansive property is vital to guarantee the uniqueness of the fix-point and the convergence of the learning algorithm . Without such property , the learning algorithm may misbehave or even diverge . Thus , it is widely believed that the Boltzmann softmax operator can not be used directly due to the violation of the non-expansive property ( Littman & Szepesvári ( 1996 ) ; Asadi & Littman ( 2016 ) ) . In fact , as far as we know , how to characterize the use of the Boltzmann softmax operator , which violates the property of non-expansion in value iteration , remains an open problem ( Littman ( 1996 ) ) . In this paper , we study the property of the Boltzmann softmax operator and propose a new technique to characterize its error bound in value iteration with fixed parameter β . To be specific , we show that for the error bound between the value function induced by the Boltzmann softmax operator and the optimal value function , there remains a term related to β that will not converge to 0 . Thus , although the Boltzmann softmax operator guarantees the approximate convergence of value iteration , it would converge to a sub-optimal policy unfortunately . Indeed , the direct use of the Boltzmann softmax operator inevitably introduces performance drop in value iteration . We then take a step further and study an essential problem , is there a way that the Boltzmann softmax operator be applied in value iteration which guarantees the convergence to the optimal policy ? Based on this technique , we propose the dynamic Boltzmann softmax operator boltzβt , termed the DBS operator , to eliminate the loss and enable the convergence of value iteration to the optimal value function . Our core idea is to dynamically change βt in value iteration and present its convergence rate analysis . Then , we propose the DBS Q -learning algorithm with the application of the DBS operator in a popular model-free reinforcement learning algorithm , i.e. , Q-learning ( Watkins & Dayan ( 1992 ) ) , and prove the convergence of the DBS Q-learning . We conduct experiments to verify the effectiveness and efficiency of our proposed dynamic Boltzmann softmax operator . We first evaluate DBS value iteration and DBS Q-learning on a tabular case , the GridWorld . Results show that the DBS operator leads to smaller error and faster convergence . We then demonstrate that the DBS operator can be extended to large scale problems , Atari games . Using DQN as baseline , we show that DQN with the dynamic Boltzmann softmax operator ( abbreviated as DBS-DQN ) substantially outperforms DQN in a suite of Atari benchmark games . 2 PRELIMINARIES . A Markov decision process ( MDP ) is defined by a 5-tuple ( S , A , p , r , γ ) , where S and A denote the set of states and actions , p ( s′|s , a ) represents the transition probability from state s to state s′ under action a , and r ( s , a ) is the corresponding immediate reward . The discount factor is denoted by γ ∈ [ 0 , 1 ) , which controls the degree of importance of future rewards . At each time , the agent interacts with the environment with its policy π , a mapping from state to action . The objective is to find an optimal policy that maximizes the expected discounted long-term reward E [ ∑∞ t=0 γ trt|π ] , which can be solved by estimating value functions . The state value of s and state-action value of s and a under policy π are defined as V π ( s ) = Eπ [ ∑∞ t=0 γ trt|s0 = s ] and Qπ ( s , a ) = Eπ [ ∑∞ t=0 γ trt|s0 = s , a0 = a ] . The optimal value functions are defined as V ∗ ( s ) = maxπ V π ( s ) and Q∗ ( s , a ) = maxπ Qπ ( s , a ) . Littman & Szepesvári ( 1996 ) proposed a general framework for reinforcement learning , where the generalized action selection operator is denoted by ⊗ . The optimal value function V ∗ satisfies the generalized Bellman equation , which is defined recursively as in Equation ( 1 ) : V ∗ ( s ) = ⊗ a∈A [ r ( s , a ) + ∑ s′∈S p ( s′|s , a ) γV ∗ ( s′ ) ] ( 1 ) Starting from arbitrary initial value function V0 , the optimal value function V ∗ can be computed by value iteration ( Bellman ( 2013 ) ) according to an iterative update : Vk+1 = T Vk , where T is the generalized the Bellman operator as defined in Equation ( 2 ) . ( T V ) ( s ) = ⊗ a∈A [ r ( s , a ) + ∑ s′∈S p ( s′|s , a ) γV ( s′ ) ] . ( 2 ) The convergence of value iteration is guaranteed if ⊗ is a non-expansion , which guarantees the unique solution of the generalized Bellman equation ( 1 ) , i.e. , T V ∗ = V ∗ . The non-expansion is defined as : ∣∣⊗ a Q1 ( s , a ) − ⊗ a Q2 ( s , a ) ∣∣ ≤ ||Q1 ( s , · ) −Q2 ( s , · ) ||∞ , ( 3 ) where || · ||∞ denotes the ` ∞-norm . The Boltzmann softmax operator is one kind of the action selection operator ⊗ , which is defined as : boltzβ ( X ) = ∑n i=1 xie βxi∑n i=1 e βxi . ( 4 ) 3 ANALYSIS OF THE BOLTZMANN SOFTMAX OPERATOR . In this section , we first analyze the property of the Boltzmann softmax operator and then propose a new technique to analyze its error bound in value iteration . It has been shown that the Boltzmann softmax operator is not a non-expansion ( ( Littman & Szepesvári ( 1996 ) ; Asadi & Littman ( 2016 ) ) ) as it does not satisfy Inequality ( 3 ) . Indeed , the non-expansive property is vital to the convergence of the learning algorithm , which guarantees the uniqueness of the fixed point . We first analyze the property of the Boltzmann softmax operator , which paves the path for studying the effect of using such operators that violates the non-expansive property in value iteration . Proposition 1 For β > 0 and X , Y ∈ Rn , the Boltzmann softmax operator satisfies the following property : |boltzβ ( X ) − boltzβ ( Y ) | ≤ ||X−Y||∞ + 2 log ( n ) β , ( 5 ) where ‖ · ‖∞ is the ` ∞-norm in Rn . In Proposition 1 , we show that although the Boltzmann softmax operator is not a non-expansive operator , the degree of the violation of the non-expansive property is controlled by β . The larger the value of β is , the closer it is to the non-expansion . Due to space limit , we put the proof of Proposition 1 in Appendix A . Next , we propose a new technique to characterize the error bound of value iteration with Boltzmann softmax operator in value iteration , where the full proof is referred to Appendix B. Theorem 1 ( Error bound of value iteration with Boltzmann softmax operator ) Let Vt be the value function computed by the Boltzmann softmax operator at the t-th iteration and V0 denote the initial value . After t iterations , ||Vt − V ∗||∞ ≤ γt||V0 − V ∗||∞ + 4 log ( |A| ) ( 1− γt ) β ( 1− γ ) . ( 6 ) Taking the limit of t in both sides of Inequality ( 6 ) , we obtain the following result : Corollary 1 For the Boltzmann softmax operator boltzβ , the error of value functions is limt→∞ ||Vt+1 − V ∗|| ≤ 4 log ( |A| ) β ( 1−γ ) . Corollary 1 characterizes the error bound of value iteration with the Boltzmann softmax operator . With a fixed parameter β , the error is upper bounded by 4 log ( |A| ) β ( 1−γ ) , which decreases with an increasing value of β . Thus , the direct use of the Boltzmann softmax operator inevitably introduces performance drop in practice . Motivated by the theoretical findings , we propose the dynamic Boltzmann softmax operator , which enables the convergence to the optimal .

Summary: This work demonstrates that, although the Boltzmann softmax operator is not a non-expansion, a proposed dynamic Boltzmann operator (DBS) can be used in conjunction with value iteration and Q-learning to achieve convergence to V* and Q*, respectively. This time-varying operator replaces the traditional max operator. The authors show empirical performance gains of DBS+Q-learning over Q-learning in a gridworld and DBS+DQN over DQN on Atari games.