Clustered Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) ( Sutton & Barto , 1998 ) studies how an agent can maximize its cumulative reward in an unknown environment , by learning through exploration and exploitation . A key challenge in RL is to balance the relationship between exploration and exploitation . If the agent explores novel states excessively , it might never find rewards to guide the learning direction . Otherwise , if the agent exploits rewards too intensely , it might converge to suboptimal behaviors and have fewer opportunities to discover more rewards from exploration . Although reinforcement learning , especially deep RL ( DRL ) , has recently attracted much attention and achieved significant performance in a variety of applications , such as game playing ( Mnih et al. , 2015 ; Silver et al. , 2016 ) and robot navigation ( Zhang et al. , 2016 ) , exploration techniques in RL are far from satisfactory in many cases . Exploration strategy design is still one of the challenging problems in RL , especially when the environment contains a large state space or sparse rewards . Hence , it has become a hot research topic to design exploration strategy , and many exploration methods have been proposed in recent years . Some heuristic methods for exploration , such as -greedy ( Silver et al. , 2016 ; Sutton & Barto , 1998 ) , uniform sampling ( Mnih et al. , 2015 ) and i.i.d./correlated Gaussian noise ( Lillicrap et al. , 2016 ; Schulman et al. , 2015 ) , try to directly obtain more different experiences during exploration . For hard applications or games , these heuristic methods are insufficient enough and the agent needs exploration techniques that can incorporate meaningful information about the environment . In recent years , some exploration strategies try to discover novel state areas for exploring . The direct way to measure novelty is to count the visited experiences . In ( Bellemare et al. , 2016 ; Ostrovski et al. , 2017 ) , pseudo-counts are estimated from a density model . Hash-based method ( Tang et al. , 2017 ) counts the hash codes of states . There also exist some work using the counts of state-action pairs to design their exploration techniques , such as explicit explore or exploit ( E3 ) ( Kearns & Singh , 2002 ) , R-Max Brafman & Tennenholtz ( 2002 ) , UCRL ( Auer & Ortner , 2006 ) , UCAGG ( Ortner , 2013 ) . Besides , the state novelty can also be measured by empowerment ( Klyubin et al. , 2005 ) , the agent ’ s belief of environment dynamics ( Houthooft et al. , 2016 ) , prediction error of the system dynamics model ( Pathak et al. , 2017 ; Stadie et al. , 2015 ) , prediction by exemplar model ( Fu et al. , 2017 ) , and the error of predicting features of states ( Burda et al. , 2018 ) . All the above methods perform exploration mainly based on the novelty of states without considering the quality of states . Furthermore , there are some methods to estimate the quality of states . Kernel-based reinforcement learning ( Ormoneit & Sen , 2002 ) uses locally weighted averaging to estimate the quality ( value ) of states . UCRL ( Auer & Ortner , 2006 ) and UCAGG ( Ortner , 2013 ) compute average rewards for choosing optimistic values . The average reward can be regarded as an estimation of the quality of states to guide the exploring direction , but there are no methods using the quality of states as an exploration technique . Furthermore , in most existing methods , the novelty and quality in the neighboring area of the current state are not well utilized to guide the exploration of the agent . To tackle this problem , we propose a novel RL framework , called clustered reinforcement learning ( CRL ) , for efficient exploration in RL . The contributions of CRL are briefly outlined as follows : • CRL adopts clustering to divide the collected states into several clusters . The states from the same cluster have similar features . Hence , the clustered results in CRL provide a possibility to share meaningful information among different states from the same cluster . • CRL proposes a novel bonus reward , which reflects both novelty and quality in the neighboring area of the current state . Here , the neighboring area is defined by the states which share the same cluster with the current state . This bonus reward can guide the agent to perform efficient exploration , by seamlessly integrating novelty and quality of states . • Experiments on several continuous control tasks with sparse rewards and several hard exploratory Atari-2600 games ( Bellemare et al. , 2013 ) show that CRL can outperform other state-of-the-art methods to achieve the best performance in most cases . In particular , on several games known to be hard for heuristic exploration strategies , CRL achieves significantly improvement over baselines . 2 RELATED WORK . Recently , there are some exploration strategies used to discover novel state areas . The direct way to measure the novelty of states is to count the visited experiences , which has been applied in several methods . In the tabular setting and finite Markov decision processes ( MDPs ) , the number of stateaction pairs is finite which can be counted directly , such as model-based interval estimation with exploratory bonus ( MBIE-EB ) ( Strehl & Littman , 2008 ) , explicit explore or exploit ( E3 ) ( Kearns & Singh , 2002 ) and R-Max ( Brafman & Tennenholtz , 2002 ) . MBIE-EB adds the reciprocal of square root of counts of state-action pairs as the bonus reward to the augmented Bellman equation for exploring less visited ones with theoretical guarantee . E3 determines the action based on the counts of state-actions pairs . If the state has never been visited , the action is chosen randomly and if the state has been visited for some times , the agent takes the action that has been tried the fewest times before . R-Max uses counts of states as a way to check for known states . In the continuous and high-dimensional space , the number of states is too large to be counted directly ( Bellemare et al. , 2016 ; Ostrovski et al. , 2017 ; Tang et al. , 2017 ; Abel et al. , 2016 ) . Bellemare et al . ( 2016 ) and Ostrovski et al . ( 2017 ) use a density model to estimate the state pseudo-count quantity , which is used to design the exploration bonus reward . Tang et al . ( 2017 ) counts the number of states by using the hash function to encode states and then it explores by using the reciprocal of visits as a form of reward bonus , which performs well on some hard exploration Atari-2600 games . Abel et al . ( 2016 ) records the number of cluster center and action pairs and makes use of it to select an action from the Gibbs distribution . These count-based methods encourage the agent to explore by making use of the novelty of states and do not take quality into consideration . Furthermore , there are some methods to estimate the quality of states . Average reward , in kernel-based reinforcement learning ( Ormoneit & Sen , 2002 ) , UCRL ( Auer & Ortner , 2006 ) and UCAGG ( Ortner , 2013 ) , can be regarded as the quality of states . Kernel-based reinforcement learning ( Ormoneit & Sen , 2002 ) is proposed to solve the stability problem of TD-learning by using locally weighted averaging to estimate the value of state . UCRL ( Auer & Ortner , 2006 ) and UCAGG ( Ortner , 2013 ) use average reward to choose optimistic values . Besides , the value of cluster space can also indicate the quality of states . Singh et al . ( 1994 ) uses the value of cluster space with Q-learning and TD ( 0 ) by soft state aggregation and provides convergence results . But these methods do not use the quality of states to explore more areas . To the best of our knowledge , the novelty and quality in the neighboring area of the current state have not been well utilized to guide the exploration of the agent in existing methods , especially in the high dimensional state space . This motivates the work of this paper . 3 NOTATION . In this paper , we adopt similar notations as those in ( Tang et al. , 2017 ) . More specifically , we model the RL problem as a finite-horizon discounted Markov decision process ( MDP ) , which can be defined by a tuple ( S , A , P , r , ρ0 , γ , T ) . Here , S ∈ Rd denotes the state space , A ∈ Rm denotes the action space , P : S×A×S → R denotes a transition probability distribution , r : S×A → R denotes a reward function , ρ0 is an initial state distribution , γ ∈ ( 0 , 1 ] is a discount factor , and T denotes the horizon time . In this paper , we assume r ≥ 0 . For cases with negative rewards , we can transform them to cases without negative rewards . The goal of RL is to maximize Eπ , P [ ∑T t=0 γ tr ( st , at ) ] which is the total expected discounted reward over a policy π . 4 CLUSTERED REINFORCEMENT LEARNING . This section presents the details of our proposed RL framework , called clustered reinforcement learning ( CRL ) . The key idea of CRL is to adopt clustering to divide the collected states into several clusters , and then design a novel cluster-based bonus reward for exploration . 4.1 CLUSTERING . Intuitively , both novelty and quality are useful for exploration strategy design . If the agent only cares about novelty , it might explore intensively in some unexplored areas without any reward . If the agent only cares about quality , it might converge to suboptimal behaviors and have low opportunity to discover unexplored areas with higher rewards . Hence , it is better to integrate both novelty and quality into the same exploration strategy . We find that clustering can provide the possibility to integrate both novelty and quality together . Intuitively , a cluster of states can be treated as an area . The number of collected states in a cluster reflects the count ( novelty ) information of that area . The average reward of the collected states in a cluster reflects the quality of that area . Hence , based on the clustered results , we can design an exploration strategy considering both novelty and quality . Furthermore , the states from the same cluster have similar features , and hence the clustered results provide a possibility to share meaningful information among different states from the same cluster . The details of exploration strategy design based on clustering will be left to the following subsection . Here , we only describe the clustering algorithm . In CRL , we perform clustering on states . Assume the number of clusters isK , and we have collected N state-action samples { ( si , ai , ri ) } Ni=1 with some policy . We need to cluster the collected states { si } Ni=1 into K clusters by using some clustering algorithm f : S → C , where C = { Ci } Ki=1 and Ci is the center of the i-th cluster . We can use any clustering algorithm in the CRL framework . Although more sophisticated clustering algorithms might be able to achieve better performance , in this paper we just choose k-means algorithm ( Coates & Ng , 2012 ) . K-means is one of the simplest clustering algorithms with wide applications . The detail of k-means is omitted here , and readers can find it in most machine learning textbooks .

Summary: This paper presents a clear approach to improve the exploration strategy in reinforcement learning, which is named clustered reinforcement learning. The approach tries to push the agent to explore more states with high novelty and quality.  It is done by adding a bonus reward shown in Eq. (3) to the reward function. The author first cluster states into clusters using the k-means algorithm. The bonus reward will return a high value for a state if the corresponding cluster has a high average reward. When the total reward in a cluster is smaller than a certain threshold, the bonus reward will consider the number of states explored. In the experiments, the authors test different models on two MuJoCo tasks and five Atari games. TRPO, TRPO-Hash, VIME are selected as baselines to compare with. Results show that the proposed bonus reward reaches faster convergence and the highest return in both MuJoCo tasks. In those five Atari games, the proposed method achieves the highest or second-highest average returns.