Title: Hierarchical W-Learning
Primary area: reinforcement learning
Keywords: Q-learning, Hierarchical reinforcement learning, Robot navigation

Abstract: Inspired by a model of the brain called projective simulation, which has attracted interest among physicists in recent years, we develop a simple and generic new method for hierarchical reinforcement learning in this article. The proposed method generalizes the action-value Q function to  W function, enabling the agent to execute actions according to a hierarchical strategy. In the first part of the article, we present a rigorous construction of the hierarchical structure, along with the W-learning algorithm and the hierarchical policy gradient theorem. In the second part, as an example, we illustrate the W-learning procedure in the context of a navigation task. Experimental results show that the introduction of the hierarchical structure can lead to better performance than traditional Q-learning, provided the strategy is well designed and the update parameters are appropriately chosen.