Using Locally Weighted Regression to Enhance Q-learning

This paper describes a two-part method of dealing with the large or continuous state spaces encountered in large reinforcement learning (RL) problems. The problem is first approached by learning the value function over a coarse quantization of states. The value function is then approximated to extend the results to the original large or continuous state space. This second step improves the problem performance by allowing the agent to use all of the originally available state information. Performance of this method is significantly improved compared to the original performance of the RL algorithm. The nature of this method makes it directly applicable to practical reinforcement learning problems where perceptual aliasing or interminable learning times are a problem. A simulator is used to demonstrate this method on the task of obstacle avoidance in a complicated environment in which a moderate amount of perceptual aliasing is present. Significant performance enhancements are observed, as well as a learned policy that is not dependent on state history. Introduction: Oftentimes, the problem faced by researchers applying reinforcement learning to a nontrivial robotics problem is that they run headon into the curse of dimensionality. This is a particular problem for those researchers using discrete-state algorithms, as the number of states exponentially increases with the complexity of the problem. Various methods of dealing with large state spaces have been proposed and experimentally demonstrated, but none of these methods have effectively dealt with practical situations where the difference between states is sometimes not perceivable by the robot. It can be shown that many useful reinforcement learning algorithms experience theoretical difficulties under these conditions, and that other algorithms which explicitly account for incomplete perception either lack generality, or are too computationally awkward for effective use. The method introduced here for dealing with this problem is a two-step procedure. First the problems’ value function is learned over a small number of states (here accomplished with Q-learning). The second step involves approximating the value function so that it can generalize from those discrete states to a continuous state space, allowing previously discarded state information to be used effectively in the problem solution. Background and theoretical underpinning: An abridged version of the problem background is presented here. Readers wishing a more thorough background are encouraged to refer to [1]. Reinforcement learning describes situations where an agent is presented with a problem along with no information about how to solve it other than a set of rewards and possible actions. For example, in the obstacle avoidance problem, the agent knows nothing about the situation other than that it gets rewards for moving and not hitting anything. Given this , the agent can develop a policy for choosing a path through the obstacles through trial and error. This policy will be based entirely on the robots current sensory input (i.e. the state S), and a memory of what reinforcements were received upon previous visits to this state. Watkins [2] derived an algorithm, named Q-learning, that iteratively improved the agents estimate of a state’s value. Q-Learning Algorithm: • Initialize Q(s,a) arbitrarily for all states and actions. • Set s to some initial start state