Policy Iteration on Continuous Domains using Rapidly-exploring Random Trees

Reinforcement Learning(RL) deals with learning optimal control strategies through interactions with the world. One of the major hindrances in extending existing RL techniques to real world scenarios, is their inability to deal with continuous valued variables. Techniques such as variable discretization and using function approximators in place of lookup tables in the original discrete algorithms exist. Least Squares Policy Iteration(LSPI) [4] is an offline method that uses a set of experiences and linear function approximators. Although LSPI is a fundamentally sound algorithm, the question of how exactly the experiences should be generated exists. Attempts have been made to address this problem of online exploration in large domains by combining LSPI with R-MAX, a technique that does efficient exploration in discrete domains[6]. Several implementation issues arise in translating this to continuous domains, such as tuning exploration parameters and generalizing state visitation counters. We propose an algorithm RRTPI that combines ideas from the domain of continuous path planning to clearly provide a method to generate sufficiently representative samples and ensure exploration. We briefly introduce rapidly-exploring random trees(RRTs)[5] and their properties that allow us to use them for solving optimal control problems in deterministic continuous domains. A RRT is grown as follows: (i) Sample a state s from a given state space S according to a sampling distribution fs (ii) Calculate nearest node snearest in the tree as measured by a distance metric | . |V (iii) Extend the tree from snearest towards s as given by the | . |V and repeat till required size is reached. RRTs are sampling based methods that have been used to solve for feasible trajectories from a given starting to goal state in kinodynamic spaces, which is a problem that is at least PSPACE-hard. RRTs have good space filling properties and possess asymptotic completeness, i.e. they eventually find a solution if one exists. RRT* an extended version of RRTs have been developed that also guarantee asymptotic optimality, i.e. they eventually converge to an optimal solution as more samples are drawn [3]. These methods tend to perform poorly in domains that are underactuated and underpowered. It has been shown that RRTs explore the space efficiently only when the distance metric | . |V reflects the true cost to go[2], or the value function as they are known in RL literature, and not the euclidean distance. Several techniques have been developed that use LQR based approximations to the value function for efficient exploration. We use fitted policy evaluation on the samples generated by RRTs to estimate the value function. We then generate close to optimal trajectories by biasing the RRT using this estimated value function and then iteratively improves this by using the new samples to re-estimate the value function and so on. Given a model that allows us to sample transitions and rewards, we can build a sample set D consisting of (s, s′, r) where s and s′ are the current and next state and r is the reward. A procedure FPE(.),