Approximate Dynamic Programming Using Random Sampling : Summary

The focus of this grant is on developing approximate dynamic programming methods for the control of high performance nonlinear systems, based on using local models of the value function and policy, optimizing trajectory segments to compute the value function, and managing sets of trajectories to find globally optimal value functions and optimal policies. We will compare our work to adaptive critic approaches that use global function approximation methods including both linear representations and nonlinear representations such as sigmoidal neural nets, radial basis functions, and wavelets. We are using humanoid robots as a domain to evaluate our nonlinear control design techniques. The basis of much of this work are randomized variants of dynamic programming. Dynamic programming generates globally optimal trajectories and finds a control law or policy that covers the relevant portion of state space. In the past dynamic programming has been viewed as limited by the “curse of dimensionality”, where the cost of computing and representing the value function increases exponentially with dimensionality. This proposal focuses on several ways to reduce the effect of the curse of dimensionality. Our preliminary results with up to eight dimensional problems indicate that a randomized state approach with thousands of states performs better than a traditional grid-based approach with millions of states. Intellectual Merit: Key contributions of this work will be: 1) We will develop an integrated approach for dynamic planning based on approximate dynamic programming for high performance systems. 2) We will show how to combine many fast greedy local planners to produce globally optimal solutions. 3) We will show how to represent knowledge along explicit trajectories, which increases the feasible sparseness of this approximate dynamic programming method. 4) We will show how to create an adaptive representation using random sampling of states. 5) We will show how to use local models, and compare them to global parametric function approximators. 6) We will show how to use fast trajectory optimization to speed up dynamic programming. 7) We will focus initially on deterministic planning, and then explore how to deal with modeling error. 8) We will evaluate our approach in simulation and on actual robots, and compare it with other adaptive critic approaches. We expect our approach to be able to discover underlying simplicity in complex high dimensional control problems. Broader impacts: The impact of this work will be to enable high performance (near optimal) transient control in areas like robotics, transportation, and energy. We will use a 50 dimensional humanoid robot as our testbed for this work. The impact if this work is not funded is that ADP will continue to have low acceptance in such high performance applications. The potential societal payoff is industrial processes, machines, and robots that perform better and waste fewer resources. The ability to plan how complex dynamic processes should work is a key step towards controlling them effectively. More autonomous optimal control methods will simplify and reduce the expense of programming robots and machine-based systems in general. We will evaluate these algorithms on humanoid robots, but we see them as broadly applicable. Another of our goals is to make humanoid robots useful elements of society, and developing an ability to do sophisticated planning on the part of these robots is a key element of making robots provide useful services.