Given recent deep learning results that demonstrate the ability to effectively optimize high-dimensional non-convex functions with gradient descent optimization on GPUs, we ask in this paper whether symbolic gradient optimization tools such as Tensorflow can be effective for planning in hybrid (mixed discrete and continuous) nonlinear domains with high dimensional state and action spaces? To this end, we demonstrate that hybrid planning with Tensorflow and RMSProp gradient descent is competitive with mixed integer linear program (MILP) based optimization on piecewise linear planning domains (where we can compute optimal solutions) and substantially outperforms state-of-the-art interior point methods for nonlinear planning domains. Furthermore, we remark that Tensorflow is highly scalable, converging to a strong policy on a largescale concurrent domain with a total of 576,000 continuous actions over a horizon of 96 time steps in only 4 minutes. We provide a number of insights that clarify such strong performance including observations that despite long horizons, RMSProp avoids both the vanishing and exploding gradients problem. Together these results suggest a new frontier for highly scalable planning in nonlinear hybrid domains by leveraging GPUs and the power of recent advances in gradient descent with highly optmized toolkits like Tensorflow. Introduction Many real-world hybrid (mixed discrete continuous) planning problems such as Reservoir Control [Yeh, 1985], Heating, Ventilation and Air Conditioning (HVAC) [Erickson et al., 2009; Agarwal et al., 2010], and Navigation [Faulwasser and Findeisen, 2009] have highly nonlinear transition and (possibly nonlinear) reward functions to optimize. Unfortunately, existing state-of-the-art hybrid planners [Ivankovic et al., 2014; Löhr et al., 2012; Coles et al., 2013; Piotrowski et al., 2016] are not compatible with arbitrary nonlinear transition and reward models. Monte Carlo Tree Search (MCTS) methods [Coulom, 2006; Kocsis and Szepesvári, 2006; Keller and Helmert, 2013] 0.600 0.750 .900 0.600 0.750 .900 0.600 0.750 0.9 00 0.600 0.750 .900 0.600 0.750 .900 0.600 0.750 .900 0.600 0.750 0.9 00 Epochs:10 Epochs:20 Epochs:40 Epochs:80 Epochs:160 Epochs:320 Figure 1: The evolution of RMSProp gradient descent based Tensorflow planning in a 2D Navigation domain with nested central rectangles indicating nonlinearly increasing resistance to robot movement. (top) In initial RMSProp epochs, the plan evolves directly towards the goal shown as a star. (bottom) As later epochs of RMSProp descend the objective cost surface, the fastest path evolves to avoid the central obstacle entirely. including AlphaGo [Silver et al., 2016] that can use any (nonlinear) black box model of transition dynamics do not inherently work with continuous action spaces due to the infinite branching factor. While MCTS with continuous action extensions such as HOOT [Weinstein and Littman, 2012] have been proposed, their continuous partitioningmethods do not scale to high-dimensional continuous action spaces (e.g., 100’s or 1,000’s of dimensions as used in this paper). Finally, offline model-free reinforcement learning (e.g., Q-learning) with function approximation [Sutton and Barto, 1998; Szepesvári, 2010] and deep extensions [Mnih et al., 2013] do not require any knowledge of the (nonlinear) transition model or reward, but they also do not directly apply to domains with high-dimensional continuous action spaces. I.e., offline learning methods like Q-learning require action maximization for every update, but in high-dimensional continuous action spaces such nonlinear function maximization is non-convex and computationally intractable at the scale of millions or billions of updates. To address the above scalability and expressivity limitations of existing methods, we turn to Tensorflow [Abadi et al., 2015], which is a symbolic computation platform used in the machine learning community for deep learning due to its compilation of complex layered symbolic functions into a representation amenable to fast GPU-based reverse-mode automatic differentiation [Linnainmaa, 1970] for gradient-based optimization. Given recent results in gradient descent optimization with deep learning that demonstrate the ability to effectively optimize high-dimensional non-convex functions, we ask whether Tensorflow can be effective for planning in hybrid (mixed discrete and continuous) nonlinear domains with high dimensional state and action spaces? Our results answer this question affirmatively, where we demonstrate that hybrid planning with Tensorflow and RMSProp gradient descent [Tieleman and Hinton, 2012] is surprisingly effective at planning in complex hybrid nonlinear domains. As evidence, we reference figure 1, where we show Tensorflow with RMSProp optimizing a path in a 2d nonlinear Navigation domain. In general, Tensorflow with RMSProp planning results are competitive with optimal MILPbased optimization on piecewise linear planning domains and directly extend to nonlinear domains, where they substantially outperform interior point methods for nonlinear function optimization. Furthermore, we remark that Tensorflow converges to a strong policy on a large-scale concurrent domain with 576,000 continuous actions spread over a horizon of 96 time steps in 4 minutes. To explain such excellent results, we note that gradient descent algorithms such as RMSProp are highly effective for the non-convex function optimization that occurs in deep learning. Further, we provide an analysis of many transition functions in planning domains that suggest gradient descent on these domains will not suffer from either the vanishing or exploding gradient problems and hence provide a strong signal for optimization over long horizons. Together these results suggest a new frontier for highly scalable planning in nonlinear hybrid domains by leveraging GPUs and the power of recent advances in gradient descent with the Tensorflow toolkit. Hybrid Nonlinear Planning via Tensorflow
Output: Scalable Planning with Tensorflow for Hybrid Nonlinear Domains