Differentiable Spatial Planning using Transformers

We consider the problem of spatial path planning . In contrast to the classical solutions which optimize a new plan from scratch and assume access to the full map with ground truth obstacle locations , we learn a planner from the data in a differentiable manner that allows us to leverage statistical regularities from past data . We propose Spatial Planning Transformers ( SPT ) , which given an obstacle map learns to generate actions by planning over long-range spatial dependencies , unlike prior data-driven planners that propagate information locally via convolutional structure in an iterative manner . In the setting where the ground truth map is not known to the agent , we leverage pre-trained SPTs to in an end-to-end framework that has the structure of mapper and planner built into it which allows seamless generalization to out-of-distribution maps and goals . SPTs outperform prior stateof-the-art across all the setups for both manipulation and navigation tasks , leading to an absolute improvement of 7-19 % . 1 INTRODUCTION ! '' ! # Why learn to plan ( explicitly ) ? • More effective than • reactive CNNs • implicit planning LSTMs • Differentiable : can learn perception end-to-end using only action supervision • Faster than analytical planning 3 Observations CNN+LSTM Actions Observations Perception Model Planning Model ActionsMap Reactive or Implicit Planning Explicit Planning Decomposing Perception and Planning ( a ) Navigation ( b ) Manipulation Figure 1 : Spatial Path Planning : The raw observations ( top left ) and obstacles can be represented spatially via top-down map in navigation ( left ) and via configuration space in manipulation ( right ) . The problem of path planning has been a bedrock of robotics . Given an obstacle map of an environment and a goal location in the map , the task is to output a shortest path to the goal location starting from any position in the map . We consider path planning with spatial maps . Building a top-down spatial map is common practice in robotic navigation as it provides a natural representation of physical space ( Durrant-Whyte & Bailey , 2006 ) . In fact , even robotic manipulation can also be naturally phrased via spatial map using the formalism of configuration spaces ( Lozano-Perez , 1990 ) , as shown in Figure 1 . This problem has been studied in robotics for several decades , and classic goto planning algorithms involve Dijkstra et al . ( 1959 ) , PRM ( Kavraki et al. , 1996 ) , RRT ( LaValle & Kuffner Jr , 2001 ) , RRT * ( Karaman & Frazzoli , 2011 ) , etc . Our objective is to develop methods that can learn to plan from data . However , a natural question is why do we need learning for a problem which has stable classical solutions ? There are two key reasons . First , classical methods do not capture statistical regularities present in the natural world , ( for e.g. , walls are mostly parallel or perpendicular to each other ) , because they optimize a plan from scratch for each new setup . This also makes analytical planning methods to be often slow at inference time which is an issue in dynamic scenarios where a more reactive policy might be required for fast adaptation from failures . A learned planner represented via a neural network can not only capture regularities but is also efficient at inference as the plan is just a result of forward-pass through the network . Second , a critical assumption of classical algorithms is that a global ground-truth obstacle space must be known to the agent ahead of time . This is in stark contrast to biological agents where cognitive maps are not pixel-accurate ground truth location of agents , but built through actions in the environment , e.g. , rats build an implicit map of the environment incrementally through trajectories enabling them to take shortcuts ( Tolman , 1948 ) . A learned solution could not only provides the ability to deal with partial , noisy maps and but also help build maps on the fly while acting in the environment by backpropagating through the generated long-range plans . Several recent works have proposed data-driven path planning models ( Tamar et al. , 2016 ; Karkus et al. , 2017 ; Nardelli et al. , 2018 ; Lee et al. , 2018 ) . Similar to how classical algorithms , like Dijkstra et al . ( 1959 ) , move outward from the goal one cell at a time to predict distances iteratively based on the obstacles in the map , current learning-based spatial planning models propagate distance values in only a local neighborhood using convolutional networks . This kind of local value propagation requires O ( M ) iterations , where M is the map dimension . In theory , however , the optimal paths can be computed much more efficiently with total iterations that are on the order of number of obstacles rather than the map size . For instance , consider two points with no obstacle between , an efficient planner could directly connect them with interpolated distance . Nonetheless , this is possible only if the model can perform long-range reasoning in the obstacle space which is a challenge . In this work , our goal is to capture this long-range spatial relationship . Transformers ( Vaswani et al. , 2017 ) are well suited for this kind of computation as they treat the inputs as sets and propagate information across all the points within the set . Building on this , we propose Spatial Planning Transformers ( SPT ) which consists of attention heads that can attend to any part of the input . The key idea behind the design of the proposed model is that value can be propagated between distant points if there are no obstacles between them . This would reduce the number of required iterations to O ( nO ) where nO is the number of obstacles in the map . Figure 2 shows a simple example where long-distance value propagation can cover the entire map within 3 iterations while local value propagation takes more than 5 iterations – this difference grows with the complexity of the obstacle space and map size . We compare the performance of SPTs with prior state-of-the-art learned planning approaches , VIN ( Tamar et al. , 2016 ) and GPPN ( Lee et al. , 2018 ) , across both navigation as well as manipulation setups . SPTs achieve significantly higher accuracy than these prior methods for the same inference time and show over 10 % absolute improvement when the maps are large . Next , we turn to the case when the map is not known apriori . This is a practical setting when the agent either has access to a partially known map or just know it through the trajectories . In psychology , this is known as going from route knowledge to survey knowledege ( Golledge et al. , 1995 ) where animals aggregate the knowledge from trajectories into a cognitive map . We operationalize this setup by formulating an end-to-end differentiable framework , which in contrast to having a generic parametric policy learning ( Glasmachers , 2017 ) , has the structure of mapper and planner built into it . We first pre-train the SPT planner to capture a generic data-driven prior , and then backpropagate through it to learn a mapper that maps raw observations to obstacle map . This allows us to learn without requiring map supervision or interaction . Learned mapper and planner not only allow us to plan for new goal locations at inference but also generalize to unseen maps . Our end-to-end mapping and planning approach provides a unified solution for both navigation and manipulation . We perform thorough experiments in both manipulation and real-world navigation maps as well as manipulation . Our approach outperforms prior state-of-the-art by a margin on both mapping and planning accuracy without assuming access to the map at training or inference . 2 PRELIMINARIES AND PROBLEM DEFINITION . We represent the input spatial map as a matrix , m , of size M ×M with each element being 1 , denoting obstacles , or 0 , denoting free space . The goal location is also represented as a matrix , g , of size M ×M with exactly one element being 1 , denoting the goal location , and rest 0s . The input to the spatial planning model , x , consists of matrices m and g stacked , x = [ m , g ] , where x is of size 2×M ×M . The objective of the planning model is to predict y which is of size M ×M , consisting of action distances of corresponding locations from the goal . Here , action distance is defined to be the minimum number of actions required to reach the goal . For navigation , m is a top-down obstacle map , and g represents the goal position on this map . For manipulation , m represents the obstacles in the configuration space of 2-dof planar arm with joint angles denoted by θ1 and θ2 . Each element ( i , j ) in m indicate whether the configuration of the arm with joint angles θ1 = i and θ2 = j , would lead to a collision . g represents the goal configuration of the arm . In the first set of experiments , we will assume that m is known and in the second set of experiments , m is not known and the agent receives observations , o , from its sensors instead . 3 METHODS . We design a spatial planning model capable of long-distance information propagation . We first describe the design of this spatial planning module , called Spatial Planning Transformer ( SPT ) , shown in Figure 3 , which takes in a map and a goal as input and predict the distance to the goal from all locations . We then describe how the SPT model can be used as a planning module to train end-to-end learning models , which take in raw sensory observations and goal location as input and predict action distances without having access to the map . 3.1 SPT : SPATIAL PLANNING TRANSFORMERS . To propogate information over distant points , we use the Transformer ( Vaswani et al. , 2017 ) architecture . The self-attention mechanism in a Transformer can learn to attend to any element of the input . The allows the model to learn spatial reasoning over the whole map concurrently . Figure 3 shows an overview of the SPT model , which consists of three modules , an Encoder E to encode the input , a Transformer network T responsible for spatial planning , and a Decoder D decoding the output of the Transformer into action distances . Encoder . The Encoder E computes the encoding of the input x : xI = E ( x ) . The input x ∈ { 0 , 1 } 2×M×M consisting of the map and goal is first passed through a 2-layer convolutional network ( LeCun et al. , 1998 ) with ReLU activations to compute an embedding for each input element . Both layers have a kernel size of 1 × 1 , which ensures that the embedding of all the obstacles is identical to each other , and the same holds true for free space and the goal location . The output of this convolutional network is of size d×M ×M , where d is the embedding size . This output is then flattened to get xI of size d×M2 and passed into the Transformer network . Transformer . The Transformer network T converts the input encoding into the output encoding : xO = T ( xI ) . It first adds the positional encoding to the input encoding . The positional encoding enables the Transformer model to distinguish between the obstacles at different locations . We use a constant sinusoidal positional encoding ( Vaswani et al. , 2017 ) : p ( 2i , j ) = sin ( j/C 2i/d ) , p ( 2i+1 , j ) = cos ( j/C 2i/d ) where p ∈ Rd×M2 is the positional encoding , j ∈ { 1 , 2 , . . . , M2 } is the position of the input , i ∈ { 1 , 2 , . . . , d/2 } , and C =M2 is a constant . The positional encoding of each element is added to their corresponding input encoding to get Z = xI + p . Z is then passed through N = 5 identical Transformer layers ( fTL ) to get xO . Decoder . The Decoder D computes the distance prediction ŷ from xO using a position-wise fully connected layer : ŷi =W T DxT , i + bD where xT , i ∈ Rd×1 is the input at position i ∈ 1 , 2 , . . . , M2 , WD ∈ Rd×1 , bD ∈ R are parameters of the Decoder shared across all positions i and ŷi ∈ R is the distance prediction at position i . The distance prediction at all position are reshaped into a matrix to get the final prediction ŷ ∈ RM , M . The entire model is trained using pairs of input x and output y datapoints with mean-squared error as the loss function .

Summary: This paper presents Spatial Planning Transformers (SPTs); neural network modules that perform spatial planning over grid-like state spaces. The paper also goes on to present the idea that differentiable mapping and differntiable planning modules could be trained end-to-end, for better performance. This is evaluated against two baselines (value iteration networks (VINs) and generative path planning networks (GPPNs)) on small-scale navigation and manipulation tasks.