Differentiable Spatial Planning using Transformers

We consider the problem of spatial path planning . In contrast to the classical solutions which optimize a new plan from scratch and assume access to the full map with ground truth obstacle locations , we learn a planner from the data in a differentiable manner that allows us to leverage statistical regularities from past data . We propose Spatial Planning Transformers ( SPT ) , which given an obstacle map learns to generate actions by planning over long-range spatial dependencies , unlike prior data-driven planners that propagate information locally via convolutional structure in an iterative manner . In the setting where the ground truth map is not known to the agent , we leverage pre-trained SPTs to in an end-to-end framework that has the structure of mapper and planner built into it which allows seamless generalization to out-of-distribution maps and goals . SPTs outperform prior stateof-the-art across all the setups for both manipulation and navigation tasks , leading to an absolute improvement of 7-19 % . 1 INTRODUCTION ! '' ! # Why learn to plan ( explicitly ) ? • More effective than • reactive CNNs • implicit planning LSTMs • Differentiable : can learn perception end-to-end using only action supervision • Faster than analytical planning 3 Observations CNN+LSTM Actions Observations Perception Model Planning Model ActionsMap Reactive or Implicit Planning Explicit Planning Decomposing Perception and Planning ( a ) Navigation ( b ) Manipulation Figure 1 : Spatial Path Planning : The raw observations ( top left ) and obstacles can be represented spatially via top-down map in navigation ( left ) and via configuration space in manipulation ( right ) . The problem of path planning has been a bedrock of robotics . Given an obstacle map of an environment and a goal location in the map , the task is to output a shortest path to the goal location starting from any position in the map . We consider path planning with spatial maps . Building a top-down spatial map is common practice in robotic navigation as it provides a natural representation of physical space ( Durrant-Whyte & Bailey , 2006 ) . In fact , even robotic manipulation can also be naturally phrased via spatial map using the formalism of configuration spaces ( Lozano-Perez , 1990 ) , as shown in Figure 1 . This problem has been studied in robotics for several decades , and classic goto planning algorithms involve Dijkstra et al . ( 1959 ) , PRM ( Kavraki et al. , 1996 ) , RRT ( LaValle & Kuffner Jr , 2001 ) , RRT * ( Karaman & Frazzoli , 2011 ) , etc . Our objective is to develop methods that can learn to plan from data . However , a natural question is why do we need learning for a problem which has stable classical solutions ? There are two key reasons . First , classical methods do not capture statistical regularities present in the natural world , ( for e.g. , walls are mostly parallel or perpendicular to each other ) , because they optimize a plan from scratch for each new setup . This also makes analytical planning methods to be often slow at inference time which is an issue in dynamic scenarios where a more reactive policy might be required for fast adaptation from failures . A learned planner represented via a neural network can not only capture regularities but is also efficient at inference as the plan is just a result of forward-pass through the network . Second , a critical assumption of classical algorithms is that a global ground-truth obstacle space must be known to the agent ahead of time . This is in stark contrast to biological agents where cognitive maps are not pixel-accurate ground truth location of agents , but built through actions in the environment , e.g. , rats build an implicit map of the environment incrementally through trajectories enabling them to take shortcuts ( Tolman , 1948 ) . A learned solution could not only provides the ability to deal with partial , noisy maps and but also help build maps on the fly while acting in the environment by backpropagating through the generated long-range plans . Several recent works have proposed data-driven path planning models ( Tamar et al. , 2016 ; Karkus et al. , 2017 ; Nardelli et al. , 2018 ; Lee et al. , 2018 ) . Similar to how classical algorithms , like Dijkstra et al . ( 1959 ) , move outward from the goal one cell at a time to predict distances iteratively based on the obstacles in the map , current learning-based spatial planning models propagate distance values in only a local neighborhood using convolutional networks . This kind of local value propagation requires O ( M ) iterations , where M is the map dimension . In theory , however , the optimal paths can be computed much more efficiently with total iterations that are on the order of number of obstacles rather than the map size . For instance , consider two points with no obstacle between , an efficient planner could directly connect them with interpolated distance . Nonetheless , this is possible only if the model can perform long-range reasoning in the obstacle space which is a challenge . In this work , our goal is to capture this long-range spatial relationship . Transformers ( Vaswani et al. , 2017 ) are well suited for this kind of computation as they treat the inputs as sets and propagate information across all the points within the set . Building on this , we propose Spatial Planning Transformers ( SPT ) which consists of attention heads that can attend to any part of the input . The key idea behind the design of the proposed model is that value can be propagated between distant points if there are no obstacles between them . This would reduce the number of required iterations to O ( nO ) where nO is the number of obstacles in the map . Figure 2 shows a simple example where long-distance value propagation can cover the entire map within 3 iterations while local value propagation takes more than 5 iterations – this difference grows with the complexity of the obstacle space and map size . We compare the performance of SPTs with prior state-of-the-art learned planning approaches , VIN ( Tamar et al. , 2016 ) and GPPN ( Lee et al. , 2018 ) , across both navigation as well as manipulation setups . SPTs achieve significantly higher accuracy than these prior methods for the same inference time and show over 10 % absolute improvement when the maps are large . Next , we turn to the case when the map is not known apriori . This is a practical setting when the agent either has access to a partially known map or just know it through the trajectories . In psychology , this is known as going from route knowledge to survey knowledege ( Golledge et al. , 1995 ) where animals aggregate the knowledge from trajectories into a cognitive map . We operationalize this setup by formulating an end-to-end differentiable framework , which in contrast to having a generic parametric policy learning ( Glasmachers , 2017 ) , has the structure of mapper and planner built into it . We first pre-train the SPT planner to capture a generic data-driven prior , and then backpropagate through it to learn a mapper that maps raw observations to obstacle map . This allows us to learn without requiring map supervision or interaction . Learned mapper and planner not only allow us to plan for new goal locations at inference but also generalize to unseen maps . Our end-to-end mapping and planning approach provides a unified solution for both navigation and manipulation . We perform thorough experiments in both manipulation and real-world navigation maps as well as manipulation . Our approach outperforms prior state-of-the-art by a margin on both mapping and planning accuracy without assuming access to the map at training or inference . 2 PRELIMINARIES AND PROBLEM DEFINITION . We represent the input spatial map as a matrix , m , of size M ×M with each element being 1 , denoting obstacles , or 0 , denoting free space . The goal location is also represented as a matrix , g , of size M ×M with exactly one element being 1 , denoting the goal location , and rest 0s . The input to the spatial planning model , x , consists of matrices m and g stacked , x = [ m , g ] , where x is of size 2×M ×M . The objective of the planning model is to predict y which is of size M ×M , consisting of action distances of corresponding locations from the goal . Here , action distance is defined to be the minimum number of actions required to reach the goal . For navigation , m is a top-down obstacle map , and g represents the goal position on this map . For manipulation , m represents the obstacles in the configuration space of 2-dof planar arm with joint angles denoted by θ1 and θ2 . Each element ( i , j ) in m indicate whether the configuration of the arm with joint angles θ1 = i and θ2 = j , would lead to a collision . g represents the goal configuration of the arm . In the first set of experiments , we will assume that m is known and in the second set of experiments , m is not known and the agent receives observations , o , from its sensors instead . 3 METHODS . We design a spatial planning model capable of long-distance information propagation . We first describe the design of this spatial planning module , called Spatial Planning Transformer ( SPT ) , shown in Figure 3 , which takes in a map and a goal as input and predict the distance to the goal from all locations . We then describe how the SPT model can be used as a planning module to train end-to-end learning models , which take in raw sensory observations and goal location as input and predict action distances without having access to the map . 3.1 SPT : SPATIAL PLANNING TRANSFORMERS . To propogate information over distant points , we use the Transformer ( Vaswani et al. , 2017 ) architecture . The self-attention mechanism in a Transformer can learn to attend to any element of the input . The allows the model to learn spatial reasoning over the whole map concurrently . Figure 3 shows an overview of the SPT model , which consists of three modules , an Encoder E to encode the input , a Transformer network T responsible for spatial planning , and a Decoder D decoding the output of the Transformer into action distances . Encoder . The Encoder E computes the encoding of the input x : xI = E ( x ) . The input x ∈ { 0 , 1 } 2×M×M consisting of the map and goal is first passed through a 2-layer convolutional network ( LeCun et al. , 1998 ) with ReLU activations to compute an embedding for each input element . Both layers have a kernel size of 1 × 1 , which ensures that the embedding of all the obstacles is identical to each other , and the same holds true for free space and the goal location . The output of this convolutional network is of size d×M ×M , where d is the embedding size . This output is then flattened to get xI of size d×M2 and passed into the Transformer network . Transformer . The Transformer network T converts the input encoding into the output encoding : xO = T ( xI ) . It first adds the positional encoding to the input encoding . The positional encoding enables the Transformer model to distinguish between the obstacles at different locations . We use a constant sinusoidal positional encoding ( Vaswani et al. , 2017 ) : p ( 2i , j ) = sin ( j/C 2i/d ) , p ( 2i+1 , j ) = cos ( j/C 2i/d ) where p ∈ Rd×M2 is the positional encoding , j ∈ { 1 , 2 , . . . , M2 } is the position of the input , i ∈ { 1 , 2 , . . . , d/2 } , and C =M2 is a constant . The positional encoding of each element is added to their corresponding input encoding to get Z = xI + p . Z is then passed through N = 5 identical Transformer layers ( fTL ) to get xO . Decoder . The Decoder D computes the distance prediction ŷ from xO using a position-wise fully connected layer : ŷi =W T DxT , i + bD where xT , i ∈ Rd×1 is the input at position i ∈ 1 , 2 , . . . , M2 , WD ∈ Rd×1 , bD ∈ R are parameters of the Decoder shared across all positions i and ŷi ∈ R is the distance prediction at position i . The distance prediction at all position are reshaped into a matrix to get the final prediction ŷ ∈ RM , M . The entire model is trained using pairs of input x and output y datapoints with mean-squared error as the loss function .

Summary: The paper provides an interesting direction in the field of spatial path planning. The method is interesting as it is fully learnt in an end to end fashion. The key idea is to use a transformer like architecture to model long range dependencies. Also the paper extend its findings to out of distribution maps and  the cases where the ground truth map is not known to the agent.