Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
This paper presents Spatial Planning Transformers (SPTs); neural network modules that perform spatial planning over grid-like state spaces. The paper also goes on to present the idea that differentiable mapping and differntiable planning modules could be trained end-to-end, for better performance. This is evaluated against two baselines (value iteration networks (VINs) and generative path planning networks (GPPNs)) on small-scale navigation and manipulation tasks.


## Strengths

**S1** The central idea presented in this paper is quite interesting. Re-formulating the planning problem as learning an attention operator over an input map is both interesting, and challenging. The approach does have a strong motivation, considering the impact transformers have had in language processing. SPTs seek to replace memory-based models, such as LSTMs with purely attention-based models such as transformers, to potentially capture long-range dependencies.

**S2** The paper presents a promising idea in attempting to integrate differentiable mapping modules with differentiable planners. This could instigate further work on the lines of what kind of inductive biases are fruitful to have in neural planners.

**S3** The paper was very-well written and easy to follow.


## Weaknesses

I do find several aspects of the paper can be improved, and I list my most important observations here. I look forward to the author response, and am open to changing my evaluation if the concerns are adequately addressed.

**W1** Discrete action spaces: It seems like the proposed approach is tailored to discrete state-spaces or maps (and perhaps to discrete actions)? It could be interesting to know whether the approach scales to continuous state and/or action spaces. Upfront, it seems dubious, as it is unclear how the transformer blocks might apply to continuous maps.

**W2** Scalability issues: The assumption of a grid-like partitioning indicates that SPTs may exhibit unfavourable scaling properties? For instance, if one were to move from a 2D (planar) space to 3D, this seems like it will dramatically increase planning complexity. (While this can also be argued for "classical" planners like RRT, they often are accompanied by an exploration-exploitation tradeoff that lends them an anytime flavour)

**W3** Better placement wrt diff planning literature: I believe that SPTs can be better placed with respect to current literature on differentiable planning. For instance a few interesting approaches in this space include references [A-J]. Some of these [F, G, J] could be construed as concurrent work (but may be worth citing in a camera-ready version, for example). It would be a great addition to incorporate interesting new work in the robotics community on neural and/or differentiable planning [H, J]. It may also be worth using some of these approaches as baselines for evaluation.

**W4** More baselines: The current draft seems to only use VINs and GPPNs as baseline planning approaches. The presented method will seem better-placed in the context of other recent contributions if more baselines were to be introduced. For instance, universal planning networks, motion planning networks, semi-parameteric topological memory are all relevant baselines for navigation and/or manipulation.

**W5** In trying to present two important ideas (SPTs, diff mapping + planning), the current manuscript appears to lack a thorough study of neither. For instance, are there various combinations of differentiable mapping and planning strategies that can be evaluated? One would also have expected an evaluating against non-differentiable mapping + planning (i.e., "classical" robotics) strategies to strengthen this claim.


## Summary

In summary, while the paper presents a very interesting idea, there are several questions to be asked of the experimental demonstrations of the idea. This could benifit from a revision, and I strongly encourage the authors to address some of the issues raised above.


## References

[A] Learning to Plan in High Dimensions via Neural Exploration-Exploitation Trees - arXiv 2019

[B] Learning to search with MCTSnets - ICML 2018

[C] TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learning - ICLR 2018

[D] Memory augmented control networks - ICLR 2018

[E] Learning to Plan in High Dimensions via Neural Exploration-Exploitation Trees - arXiv 2019

[F] Neural Enhanced Belief Propagation on Factor Graphs - arXiv 2020

[G] Neural Manipulation Planning on Constraint Manifolds - arXiv 2020

[H] Learning sampling distributions for robot motion planning - ICRA 2018

[I] Semi-parametric topological memory for navigation - ICLR 2018

[J] Graph neural networks for motion planning - arXiv 2020

review_title: Interesting idea; need more experiments

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a method to infer goal distance maps for use in path planning using transformer networks. These 2D distance maps are evaluated on both planning and combined mapping and planning experiments on synthetic data.

While the proposed method seems technically sound and based on the experiments works as intended, it is not clear how the proposed system would be applicable in practice. Additionally, the motivations for the proposed approach over classical planning methods, such as run-time and incomplete environments, were not explored.

From a machine learning point of view, there is nothing particularly novel about the proposed method which uses well known and existing methods to build the planning framework. This is not a negative point for methods which develop interesting algorithms for applications. However, in the case of this paper, there are many questions regarding the applicability of the resulting system.
The proposed method produces a distance to goal map based on an obstacle representation. While such a distance map can be used with search methods to find a path, it does not constitute a path planning method itself. Furthermore, the proposed method is by design limited to 2D spaces. While this to some extent might be justifiable for ground-based robots, it precludes the use of the approach in challenging outdoor environments or on flying robots. More importantly, this reduces the applicability this method has in manipulation severely as the vast majority of manipulators used these days have six or seven degrees of freedom. These limitations are furthermore exaggerated by the very coarse environment discretization used in the examples of 0.25 cm or 10 degrees. While 0.25 cm grid sizes for navigation can be acceptable in large outdoor environments, the 10 degree discretization used would be unsuitable for most manipulation tasks.

One of the motivations given by the paper for the development of an end to end learning framework is the computational efficiency of having to only perform a single forward pass over the input. This, in theory, enables fast recomputation of a solution should the environment change. Given the 2D grid structure of the input and outputs, one would expect a comparison to widely used search methods such as A* and R*. These methods would be very fast to compute on the problem setups used in the experiments while being provably optimal. Providing a run-time comparison against such methods would have been good to see. Similarly, comparisons to RRT based methods, which in the limit are also provably optimal, would be appreciated.
While the combination of planning and mapping was interesting, it is not clear why this was needed, as methods capable of accurately mapping large scale environments exist without relying on learning systems. As the focus of the paper was on planning, this also caused some confusion on their relation.

Overall, while the proposed method works, it is not clear what practical benefit it has. As a deep learning approach, it requires a significant amount of training and has no guarantees to provide predictable results at all times. This is in stark contrast with decade-old methods such as A*, RRT*, and trajectory optimization which have such guarantees. The major limitation of the proposed method to 2D spaces severely limits its applicability with questions regarding scalability given the sizes used in the experimental section.


=== Post rebuttal ===

The inclusion of runtime comparisons with search based methods is a welcome addition, as it showcases that the method provides an interesting avenue to utilize GPUs for this kind of search task. Effectively trading up-front training time for deployment computational time. However, the question regarding the ability to handle larger state spaces is only touched on in the appendix and it is not clear from that description what the state space was. If the proposed approach works for planning in 6D environments or control of 7DoF manipulators then this is very interesting and should not be hidden away, so to speak, in the appendix but be highlighted in the main paper and showcased.

In light of the considerable amount of work that has been put into the revision I changed my score from 3 to 4. While I could see improvements and clarifications I could not see the main concern I had, handling of high dimensional problems, being addressed.
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: The paper correctly identifies some of the shortcomings of classical planning methods. The idea to use a transformer for planning seems promising and that is backed up by experimental results where the method outperforms previous work. I think the paper has some interesting ideas and the experimental section seems promising, but some assumptions are not stated clearly enough (I will discuss specific points under "Cons"). I recommend accepting the paper on the basis that the ideas presented are relevant to the community.

### Pros

* To the best of my knowledge, this is the first paper to propose combining transformers with neural planning in the style of value iteration networks.
* The paper deals with both robotic manipulation and navigation in the same framework. It is nice to see these related problems being tackled jointly.
* The method has an explicit component for creating maps, which is generally omitted in the framework of value iteration networks and their derivatives.
* The experimental results clearly demonstrate an edge over other VIN-type approaches.

### Cons

* The section on the mapper makes it sound like it maps one observation to an entire map, though in almost all cases, one observation won't be enough to do so. No mechanism for aggregating observations over time is described. The only way I can see this working is if the method gets an exhaustive dictionary of images from many locations and orientations in the environment, such as the 3D experiments by [Lee et al.](https://arxiv.org/pdf/1806.06408.pdf) The experiments section also gives this impression. This does not invalidate the approach but I would like this to be stated more clearly and earlier on in the text.

* The end-to-end module is trained using action-distance labels. This is basically assuming we have access to shortest-distance labels or a full value function. I don't understand how this is preferable to assuming we have full knowledge of the obstacles, which the paper seems to imply by saying:

  > This essentially allows us to train a mapping module without any map-level supervision using a pretrained planner and action-level supervision.

  My issue here is that I don't understand how we could get exhaustive shortest-distance labels without having access to an occupancy map or a list of obstacles. It's fine to assume we have access to such things during training, I just want this to be stated clearly.

* Some metric that is a direct measure of navigation success would be beneficial (e.g. percentage of successful attempts or [success-path-length](https://arxiv.org/pdf/1807.06757.pdf)). The action-prediction-accuracy is a proxy to this but one can imagine situations where a model can still have high navigation success while taking sub-optimal actions some of the time.
review_title: This paper aims to solve spatial planning using a neural network. The idea is to encode inductive biases about spatial environments into a neural planner by learning on a set of environments. A multi-head attention mechanism is used to propagate information across long distances.

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
Summary:
 
The paper provides an interesting direction in the field of spatial path planning. The method is interesting as it is fully learnt in an end to end fashion. The key idea is to use a transformer like architecture to model long range dependencies. Also the paper extend its findings to out of distribution maps and  the cases where the ground truth map is not known to the agent. 

##########################################################################

Reasons for score: 
 
Overall, I think the paper is slightly below the acceptance threshold. The motivations and the architecture are clearly presented, but I think there is a lack of clarity in the way the methods and the results are presented, especially in the most interesting section, the one about the mapper. Also the analysis are limited to reconstructions of the map and some attention weights in the appendix. 

##########################################################################

Pros: 
 
1. Interesting new approach with potential applications in both navigations and manipulations task.
2. The out of distribution results are convincing.

Cons:

1.  Although the method is a clear advantage over previous ones it still requires to know the ground truth distance to get the map. 

2. Instead of using a paragraph to explain the transformer architecture, which is relatively well know, my suggestion would be clarifying the mapper section with more notation to clearly state the loss. This would already help to most likely increase my score.

3. In section 3.2 the authors claim that “While it is possible to train a separate mapper model to predict maps from observations, this requires map annotations which are expensive to obtain and often inaccurate”. I don’t think this is true: e.g. Gregor K. et al., 2019 ->  https://arxiv.org/pdf/1906.09237.pdf present an example of how a map can be learnt.

4. It is not clear how much the model relies on having perfect distances as supervision vs. noisy ones. An analysis on this topic I think would be important.

5. No Analysis of computational complexity

review_title: Transformer for planning: interesting, but few points need clarification.

Meta-review: This paper proposes to jointly learn a mapper and planner for navigation or manipulation in a 2D space represented by an MxM grid, with the mapper taking raw observations as inputs and producing a 2D MxM occupancy and goal location map, and the planner -- pretrained on generic 2D maps of same size MxM -- produces an MxM action distance image (the plan). The whole system is trained end-to-end, with the mapper trained on the specific navigation or manipulation task, and the planner frozen. The Spatial Planning Transformer network can predict the MxM action plans faster than baselines (Value Iteration Networks and Gated Path Planning Networks) because of the attention mechanism in the transformer architecture, as opposed to the local information propagation through the convolutional encoding of the Bellman equation in VIN or GPPN. The differentiable approach is motivated by exploiting regularities in 2D maps and a faster inference time (as opposed to classical Dijkstra planners or VIN / GPPN), and is demonstrated on a simple room navigation task in the Gibson environment.

Reviewers have praised how a simple idea (transformer-based planning à la VIN) can be applied as a unifying approach to 2 DOF manipulation and 2D navigation, and the out-of-sample evaluation. The critique was about:
* the lacunary explanation of the mapper (corrected by the authors),
* the limits of non-recurrent mappers that cannot handle occlusions in the observations when building maps,
* confusion about where the full MxM image of action labels (necessary for supervision) can come from (the authors added experiments with sparse labels),
* claims about this approach being preferable to A* (countered by the authors, who conducted extensive experiments at the request of R4),
* focus on two problems at once (mapping and planning) rather than more ablation analysis of the planner, with a potential comparison to Active Neural SLAM,
* missing evaluation using navigation-specific metrics, such as SPL.

The authors have a point in their defense of differentiable / learning-based methods for planning (including VIN and GPPN) as opposed to classical planning such as A*, and there is value in investigating how an end-to-end differentiable method for mapping and planning could be designed. At the same time, several reviewers raised concerns about scalability and about the pertinence of combining two learnable modules (mapper and planner) rather than investigating and demonstrating the advantage of transformers for planning.

Given these reviews, rebuttal, and remaining concerns, I am sorry to reject this paper. I hope that with the proposed modifications it will be quickly accepted at another venue.
Decision: Reject