Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents an approach to leverage problem-domain symmetry for planning problems defined over small 2D lattices. The key idea is to extend value iteration networks by using steerable convolutions to exploit symmetric structure (e.g., translational/rotational equivariance). Evaluation is carried out on three problem domains, and the presented approach performs better than VIN (value iteration networks) and GPPN (gated path planning networks).
strengths_and_weaknesses: Strengths
=========

**S1** The paper is very well-written. Differentiable path planning is an area of great interest to the Neurips(/ICLR/ICML) community, as evidenced by several similar papers in the past (e.g., VIN, GPPN, [35-38]).

**S2** The paper is largely self-contained. As a reader who was not an expert in symmetries and VIN, I appreciated the pointers to relevant readings. This section helped establish context for the technical contributions sections.

**S3** At a technical level, the paper looks well-executed. The core hypothesis is sound -- the 2D gridworld domain (with the considered transition) exhibits symmetries that can be leveraged by learning-based planners to both perform well on the planning problem and also to generalize better to novel problem instances. Building on VINs, which reformulate value iteration as a series of convolutional operators, the proposed approach additionally leverages group convolutions (specifically steerable CNNs) to induce symmetry. The formulation is sound, and achieves better performance compared to variants that do not explicitly assume symmetric structure in the problem domains.

---

Weaknesses
==========

**W1** *Experiments*: Currently, analysis is only carried out on three 2D problem domains (if counting the C-space and workspace manipulation environments as distinct). In table 1, ConvGPPN seems to already achieve stellar performance on all tasks (89.88 success rate % on the workspace manipulation env, >97% on all other envs). The gains due to SymGPPN, while consistent, do not appear to be significant. This might well be because of the inherent task complexity (ConvGPPNs essentially seem to 'solve' the task); to better investigate the benefits of SymGPPN either larger problem instances or more complex problem domains are necessary.

**W2** *Inherent limitations*: One potential reason for choosing low-dimensional (2D) problem domains and further, small problem sizes (largest problem involves 50 x 50 grid) is the (apparent) poor scalability of the approach. Is this because value iteration scales poorly with problem size, and that it is a core component of the approach? Would(n't) this also impact scalability to more complex problem domains (e.g., 3D environments involving larger action spaces, for example)? (While I appreciate the fact that the manuscript lists this as an avenue for future work, I also believe this shortcoming greatly limits the problems this technique can be applied to, thereby affecting the perceived impact of this work).

---

In summary, I think this work tackles an exciting direction; but I am of the opinion that it needs more experimental analysis and some strategies to mitigate the inherent limitations it brings along.
questions: I would like to see **W1** and **W2** discussed
limitations: please see **W2** (while authors explicitly list a limitation as an avenue for future work, I believe that limits the scope of the current submission -- this is factored into my eventual score)
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper identifies the symmetry in 2D path planning problem and proposes a framework for incoporating the symmetry into an end-to-end differentiable planning framework. This is done mainly by extending the CNN in value iteration network to steerable CNN. The paper demonstrates the advantages of such approaches in two 2D path planning domains.
strengths_and_weaknesses: To say it upfront, with a background in robotic manipulation and reinforcement learning, I do not have background in steerable feature fields and consider the paper outside my area of expertise. I find it difficult to understand the technical details of the paper. As such, I can only provide some high-level feedback to the paper and hope the AC and other reviewers can provide more detailed evaluations.

Here are some suggestions on writing:
1. The explaination of steerable CNN in Sec. 3 heavily references [14, 15, 16], making it hard for me to understand the concept from reading this paragraph alone. Some notations and jargons are also not explained. Figure 2 does not help much as multiple concepts are squeezed into one figure. 
2. The benefit of differentiable planning may not be well known in the robotics community. The two tasks done in the paper seems almost trivial and can be easily solved using other path planning techniques like RRT or A*. The paper could benefit from a better motivation on why working on differentiable planning.
questions: 1. My understanding is that the symmetry group is manually defined instead of being learned through data. In such case, is there a simpler way of incoporating the symmetry? For example, one way could be applying each transformation in the group, run the planner and returns the plan with the lowest cost.

2. Can the approach generalize to continuous actions?
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes using steerable convolutions in the value iteration networks framework to incorporate equivariance under rotations and reflections.
strengths_and_weaknesses: ### Strengths

* The proposed approach brings quite a bit of improvement over a regular VIN.
* Equivariance under rotation and reflection are valuable inductive biases.

### Weaknesses

* I believe the paper has an accessibility issue. The main audience for the paper will be researchers interested in VINs. I would expect these to generally not be too familiar with steerible CNNs and the theory that goes into them. Yet, the paper is quite heavy with mathematical notation and terminology from this area. I understand that the authors need to use the math that goes into steerible CNNs to be able to explain their ideas. Though at the moment specificity is getting in the way of clarity. Sections 4 and 5 currently contain too much jargon that is spread throughout the text in a way that makes it genuinely hard to follow the narrative through-line. I would like to give some passages as examples, just to make it clear, what exactly I mean: lines 104-115, 123-128, 171-175, 194-198, 243-249. I think it is in the best interest of the authors to rethink the presentation of sections 4 and 5, and present their ideas in a way that requires a minimum of knowledge about steerible CNNs. Otherwise, you are limiting your reach by creating a hurdle of mathematical prerequisites. Of course, the reader needs to eventually familiarize themselves with the math to truly understand the paper, but the first time reader shouldn't be completely lost either. Again, most people who read your paper won't be familiar with this math, and currently I find it unlikely they will have an easy time.
* The gains in performance over ConvGPPN are somewhat marginal. This would be less of an issue if it weren't for my previous point. If I was a researcher or practitioner interested in using VINs, I believe Table 1 might convince me to use ConvGPPN and accept a slight drop in performance in exchange for the relative algorithmic and theoretical simplicity.
questions: ### Questions

* Does the method extend to rotations that are not $90^\circ$?

* Do you think your method will bring improvements over data augmentation techniques that could extend the data set with rotated/reflected versions of the normal environments?

limitations: These points are addressed well.
review_title: null

Meta-review: metareview: The paper addresses path planning with RGB inputs by leveraging the workspace symmetry. To that end, the authors propose a end-to-end differentiable planner  that builds on top of VINs and evaluate the method of several 2D-grid planning tasks.

The reviewers recognized that the method presents a performance improvement compared to VINs-like methods, but have raised the questions around the accessibility, scalability, and overall benefits of the method. During the rebuttal, the authors added new experiments to show the method's efficiency in larger environments, and reorganized the manuscript for the better accessibility.

I have read the final version of the manuscript. Based on the current state of the manuscript and the reviewers feedback, I do not believe that it is ready for the publication. My main concerns are around the positioning of the paper and the accessibility.

Positioning of the work -- the authors present the work as addressing robot path planning. The environments and evaluation tasks, even with the new experiments, are toy for the robotics. The 2D C-Space with image observations and no robot dynamics is not a suitable robotics problem. See PRM-RL [Faust et al., ICRA 2018], RL-RRT [Chiang et al., RA-L 2019], Critical PRMs [Ichter et al., ICRA 2020], optimal control w/ visual navigation [Bansal et al, CORL 2020] for methods that combine motion planning, controls and perception (ego sensors, motion planning, and non-trivial robot dynamics and geometry). Granted, they are not differential planning, but they solve more path planning more realistic and complex settings. (In the rebuttal the authors comment that the differentiable planning is capable of jointly training perception with the transition model is intractable for RRT or A*. However, the transition model is trivial here -- there are no kinodynamic constraints, or complicated geometry.) Perhaps a better framing for the presented work is as incorporating symmetry into latent planning, instead of framing it around the robotics. 

It is not clear what problem the paper is seeking to solve. Please add a clear definition of the path planning problem. Are the policies goal-conditioned? Is the generalization over the workspaces, or the initial configurations or both? What is in the training set? Are the connections between the planning points known or not? And are there any other constraints on the transition function (beyond the workspace constraints)?

Accessibility --  Even after the rebuttal, Sections 3 and 4 are not clear. The symmetry is not introduced well for a non-expert. Some questions -- If I understand correctly, the symmetric NNs maps inputs to equivalent states. How is that different from latent spaces? Is the proposed method too specific for CNNs, which are rapidly becoming obsolete, in favor of newer models? How would the method compare to VAE? I suggest that authors take an intuitive example of the symmetry (for example, we expect the planner to learn when it sees the wall in a given direction, that the transition in that direction is not possible. The same will hold for left, right, top or bottom. So we hope that by exploiting symmetry, we can speed up the learning since the agent would need to learn only on a single instance of the equivalence class, and generalize to the others.) Lastly, the paper would be stronger, with a more in-depth analysis of the method. Where and how exactly did the symmetry help?

Overall, the exploitation of workspace symmetry in E2E differential planning has merit. But the framing around the robotics, VINs, and CNNs is too specific, yielding results which significance is not clear. With more generalized framing rooted into the current ML trends this paper can make a strong a valuable contribution.
confidence: Certain
Decision: Reject