Integrating Symmetry into Differentiable Planning
Authors: Linfeng Zhao
Xupeng Zhu
Lingzhi Kong
Robin Walters
Lawson L.S. Wong
Venue: NeurIPS 2022 Submitted (2022)
Keywords: planning
equivariance
symmetry
reinforcement learning
model-based reinforcement learning

Abstract: We study how group symmetry helps improve data efficiency and generalization for end-to-end differentiable planning algorithms, specifically on 2D robotic path planning problems: navigation and manipulation. We first formalize the idea from Value Iteration Networks (VINs) on using convolutional networks for path planning, because it avoids explicitly constructing equivalence classes and enables end-to-end planning. We then show that value iteration can always be represented as some convolutional form for (2D) path planning, and name the resulting paradigm Symmetric Planner (SymPlan). In implementation, we use steerable convolution networks to incorporate symmetry. Our algorithms on navigation and manipulation, with given or learned maps, improve training efficiency and generalization performance by large margins over non-equivariant counterparts, VIN and GPPN.

28 Nevertheless, symmetry in model-free deep reinforce- ment learning (RL) has been studied recently [9, 10]. However, it can only handle pixel-level "element-wise" symmetry, such as flipping or rotating state and action together. However, critical benefit of model-fir -ree RL agents that enables great asymptotic performance its end-to-end differentiability. This motivates us to com- Integrating Symmetry into Differentiable Planning
Anonymous Author(s)
Affiliation Address
email
Abstract
We study how group symmetry helps improve data efficiency and generalization for end-to-end differentiable planning algorithms, specifically on 2D robotic path planning problems: navigation and manipulation. We first formalize the idea from Value Iteration Networks (VINs) on using convolutional networks for path plan- ning, because it avoids explicitly constructing equivalence classes and enable end- to-end planning. We then show that value iteration can always be represented as some convolutional form for (2D) path planning. and name the resulting paradigm Symmetric Planner (SymPlan), In implementation, we use steerable convolution networks to incorporate symmetry. Our algorithms on navigation and manipula- tion, with given or learned maps, improve training efficiency and generalization performance by large margins over non-equivariant counterparts, VIN and GPPN.
2
11
M
9.M
Introduction
12
Model-based planning usually struggles in complex prob- 14 lems, and planning in more structured and abstract space
is major solution [1, 2, 3, 4]. Symmetry is ubiquitous
16 in learning and decision-making problems and can effec- tively reduce search space for planning. However, isting planning algorithms using symmetry assumes per-
SymPPhu(AA)
SymPlun(af
17
18
19 feet dynamics knowledge, needs to explicitly build equiv- alence classes, or does not consider problem structure A on path plan- 22 ning, we cannot specify visually obvious rotation sym-
20
21 [5, 4, 7, 8]. For example,
23 metry in Figure and need to detect in manually from
24 the provided dynamics model. This would be even more challenging detect in differentiable planning.
25
The path planning problem has symmetry, so we study how to exploit its symmetry in (differentiable) planning. Red dots goal. The optimal actions (ar- rows) = SymP1am(M) (bottom row) for the maps M (top row) are guaran- teed to be equivariant SymPlan(g.M
Figure 1:
27
28
29
30
31
g. y.SymPlan Lan( (M) under O rotations for (2D) path planning.
32
example, the action in the NW corner of A is the same as the action in the SW comner of g.A, after also rotating the arrow O 90"
33 bine the spirit of both: possible to design an end-to-
34 end differentiable planning algorithm that makes use of symmetry in environments?
35
38 In this work, we propose to (1) avoid explicitly building equivalence classes for symmetric states
37 while (2) realize planning in an end-to-end differentiable manner. We are motivated by work in
38 the equivariant network and geometric deep learning community [11, 12, 13, 14, 15, 16), which
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

39 treat an RGB image as mapping 2² + I3 and apply equivariant convolutions between feature maps. It satisfies our desiderata: equivariant networks on images do not need explicitly consider "symmetric pixels" while guarantee symmetry properties. Based on the intuition, we propose a framework, Symmetric Planning (SymPlan), to understand a straightforward but general problem, path planning, as operating like images, called steerable feature fields [14, 16]. We focus on 2D) grid and prove that value iteration (VI) for 2D path planning is equivariant under the isometries of z²: translations, rotations, and reflections, and further show that VI here is special form of steerable convolution network [14]. This provides us a foundation to equip Value Iteration Network (VIN, [17]) with steerable convolution. We implement the equivariant steerable version of VIN, named VIM,IN, and use a variant, GPPN, to build SymGPPN. Both SymPlan methods achieve great improvement training efficiency and generalization performance to unseen random maps, which showcases the advantage of exploiting symmetry from environments for planning.
40
41
42
48
44
45
46
48
50
51
Our contributions are as follows:
Understand the inherent symmetry in path planning problems (on 2D grids), formulate value iter- 52
53
ation in as steerable convolution network, and connect both to incorporate symmetry into VI.
54
Based on the formulation, implement equivariant steerable version of VIN and GPPN.
55
Show significant improvement in training and generalization on 2D navigation and manipulation.
2
Related work
56
57
Planning with symmetries (Symmetric Planning). Symmetries widely exist in various domains, and have been exploited in classic planning algorithms as well as model checking [5, 4, 6, 18, 19,
58
59 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]. Zinkevich and Balch [7] show the invariance of value function
60 for an MDP with symmetry. Narayanamurthy and Ravindran [8] prove that finding exact symmetry
61 in MDPs is graph isomorphism complete. However, they are based on classic planning algorithms, such A#, and have a fundamental issue with exploitation of symmetries: they explicitly construct 63 equivalence classes of symmetric states, which explicitly represents states and introduces symmetry breaking. Therefore, they are intractable (NP-hard) in maintaining symmetries in trajectory rollout and forward search (for large state space and symmetry group) and incompatible with differentiable pipelines for representation learning, hindering it from wider applications in RL and robotics.
64
65
67
State abstraction for detecting symmetries. Coarsest state abstraction aggregates all symmetric states into equivalence classes, studied in MDP homomorphisms and bisimulation [3, 30, 2]. How- ever, they usually require perfect MDP dynamics knowledge and do not scale up well, because of the complexity in maintaining abstraction mappings (homomorphisms) and abstracted MDPs. van der Pol et al. [31] integrate symmetry into model-free RL based on MDP homomorphisms [3], which avoids the challenges in handling symmetry in forward search. Park et al. [32] learn equivariant transition models, but do not consider planning. Additionally, the formulation in commonly defined
68
69
70
71
73
74 symmetric MDPs [3. 9, 6, 7] is different from our symmetry formulation for path planning, since symmetry for every state-action pairs and require reward to be symmet- 76 ric. Our reward is not symmetric and we mainly study symmetry of the underlying domain (2D grid), as further discussed in Section B.
75 they study "element-wize
77
78
Symmetries and equivariance in deep learning. Equivariant neural networks are used to incor- porate symmetry in supervised learning for different domains (e.g. grid and sphere), symmetry groups (e.g. translations and rotations), and group representation on feature spaces [33]. Cohen and
79
80
81 Welling [15] introduce G-CNNs, followed by Steerable CNNs [14] which generalizes from sealar
feature fields to vector fields with induced representations. Kondor and Trivedi [13], Cohen et al.
83
[12] study theory on the relation between equivariant maps and convolutions. Weiler and Cesa [16]
propose to solve kernel constraints under arbitrary representations for E2) and its subgroups by decomposing into irreducible representations, named ((2)-CNN.
85
86
Differentiable planning. Our pipeline is based on learning to plan in neural network in a differ- entiable manner. Value iteration network (VIN) [34] is representative work that performs value iteration using convolution on lattice grids, and has been further extended [35, 36, 38). Other than using convolution network, works on integrating learning and planning into differentiable networks include [39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]. In the theoretical side, Grimm et al. [50, 51] propose to understand the differentiable planning algorithms from value equivalence perspective, while Gehring et al. [52] study its gradient dynamics.
87
88
89
90
91

---

93
3
Preliminaries
94
Markoy decision processes. We model the path planning problems as Markov decision processes
(MDP) [1]. An MDP is 5-tuple M (S. A, R, with state space S, action space A, transition probability function P: : S x A x S + Rt, reward function R : S x A + R, and discount factor
95
96
7 [0, I]. Value functions V S R and Q: S x - R represent expected future returns [1].
97
98
Symmetry groups and equivarance. A symmetry group is defined as a set Gi together with a binary composition map satisfying the axioms of associativity, identity, and inverse. A (left) group action of G on a set X' is defined as the mapping (g. + g.z which is compatible with composition. X' and y, then f is G-equivariant if it commutes
99
100
Given a function
XX + y and G acting with group actions: 9. (x)( = f(g.z), Vg E G, Vx E X. In the special case the action on y is trivial g-y = y. then f()) = f(g.x) holds, and we say is G-invariant.
102
103
104
Group representations. We mainly use two groups: dihedral group D4 and cyclic group C4. The cyclic group of elements is C4 (r 1 1), symmetry group of rotating square. The dihedral group D4 (r,8 1 = 22= (sr)² = 1) includes both rotations r and reflections 8, and has size Dal 8. A group representation defines how group action transforms a set G G + S. These groups have three types of representations of our interest: trivial, regular, and quotient representations, see [16]. The trivial representation Puiv maps each g G to I and hence fixes all & E S. The regular representation Preg of C4 group sends each g E C4 to a 4 x permutation matrix that cyclically permutes 4-element vector, such as a one-hot 4-direction action. The regular representation of D4 maps each element to an 8 x 8 permutation matrix which does not act d-direction actions, which requires the quotient representations (quotienting out fr²) and forming a x 4 permutation matrix. It is worth mentioning the standard representation of the cyclic groups which are 2 x 2 rotation matrices, only used for visualization (Figure 2 middle).
106
=
108
110
112
113
114
115
116
Steerable feature fields and Steerable CNNs. The concept of feature fielss is used in (equivariant) CNNs [11, 12 13, 14, 15, 16]. The pixels of an 2D RGB image 2² - I23 on a domain Ω z² is feature field. In stecrable CNNs for 2D grid, features are formed as steerable feature fields z² + _º that associate C-dimensional feature vector f(x) E HC to each element on base space, such as Z². Defined like this, we know how to transform a steerable feature field and also the feature field after applying CNN on it, using some group [14]. The type of CNNs that operates on steerable feature fields is called Steerable CNN [14), which is equivariant to groups including translations as subgroup (Z², +), extending [15]. It needs satisfy kernel steerability constraint, where the 2 and 222 cases are considered in [16]. We consider the 2D grid as our domain
118
120
122
123
124
125 Ω = S Z and use G p4m group as the running example. The group p4m- = (Z2 +) x D4 (wallpaper group) is semi-direct product of discrete translation group 2² and dihedral group D1, see [15, 14]. We visualize the transformation law of p4m on a feature field on ? -² in Figure (iidlle), usually referred as induced representation [14, 16]. Additional details in Section C.
128
127
128
Planning convolution. The core component behind dynamic programming (DP) based algo- rithms in planning or reinforcement learning is Bellman (optimality) equation [53, 1]: (ss) = max, R( a(s.a) + +.E,, P(s'|s s, a)) (ss). Value Iteration (VI) iteratively applies Bellman operator and converges to fixed points [1, 53]. The key component of our interest is E.' P(s'| (s, (s') that aggregates values V(s') from adjacent states by expectation using transition probabilities, here referred as expected value operation. Tamar et al. [17] propose Value Iteration Network (VIN) that uses convolution (networks) for planning, as an instance of differentiable planning, by recursively applying planar convolutions and max-pooling over feature spaces on 2D grid E?2
130
132
133
134
135
136
4
Symmetric Planning Framework
137
138
This section formulates the notion of Symmetric Planning (SymPlan). We expand the understanding of path planning in neural networks by planning as convolution on steerable feature fields (steerable planning). We use that to build steerable value iteration and show it is equivariant.
139
140
141
4.1
Steerable Planning: planning on steerable feature fields
142
We start the discussion based on Value Iteration Networks (VINs, [17]) and use a running example of planning on the 2D grid Z22 We aim to understand (1) how VIN-style networks embed planning
148
3

---

6%
Special
Figure 2: (Left) Construction of spatial MDPs from path planning problems, enabling G- G-invariant transition. (Middle) The group acts feature field (MDP actions). We need to find the element in the original field by f(r-1x), and also rotate the arrow by p(r). where r E D4- We represent one-hot actions as arrows (vector field, using Paa) for visualization. (Right) Equivariance of V + Q in Bellman operator on feature fields, under O 90 E C₁ rotation, which visually explains Theorem 4.1. The example simulates VI for one step (see red circles; minus signs omitted) with true transition P using O N- N-W-S-E actions. The Q-value field are for actions and can be viewed as either 2² Ra ((14, 16)) Z C2 R (on pd group, [15]). See Appendix D for more details.
and how its idea generalizes, (2) how is symmetry structure defined in path planning and how could it be injected into such planning networks.
145
146
Constructing G-invariant transition: spatial MDP. Intuitively, the embedded MDP in VIN different from the original path planning problem, since (planar) convolutions are translation equivariant but there are different obstacles in different regions.
147
149
We found the key insight in VINs is that implicitly uses MDP that has translation equivariance. The core idea behind the construction is that it converts obstacles (encoded in transition probability P. by blocking) into "traps" (encoded in reward R, by -oo reward). This allows use planar con- volutions with translation equivariance, and also enables use to further use steerable convolutions.
150
151
152
The demonstration of the idea is shown in Figure 2 (Left). We call it spatial MDP, with different transition and reward function . = AS. P, Rm, R), which converts the "complexity" in the transition function P in M the reward function Rm, in M. The state and action space are kept the same: state = Z and action A C Z to move As in four directions a 2D grid. We provide the detailed construction of the spatial MDP in Appendix D.I.
154
156
157
158
Steerable features fields. We generalize the idea from VIN, by viewing functions (in RL and planning) steerable feature fields, motivated by [11, 12, 14]. This is analogous to pixels on images [255] and would allow us to apply convolution on The state value function is expressed as a field : + R, while the Q-value function needs a field with |A| channels: Q S + R|4|, Similarly, policy field has probability logits of selecting [A| actions. For the transition probability
160
161
163
(ss|ss, a), we can use action to index as P" (s'(ss, similarly for reward R" (s). The next section convert the transition function to field and even convolutional filter. Additional details are in Appendix D.
will show that
165
186
Symmetric Planning: symmetry by equivariance
The seemingly slight change in the construction of spatial MDPs brings important symmetry struc- ture. The general idea in exploiting symmetry in path planning is to use equivariance to avoid explicitly constructing equivalence classes of symmetric states. To this end, we construct value iteration over steerable feature fields, and show it equivariant for path planning.
168
170
171 In VIN, the convolution is over 2D grid Z², which is symmetric under D₂ (rotations and reflections). However, also know that VIN already equivariant under translations. To consider all symme- tries, as in [14, 16], we understand the group p4m G B H constructed by base space D4, which is a stabilizer subgroup that fixes the origin 0 E Z². We could then formally study such symmetry in the spatial MDP, since we construct it to
172
178
B G/ H (Z², +) and a fiber group
174
175
We avoid the symbol for policy since it is used for induced representation in [14, 16).

---

178
ensure that the transition probability function in M is G-invariant. Specifically, we can uniquely decompose any g E z² D4 t E z² and E D4 (and translations act "trivially" on action), so
177
P(s' s, P g.a) P ((tr).s' (tr).,,r.a), Vg= tr E z² D4, Vs, (1)
178
Expected value operator as steerable convolution. The equivariance property can be shown step- by-step: (1) expected value operation, (2) Bellman operator, and (3) full value iteration. First, we use G-invariance to prove that the expected value operator Es P(8 ((5)(s,)V((( (6'))r
178
as') is equivariant.
180
181
Theorem 4.1. If transition is G-invariant, the expected value operator E over 2² is G-equivariant.
182
The proof in Appendix E. I and visual understanding in Figure 2 left. However, this provides intuition but is inadequate since we do not know: (1) how to implement it with CNNs, (2) how to use multiple feature channels like VINs, since it shows for scalar-valued transition probability and value function (corresponding to trivial representation). To this end, we next prove that we can implement
183
184
186
value iteration using steerable convolution with general steerable kernels.
187
Theorem 4.2. If transition is G-invariant, there exists a (one-argument, isotropic) matrix-valued
188
steerable kernel Pa (s 1 s') (for every action), such that the expected value operator can be written as a steerable convolution and is G-equivariant:
189
E pa V, [g- VII(s) [ [g.V]](s), Vs N2 Z2 D1.
(2)
190
The full derivation is provided in Appendix E. We write the transition probability as (s, s'), and show it only depends on state difference (* s') (or one-argument kernel [12]) using G- invariance, which is the key step to show it is some convolution. Note that we use one kernel P for each action (four directions), and when the group acts on E, it also acts on the action pg.a (and state, so technically acting on S x A). Additionally, if the steerable kernel also satisfies the D4- steerability constraint [16, 54], the steerable convolution is equivariant under p4m = Z2 D4. We can then extend VINs from 2² translation equivariance to p-m-equivvriance (translations, rotations, reflections). The derivation follows the existing work on steerable CNNs [15, 14, 16, 12]. while this is our goal: to justify the close connection between path planning and steerable convolutions.
192
183
194
195
198
197
198
Steerable Bellman operator and value iteration. We can now represent all operations in Bellman (optimality) operator on steerable feature fields over Z² (or steerable Bellman operator) as follows:
200
V (+) = max (x) + x [Pe * (+),
where V, R", Pa are steerable feature fields over Z². As for the operations, max, is (max) pooling (over group channel), +, X are point-wise operations, and * is convolution. As the second step, the main idea is prove every operation in Bellman (optimality) operator on steerable fields is equivariant, including the nonlinear max, operator and +, x. Then, iteratively applying Bellman
203
204
205
208
operator forms value iteration and is also equivariant, as shown below and proved in Appendix E.4.
Proposition 4.3. For spatial MDP with G-invariant transition, the optimal value function can be found through G-steerable value iteration.
208
Remark. Our framework generalizes the idea behind VINS and enables us understand its appli- cability and restrictions. More importantly, this allows us to integrate symmetry but avoid explicilly building equivalence classes and enables planning with symmetry in end-to-end fashion. We em- phasize that the symmetry in spatial MDPs is different from symmetric MDPs [7, 3, 9], since our reward function is not G-invariant. Although we focus on Zz, we can generalize path planning on higher-dimensional or even continuous Euclidean spaces (like I& space [54] or spatial graphs in R)3 [55]), and use equivariani operations on steerable steerablefeatur feature fields (such as steerable convolutions,
210
212
213
214
215
216
pooling, and point-wise non non-linee linearities) from steerable CNNs. We refer the readers to Appendix D and to [15. 14, 56, 16] for more details.
217
Symmetric Planning in Practice
218
In this section, we discuss how to achieve Symmetric Planning on 2D grids with E(2)-steerable CNNs [16]. We focus on implementing symmetric version of value iteration, SymVIN, and gener- alize the methodology to make symmetric version of a popular follow-up of VIN, GPPN (36).
220
5

---

M
V/
Quit
Ru
U
Ry Freh
PC soltoa
Li
90'
trival
(ebowed
Bar B
g. 5.M
g.Qu
V.
9.4
Figure 3: Commutative diagram for the full pipeline of SYmVIN on steerable feature fields over Z² (every grid). If rotating the input map M by TM (g) of any 9, the output action A = SymVIN is guaranteed to be transformed by т (g). i.e. the entire steerable SymVIN is equivariant under induced representations иM and AA: SymVIN(MM ture fields to emphasize that SymVIN supports direct-sum of representations beyond scalar-valued.
= a (g) SymVII M). We use stacked fea-
Steerable value iteration. We have showed that, value iteration for path planning problems Z² consists of equivariant maps between steerable feature fields. It can be implemented as an equivari- steerable CNN, with recursively applying two alternating (equivariant) layers:
223
224
Q% R (6) + x [Pg Vol (+); Vs+1(8) max (%))), 8 E Z
(4)
225
where k [K] indexes iteration, Vk. 2i R are steerable feature fields over Z² output by equiv- ariant layers, Pg is learned kernel in neural network, and +, x are element-wise operations.
228
Pipeline. We follow the pipeline in VIN [17]. The commutative diagram for the full pipeline is shown in Figure 3. The path planning task is given by m x m spatial binary obstacle occupancy map and one-hot goal map, represented
228
a feature field M : z² + {0,1) 1}2. For the iterative process 0% + V& + Qi+ the reward field RM is predicted from map M (by a 1 x convolution layer) and the value field Vo is initialized as zeros. The network output is (logits of) planned actions A 2² RIAI, predicted from the final Q-value field QK (by
229
230
231
for all locations represented another x convolution layer). The number of iterations K and the convolutional kernel size F m is kept consistent.
233
of PΦ are set based map size M, and the spatial dimension
235
Building Symmetrie Value Iteration Networks. Given the pipeline of VIN fully on steerable feature fields, we are ready to build equivariant version with E(2)-steerable CNNs [16]. The idea to replace every Conv2d with steerable convolution layer between steerable feature fields, and associate the fields with proper fiber representations p(h).
237
239
VINs use ordinary CNNs and can choose the size of intermediate feature maps. The design choices in steerable CNNs is the feature fields and fiber representations (or type) for every layer [14, 16). The main difference" in steerable CNNs is that we also need to tell the network how transform every feature field, by specifying fiber representations, as shown in Figure 3.
240
241
243
Specification of input map and output action. We first specify fiber representations for the input and output field of the network: map M and action A. For input occupancy map and goal M 2 {0, 1]22, it does not D4 to on the channels, so we use two copies of trivial representations
244
245
PM Paiv ① Puiv- For action, the final action output A Z22 RRAA is for logits of four actions
246
247
A = (north, west, south, east) for every location. If we use H = C2, it naturally acts on
the four actions (ordered () by cyclically 3 permuting the E channels. However, since the DA group has 8 elements, we need quotient representation, see [16] and Appendix F.
248
249
Specification of intermediate fields: value and reward. Then, for the intermediate feature fields: V-lluuss Qk. state value Vi, and reward Ry, we are free to choose fiber representations, as well as the width (number of copies). For example, if we want copies of regular representation of D4, the feature field has 2 x 8 = 16 channels and the stacked representation is 16 x 16 (by direct-sum).
251
253
For the Q-value field (s), we use representation PQ and its size as Cq- We need at least CA 2 [A] channels for all actions of Q(s, a) as in VIN and GPPN, then stacked together and denoted as
255
Qk A + a Qi with dimension Qk z R CA Therefore, the representation is direct-sum
256
Technically, it also includes values or actions for obstacles, since the network needs to learn to approximate the reward RM As) with enough small reward and avoid obstacles.

---

+ PQ for CA copies. The reward is implemented similarly as Ra & D. R and must have same dimension and representation add element- demely-wisely. For state value field, we denote the choose as fiber representation as PV and its size Cv. It has size V& 2 - BCv Thus, the steerable kernel is matrix-valued with dimension Po : 2² + C(Cg.Ca)xCv, In practice, we found using regular representations for all three works the best. It can be viewed as "augmented" state and is related to group convolution, detailed in Appendix F.
257
258
280
261
262
Other operations. We now visit the remained (equivariant) operations. (1) The max operation in
Qk + V6+1. While we have showed the max operation in ( max, (x) is equivariant in Theorem 4.3, we need to apply max(-pooling) for all actions along the "representation channel" from stacked representations CA * Ce to one Co- More details are in Appendix F.5. (2) The final output layer QK + A. After the final iteration, the Q-value field Qx is fed into the policy layer
264
265
266
267
with 1 1 convolution to convert the action logit field 2 RIA
268
269
Extended method: Symmetric GPPN. Gated path planning network (GPPN [36]) proposes to use LSTM to alleviate the issue of unstable gradient in VINs. Although it does not strictly follow value iteration, it still follows the spirit of steerable planning. Thus, we first obtained a fully convolutional variant of GPPN from [Redacted for anonymous review], called ConvGPPN. It replaces the MI. in the original LSTM cell with convolutional layers, and then replaces convolutions with equivariant steerable convolutions, resulting in a fully equivariant SymGPPN. See Appendix F.3 for details.
271
272
273
275
Extended tasks: planning on learned maps with mapper networks. We consider two planning tasks on 2D grids: 2D navigation and 2-DOF manipulation. To demonstrate the ability of handling symmetry differentiable planning, we consider more complicated state space input: visual nav-
276
277
igation and workspace manipulation, and discuss how to use mapper networks to convert the state input and use end-to-end learned maps, as [36, 37J. See Appendix F.2 for details.
278
279
Experiments
280
281
We experiment VIN, GPPN and our SymPlan methods on four path planning tasks, including using given learned maps. The additional experiments and ablation studies are in Appendix G.
282
283
inviromennss and datasets. We demonstrate the idea in two major robotics tasks: navigation and manipulation. We focus the 2D regular grid setting for path planning, as adopted in prior work [17, 36. 37J. For each task, we consider using either given (2D navigation and 2-DOF configuration- space manipulation) or learned maps (visual navigation and 2-DOF workspace manipulation). In the latter case, the planner needs to jointly learn a mapper that converts egocentric panoramic images (visual navigation) workspace states (workspace manipulation) into plannable loss, as in [36, 37]. In both cases, we randomly generate training, validation and test data of 10K/2K/2 K/2K maps for all map sizes, demonstrate data efficiency and generalization ability of symmetrie planning. Note that the test maps are unlikely to be symmetric to the training maps by any transformation from the symmetry groups G. For all environments, the planning domain is the 2D) regular grid S f=
285
286
287
289
280
281
282
293
and the action space is to move in 4 C directions A (north, west, south, east).
planner networks. We compare five planner methods, where two are SymPlan version of their non-equivariant counterparts. Our equivariant implementation is based on Value Iteration Networks (VIN, [17]) and Gated Path Planning Networks (GPPN, [36]). We implement the equivariant version of VIN, named SymVIN. For GPPN, we first obtained a fully convolutional version, named ConyGPPN [Redacted for anonymous review], and furthermore SymGPPN with steerable CNNs. All methods use (equivariant) convolutions with circular padding in planning in configuration spaces for the manipulation tasks, except GPPN that is not fully convolutional. Chaplot et al. [37] propose SPT based on Transformers, while integrating symmetry to Transformers beyond steerable convolutions, thus we do not consider it but still adopt some useful setup.
Methods:
295
297
299
300
301
302
303
Training and evaluation. We report successful rate and its training curves over 3 seeds for each setup. The training process of the given map setup follows [17, 36), where we train 30 epochs with batch size 32, and use kernel size F = 3 by default. The gradient clip threshold is set to 5. The
305
'Note that the MDP action space A needs to be compatible with the group action G x A + A. Since the E2CNN package [16] uses counterclockwise rotations O as generators for rotation groups C.,, the action space needs to be counterclockwise O
7

---

Figure 4: (Left) A visual navigation environment rendered from randomly generated 7 x 7 maze (Middle), where the hover is the visualization of four views at position (5.3). 3). (Right) 2-joint manipulation task in workspace (topdown) and configuration space (2 DOFs) in 18 x 18 resolution.
1.0
1.8
0.8
model
VIN
0.4-
model
0.6-
SymVIN
SymMMh
0.4
GPP
SPm
0.2
ConvGINN
SymGSPP
0.9
Epocrs
Epocts
Figure 5: Training curves on (Left) 2D navigation with 10K of 15 x 15 maps and on (Right) 2DoFs manipulation with 10K of 18 18 maps in configuration space. Faded areas indicate standard error.
308
default batch size is 32, while we need to reduce for some GPPN variants, since LSTM consumes much more memory.
307
6.1
Planning given maps
Environmental setup. In the 2D navigation task, the map and goal are randomly generated, where the map size is (15, 28, 50}. In 2-DOF manipulation in configuration space, we adopt the setting in [37] and train networks to take as input of configuration space, represented by two joints. We randomly generate to 5 obstacles in the manipulator workspace. Then the
310
312
of-eredmm (DOF) configuration space is constructed from workspace and discretized into 2D grid with sizes {18, 36). corresponding to bins of 20 and 10", respectively. All methods are trained using the same use regular representations for all layers, which has size Dal = 8. We keep the same parameters for all methods, so all equivariant convolution layers with regular representations will have higher embedding sizes. Due to memory constraint,
314
network size, where for equivariant versions,
316
318
K = 30 iterations for 2D maze navigation, and K = 27 for manipulation. We use kernel sizes
319 F (3,5,5) for m (15, 28, 50) navigation, and F {3,5) for m (18, 36} manipulation.
320
Results. We show the averaged test results for both 2D navigation and C-space manipulation tasks on generalizing to unseen maps (Table 1) and the training curves for all methods (Figure 5). For VIN series, our VIN is much better than the vanilla VIN in terms of generalization and training performance in both environments, which learns much faster and achieves almost perfect asymptotic performance. As for GPPN, we found the fully convolutional variant ConyGPPN actually works bet- ter than the original one in [36], especially in learning speed. SymGPPN further boosts ConvGPPN and outperforms all other methods, including our SymVIN, One exception is GPPN learns poorly in C-saace manipulation. For GPPN, the added circular padding in the convolution encoder leads to gradient vanishing problem.
322
324
326
328
Additionally, we found using regular representations (for Da or C4) for state value V 2 - RCV
330
(and for Q-value) works better than trivial representations. This is counterintuitive since we expect the V value to be scalar 2² + R. One reason is that switching between regular (for Q) and trivial (for V) representation introduces unnecessary bottleneck. Depending on the choose of representa- tions, we implement different max-pooling, with details in Appendix F.5, We also empirically found using FC only in the final layer QK + A helps stabilize the training a bit. The ablation study on this and more are in Appendix G.
332
334
336
Remark. Two symmetric planners are both significantly better than their counterparts. Notably, we did not include any symmetric maps to the test data that symmetric planners would perform much better. There are several potential sources of advantages: (1) SymPlan allows parameter sharing across positions and maps and implicitly enables planning in reduced space: every (s, s')
338

---

Table 1: Averaged test success rate (%) for using 10K/2K/2K dataset for all four types of tasks.
Method (10K Data)
Navigation
Manipulation
15 x 15
28 28
50 x 50
Visual
18 18
36 x 36
Workspace
VIN
66,97
67.57 98.14
57.92
50.83
77.82 99.98
84.32
80,44
SymVIN
98.99
86.20
95.50
99.36
91.10
GPPN
96.36
95.77
91.84
93.13
2.62
1688
3.67
ConvGPPN
99.75
99.09
97.21
98.55
99.98
99.95
89.88
SymGPPN 99.98
99.86
99.49
99.78
100.00
99.99
90.50
seamlessly generalizes to (g.s.g.a,g..
g.a. g.a.g.s')) g.s") for any g E G, (2) thus it uses training data more efficiently, (3) it reduces the space of hypothesis class and facilitate generalization to unseen maps.
341
342
6.2
Planning on learned maps: simultaneously planning and mapping
343
Environmental setup. For visual navigation, we randomly generate maps using the same strategy as before, and then render four egocentric panoramic views for each location from produced 3D environments with Gym-MiniWorld [57], since allows to generate 3D mazes with any layout. For m m maps, all egocentric views for a map is represented by m m X 4 RGB images. For workspace manipulation, we randomly generate 0 to obstacles in workspace as before. We use a mapper network convert the 96 x 96 workspace (image of obstacles) to the x m 2 degree-of- freedom (DOF) configuration space (2D occupancy grid). In both environments, the setup is similar to Section 6. while we only use m = 15 maps but longer 100 epochs for visual navigation and 18 maps still with 30 epochs for workspace manipulation.
345
347
349
351 m-
352
Methods: mapper networks and setup. For visual navigation, we follow the mapper network setup in [36]. A mapper network converts every image into a 256-dimensional embedding m x m x x 1. For workspace manipulation, we use U-net [58]
353
4 x 256 and then predicts map layout
with residual-comnection [59] as a mapper. See Section G for details.
355
356
The results are also shown in Table 1, denoted as Visual (navigation, 15 x 15) and Workspace (manipulation, 18 x 18). In visual navigation, the trends are similar to 2D case: two symmetric planners both train much faster. Besides vanilla VIN, all approaches finally converge to near-optimal successful rate (around 95%), while the validation and test results show large gaps. SymGPPN has almost no generalization gap, while VIN does not generalize well to new 3D visual navigation environments. Our SymVIN improves test successful rate from less than 50% to 90% and is comparable with GPPN. Since the input is raw images and mapper is used to learn end-to- end, it potentially causes one major source of generalization gap for some approaches. In workspace manipulation, the results are also analogous to C-space, while ours advantages over baselines are smaller. our inspection, we found the mapper network is the bottleneck, since the mapping for obstacles from workspace to C-space is nontrivial to learn
Results.
357
358
359
361
363
365
367
Remark. The SymPlan models demonstrate end-to-end planning and learning ability, potentially enabling further applications to other tasks as a differentiable component for planning. The addi- tional results and ablation studies are provided in Appendix G.
369
Discussion
370
371
In this work, we study the symmetry in 2D path planning problem, and build a framework using the theory of steerable CNNs to prove that value iteration path planning is actually form of steerable CNN (on 2D grids). Although we focus on z², we can generalize to path planning on higher-dimensional even continuous Euclidean spaces [54, 55], and use equivariant operations on steerable feature fields (such as steerable convolutions, pooling, and point-wise non- con-linearssies) from steerable CNNs. We practically show that the SymPlan algorithms exactly motivated by the theory provide great improvement. We hope the framework along with the design of practical algorithm
372
373
374
375
376
377
378
can enable new perspective to exploit the symmetry structure path planning problems. Although it still has some limitations, such as (1) that the action needs to be known as moving on the domain (2D), and (2) that it is not sampling- based and may struggle for high-dimensional problems, we
380
believe that it has potential extensions to more general formulation.

---

References
382
383
[1] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: an introduction. Adaptive computation and machine learning series. The MIT Press, Cambridge, Massachusetts, second edition edition, 2018. ISBN
384
385
386
[2]
Lihong Li, Thomas J. Walsh, and M. Littman. Towards a Unified Theory of State Abstraction for MDPs. In AI&M, 2006.
388
[3] Balaraman Ravindran and Andrew G Barto. An algebraic approach to abstraction in rein- forcement learning. PhD thesis, University of Massachusetts at Amherst, 2004.
389
[4] Maria Fox and Derek Long. Extending the exploitation of symmetries in planning. In In Proceedings of AIPS'02, pages 83-91, 2002.
381
382
[5] Maria Fox and Derek Long. The Detection and Exploitation of Symmetry in Planning Prob- lems. In In IJCAI, pages 956-961. Morgan Kaufmann, 1999.
394
[6] Nir Pochter, Aviv Zohar, and Jeffrey S. Rosenschein. Exploiting Problem Symmetries in State- Based Planners. In Twenty-Fifth AAAI Conference on Artificial Intelligence, August 2011. URL attpps//wwet aaai. org/ ocs/ index. php/ Ttt////AA///ppperiee//////////
395
396
397
[7] Martin Zinkevich and Tucker Balch. Symmetry in Markov decision processes and its impli- cations for single agent and multi agent learning. In In Proceedings of the 18th International Conference on Machine Learning, pages 632-640. Morgan Kaufmann, 2001.
399
[8] Shravan Matthur Narayanamurthy and Balaraman Ravindran. On the hardness of finding symmetries in Markov decision processes. In Proceedings of the 25th international confer- ence on Machine learning ICMI. '08, pages 688-695, Helsinki, Finland, 2008. ACM Press. ISBN 978-1 778-1-60558-200-4. doi: 10/bkswe2. URL htttp://portall acm org/ org/citation.
400
402
404
cfm 1390243.
405
[9]
Elise van der Pol, Daniel E. Worrall, Herke van Hoof, Frans A. Oliehock, and Max Welling, MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning. arXiv:2006. 16908 [cs, stat], June 2020. URLhttp://arzt orgo www.a///..... 16908. arXiv: 2006.16908.
408
407
408
409
[10] Dian Wang, Robin Walters, and Robert Platt. Learning. September 2021. URL https:://operrevieee..o http://www...ttoooooor/e////nooe
Reinforcement
[11] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learn- ing: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104. 13478, 2021.
412
[12] Taco Cohen, Mario Geiger, and Maurice Weiler. A General Theory of Equivariant CNNs Homogeneous Spaces. arXiv: 1ete/8111.000 1.02017 [es, stat), January 2020. URL. http://arxiv..
414
415
/////////1111..011111// arXiv: 1811.02017.
[13] Risi Kondor and Shubhendu Trivedi. On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups. arXiv:1802.000 100.00360 [cs, stat], November 2018. URL http: // /arxiv..vvv org/ abs//802..0l 03690. arXiv: 1802.03690.
417
419
[14] Taco S. Cohen and Max Welling. Steerable CNNs. November 2016. URL https:
420
//openreview wtt.://ooorr/dddd11/000555
[15] Taco S. Cohen and Max Welling. Group Equivariant Convolutional Networks.
422
Xrvv 07577 [cs, stat], June 2016. URL BtLhtt////arxie http::///axxxv.o//a///////0// httgaabb//102.077777777 075766 1602.07576.
arXiv:
[16] Maurice Weiler and Gabriele Cesa.
General SE(2)S-Equivarian) Steerable CNNs.
425
19Xiv:191 191119821111 [cs, eess), April 2021. URL http: wttp://wwww/////.1....... 08251.
arXiv: 1.02251..
10

---

427
[17] Aviv Tamar. Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value Iteration Net- works. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial In- relligence, pages 4949-4953 Melbourne, Australia, August 2017. International Joint Confer- ences on Artificial Intelligence Organization. ISBN 978-0-9992411--033 doi: 10/ggjfst. URL, attps://wwe 1j jaci....... org/ httpp//ceedng//200070000000
428
430
432
[18]
Carmel Domshlak, Michael Katz, and Alexander Shleyfman. Enhanced Symmetry Breaking in Cost-Optimal Planning Forward Search. page 5.
434
[19] Alexander Shleyfman. Symmetry Breaking: Satisficing Planning and Landmark Heuristics.
435
page 5.
436
[20] Alexander Shleyfman, Michael Katz, Malte Helmert, Silvan Sievers, and Martin Wehrle. Heuristics and Symmetries in Classical Planning. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1), March 2015. ISSN 2374-3468. URL https: //ojs./jj/...//// aaai. index.
437
439
Number: 1.
[21] Silvan Sievers, Martin Wehrle, Malte Helmert, Alexander Shleyfman, and Michael Katz, Fac- tored Symmetries for Merge-and-Shrink Abstractions. page 8.
441
442
[22] Martin Wehrle, Malte Helmert, Alexander Shleyfman, and Michael Katz. Integrating Partial Order Reduction and Symmetry Elimination for Cost-Optimal Classical Planning, page 7.
443
444
[23] Mohammad Abdulaziz, Michael Norrish, and Charles Gretton. Exploiting Symmetries by Planning for a Descriptive Quotient. page 8.
446
[24]
Silvan Sievers, Martin Wehrle, Malte Helmert, and Michael Katz. An Empirical Case Study on Symmetry Handling in Cost-Optimal Planning as Heuristic Search. In Steffen Hölldobler, Rafael Peñaloza, and Sehastian Rudolph, editors, K/ 2015: Advances in Artificial Intelligence, volume 9324, pages 166-180. Springer International Publishing, Cham, 2015. ISBN 978- 3-319-24488-4 978-3-319-24489-1. doi: 01-10007/97-33-11-244493 1_33. URL http:// //
448
450
link. spr inger.e.... www.co//////// wwwm/1..100///78...0 --0000077777-----9---- 244-9- 1113. Computer Science.
Series Title: Lecture Notes in
452
453
[25]
Silvan Sievers. Structural Symmetries of the Lifted Representation of Classical Planning Tasks.
page 8.
455
[26] Dominik Winterer, Martin Wehrle, and Michael Katz. Structural Symmetries for Fully OB- servable Nondeterministie Planning. page 7.
457
[27] Gabriele Röger, Silvan Sievers, and Michael Katz. Symmetry-Based Task Reduction for
458
Relaxed Reachability Analysis. In Twenty-Eighuh International Conference on Automated Planning and Scheduling, June 2018. URL https: / //waaiaaaaa/aaa///aa/aaaa org/ocs/////ooooooooooooooooo wwcs/index.../eee indix..pp////iiiiie.i.mm.p....tt ICAPS/
459
480
SaAPS18/papprrvies//rr 1177//72...
(28)
Silvan Sievers, Gabriele Röger, Martin Wehrle, and Michael Katz. Theoretical Foundations for Structural Symmetries of Lifted PDDL Tasks. Proceedings of the International Conference on Automated Planning and Scheduling, 29:446 454, 2019. ISSN 2334-0843. URL https: //ojs. //ajs.aaii. org/ index. wttp:Icaa//attiilleeettteeeeeeee
462
464
465
[29]
Daniel Fižer, Álvaro Torralba, and Alexander Shleyfman. Operator Mutexes and Symmetries for Simplifying Planning Tasks. Proceedings of the AAAI Conference on Artificial Intelligence,
467
33(01):7586-755 7593, July 2019. ISSN 2374-3466 44668. doi: 10.1609 w33iaai.ii.3...... 33017586. URL
https://ojs.. aaai org/ indexidxxxx.. php/
4751.
Number: 01.
489
[30] N. Ferns. P. Panangaden, and Doina Precup. Metrics for Finite Markov Decision Processes. In AAAI, 2004.
470
[31] Elise van der Pol, Daniel Worrall, Herke van Hoof, Frans Oliehoek, and Max Welling. Mdp homomorphic networks: Group symmetries in reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
472

---

474
[32] Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan Willem van de Meent, and Robin Walters. Learning Symmetric Embeddings for Equivariant World Models, arXiv:2204.11377 [cs], April arXiv: 2204.1137)
475
476
2022. URL /ttttp://ariiv org/ /2044. 11371.
477
[33] Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric Deep Learn- ing: Grids, Groups, Graphs, Geodesies, and Gauges. arXiv:2104. 13478 [cs, stat], April 2021. URL
479
org/ wtrggabs/21044 13478. arXiv: 2104.13478.
480
[34] Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Picter Abbeel. Value iteration net- works. arXiv preprint arXiv: 1602i 028677 2016.
481
482
[35] Sufeng Niu, Siheng Chen, Hanyu Guo, Colin Targonski, Melissa C. Smith, and Jelena Ko- vačević. Generalized Value Iteration Networks: Life Beyond Lattices. urXiv:1706.022. 02416 [cs), October 2017. URL http: / www.aawwwwaaaaaaaaa...ooowoeeeee absa /106 02416, arXiv: 1706.02416.
484
485
[36] Lisa Lee, Emilio Parisotto, Devendra Singh Chaplot, Eric Xing, and Ruslan Salakhutdi- nov. Gated Path Planning Networks. arXiv: 06408 [cs, stat], June 2018. URL
486
487
http://arxiv. wwwp:/oogg/bb////eeeee 06408. arXiv: 1806.06408.
[37] Devendra Singh Chaplot, Deepak Pathak, and Jitendra Malik. Differentiable Spatial Planning using Transformers. arXiv:21 [cs). December 2021. URL http://arxizi org abs/2112. 01010.
489
arXiv: 21 12.00101..
491
[38] Andreea Deac, Petar Veličković, Ognjen Milinković, Pierre-Luc Bacon, Jian Tang, and Mladen Nikolić. Neural Algorithmic Reasoners Implicit Planners. October 2021. URL https:
493
//arxiv.org/abbe
494
[39] Junhyuk Oh, Satinder Singh, and Honglak Lee. Value Prediction Network. 1xXiv:1777.03997 [cs), November 2017. URL http:://axxxi org/ absaabss/////// 03497. arXiv: 1707.03497.
495
496
[40] Peter Karkus, David Hsu, and Wee Sun Lee. QMDP-Net: Deep Learning for Planning under Partial Observability. arXiv:1703.0660 066966662/ [cs, stat], November 2017. URL org/abs/1705 06692.
498
arXiv: 1703.06692.
499
[41] Théophane Weber, Sébastien Racanière, David P. Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomènech Badia, Oriol Vinyals, Nicolas Heess, Yu- jia Li, Razvan Pascanu, Peter Battaglia, Demis Hassabis, David Silver, and Daan Wierstra. Imagination- imagination-Augmennee Augmented Agents for Deep Reinforcement Learning. arXiv:1707.066: [cs, stat], February 2018. URL http: httt:::/areee //arxiva..vv//// org/ aat//b///11000 06203.
500
501
502
503
arXiv: 7707.06203.
[42] Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine. and Chelsea Finn. Universal Planning Networks. arXiv:1804. 100.0064400 [cs, stat], April 2018. URL httt:///rxiivv org/ abs/1804. 18044.0044.......1
505
arXiv: 1804.00645.
507
[43]
Julian Schrittwieser, Joannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering Atari, Go, Chess and Shogi by Planning with Learned Model. arXiv: 1911. 1.22665866// [cs, stat], November 2019. URL http:// /tap::///axxvv.eeeeeeeeeeeeeea www.ii.or//abb......oo.o abs/191s111111 08265. arXiv: 191 1. 108265.
508
509
510
511
512
[44] Brandon Amos, Ivan Dario Jimenez Rodriguez, Jacob Sacks, Byron Boots, and J. Zico Kolter. Differentiable MPC for End-to-end Plamning and Control. arXiv:1810. 1..0..... 13400 [cs, math stat), October 2019. URL http: www.aa.....a.a...aaaaa.aa. attpaa/s/....eeeaaa////aaaaaaaaa 18100 13400. arXiv: 1810.13400.
514
515
[45] Tingwu Wang and Jimmy Ba. Exploring Model-based Planning with Policy Networks. June 2019. URL https: http:://axxvvvoo/e aaaaaaaaa/900 08649v1.
516
517
[46] Arthur Guez, Mehdi Mirza, Karol Gregor, Rishabh Kabra, Sébastien Racanière, Théophane Weber, David Raposo, Adam Santoro, Laurent Orseau, Tom Eccles, Greg Wayne, David Silver, and Timothy Lillierap. An investigation of model-free planning. arXiv: 1101199035500 (cs, stat),
519
May 2019. URL http: // /arxiv. org/ /900 03559.
arXiv: 1901.03559,
12

---

521
[47] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to Control: [cs], March 2020. URL http:
522
Learning Behaviors by Latent Imagination.
523
//arxiv. .rgg aaaaaaa//999 /912. 1912.01603. 01603.
arXiv: 1912.01603.
[48] Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal Difference Models: Model-Free Deep RL for Model- Based Control. arXiv: /00ivv8802.99900 09081 [cs], February 2018. URL
625
http://arxiv.. org/ abs/ 1802. 09081. arXiv: 1802.09081.
527
[49] Ignasi Clavera, Violet Fu, and Pieter Abbeel. Model-Augmented Actor-Critic: Backpropagat- ing through Paths. urXiv:2005.088 08068 (cs, stat), May 2020. URL
528
org/abs/
529
2005. 08068. arXiv: 2005.08068.
530
[50] Christopher Grimm, André Barreto, Satinder Singh, and David Silver. The Value Equivalence Principle for Model-Based Reinforcement Learning, arXiv:201 1.03506 [cs], November 2020. URL http: /ttp://aaxvvvvoo http://aaiii..oraa/////o..../ httpg//ab///11..0///2/
532
arXiv: 1.05506
539
[51] Christopher Grimm, André Barreto, Gregory Farquhar, David Silver, and Satinder Singh. Proper Value Equivalence. arXiv:2106.003 [cs), December 2021. URL http://arxiv. org/ Ttrg/bbs/1100 10316.
535
arXiv: 2106.10316
536
[52] Clement Gehring, Kenji Kawaguchi. Jiaoyang Huang, and Leslie Pack Kaelbling, Understand- ing End-to-End Model-Based Reinforcement Learning Methods as Implicit Parameterization. May 2021. URL https://openrevieee
537
538
[53] Steven M. Valale Planning Algorithms. Cambridge University Press, May 2006. ISBN 978-1-139-45517-6.
540
541
[54] Maurice Weiler, M. Geiger, M. Welling, Wouter Boomsma, and Taco Cohen. 3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data. In NeurIPS, 2018.
543
[55] Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik Bekkers, and Max Welling. Geometric and Physical Quantities Improve E(3) Equivariant Message Passing. 10000:2110.02905 [cs, stat], December 2021. URL http:///axiyi
545
arXiv: 2110.02905.
[56] T. S. Cohen. Equivariant convolutional networks. 2021. URL https://dare. nl/ search searchiidaat.fiorr000011eee htt.://err0000701444----------30 -094-4300--0 a5d8- 37d03d8d10e6.
548
549
[57] Maxime Chevalier-Boisverr. Miniworld: Minimalistic 3d environment for rl & robotics re- search. https://gitie cww wwwom//aaiinee//eeoooommooooaeee wtwp///wwmmmmiddddmmmm.mmmmm 2018.
550
551
[58] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. CoRR, Abs//1505.0459,, 2015. URL http://rxxivoorr
553
abs/1505. 044997775.00....
554
[59] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03388 2015. URL http://arxiv... .rg/
Checklist
557
1. For all authors...
558
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
559
contributions and scope? [Yes]
560
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
562
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
583
964
2. If you are including theoretical results...
565
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Briefly in Section 4, and in full in the supplementary material.
566
13

---

587
(b) Did you include complete proofs of all theoretical results? [Yes] See supplementary material.
588
569
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [No] The code and data will be cleaned and released prior to final publication.
571
573
(b) Did you specify all the training details (e.g., data splits, hyperparameters. how they were chosen)? [Yes] Briefly in Section 6, and in full in the supplementary material. (c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] Briefly in Section 6, and in full in the supplementary material.
574
576
577
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplementary material.
579
580
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
582
(b) Did you mention the license of the assets? [No]
583
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
585
(d) Did you discuss whether and how consent was obtained from people whose data you'l re using/curating? [N/A]
586
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
588
509
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
581
592
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
593
594
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "s-zxjhwEIAI", "reviewer": "Reviewer_qXsg", "summary": "The paper proposes using steerable convolutions in the value iteration networks framework to incorporate equivariance under rotations and reflections.", "questions": "### Questions\n\n* Does the method extend to rotations that are not $90^\\circ$?\n\n* Do you think your method will bring improvements over data augmentation techniques that could extend the data set with rotated/reflected versions of the normal environments?\n", "limitations": "These points are addressed well.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n* The proposed approach brings quite a bit of improvement over a regular VIN.\n* Equivariance under rotation and reflection are valuable inductive biases.\n\n### Weaknesses\n\n* I believe the paper has an accessibility issue. The main audience for the paper will be researchers interested in VINs. I would expect these to generally not be too familiar with steerible CNNs and the theory that goes into them. Yet, the paper is quite heavy with mathematical notation and terminology from this area. I understand that the authors need to use the math that goes into steerible CNNs to be able to explain their ideas. Though at the moment specificity is getting in the way of clarity. Sections 4 and 5 currently contain too much jargon that is spread throughout the text in a way that makes it genuinely hard to follow the narrative through-line. I would like to give some passages as examples, just to make it clear, what exactly I mean: lines 104-115, 123-128, 171-175, 194-198, 243-249. I think it is in the best interest of the authors to rethink the presentation of sections 4 and 5, and present their ideas in a way that requires a minimum of knowledge about steerible CNNs. Otherwise, you are limiting your reach by creating a hurdle of mathematical prerequisites. Of course, the reader needs to eventually familiarize themselves with the math to truly understand the paper, but the first time reader shouldn't be completely lost either. Again, most people who read your paper won't be familiar with this math, and currently I find it unlikely they will have an easy time.\n* The gains in performance over ConvGPPN are somewhat marginal. This would be less of an issue if it weren't for my previous point. If I was a researcher or practitioner interested in using VINs, I believe Table 1 might convince me to use ConvGPPN and accept a slight drop in performance in exchange for the relative algorithmic and theoretical simplicity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PoRrJJt9VjR", "reviewer": "Reviewer_duzU", "summary": "This paper presents an approach to leverage problem-domain symmetry for planning problems defined over small 2D lattices. The key idea is to extend value iteration networks by using steerable convolutions to exploit symmetric structure (e.g., translational/rotational equivariance). Evaluation is carried out on three problem domains, and the presented approach performs better than VIN (value iteration networks) and GPPN (gated path planning networks).", "questions": "I would like to see **W1** and **W2** discussed", "limitations": "please see **W2** (while authors explicitly list a limitation as an avenue for future work, I believe that limits the scope of the current submission -- this is factored into my eventual score)", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n=========\n\n**S1** The paper is very well-written. Differentiable path planning is an area of great interest to the Neurips(/ICLR/ICML) community, as evidenced by several similar papers in the past (e.g., VIN, GPPN, [35-38]).\n\n**S2** The paper is largely self-contained. As a reader who was not an expert in symmetries and VIN, I appreciated the pointers to relevant readings. This section helped establish context for the technical contributions sections.\n\n**S3** At a technical level, the paper looks well-executed. The core hypothesis is sound -- the 2D gridworld domain (with the considered transition) exhibits symmetries that can be leveraged by learning-based planners to both perform well on the planning problem and also to generalize better to novel problem instances. Building on VINs, which reformulate value iteration as a series of convolutional operators, the proposed approach additionally leverages group convolutions (specifically steerable CNNs) to induce symmetry. The formulation is sound, and achieves better performance compared to variants that do not explicitly assume symmetric structure in the problem domains.\n\n---\n\nWeaknesses\n==========\n\n**W1** *Experiments*: Currently, analysis is only carried out on three 2D problem domains (if counting the C-space and workspace manipulation environments as distinct). In table 1, ConvGPPN seems to already achieve stellar performance on all tasks (89.88 success rate % on the workspace manipulation env, >97% on all other envs). The gains due to SymGPPN, while consistent, do not appear to be significant. This might well be because of the inherent task complexity (ConvGPPNs essentially seem to 'solve' the task); to better investigate the benefits of SymGPPN either larger problem instances or more complex problem domains are necessary.\n\n**W2** *Inherent limitations*: One potential reason for choosing low-dimensional (2D) problem domains and further, small problem sizes (largest problem involves 50 x 50 grid) is the (apparent) poor scalability of the approach. Is this because value iteration scales poorly with problem size, and that it is a core component of the approach? Would(n't) this also impact scalability to more complex problem domains (e.g., 3D environments involving larger action spaces, for example)? (While I appreciate the fact that the manuscript lists this as an avenue for future work, I also believe this shortcoming greatly limits the problems this technique can be applied to, thereby affecting the perceived impact of this work).\n\n---\n\nIn summary, I think this work tackles an exciting direction; but I am of the opinion that it needs more experimental analysis and some strategies to mitigate the inherent limitations it brings along.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BkMT4Vef6nz", "reviewer": "Reviewer_YbC2", "summary": "The paper identifies the symmetry in 2D path planning problem and proposes a framework for incoporating the symmetry into an end-to-end differentiable planning framework. This is done mainly by extending the CNN in value iteration network to steerable CNN. The paper demonstrates the advantages of such approaches in two 2D path planning domains.", "questions": "1. My understanding is that the symmetry group is manually defined instead of being learned through data. In such case, is there a simpler way of incoporating the symmetry? For example, one way could be applying each transformation in the group, run the planner and returns the plan with the lowest cost.\n\n2. Can the approach generalize to continuous actions?", "limitations": "Yes", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "To say it upfront, with a background in robotic manipulation and reinforcement learning, I do not have background in steerable feature fields and consider the paper outside my area of expertise. I find it difficult to understand the technical details of the paper. As such, I can only provide some high-level feedback to the paper and hope the AC and other reviewers can provide more detailed evaluations.\n\nHere are some suggestions on writing:\n1. The explaination of steerable CNN in Sec. 3 heavily references [14, 15, 16], making it hard for me to understand the concept from reading this paragraph alone. Some notations and jargons are also not explained. Figure 2 does not help much as multiple concepts are squeezed into one figure. \n2. The benefit of differentiable planning may not be well known in the robotics community. The two tasks done in the paper seems almost trivial and can be easily solved using other path planning techniques like RRT or A*. The paper could benefit from a better motivation on why working on differentiable planning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject