Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learning
Authors: Hyunwoo Ryu
Hong-in Lee
Jeong-Hoon Lee
Jongeun Choi
Venue: ICLR 2023 poster (2023)
Keywords: Robotics
Manipulation
Robotic manipulation
Equivariance
SE(3)
SO(3)
Energy-based model
Lie group
Representation theory
Equivariant robotics
Roto-translation equivariance
End-to-end
Point clouds
Graph neural networks
Imitation learning
Learning from demonstration
Sample efficient
Few shot
Unseen object
Category-level manipulation
MCMC
Langevin dynamics

Abstract: End-to-end learning for visual robotic manipulation is known to suffer from sample inefficiency, requiring large numbers of demonstrations. The spatial roto-translation equivariance, or the SE(3)-equivariance can be exploited to improve the sample efficiency for learning robotic manipulation. In this paper, we present SE(3)-equivariant models for visual robotic manipulation from point clouds that can be trained fully end-to-end. By utilizing the representation theory of the Lie group, we construct novel SE(3)-equivariant energy-based models that allow highly sample efficient end-to-end learning. We show that our models can learn from scratch without prior knowledge and yet are highly sample efficient (5~10 demonstrations are enough). Furthermore, we show that our models can generalize to tasks with (i) previously unseen target object poses, (ii) previously unseen target object instances of the category, and (iii) previously unseen visual distractors. We experiment with 6-DoF robotic manipulation tasks to validate our models' sample efficiency and generalizability. Codes are available at: https://github.com/tomato1mule/edf

Published as a conference paper at ICLR 2023
EQUIVARIANT DESCRIPTOR FIELDS: EQUIVARIANT ENERGY-BASED MODELS FOR END- TO-END VISUAL ROBOTIC MANIPULATION LEARNING
SE(3)-
Hyunwoo Ryu*
Hong-in Lee-
Jcong-Hoon Lee
Jongeun Choit.3
l DDpartment of Artificial Intelligence, Yonsei University
Samsung Research ftomato lmule, theorist17, jongeunchoi Serraanchoooeooooaooooaoooo @yonsei.... .ac..ac...... .r. jh_0921. 1eel amsung.... com.
School of Mechanical Engineering, Yonsei University
ABSTRACT
End-to-end learning for visual robotic manipulation known to suffer from sam- ple inefficiency, requiring large numbers of demonstrations. The spatial roto- translation equivariance, or the -3)-ivvaraaaraane can be exploited to improve the sample efficiency for learning robotic manipulation. this paper, we present (E33)-equivarian) models for visual robotic manipulation from point clouds that can be trained fully end-to-end. By utilizing the representation theory of the Lie group, we construct novel S FF((3))equivara) low highly sample efficient end-to-end learning. We show that our models can learn from scratch without prior knowledge and yet are highly sample efficient (5~110 demonstrations are enough). Furthermore, we show that our models can generalize tasks with (i) previously unseen target object poses, (ii) previously unseen target object instances of the category, and (iii) previously unseen vi- sual distractors. We experiment with 6-DoF robotic manipulation tasks vali- date our models' sample efficiency and generalizability. Codes are available at: https:: /ttts:://githute httpthttttuhgggo//////// www./oo.oomaaaee wwwell/eommeeeeeeeem
energy-based models that al-
INTRODUCTION
Learning robotic manipulation from scratch often involves learning from mistakes, making real- world applications highly impractical (Kalashnikov et al., 2018 Levine et al. 2016 Lee & Choi 2022). Learning from demonstration (LfD) methods (Ravichandar al. 2020 Argall et al. 2009) are advantageous because they do not involve trial and error, but expert demonstrations are often rare and expensive to collect. Therefore, auxiliary pipelines such as pose estimation (Zeng et al. 2017 Deng et al. 2020). segmentation (Simeonov et al., (202I). or pre-trained object representations (Flo- rence al. 2018 Kulkarni al. 2019) commonly used improve data efficiency. However, collecting sufficient data for training such pipelines often burdensome or unavailable in practice.
Recently, roto-translation equivariance has been explored for sample-efficien robotic manipulation learning. Transporter Networks (Zeng et al. 2020) achieve high sample efficiency in end-to-end vi- sual robotic manipulation learning by exploiting SE (eqiiviavve (planar roto-translation equiv- ariance). However, the efficiency of Transporter Networks is limited to planar tasks due to the lack of the full -qqiivrrria (spatial
contrast, Neural Descrip- trooosatnlltionn equivariance). tor Fields (NDFs) (Simconov et al. 2021) can achieve few-shot level sample efficiency in learning highly spatial tasks by exploiting the -qqiivaraaa Moreover, the trained NDFs can gener- alize to previously unseen object instances (in the same category) in unseen poses. However, unlike Transporter Networks, NDFs cannot be end-to-end trained from demonstrations. The neural net- works of NDFs need to be pre-trained with auxiliary self-supervised learning tasks and cannot be fine-tuned for the demonstrated tasks. Furthermore, NDFs can only be used for well well-segmented point cloud inputs and fixed placement targets. These limitations make it difficult to apply NDFs when 1) no public dataset is available for pre- pre-training on the specific target object category, 2) when well-segmented object point clouds cannot be expected, when 3) the placement target not fixed.

---

Published as a conference paper at ICLR 2023
Small Amount (-10) of Demonstrations
A) Unseen Poses
B) Unseen Instances
C) Unseen Distractors
Fully End-to-End Learning
Pick
SellSSupesttedd Learning
Objecct Segmentation
Object Pose Estimation
Place
Figure 1: Given few (5~10) demonstrations of mug bock-and-place task, EDFs can be trained fully end-to-end without requiring any pre-training object segmentation, or pose estimation pipelines. In addition, show that EDFs can generalize to A) unseen poses, B) unseen instances of the target object category, and C) the presence of unseen visual distractors.
To overcome such limitations, present Equivariant Descriptor Fields (EDFs), the first end-to-end trainable and SE(3) -equivariant visual robotic manipulation models. EDFs can be fully end-to-end trained solve highly spatial tasks from only few (5~10) demonstrations without requiring any pre-training, object keypoint annotation, or segmentation. EDFs can generalize to previously unseen target object instances in distracting objects and unseen placement poses (See Figure Our contributions are as follows:
(3)
poses NDFs. Furthermore, EDFs generalize to unseen
1. To enable end-to-end training, we reformulate the energy minimization problem of NDFs into probabilistic learning framework with energy-based models on the (E(3)-manifol
2. We generalize the invariant descriptors of NDFs into representation-theoree equivariant descriptors. Using equivariant descriptors significantly improves generalizability owing their orientational sensitivity.
3. We propose novel energy function and end-to-end trainable query point models to achieve the -equivvaaae regarding both the target object and placement target.
4. EDFs do not resort to non-local mechanisms to achieve the -buuueeequivaainneee This specific design enables our method to work well without object segmentation pipelines.
BACKGROUND AND RELATED WORKS
Equivariant Robotic Manipulation Equivariant models have emerged as a promising approach for robotic manipulation learning. with growing evidence indicating they can significantly improve both sample efficiency and generalizability (Wang Walters 2022 Wang et al.) (022). Trans- porter Networks and their variants (Zeng et 20200 (Seita al. 1021 are end-to-end models for visual robotic manipulation tasks that exploit the planar roto-translation equivariance, or the SE)))- equivariance for the sample efficiency. Equivariant Transporter Networks (ETNs) (Huang et al. 2022) exploit the representation theory of discrete rotation groups to further improve the sample effi ciency. However, the efficiency of (E())cqquivarian models is limited to planar tasks and cannot be extended to highly spatial tasks. Neural Descriptor Fields (NDFs) (Simeonov et al. 2021) overcome this limitation by leveraging the spatial roto-translation equivariance, the SE(3) -equivariance.
Energy-Based Models Energy-based models (EBMs) are probabilistic models that are derived from energy functions. EBMs are widely used for image and video generation (Zhu & Mumford
1998 Xie et al., 2016 2017: Du & Mordatch 2019). 3D geometry generation (Xic et al. 20 1b
2021a). internal learning (Zheng et al.] | [202]) and control (Xu al. 2022 Florence et al. 2022).
Due to the intractability of the integral in the denominator of EBMs, Markov chain Monte Carlo
(MCMC) methods are commonly used to estimate the gradient of the log-denominator to maximize
the log-likelihood (Hinton] 2002 Carreira-Perpinan & Hinton 2005). The Metropolis-Plasting al- gorithm (MH) (Hastings] 1970) and the Langevin dynamics (Langevin 1908 Welling Teh Teh] 201T) widely used MCMC methods for EBMs Euclidean spaces. However, typical Langevin dy- namics cannot be used for non-Euclidean manifolds such as the SE manifold. The Langevin dynamics the (3) group and general Lie groups are studied by Brockett (1997); Chirikjian
2

---

Published as a conference paper at ICLR 2023
2011;; Davidehack al. (2017). The Langevin dynamics and their convergence general Rie-
mannian manifold have been studied by Girolami & Calderhead 2011); Gatmiry & Vempala (022)) In this paper, propose SE(3) -equiivaaaai EBMs the SE manifold, which should be dis-
tinguished from (333))equivaaa EBMs on Fuclidean spaces (Jaini et al. 2021 Wu et al. 2021
Representation Theory of Lie Groups A representation D of a group G is map from G' to the space of linear operators acting on a vector space V that has the following property:
(igh) Vg, h G
Any representation of SO(3) group D (R) for R E SO(3) can be block block-diggonalized into the direct [222++1)(((+11) of degree 1 E {0, 1, 2.. J. which are sum of (real) Wigner D-matrices (R) E
orthogonal matrices (Aubert 2013). The (2l+1) dimensional vectors that are transformed by (R) are called rype-l (or spin-I) vectors. Type-O vectors are invariant to rotations such that D(R) = I. Type-1 vectors are the familiar 3-dimensional space vectors with D(R) R. Type-l vectors are identical to themselves when rotated by 0 = 2T /l.
type-l vector field f IR3
2 is -equivraqu if
(i(R) [['xTxXX VI (R. E SE((), X
(2)
where X' is some set equipped with group action and Tx Rx +v. Tensor Field Networks (TFNs) (Thomas et al., 2018) and $ (333))Trnnsforme (Fuchs et al. 2020) are used implement vector fields in this work. We provide details of these networks in Appendix G
S
3 PROBLEM FORMULATION
Let colored point cloud with M points given by X (i).,,.......... (xxM; CM)) E P where x; IR is the position, C; R3 is the color vector of the i-th point, and P is the set of all possible colored point clouds. The action of (3) on point clouds - SE(3) x P - P is then defined as T.X- (TTx1, c1), (Tx2, C2), ... (TxM, cM)) VT (R,
(3) Consider a problem where a robot has to learn to grasp and place objects in specific locations from human demonstrations. To solve this problem, the placement pose T SE(3) must be inferred such that the relative pose between the grasped object and placement target remains consistent with the demonstrations, regardless of changes in their postures. This can be achieved by requiring the bi- ri-equivvriance to T such that 1) it follows changes in the posture of the placement target, and 2) compensates for changes in the posture of the grasp (See Appendix C for more explanation). In contrast, uni-equivariant methods such as NDFs, which can only account for changes in one object, are not well-suited for problems where both postures may change.
SE(3)
Now let X be the point cloud of the scene (where the placement target belongs), and Y be the point cloud of the end-effector (with the grasped object) observed in the end-effector frame T E SE(3).
The formal definition of bi-equivariance is as follows:
Definition 1. differential probability distribution dP X. Y) on SE(3) conditioned by two point clouds X, Y E P is bi bi-qquivariant if for all Borel subsets Ω C SE(3),
J Jrea dP(TIX,Y) | frEss dP(T)S.x.Y)
S dP(TIX..-1o8) VS E (4)
where SQ = (ST E Q1) ΩS (TS E R.), and S-1 denotes the group inverse of Definition 2. A scalar function f SE(3) x P x P + R is bi-equivariant if
F(T(X,Y) = f(STSS X, = f(TS)X, .Y) VS E SE(3)
(5)
Proposition A probability distribution P Y)dT is bi-equivariant if dT is the bi-invariant volume form (See Appendix on the SE(3) manifold and P(T)X, Y) is bi-equivariant probabil- ity density function (PDF).
The proof of Proposition T can be found in Appendix F.I
The bi-equivariance condition of Defi- generalization of Huang et al. (2022) to non-commutative groups like
nition T can be viewed
SE(3) and a probabilistic generalization of Ganea et al. (2021) By Proposition I our goal boils
down constructing a bi-equivariant PDF X, Y) such that
X) P(ST X,Y) P(TS VS E SE(3)
3

---

Published as a conference paper at ICLR 2023
A) Global Equivariance
B) Local Equivariance
The grasp pose is equivariant to the whole scene
The grasp pose is equivariant only to the target object
Figure 2: A) The model is globally equivariant if the grasp pose is equivariant the transformations of the whole scene (the target object and background). B) The model is locally equivariant to the target object if the grasp pose is equivariant the localized transformations of the target object.
However, we want our policy to be not only globally equivariant as Equation 6 but also to be locally equivariant. That is, we want our models to equivariant only to the target object, the backgrounds. We illustrate the local equivariance and the global equivariance in Figure To achieve the local equivariance, the model should only rely on local mechanisms to achieve the equivariance. For example, Transporter Networks achieve translational equivariance by using convolutional neural networks, which are well-known for their locality (Battaglia et al., 20188] Goodfellow et al. 2016). On the other hand, NDFs (Simeonov et al. 2021) rely the centroid subtraction method to obtain translational equivariance, which is highly nonlocal. As a result, unlike Transporter Networks, NDFs cannot be used without object segmentation pipelines.
B1-EQUIVARIANT ENERGY BASED MODELS ON SE(3)
In this section, present EDFs and the corresponding bi-equivariant energy- models SE(3). We also provide practical implementations for the proposed models. We illustrate the overview our method in Figure
4.1 EQUIVARIANT DESCRIPTOR FIELD
We define the EDF (x direct sum of N vector fields
(xx|X)
sp (m) (x
(7)
where (0()) (x(X) IR3 x P - 21 +1 is an (E -equivvrra type-l,, vector field. Therefore, the EDF xx) transforms according to a rigid body transformation SE(3) as
(Txx/T . X)= (xR)p(x(XX VT (R, v) E SE(3)
(8)
where D(R) θ Di, (R) is the direct sum of the real Wigner D-Matrices of degree l, in the real basis. Therefore, D(R) an orthogonal representation the SO(3) group.
Note that NDFs (Simeonov al. 2021 only use type-O descriptors, which are invariant to rotations such that D(R) I. contrast, EDFs also use type-1 or higher descriptors, which are highly sensitive to rotations. As a result, NDFs require at least three non-collinear query points (and much more in practice) to represent the orientation of a rigid body, whereas EDFs require only one point.
4.2 EQUIVARIANT SABRGY-BASED MODEL ON SE(3)
Naively minimizing the energy function like Simeonov ously train the descriptors, as this would result in all the descriptors collapsing to zero (or some other constants). Therefore, we use the EBM approach for the end-to-end training of descriptors.
(022) cannot be used simultane-
An energy- based model on the SE(3) manifold conditioned by X, P can be defined as
cxp E Y)]
P(T[X,Y)
(9)
SS dT exp [ Y)]
4

---

Published as a conference paper at ICLR 2023
A) Input y
Query Points
Query Descriptors
Query-Key Aligument
Low Energy
P
poxi
P
I.
Query EDF
(
Type-1 Descriptoes
Type-O Descriptons
Query-Key Misalygament
B) Input X
Key Descriptors
(x
High Energy
Figure 3; A) Query points and query EDF are generated from the point cloud of the grasp. Query EDF values at the query points used as the query descriptors. We visualized three type-0 de- scriptors in colors (RGB) and type-1 descriptors as arrows. We only visualized type-1 descriptors in important locations. We did not visualize higher-type descriptors. B) The key descriptors generated from the point cloud of the scene. Cy The query descriptors are transformed and matched the key descriptors produce the energy of the pose. For simplicity, only visualized the query descriptor for single query point. Note that the query and key descriptors are better aligned in the low energy case than in the high energy case for both the type-O and type-I descriptors (The orange query points are near the orange region, and the black arrow is well aligned to the gray arrows).
Proposition 2. The EBM P(T)X,Y) in Equation 9 is bi-equivariant if the energy function [( Y) is bi-equivariant.
We prove Proposition 2 in Appendix
F2
We now propose the following energy function:
E(TIX,Y)
("xxxx())]<<((xxXX) D(R))(((Y))
(10)
where XX is the key EDF, (x()) is the query EDF, and p(x Y) is the query density. Note that T = (R. v). The query density is an S (3( non-negative scalar field such that
p(x|Y)= = [T . Y) VT SE(3)
(11)
Intuitively, the energy function in Equation TO can be thought as a query-key matching between the key EDF and the query EDF which analogous to (Zeng et al. 2020 Huang et al. 2022).
Proposition 3. The energy function ETT(X,Y) in Equation 10 is bi-equivariant.
prove Proposition 3 in Appendix
As result, the EBM in Equation
4 with the energy
function in Equation (10) is also bi-equivariant. Lastly, provide an important consequence of Equation (TT). whose proof can be found in Appendix[F.4
Proposition 4. Non-constant query densities that satisfy Equation 77 must be grasp-dependent.
4.3 IMPLEMENTATION
Our method consists of two models, viz. the pick-model and the place-model. The pick-model is a simplified version of the place-model. Therefore, we only demonstrate here the components of the place-model. The mode- is demonstrated in Appendix E We show in Appendix that the energy function used in [Smmeooov et al. (202)) is special case of our pick-model's energy function. For the following sections, we denote all the learnable parameters as O. Therefore, all the functions with as a subscript are understood as trainable models. We visualized the key EDF of trained mock-model in Figure
Query Density make the integral Equation 10 tractable, we model the query density as
weighted query points by taking the weighted sum of Dirac delta functions ((3) (x) = I[ 8(x,)
No
(x Σ we (q:: ((3) (x- (Y))
(12)
5

---

Published as a conference paper at ICLR 2023
B)
(xxX))
(x
Figure 4: The key EDF of trained pick-model is illustrated for the scenes with mug in A) upright pose and B) lying pose. Note that the colors (type-0 descriptors) are invariant to the rotation of the mug. On the other hand, the arrows (type-1 descriptors) are equivariant the rotation. We only visualized type-1 descriptors in important locations. Higher-type descriptors not visualized.
where 9::0 (Y): P R& is the i-th query point function and we (x(xxYY: R x + R the query weight field. These maps are ( such that
q:o We (Tx Y) wo(x({)
(T' o Y) = TQ:e(Y)
(13)
Proposition 5. The query density Y in Equation (2) is SE(3)-equivarian.
We prove Proposition in Appendix F.5
Note that as a consequence of Proposition 4 the grasp
dependence is inevitable for non-constant query point models like Equation (12). In this case, the integral in Equation 10) can be written in the following tractable summation form:
N2
XX = Σ Fo (T we (q:0 (Y)I (V))
(14)
ET w. q) = D(R)
(15)
In Appendix B we provide practical implementations of q)) and we (xlY) that are continuously parameterized and differentiable.
EDFs As was argued in Section only the local operations should be used in our models for the local equivariance. We use Tensor Field Networks (TFNs) (Thomas et al.) (018) for the last layer and (/t[)))Tranformerss (Fuchs et al., (220)) for the other layers. The convolution operations that are used in these networks are highly local when their radial functions (See Appendix G have short cutoff distances. We used simple radius clustering, which is locally (E33)-equivariant within the clustering radius, to make the point clouds into graphs. For computational efficiency, only the last layer is used to evaluate the field values at the query points. All the other layers' outputs only depend on the point cloud and not the query points. Therefore, during the MCMC steps, only the last layer (TFN) has to be recalculated, and the outputs of the other layers (ran))-Tranfoommrss be reused. We use the E3NN package (Geiger et al. 2022) to implement the equivariant layers.
5 SAMPLING AND TRAINING
For the sampling, we first run the Metropolis- Haatopooii-Hastings algorithm (MH) with IG distribution Nikooaaevv & Savyolov 1970; Savyolova, 1994 Leach al.) (022) for the orientation proposal and typical Gaussian distribution for the translation proposal. Next, we run the Langevin dynamics on the SE(3) manifold using the samples from MH as initial seeds. The Lie derivatives (Brockett) 1997 Chirikjian 2011) for the Langevin dynamics are calculated in inniermmioontaanlattionn parame- terization Davidchack et al. (2017) avoid singularity issues. We provide details in Appendix
For the training, we estimate the gradient of the log- lieelikld of Equation
9 at Tiarget as
N
1
(Target X,Y) Eo Ve log Po (Ttarget X, = -
VoEo(T,)XXY))
(16)
N
where T,, ~ P(T Y) is the n-th negative sample (Carreira- Perpinan & Hinton
2005 However, naively maximizing the log-likelihood is highly unstable. the query EDF and the key EDF

---

Published as a conference paper at ICLR 2023
initially very different at some important sites, the learning algorithm tends lower the query den- sity of these sites rather than make the two EDFs closer. Therefore, all the query points in essential locations (such as contact points) are being pushed away. As a result, the training diverges.
To avoid this instability, we propose using the following surrogate query model during the early stage of training. We first decompose the EBM P(T]X, Y) in Equation (14) into
P(TIX,Y) dw dQP(T X, w, Pw,,, Q
(17)
- E Y, wis q:)
exp
Y, Q,
(18)
SSE(3) dT exp
En E(T)X,, Y wi, w; 9:)
No
N₂
P(w, Q(Y = II P. (, 4 = II
6(w; w(q x - -
(19)
where Q = -q11 AN₂) and w (Wi,". WN,). We temporarily hide θ for brevity. Proposition The marginal EBM P Y) Equation [7 bi-equivariant if
P(w, Q P(w, SQIS.Y) vS E SE(3)
We now relax this deterministic query model into stochas- tic model by adding Gaussian noise the logits of the query weights 4 log w, follows.
We prove Proposition Appendix F.6
dl N(4 w(q: o), (q: qY)
P(w, Q II P, q YY
II
(20)
dw,
Now we propose the following surrogate query model
N.
H(w, QIX,Y,X) II q,
(21)
(xx,,,,)lX,YT)))) = (1 (il
if dmin (Tq;, <r (n) q.)) else
where 0g E and E R are hyperparameters and dmsi (x, X): R3 p R+ is the shortest Euclidean distance between x and the points X. We set a to be sufficiently small so that query points without neighboring points in X can be suppressed.
To train our models using the surrogate query model in Equation 21 we maximize the following variational lower bound (Kingma Welling 2013) instead of the marginal log- likellioooodd
(o Ew (log Po Po Q) DKL Ho(w QIX,Y, |Pow
[I QI))
(22)
Proposition 7. The variational lower bound (e Y) in Equation 22 is bi-equivariant.
We provide the proof of Proposition in Appendix F.7 The Kullback-I Leibler divergence term in
Equation (22 is provided in Appendix B.2 Once the query model has been sufficiently trained,
remove the surrogate query model and return to the maximum likelihood training in Equation (16).
EXPERIMENTAL RESULTS
We design the experiments to assess the generalization performance of our approach (EDFs) when the number of demonstrations is very limited. Similar to Simeonov et al. (2021). we evaluate the gen- eralization performance of models with a mug-hanging task and bowl/bottle In the mug- mug-hanging task, mug has be picked by its rim and then hung dle. In the bowl/bottle lork-and--lacee task. a bowl/bottle should be picked and placed on a tray. As opposed to Simeonov al. (202T), randomize the pose of the hanger and tray evaluate
tasks.
hanger by its han-

---

Published as a conference paper at ICLR 2023
A) Trained with demonstrations with upright only
B) Evaluated with unseen instances, & distracting objects
Figure 5: A) Only ten demonstrations with objects in upright poses are provided during the training. B) The models are evaluated with unseen object instances in unseen poses with unseen distractors.
Table 1: Pick-and-ples -lace success rates in various out-of-distriutton settings.
Mug Place
Bowl
Bottle
Pick
Total
Pick
Place
Total
Pick
Place
Total
Unseen Instances SE(3)-TNs (Zeng et al. 2020)
1.00
0.36
0,36
0.76
1.00
0.76
0.20
1.00
0.20
EDFs (Ours)
1.00
0.97
0.97
0.98
1.00
0.98
1.00
1.00
1.00
Unseen Poses
SE(3)-TNs (Zeng al. 2020)
0.00
N/A
0.00
0.00
N/A
0.00
0.00
N/A
0.00
EDFs (Ours)
1.00
1.00
1.00
1.00
1.00
1.00
0.95
1.00
0.95
Unseen Distracting Objects
SE(3)-TNs (Zeng et al. 2020
1.00
0.63
0.63
1.00
1.00
1.00
0.96
0.92
0.88
EDFs (Ours)
1.00
0.98
0.98
1.00
1.00
1.00
0.99
1.00
0.99
Unseen Instances, Arbitrary
Poses & Distracting Objects
SE(3)-TNs (Zeng et al. 2020)
0.25
0.04
0.01
0.09
1.00
0.09
0.26
0.88
0.23
EDFs (Ours)
1.00
0.95
0.95
0.95
1.00
0.95
0.95
1.00
0.95
with changing target placement poses. We use ten demonstrations upright poses generated by probabilistic oracles for the training. We then evaluate the success rate for (1) unseen instances, (2) unseen poses (lying poses), (3) unseen distracting objects, and (4) unseen instances arbitrary poses (50% lying, 50% upright but in arbitrary elevation) with unseen distracting objects. We illustrate the experimental setups in Figure 5 Details are provided in Appendix
For baselines, use -Transsrtt Networks (SE(3)-TNs) (Zeng et al. 2020) as the state-of-the-ar method for end-to-end vi- sual manipulation. However, for the mug hanging task, find that (E(3)-TNs entirely fail due to the multimodality of the demon- strations. Therefore, instead train (F(3))TNN using unimodal, low-variance demonstrations only for the mug-hanging task. For fair comparison, we provide the result for EDFs trained with the
Success Rate of Mug hanging Task
Success Rate
Unseen Instances
same demonstrations in Table 3 of Appendix The experimental results for (E(3)-TNs and EDI' are summarized in Table For EDFs, we provide the learning curves for the mug task in Figure 6
Unseen Poses
Unseen Distracting Objects
Disttrcctng Objects
Task Demonstrations 10
Note that we do not directly compare EDFs with NDFs because (1) NDFs require the target placement poses to be fixed, and (2) NDFs require object segmentation pipelines (Performance out object segmentation is provided in Appendix]].. Instead, we per- form an ablation study by using only type-O descriptors as NDFs. One may consider this the end-to-end trainable and bi-equivariant modification of NDFs (See Section 4.1 and Appendix
Figure 6:
The success rate
NDFs with-
of EDFs for the mug-hanging task with respect the num- ber of demonstrations.
E We con-
trol the ablated model's query point number such that the wall-clock inference time is similar slightly longer than EDFs, Note that the query point number serves a similar role to the batch size of inputs. Details the ablated model and EDFs are provided in Appendix H All the times were
8

---

Published as a conference paper at ICLR 2023
Success rate and inference time of the ablated model and EDFs. All the evaluations are done in the unseen instances, poses & distracting objects setting.
Table 2:
Mug
Bowl Place
Bottle Place
Descriptor Type
Pick
Place
Total
Pick
Total
Pick
Total
NDF-like (Type-O Only)
Inference Time
5.7s 0.84
8.6s
14.3s 0.65
6.1s
9.9s
16.0s 0.57
5.8s
17.38
23.0s
Success Rate
0.77
0.60
0.95
0.66
0.95
0.63
EDFs (Type-0~3)
Inference Time
5.1s
8.3s
13.4s
5.2s
10.4s
15.6s
5.2s
11.58
16.7s 0.95
Success Rate
1.00
0.95
0.95
0.95
1.00
0.95
0.95
1.00
measured using Intel 19- 12-12900k CPU (P-core only) with an Nvidia RTX3090 GPU. Experimental results for the ablated model with more query points can be found in Table of Appendix
Analysis As can be seen in Table 1 EDFs outperform SE(3)- TNs (Zng al., 2020) for all three tasks. It is obvious that EDFs generalize much better than SE(3)-TNS for unseen poses, EDFs are SE(3) -qquivarnan whereas SE(3)-TNs
only SEQu)eequivariant We provide qualitative example in Figure EDFss also generalize much better than (E(3)-TNs for unseen stances and/or unseen distracting objects. For example, (E(3)- TNs often fail correctly regress the z-axis of the grasp pose of an unseen instance when the object height differs a lot from the trained objects. In contrast, EDFs rarely fail under such height differences due to the -e3uivqaurai For the distracting objects, we presume that the reason why EDFs perform better is that the point cloud inputs provide better geometric information than the orthographic RGB-D inputs that $ (3)- take.
Figure 7: A) SE(3)-TNs fail pick the object in an unseen pose due to the lack of SE(3)- equivariance. B) Type-0 only descriptors fail to place the ob- ject in proper orientation due to the lack of orientational sen- sitivity.
The ablation study shows that using higher-type descriptors sig- nificantly increases generalization performance when the number of query points is highly limited due to computational constraints. As can be seen in Table 2 EDFs outperform the ablated model that only uses type-0 descriptors as NDFs. As was discussed Section 4.1 we presume that this is due to the orientational insensitivity of type-0 descriptors. As type-0 descriptors cannot repre- sent orientations alone, query points are crucial in representing orientations for the ablated model. In contrast, the higher-type descriptors of EDFs can represent the orientation without the help of query points. As a result, EDFs can maintain orientational accuracy in unseen situations in which low-quality query points expected. We illustrate the failure case of the ablated model in Figure
DISCUSSION AND CONCLUSION
There are several limitations to EDFs that should be resolved in future works. First, faster sampling methods are required for real-time manipulations. Cooperative learning (Xie et al. 2201 20 2021b 2022) and amortized sampling (Wang & Liu 2016 Xie al., 2021c) can be applied accelerate the MCMC sampling. In addition. EDFs are not intended for tasks with significant occlusions to the target object. Future work may also encompass 3D reconstruction methods. Lastly, EDFs cannot solve problems at the trajectory level, which is shared problem with NDFs. Future work should define the adequate equivariance condition for full trajectory-lever manipulation tasks.
To summarize, introduce EDFs and the corresponding energy energy-based models, which are SE(3)- equivariant end-to-end models for robotic manipulations. We propose novel bi-equivariant energy- based models, which provably allow highly sample efficient and generalizable learning. We show by experiment that our method is highly sample efficient and generalizable to unseen object instances, unseen object poses, and unseen distracting objects. Lastly, show by the ablation study that higher-degree equivariance (type1 or higher) is important for generalizability.

---

Published as a conference paper at ICLR 2023
Acknowledgement This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 201RIA223BB110262))) This work was partially supported by the Korea Institute of Science and Technology (KIST) intramural grants (2731570).
REFERENCES
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-453 2009.
G Aubert. An alternative to wigner d-matrices for rotating real spherical harmonics. AIP Advances, 3(6):062121, 2013.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanzhez- Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 01261, 2018.
Roger Brockett. Notes on stochastic processes on manifolds. In Systems and Control the Twenty- first Century, pp. 75-100. Springer, 1997.
Miguel A Carreira- PereiraaPerpinan and Geoffrey Hinton. On contrastive divergence learning. Interna- fional workshop artificial intelligence and statistics, pp. 33-40. PMLR, 2005.
Gregory S Chirikjian. Stochastic models, information theory, and Lie groups, volume 2: Analytic methods and modern applications, volume 2. Springer Science & Business Media, 2011.
Gregory S Chirikjian. Partial bi-invariance of se (3) metries, Journal of Computing and Information Science in Engineering, 15(1), 2015.
Ethan Chun, Yilun Du, Anthony Simeonov, Tomas Lozano-Peree and Leslie Kaelbling. Local ral descriptor fields: Locally conditioned object representations for manipulation. arXiv preprint arXiv:2302.03577. 2023.
Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybbll http:///yylllet.or... let.org 2016-2021.
Ruslan Davidchack, Thomas E Ouldridge, and Michael Tretyakov. Geometric integrator for langevin systems with quaternion- rotational degrees of freedom and hydrodynamic inter- actions. The Journal of chemical physics, 147(22):224103, 2017.
Xinke Deng, Yu Xiang, Arsalan Mousavian, Clemens Eppner, Timothy Bretl, and Dieter Fox. Self- supervised 6d object pose estimation for robot manipulation. In 2020 IEEE International Confer- ence on Robotics and Automation (ICRA), pp. 3665-367. IEEE, 2020.
Rosen Diankov. Automated Construction of Robotic Manipulation Programs. PhD the- sis, Camegii Mellon University, Robotics Institute, August 2010. URL. http://wwe
programiingvisionn ctt.:atooooooooaaeaeeee diankov thesis.pdff
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems, 32, 2019.
Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning, In Conference on Robot Learning. pp. 158-168. PMLR, 2022.
Peter R Florence, Lucas Manuelli, and Russ Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. arXiv preprint 106.0:1 2018.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. transforrr, 3d roto- translation equivariant attention networks. Advances in Neural Information Processing Systems, 33: 33:1970-1991, 2020.
10

---

Published as a conference paper at ICLR 2023
Octavian- Ganea, Xinyuan Huang, Charlotte Bunne, Yatao Bian, Regina Barzilay, Tommi Jaakkola, and Andreas Krause. Independent se equueurrari models for end-to-end rigid protein docking. arXiv preprint arX1v:2111.07786, 2021.
Caelan Reed
Garrett.
Pybullet planning.
httpe://ppyii.ong/proooes
pybullet -aa.nnetpplanngg///e 2018.
Khashayar Gatmiry and Santosh S Vempala. Convergence of the riemannian langevin algorithm. arXiv preprint arXiv:2204. 10818, 2022.
Mario Geiger, Tess Smidt, Alby M., Benjamin Kurt Miller, Wouter Boomsma, Bradley Dice. Kos- tiantyn Lapchevskyi, Maurice Weiler, Michal Tyszkiewicz, Simon Batzner, Dylan Madisetti, Martin Uhrin, Jes Frellsen, Nuri Jung, Sophia Sanborn, Mingjian Wen, Josh Rackers, Mar- Rød, and Michael Bailey. Fuclidean neural networks: e3nn, apr 2022. URL, https:
//doi. 10. 5281/ www.1/eeoozzooo.... 6459381
Mark Girolami and Ben Calderhead. Riemann manifold langevin and hamiltonian monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73(2):
123-214, 2011.
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
D Griffiths and Darrell Schroeter. Introduction to quanium mechanics. Cambridge university press, 2018
W. K. Hastings. Monte Carlo sampling methods using Markov chains and their applications. Biometrika, 57(1):97-109, 04 1970. ISSN 0006-3444. doi: 10.1003/biomec55...99 URL
https: / http://o...o/111.11.0.1 http://..mmm//77......
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-100 2002.
Haojie Huang. Dian Wang, Robin Walter, and Robert Platt. Equivariant transporter network. arXiv preprint X2X1:2202.094000.0 2022.
Priyank Jaini, Lars Holdijk, and Max Welling. Learning equivariant energy based mod- els with equivariant stein variational gradient descent. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 16727-16737. Curran Associates, Inc., 2021. URL https://procedings hewpeeeeeiing..eeri..........e.s wttp:////pppe/2//11eeee 7570555172111111222227011
pdf
Kalashnikov, A Irpan, P Pastor, Ibarz, A Herzog, Jang, D Quillen, Holly, M Kalakrish- nan, V Vanhoucke, al. Qt-opt: Scalable deep reinforcement learning for rision-based robotic manipulation (2018). arXiv preprint arXiv:1806.1023 2018
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.696 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 1312v 2013.
arXiv preprint
Tejas D Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. Advances in neural information processing systems, 32, 2019.
Paul Langevin. Sur théorie du mouvement brownien. Compt. Rendus, 146:530-533, 1908.
Adam Leach, Sebastian M Schmon, Matteo T Degiacomi, and Chris G Willcocks. Denoising diffu- sion probabilistic models on SO(3) for rotational alignment. In ICLR 2022 Workshop on Geomet- rical and Topological Representation Learning, 2022.
Jeong- Hoon Lee and Jongeun Choi. Hierarchical primitive composition: Simultaneous activation of skills with inconsistent action dimensions in multiple hierarchies. IEEE Robotics and Automation Letters, 7(3):7581-777 -75888 2022.
11

---

Published as a conference paper at ICLR 2023
Sergey Levine. Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo-
motor policies. The Journal of Machine Learning Research, 17(1):133-1173. 2016.
Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. arXiv preprint arXiv:2206. / 19900, 2022.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances neural information processing systems, 29, 2016.
Nicholas Metropolis, Arianna Rosenbluth, Marshall N Rosenbluth, Augusta Teller, and Edward Teller. Equation of state calculations by fast computing machines. The journal of chemical physics, 21(6):1087-1092 1953.
Mikio Nakahara. Geometry, topology and physics. CRC press, 2018.
Dmitry Nikolayev and Tatjana I Savyolov. Normal distribution the rotation group SO(3). Tex- tures and Microstructures, 29, 1970.
Harish Ravichandar, Athanasios S Polydoros. Sonia Chernova, and Aude Billard. Recent advances in robot learning from demonstration. Annual Review of Control, Robotics, and Autonomous Systems, 3:297-330, 2020.
TM Ivanova Savyolova. Normal distributions on SO(3). In Programming And Mathemati- cal Techniques In Plysics-Proceedings Of The Conference On Programming And Mathematical Methods For Solving Physical Problems, pp. 220. World Scientific, 1994.
Daniel Seita, Pete Florence, Jonathan Tompson, Erwin Coumans, Vikas Sindhwani, Ken Gold- berg, and Andy Zeng. Learning rearrange deformable cables, fabrics, and bags with goal- conditioned transporter networks. In 2021 IEEE International Conference on Robotics and Au- tomation (ICRA), pp. 4568 4575. IEEE, 2021.
Weijing Shi and Raj Rajkumar. Point-gnn: Graph neural network for 3d object detection in a point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1711-1719, 2020.
Anthony Simconov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: SE()vevquivarian object rep- resentations for manipulation. arXiv preprint arXiv:21 2021.
Anthony Simeonov, Yilun Du, Lin Yen-Chen, Alberto Rodriguez, Leslie Pack Kaelbling, Tomas Lozano-Perez, and Pulkit Agrawal. Se (3)-quuivariant relational rearrangement with neural de- scriptor fields. arXiv preprint arXiv:2211.0977 19.9766, 2022.
Michael Spivak. Calculus on manifolds: modern approach to classical theorems of advanced calculus. CRC press, 2018
Gusi Te, Wei Hu, Amin Zheng, and Zongming Guo. Rgenn: Regularized graph cnn for point cloud segmentation. In Proceedings of the 26th ACM international conference Multimedia, pp. 746- 754, 2018.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and -qquivaiinn neural networks for 3d point clouds. arXiv preprint 1 2018.
Dian Wang and Robin Walters. So (2) equivariant reinforcement learning. International Confer- ence on Learning Representations, 2022.
Dian Wang, Jung Yeon Park, Neel Sortur, Lawson LS Wong, Robin Walters, and Robert Platt. The surprising effectiveness of equivariant models in domains with latent symmetry. arXiv preprint arXiv:2211.09231. 2022.
Dilin Wang and Qiang Liu. Learning to draw samples: With application to amortized mle for generative adversarial learning. arXiv preprint arXiv: 161101122, 2016.
12

---

Published as a conference paper at ICLR 2023
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. Acm Transactions On Graphics (tog), 38(5): 1-12, 2019.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681-688. Citeseer, 2011.
Jiaxiang Wu, Tao Shen, Haidong Lan, Yatao Bian, and Junzhou Huang. SE(3) energy- based models for end-to-end protein folding. bioRxiv, 2021.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning, pp. 2635-2644. PMLR, 2016.
Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu. Synthesizing dynamic patterns by spatial- temporal generative convnet. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 7093-7101, 2017.
Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of de- scriptor and generator networks. IEEE transactions on pattern analysis and machine intelligence, 42(1):27 2018a.
Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chus Chgn Zhu, and Ying Nian Wu. Learning descriptor networks for 3d shape synthesis and analysis. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8629-8638, 2018b.
Jianwen Xie, Yifei Xu, Zilong Zheng, Song-Chun Zhu, and Ying Nian Wu. Generative pointnet: Deep energy-based learning unordered point sets for 3d generation, reconstruction and classifi- cation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14976-14985, 2021a.
Jianwen Xie, Zilong Zheng, Xiaolin Fang. Song-Chun Zhu, and Ying Nian Wu. Cooperative training of fast thinking initializer and slow thinking solver for conditional learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):3957-3973 2021b.
Jianwen Xie, Zilong Zheng, and Ping Li. Learning energy-based model with variational auto- encoder as amortized sampler. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12):10441-0051 2021c.
Jianwen Xie, Yaxuan Zhu, Jun Li, and Ping Li. A tale of two flows: cooperative learning of langevin flow and normalizing flow toward energy- model. arXiv preprint arXiv:2205.069924 2022.
Yifei Xu, Jianwen Xie, Tianyang Zhao, Chris Baker, Yibiao Zhao, and Ying Nian Wu. Energy- based continuous inverse optimal control. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Anthony Zee. Group theory in nutshell for physicists, volume 17. Princeton University Press, 2016.
Andy Zeng, Kuan-Ting Yu, Shuran Song, Daniel Suo, Ed Walker, Alberto Rodriguez, and Jianxiong Xiao. Multi-view self-supervised deep learning for 6d pose estimation in the amazon picking challenge. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 1386- 1383. IEEE, 2017.
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, et al. Transporter networks: Rear- ranging the visual world for robotic manipulation. arXiv preprint arXiv:2010. 14406, 2020.
Zilong Zheng, Jianwen Xie, and Ping Li. Patchwise generative convnet: Training energy- based models from a single natural image for internal learning. In Proceedings of the IEEE/CVE Con- ference on Computer Vision and Pattern Recognition, pp. 2961-2970, 2021.
Song Chun Zhu and David Munford. Grade: Gibbs reaction and diffusion equations. In Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271), pp. 847-854, IEEE, 1998.
13

---

Published as a conference paper at ICLR 2023
A BI-INVARIANT VOLUME FORM
bi-invariant volume form dg of a n-dimensional Lie group G a differential n-form that satisfies
dg d(hg) d(gh) vh G
(23)
such that for all Borel subsets _ G and for all well-behaved function f(g G
/ dgf(g)
Jngens d(hg)f (hg)) E
(Left invariance)
JEN
(24)
[ dgf(g)
/ d(gh)f ((gh))=1) VhEG
(Right invariance)
DER
ghEan
where h?? {hglg [2} and Uhh (ghlg Ωt. Let Φ(g) be some coordinatization of G that for some function J(z) R + R. dg can be explicitly written as
dg (z)ddzz
Let the left/right group translations in coordinates be (z) (Lh((z)) and (z) ((g())h).
The bi-equivariance condition in Equation (23) can then be expressed in the coordinate form as
(1)
d(hg) [(z (dz = ((z()) det
d' J(z)d'' = dg
Oz
(25)
(z)(1)
d(gh) (d = =1( (1) del
d= J(z)dd" dg
Oz
thus leading to the following equations:
Üz Oz
(r())
[z)( det
J(z) ()) det
det
(26)
Oz
Detailed introduction to invariant volume forms on Lie groups can be found in (Chirikjian)
2011
Zee
2016
Readers interested differential forms may find differential geometry textbooks (Spi- Nakahara 2018) useful.
vak
2018
For example, the translation group (IR +) admits a bi-invariant volume form de because
J dxf(x) = MEn
S - Ja d(x + E) yit + f((x f((x)+)) -c) = d(x + +))((f(( +c)
dr
-)
(27)
Note that the right invariance in Equation (7 is sufficient to prove the bi-invariance of dr because (R, +) is commutative, that is x + = + VI, E R.
On the other hand, for the multiplicative group (Ryo, x), dx not a bi-invariant volume form:
[ de dxl(x)
[ d(ex)) dx ereen d(xx) ((E (ex)
= -1 exEeR
(28)
d( (ex)) # Jereen ((r)))(()((er))
Ve E Ryo
However, d(log (x|) = dx/ is a bi-invariant volume form:
f x f(x) f(x)= exeee
dx
d(ex) da d(ez) 1( / JeseaR
d(ex) S(c ((x)
(29)
Again, we only show the left invariance because the multiplicative group is commutative.
Note that not every Lie group does admit bi-invariant volume form. Nevertheless, the Lie groups that we are concerned in this paper, the 50(3) group and the SF)) group, have bi-invariant volume forms. We reproduce here the bi-invariant volume forms of SO(3) and SE(33 in coordinate forms provided in CCirikjiann (2011). The bi-invariant volume form on SO(3) can be written in Euler angles o, 3.and 7 as
dR
sin 3dad3dy
8
14

---

Published as a conference paper at ICLR 2023
The bi-invariant volume form on (E(3) can be written in
coordinate
dRdv
where v E IR&S denotes the translation vector with respect to the space frame. Lastly, the bi-invariant volume form bi-invariant integral measure of SE(3) should not be confused with the bi-invariant metric, which does not exist for (3) (Chirikjian 2015).
B QUERY MODELS
B.1 EQUIVARIANT QUERY POINTS
We use Stein variational gradient descent (SVGD) LLiu & Wang 2016 Jaini et al. (202T) method to equivariantly draw query points 19:0 (Y))) in Equation (3) from the query weight field
we ()) . In this case, (x(x) can be interpreted as an unnormalized probability distribution
on R The SVGD equation (Liu & Wang 2016) is given by
af+ qi
Mas. af) Vx log wog Vx a x=q
(30)
Ng
x(x,x) = exp
x x21
(31)
Note that SVGD is fully deterministic given initial points q= We use hG med? / log N₂ as Liu & Wang 2016) where med, denotes the median of the distances between all the points in qi. Φ.. ...
Iv. We take the final output of SVGD as the query points, that is
y q Jin
(32)
for some tn 2 1. We take and tyi as hyperparameters. In our work, we uses = 0.005 and tyin 100.
We now show that the query point q, (Y) in Equation 32 is
Proposition 8. The query points (Y) are (E(3) if the initial query points
(Y)) No are SE((3)-qqivariin that is
Tq -;
Vi = Vi
(33)
To prove Proposition we first explicitly denote the query points' dependence on as qf (Y)
We then propose the following lemma.
Lemma 1. If (T . Y) = (Y) VT E SE(3), the following equations hold:
(()) Vx (xg xX)
(34)
= =(qq)(T. q [ Y)) log Y) (1x=)((TT))
R V (X) Vxk(x, of (T (T) (Tq
(35)
Since ||x-
22 x
|Tx Tx 2 VT E SE(3), is straightforward prove that
x(x,''= = k(Tx,Tx))))
SE(3)
(36)
To prove the equivariance of the gradient terms in Equation 34 and Equation (35 we prove the following lemma.
Lemma 2. If f(x(Y) f(Tx/T Y) for some function f R x P + R. the following holds.
Vx](x(T xY) x R. VxJ(x(Y)
(37)
15

---

Published as a conference paper at ICLR 2023
Proof.
VxJ(T- 7x0
(. -
= Vxf(x(() xx==x
(Change of variables x' T T-1xx)
= R
= =R Vxf(x(( X=X
(x' - x)
where in the last line we used
0x
1 @ -
V
xx
Vx
V
0x
8x
One can prove Lemma [siig Lemma 2
and Equation (36).
We now prove Proposition using Lemma be s(Y) such that
Let the sum on the right-hand side of Equation (30)
N₂ Σ Ø=1
s sY)
(x) Vx log vxlogmo((1))))xef + Vxk(x, ((Y)) x=000)))
(38)
We first consider the case with f = 0. One can prove that s(T Y) = Rs(Y) using Lemma 1 and the equivariance of the initial points Tq. (Y) qf (T Y), which was assumed in Proposition It is then straightforward to prove that q (Y) is also equivariant.
8
Proof.
q (T o Y) = (T. c
= =T + RRs(Y)
+
(39)
= + + +v
=Tq=
We now recursively apply this relation to 1, .tnmn to conclude that the final query point also equivariant, that is q: /- (Y) q. _-t/in (T Y). Therefore, the only requirement for our query points to be equivariant is the equivariance of the initial points: =) = "T Y). (T We provide simple (and clearly not the best) deterministic method that we used sample the
initial point q (Y) in Algorithm
Algorithm Simple algorithm for deterministic and equivariant initial point sampling
Input: Y ((y1, (yM, cu)), w(x(Y), Nmans Toluster Output: qı,92, "2.......
i
Q - (y1, YM} while < Nmar do
> Initialize set Q with the set of all the points in Y
if Q is not empty then
q: + arg max YY
Take the point with largest weight in Q
yEQ
Q + Q z Q
1z-
rdasser
Remove the neighbors from Q
end if
i + i+1
end while
16

---

Published as a conference paper at ICLR 2023
B.2 SURROGATE QUERY MODEL
Let A(r) be the set of all the indices of the query points whose shortest distance to the point cloud X is farther than some radius such that A(r) E (1, N [dmin (T X) r). The
Mullback-Leible divergence term in Equation (22 can be calculated as follows:
(wr(H(w, Q (T)), Q(Y))
No
No
H, 9 1,1X,Y,T)
dwy
dq. Cq,:H, q X,Y,T)
log
Ps(w,,9b(Y)
dw,; / dq. 9: 9: Hi(w;, 9,,X,
JRt
JR3
Pi(wr, qilY)
iEA(r)
den dlg
Σ iEA(r)
-4 ((Y))W((aa
Bet
dan
N(lia,o)) (q:
x log
(14;0og Y), OH (000)000000 (4-
(40)
a(4;a,0H)
= E
dl (.N(Ha;, log N(4; log (o: (Y()))
iEA())
[ d(N((1;,,0)
Σ iEA(r)
[4 a)) - (4 - log wo [ 200
Σ Be
[
{(coH)² -(co + a log (V: +
200
iEA(r)
1
log w( - a
Σ
2
BEA(r)
where the third line we used
P P,lun 0 Vj & A(r)
log
log P
q,
and in the last line, we used the reparameterization trick for Gaussian distributions Kingma & Welling 2013).
B.3 QUERY ATTENTION
Due to the computational limitations, it is desirable have as few query points possible during the inference time. Therefore, instead of directly taking w; = we we( we normalize the query weights by taking
we wo (Y))
W7
Σ No wo (Y))
(41)
such that the query points compete with each other during the training. As result of this compe- tition, only few query points have non-negligible weights. Therefore, during the inference time, we can calculate for only few query points with nen-ngglig weights instead of calculating the whole query points to save the computation. Note that the normalized query weight is still equivari- ant because only scalar addition and division were used in Equation
C INTUITION BEHIND THE BI-EQUIVARIANCE CONDITION
To illustrate the bi-qquivariance condition in Equation 66. consider an object placing task where
Tyo E S3) is the object pose (o) in the gripper frame (g) and Ted SSF(3) the desired object
17

---

Published as a conference paper at ICLR 2023
A) Left Equivariance
B) Right Equivariance
Equivariant to the placement target
Contravariant to the grasped object
Figure 8: A) The end-effector should follow the transformation of the placement target keep the relative pose between the object and the placement target invariant. B) The end-effector should transform contravariantly compensate for the transformation of the grasped object such that the relative pose between the object and the placement target is invariant.
pose (d) that to be placed in the scene frame (s). Consequently, the gripper pose in the scene frame Teg should satisfy the following equation
Tsd = To
(42)
Now, let the desired pose of the object be placed has been transformed as Ted + T" ST.d for some transformation S' E SE(3). In order to keep the relation in Equation (42) invariant such that for the new gripper pose Th the equation T Similarily, if the pose of the grasped object is transformed as Tgo + To ST the gripper pose should also be transformed as Teg - Th = S -1 to keep the relation in Equation (42) invariant. Since T.a and yo implicitly encoded in X and Y individually, and Tgo into X and Y to get the equation
aq Too holds, should be that Th sg ST
can naively substitute Ted
P(T,glX,Y) = P(T = ST.g X,Y) This is the intuition behind Equation
=
(6). We illustrate the bi-equivariance condition in Figure
D SAMPLING DETAILS
For the sampling, we use the Metropolis-Hastigg (MH) algorithm (Hastings 1970: Metropolis et al. 1953) and Langevin algorithum on the SE(3) manifold (Brockett 1997: Chinkjian, 2011 Davidchack et al. 2017). Unlike the MH, the Langevin algorithm does not suffer from high rejection ratios and converges with much fewer iterations. However, the Langevin algorithm requires the gradient of the energy function and thus computationally inefficient. addition, the time step for the Langevin algorithm cannot be arbitrarily high maintain the precision of the dynamics. Therefore, we first run MH for rapid exploration and then run the Langevin algorithm using the MH samples initial seeds. Note that the differential geometric aspects of the SE) manifold must be considered in implementing these methods.
For the following sections, we provide details of the sampling methods that used. We first explain the proposal distributions that we used to run the MH algorithm on the S SE(3) manifold. We then introduce the Langevin algorithm SE(3). We calculate the Langevin dynamics in quaternion- translation parameterization as Davidchack et al. (2017) to avoid singularity while benefiting from commonly used autograd packages.
D. PROPOSAL DISTRIBUTION FOR MH
The Metropolis-Hastings (MH) algorithm (Hastings) 1970) is a propose-and-reject algorithm used for sampling from some probability distribution dT) First, a proposal point T, sampled from the proposal distribution dQ(T, The proposed point Tp is stochastically accepted or rejected (P(T,)AQ(TIIT)
by the acceptance ratio A min 1, aP(I,)aQ1I, 7.J the proposed point, that is T =T, If rejected, the point remains the same, that is T4t1 = T,. known that the steady-state distribution d.P.a (Tx)) converges to (T).
If the proposal is accepted, the next point is
18

---

Published as a conference paper at ICLR 2023
We decompose the proposal distribution dQ(T(T)) = Q(T into 1) the orientation proposal RRRd, R and 2) the position proposal distribution Qv(vive)d's such that distribution QR (R|R,)
Q(T]Zi)dT'
(R)RR x Qu(v/v))dVv
where d'vv is the Euclidean volume element and dR the bi-invariant volume form of SO(3) (See Appendix AA. We use Gaussian distribution for the position proposal, that is Qv NVpp: Vt, oT.. For the orientation proposal QR (R,) we used IG which is the normal
Vv)) (vplvv)
distribution on S3(3) (Nikolayev & Savyolov 1970 Savyolova 1994 Leach et 2022). Concrete
calculation and sampling methods for I9 (3 provided in Appendix D.2
D.2 NORMAL DISTRIBUTION SO(3)
We follow the method in Leach et al. 2022) to calculate and sample from sora)), the normal
distribution on SO(3) (Nikolayev & Savyolov 1970 Savyolova 1994). In the axis-angle parame-
terization, our orientation proposal distribution QR RR which is IG so(3)) can be written as
(R, (R)RRRRe)dR = - cosw)/ x fe(w)dwds)
where w (0, TT) is the rotation angle, dl sin 0/T x d0dø is the uniform volume element over the sphere S² where the rotation axis lies, and fe(w) is as follows:
fe(w) = Σ (21 + 1)e 1) 1(+ sin ((21 + 1 /2) 1
(43)
sin w/2
We approximate the infinite sum in Equation 443 by summing up to sufficiently high I. Note that the summand in Equation (43) decays exponentially fast to the square of I. Therefore, the approximation is justified. The rotation axis vector can be easily sampled by first sampling from three-dimensional Gaussian and then normalizing it. For the sampling of w, one may use numerical inverse transform sampling. As noted by Leach et al. (2022). the volume element (1 cos w:) should be multiplied to fe(w) for the inverse transform sampling.
D.3 LANGEVIN MCMC ON SE(3)
Let V; be the i-th basis of the Lie algebra of an unimodular Lie group G. Consider the following stochastic process g(t) E G generated by a Lie algebra (x = E, (t), E T.G such that g(t) = 9(0) exp (8XX (0)] exp (dt) ... exp (t dt)]. The Langevin dynamics for G is then
sx,(t) - Lv, [E(g)] dt + V2dw;
(44)
where dw;; Nova is the standard Wiener process and Lv (# (g exx(ss)))) a-n is the (left) Lie derivative of a function f G along V. It is known that this process converges to
dP (g)
exp Tapp-E((g)) F(g)) dg
when / + oo where dg is the (left) invariant volume form of G (Brockett 1997 Chirikjian 2011
Davidchack et al. (2017) provide concrete ways calculate the Lie derivative and the Langevin parameterization.
dynamics on SE(3) in quatermion-translene
ttttemmiiontaaaslatiion parameteri- zation is convenient because has no singularities. Therefore, the gradients from commonly used autograd packages can be easily used to calculate the dynamics. For SE(3), V, is the Lie algebra basis of S0(3) for 1, 2,2 and the translation generator for translation parameterization be vq. E S x RR3 C R². Let L E 7x6 be the Lie derivative matrix whose (14, i) 'tt element is [L] Zt The matrix can be calculated as
4.5, 6. Let the quaternion-
LSo(3)
04x3 Iaxa
L
0ax3
(45)
Lgo(3)
- 93
q²2
19

---

Published as a conference paper at ICLR 2023
where q q¹ +2 Derivations of Equation (45) be found in (Davidchack al. 2017). Since the chain rule holds for the Lie derivatives (Chrikjjaa 2011), the Equation (44) can be written in the parameterized form
dq dv
dz
-
V2Ldw
(46)
where - We calculate the gradient of the energy V (z) using typical autograd pack- ages. Note that dq in Equation (46) satisfies the unit-quaternion constraint q dq such that dq E 5 (Davidchack et al., (017). In practice, however, we reproject q + dq onto 5 by normalization because the inaccuracy in numerical integration.
E PICK-MODEL AND THE RELATIONSHIP TO NDFS
Pick-model For the place model, the point cloud of the end-effector Y is always different because of the grasped object. On the other hand, Y is always the same for the pick-model because no object has been grasped yet. Therefore, we remove the Y-dependence of the query EDF and the query density by taking the (x (Y) to (x) and po (x() to (x)).. In this case, the energy function Eo ( Y) in Equation 14 becomes
N₂
Eo ET Eo
Σ wy
DR
(47)
where wi, qi and sb are not the outputs of some functions anymore but just parameters that are either predefined or learned.
Relationship to NDFs We now illustrate the relation of Equation (47) to the energy function of NDFs (Simeonov et al. 2021). Let the energy function in Simeonov et al. (2021) 1 be
N₂
ENDF(T|X) = Σ
(TT X) -bil1
(48)
1=1
Ndemo
1;
(49)
N Nemmo
where T, and Xx are the grasp pose and the point cloud input of the n'th demonstration. The Equation (48) can be understood as a special case Equation (47} with (i) the L1 error instead of the square error, (ii) all the query weights being constant w, 1, and (iii) all the components of the feature EDF Xx) being the invariant scalars (type-O vectors) such that D(R) = I.
Relationship to other variants of NDFs Other recent works derived from NDFs (Simeonov et al. 2022 Chun et al. 2023) also are closely related to EDFs. Relational Neural Descriptor Fields (R- NDFs) (Simeonov al. 2022) extend NDFs' fixed placement target tasks to object rearrangement tasks in which placement target is also an object with varying poses. Instead of using fixed target descriptors NDFs, R-NDFs utilize another descriptor field to represent the target placement object. The resulting R-NDFs energy function is very similar to EDFs' bi-equivariant energy function. This a natural consequence due to the bi-equivariant nature of object rearrangement tasks.
On the other hand, Local Neural Descriptor Fields (L-NDFs) (Chun et al. 2023) focus on imposing locality on the descriptor fields. While EDFs and L-NDFs both try to exploit locality, the motivation is largely different. Chun et al. (2033 focus on locality to improve generalization and transferabiliity to novel objects. This distinguished from the major motivation for imposing locality on EDFs: removing the necessity of object segmentation.
One thing to note is that these studies are not mutually exclusive but complement each other. EDFs can be pre-trained with similar method to NDFs and R-NDFs. The strict SE(3)-equivarian) of EDFs can be relaxed by imposing the -qqiivariie on the loss function as L-NDFs instead of imposing it on the model itself. Conversely, these methods may benefit from the end-to-end trainability and generative nature of EDFs' energy-based model. The orientational sensitivity of higher-type (equivariant) descriptors should also be beneficial to these methods.
20

---

Published as a conference paper at ICLR 2023
Irrepwise L1 Norm For closer analogy with the energy function of NDFs, propose using irrepwise I norm. Let equivariant vector f be given by N f(0) where f(n) is type-l, vector. We then define the irrepwise L1 norm as
n=!
| Σ | 12
If we use irrepwise L1 norm in Equation (47 and confine all the vectors in EDFs to be of type-0, Equation (47) and Equation (48) are exactly identical. Although we did not use irrepwise L1 norm in our work, we expect that this modification would be more robust to outliers than using the square error term.
F PROOFS
F.1 PROOF OF PROPOSITION I
We first prove the left equivariance.
Proof.
J aP(TTS.X,Y) = / (TP(TTS.X,Y)
TESh
(IP(S
(
TESh
((S-1))((--111111
(: bi-invariance of d'')
S-ITEE
[ (TP((T(.Y)
(S T)
JEA
/ dP(T)X,Y)
TEA
The right equivariance can be similarly proved.
Proof.
[
Jreas
dP(TIX,S-'oP) .Y)
TEnS (TP(T(X,S-'01)
JTP((SS- ( P(TS)X,55-1))) P(TIX,Y)
TENS
PIS
(:: bi-iivvariince of df')
JTS-EE
[ (TP(TIX,Y)
(T---
JTEA
[ dP(T[X,Y)
TEa
F.2 PROOF OF PROPOSITION 2
Let the partition function (the denominator) of Equation 9 be Z(X,Y). Lemma 3. For a bi-equivariant energy function ETT Y), the following equation holds.
7(X,Y) 7(S X,Y) 2(X,5-1oY)
(50)
21

---

Published as a conference paper at ICLR 2023
Proof.
7(SxX,Y) = JSF(3)
d7 exp [-B(T]S.XXX))
dl exp [-E(S-
6. E:STSS) = E(T
SB(3)
(T) exp (-E(S-TTXX,))
(: bi-invariance of df)
SF(3)
dT exp |- Y(X,Y)
(S-T
SE(3)
7(X,5-1
dT exp -((T(XXS--oY1)
SF(3)
dT exp
(: E()) =(TIX()
SE(3)
d(TS exp [- E-E(TS [X,)]
(: bi-iivvarriance of d'')
(S((3)
dT exp |- Z(X,Y)
(TS-1 T)
(SE(3)
Now we prove the bi-equivariance of Equation ( using Lemma 3
Proof.
P cxp -
=xxp Y31/(XX)
X)
=exp ( (FTT,))),."..))2,,55551 -(TSS)XS--o0)
13
F.3 PROOF OF PROPOSITION 3
We first show that the energy function in Equation 10 satisfies E(STIS X,Y) E(T(X,Y) where T (R, v) SE() and = (Rs, vs) SE(3).
Proof.
SSSTS.X.YY
(xx0xx11)1|((SSXSSSxx - -
[ - (AA
( Equation X
/ 2200)/Y))DD(RR))(/ZXYYY D(R))
Equation
/ ( ||)) F(T)X.Y)
where the orthogonality the representation D(R) is used in the last line. Note that the inner product of two vectors is invariant orthogonal transformations.
22

---

Published as a conference paper at ICLR 2023
We now prove that E .Y) = E(T)X,
Proof.
E(TSIX,S-108)
-WP Y)1?
[
(: Equation
((3x))((xXX1]]]((TXxxx - - 6: (Tx) dx VT SE(3))
('xx(x(V))<(((xxx) - D(
(Sx x)
In the fourth line, used PT Y) p(x(Y) and (Tx Y) (x)) by the definition of the query density and the query EDF. Note that in the fifth line we used the SE invariance of the Euclidean volume element df x, that is
d3 (Tx) del [a(Rx + v)/Oxld
det (O()))/xx
(51)
= det I d'x d²x =T= (R, E SE(3)
Therefore, the energy function E([ in Equation I0 is indeed bi-equivariant.
F.4 PROOF OF PROPOSITION 4
Proof. Let a query density satisfies Equation [1 such that p(x(Y) p[Tx| Y) VT E SE(3). If this query density is grasp-independent such that p(x(Y) p(x), then p(x) p(Tx) VT SE(3) by Equation (11). Since there always exists some T E SE(() such that Tx = x' for any x' R p(x) must be constant function. In other words, there exists no grasp-independent and non-constant query density that satisfies Equation [1]
F.5 PROOF OF PROPOSITION 5
Proof.
No
Po [Tx
we
(Tx (To(ToY))
No
we (T
(Tq:0 813) - -T:o
(52)
we (q:0 (Y) (Tx-Tq;;(Y))
No
we (q: (Y) (x (i: = (x
where Equation
13 used in the second and the third lines.
23

---

Published as a conference paper at ICLR 2023
F.6 PROOF OF PROPOSITION 6
Let the query model P(w, Q be equuiaaquv such that
P(w. Q P(w,SQISo> AS SE(33
(53)
We first show that X,TiX,Y, w, Q) satisfies
P Q) = So Q)
(54)
= PPTSSX o Y, w, 5
SA = (Rs, Vs) E SE(3)
To prove Equation 54 we first show that ET Y. w. q) in Equation 15
satisfies the following:
E(ST X.ST,S.XYYYKK w, q) E(TIX, w. q) E(TS|X, o u. g
Proof. We first prove the left equivariance.
: q)
= =wl @ x) - DDR
= w # (e D(s) (B
(: Equation (8)
(: D(R)" D(R) -1
We now prove the right equivariance.
ET u.
wwlpo [TSS ((
wll DR " "oo Y\
(: Equation (8)
w]lpo(Tq:iX)) = Y,,,
(:: D(R-1)= D(R)
One may simply replace the energy function E(T|X,Y) in Appendix F22 with the energy
function E(T[X,,, Q) = Σm E(T Y, wi q:) to find that Equation 54 indeed holds.
Now show that the marginal PDF P(T is bi-equivariant,
Proof.
P(STIS.X,Y)
X...PP(STSSxX....PPPP..
/ dww.OOP(TXX,Y,w..IFF... Q) (IP((QQ))) = P(T[X,Y)
(:: Equation 53 and Equation 54)
dwdQP(TTSIX.S .Y.w .Y. Q)P S
/
dwil(S YY 5
(: Equation
dwwIQP(TSIX,5 Q)P(w. CAPP(/QQS--- Y) P
In the fourth line, used the SE(3)-invariinn invarrincc of the Eulcidean volume element in Equation 51
N₂
N₂
/ 1QQ
ddo=
8 q
I
d(qq)) = ((TQ) VT SE(3)
(55)
24

---

Published as a conference paper at ICLR 2023
F.7 PROOF OF PROPOSITION 7
We first show that (ww,; qi () in Equation (20 is Ee())--quivariant
Pu(ws, Tq. T Y) = Pw,, q Y) 47 S
(56)
Proof.
Pi(w;; Tq:
dl, ox) qi(T
dw;
dl; N1t:loo((al7))0))000000 IV)(
(: Equation
dw;
P,(w, Y:lY)
As result, P(w, Q I P, (\ in Equation 20 also satisfies
P(w, TQ = P(w, Q 4A E SF(3)
(57)
We now show that H;(w,
in Equation
21 satisfies the following equation:
Hi(ws, iXX,Y, H: ws, q X, YTST) H,(w;, S a ., T5)
(58)
Proof.
Hi(ws, X,
[Pilw Y)
if dmin So <r
(ell:,,,WWii; qY()) else
Pilwer Y)
if dmia X) <
(A)
((; q.)) else
HH(wi, q:l,i,
if dmin (T:) <r
(B)
[ddlddus))(: --- else
[.,(w,,S--0 (1
if dmin ((TS) (C)
(a(S-1o))
else
SH Xx Y,TS)
We used dmii (Tx, T X) dmmi (x, X) VA SE(3) in (A). This is because the Euclidean
distance is preserved under SE(3) transformations, We used (x(x -x2) = 58x - Tx2) VT' e
SE)) and Equation (56) in (B). Lastly, used Equation in (C).
Therefore, H(w, Q)
= -1 - H,W,, (XY,TT also satisfies
H(w, Q
X Y,Y, Hww, SX Y, S, = HW,, H[
(59)
We now propose the following lemma.
Lemma 4. Let scalar function f[T|X,Y ) be defined as follows:
f(TIX,Y)= = -f dw / dQ (T QUL,(T.X,YYw...)
J(TXXY) Y) is bi-equivariant hf (T, X, Y, w, Q) and h2 (T, X, w, Q) satisfies X,Y, h,T, W. Q) = (ST S X, Q)
(60)
hTS, X, W,W, S VS E SE(3), EE (1,2)
25

---

Published as a conference paper at ICLR 2023
Proof.
((ST|SXYY)
dw
/
f(T(x)
c: Equation (60)
/ dw
( W, S Q) (: Equation (60
J ( .Y, w, S
(: Equation 55
= fdw dw
(S Q
((TS)X,S-'Y)
We now prove the bi-equivariance of Lo(T|X,Y) Y) in Equation 22
Proof. We first define (T, w, Q) as follows:
h(T, X, Y, w. Q) = log Po (T w, Q) Po Q - Ho Q(,,Y,7)
(61)
Using Equation [54] Equation (57) and Equation (59), one can prove that h(T, w, Q) satis-
fies Equation (60p. In addition. Ho (w, QI Y") satisfies Equation (60) was shown Equa-
tion (9). Because Lo (TIX,Y Y) can be written
Co XT) =EE Ho [log (T) Q)] DRL Ho Q| YI (/xxx.1)
=f dw / dQ Ho Q www.QXXXYY.MMTXXYYY..ooo
(62)
prove the bi-equivariance of (o),)X,Y)) Y) in Equation 22 using Lemma
G EQUIVARIANT GRAPH NEURAL NETWORKS
Graph neural networks are often used to model point cloud data (Wang et al. 2019 Te et al. 2018
Shi & Rajkumar 2020). (E)3) equiivarrt graph neural networks TTomas et al.1 Fuchs al.,
2020 Liao & 2mid[[2022 exploit the trontranslttion symmetry of graphs with spatial structures. In this work, we use Tensor Field Networks (TFNs) (Thomas et al.) 2018) and the transferrs
Fuuhs et al. 2020) the backbone networks for our models.
Tensor Product and Spherical Harmonies Given two vectors u and v of type-l, and -12, the tensor product u @ transforms according to rotation R E SO(3)
X v (Di, (R)u)
(R)v)
(63)
Tensor products are important because they can be used construct new vectors of different types. By change of basis the tensor product u@v can be decomposed into the direct sum of type-/ vectors using the Clebsch- Gordann coefficients (Thomas et al. 2018 Zee 2016 Griffthh Schroeter 2018). Let this type-/ vector be (u (1), The m th components of this vector calcuated as:
f2
(
(l.m) (lamm))(amma) Umn Umz
(64)
where (x(..m) (Is (1smma)002mmm) is the Clebsch-Gordan coefficients in real basis, which can be nonzero only for [/ l2]
< < 4 lz.
26

---

Published as a conference paper at ICLR 2023
The (real) spherical harmonics Ym (x/ xII) orthonormal functions that form the complete basis of the Hilbert space on the sphere 5² E {0, 1.2..... is called the degree and m E {-1,- ... , is
called the order of the spherical harmonic function.
Consider the following (21 + 1)-dimensional vector field Y (Y - Y 1). By 3-
dimensional rotation R E 5O(3), Y transforms like type-I vector field such that
Y (R(x/|X) (R) ( (x/[[x]]]
(65)
Tensor Field Networks Tensor field networks (TFNs) (Thomas al. 2018 are SE(3)- equivariant models for generating representation-thoorriee vector fields from a point cloud input. TFNs construct equivariant output feature vectors from equivariant imput feature vectors and spher- ical harmonics. Spatial convolutions and tensor products are used for the equivariance.
Consider featured point cloud input with M points given by X f1),, f.).... (xM- fM where X; E R is the position and f; is the equivariant feature vector the i-th point. Let fo be decomposed into N vectors such that f. + p(()) where r!n) is type-l, vector, which is (2l, + 1) dimensional. Therefore, we define the action of = (R, E SE(3) on X as
T X = ( D(R)f,, ... (TxM. D(R)(s)
N where E SO(3), E 3 and D(R) 1n11 DI, (R).
Consider the following input feature field f(im) (x X) generated by the point cloud input X as
M
(xnn(x(X X) = Σ j=1
(55(3) (x x;)
(66)
where 5(3) (x = IT 6(x; y4,) is the dirre--dmensional Dirac delta function centered at xj-
Note that this input feature field is an 553())-quivariin) field, that is:
f(im) (Tx[T X) = (()) (x|X) T (R, v) SE(3)
Now consider the following output feature field by a convolution
N'
(ouu) (x|X)=
A
(ouu) (x|X)
n'=1
(67)
= -
d (Y(oxyy(n W(x-x)))
with the convolution kernel W(x- y) (d whose (n', n)-th block Wn'n(x y) +1x is defined follows:
th
W (xn'n(x)
([x(() Σ
mll y( (x/[|x([) (J)))
(68)
m'm
J=l
Here, d[x()[[x]]= IR + R is some learnable radial function. The output feature field fourt (x X)
in
Equation 67 is proven to be S -equiqaaaaa (Thomas et al.
2018 Fuchs et al. 2020).
Trans-Trasss The
2020 variants of TFNs with self-attention. Consider the case in which the output field is also a featured sum of Dirac deltas
(Fuchs et al.
M
(ouut (x(X)=
Σ (ont) (33) (x x,)
(69)
where x; is the same point as that of the point cloud input X. The SE(3) apply type-0 (scalar) self-attention oi to Equation (67):
N
(out),+ i Σ Eoy W(x Xx
w wo'ne!()
(70)
1#1
27

---

Published as a conference paper at ICLR 2023
where the W term is called the self- interattteaatnon (Thomas et al. 2018 W is nonzero only (S)
when 12 In. The self-interaction occurs where = such that (z; - 23y = W()). The self- interaction term is needed because W is a linear combination of the spherical harmonics, which are not well defined in x = O. Details about the calculation of the self-attention aiy can be found in
Fuchs et al. (2020).
H EXPERIMENTAL DETAILS
For the test environment, we use PyBullet (Coumans & Bai 2016-2021) simulator for the experi- ments. We use the Franka Panda manipulator with a custom end-effector. We use IKFast (Diankov) 2010) with Pybullet-Plannie cameras are used to observe the point cloud of the scene. Six simulated depth cameras to observe the point cloud of the grasp. We downsample the point clouds using a voxel filter. We illustrate the downsampled point clouds in Figure ? our work, we assume no collision between the environment and the robot links except for the hand link. We also allow the robot to teleport to reach pre-grasp and pre-place poses to eliminate the unnecessary influence of the motion planners, However, we fully simulate the trajectories of all the task-relevant primitives (e.g.. grasping, releasing, lifting).
(Garrett 2018) for the inverse kinematics. Three simulated depth
used
Since motion planning is not in the scope of
We experiment with three tasks, the mug-hanging task and the bowl/bottle pick-and-place task. For the mug hanging task and the bowl teek-an--llae task, we demonstrate with a single target object instance in upright poses only. For the bottle pick-and-place task, we use five object instances in upright poses only. For evaluations, we tested in four different unseen setups: (A) Unseen instances, (B) Unseen poses. (C) Unseen distractors, and (D) Unseen poses, instances, and distractors. Note that in the unseen poses setup (B), we only use lying poses, which were not presented during the training. On the other hand, in the unseen poses, instances, and distractors setup (D), we both use lying and upright poses but in unseen elevations. The reason why we also use upright poses in setup (D) is that the baseline model already completely fails in setup (B). Therefore, the result would be trivial if we only use lying poses in (D). Therefore, we mix both upright and lying poses in setup (D). Note that the upright poses are also unseen poses because of the unseen elevations.
For the inference, we run the MH for 1000 steps and then run the Langevin algorithm for 300 steps. Lastly, we optimize the samples for 100 steps using Equation 446) but without noise. We empirically find that only at most three query points have significant weights after training. Therefore, we only use three query points with the highest weights to save computations. Instead of directly taking the lowest-energy sample pose, we check the feasibility of the pose before going into action. For example, if collision is found or no inverse kinematics solution can be found for the sample pose, we deny that pose and move to the next best sample. We provide the details in Algorithm 2
Ten demonstrations are generated by a probabilistic oracle for each task, with the default instances in upright poses, with only the x. y. and yaw being randomized. In the unseen poses setup, the de- fault (trained) instances are provided in unseen (lying) poses. The poses are completely randomized, including the elevation In the unseen instances setup, ten unseen instances of target objects are provided in the trained poses (upright), again with only the x, V, and yaw being randomized. In the unseen distractors setup, four unseen visual distractors are located near the target objects. We randomize the poses and the colors of these distractors. To separate the effect of motion planning, disable the collision between the distractors and the robot. Lastly, in the unseen instances, poses, and distractors setup, we combine all the prior setups. We experiment with ten unseen instances in unseen poses (50% upright and 50% lying, arbitrary elevation) with four randomized visual distrac- tors. In this setup, we use supports to give arbitrary elevation to the target objects. Note that we both used upright and lying poses, unlike the unseen poses setup. This is to test the case with unseen distractors and unseen instances for not only the lying poses but the upright poses as well. Note that upright poses are also unseen poses because we give arbitrary elevations. For the mug and bowl tasks, only a single instance was used as the default instance for training. For the bottle task, five instances were used due to the high variance of the shape. All the models are sufficiently trained such that the total success rate in the trained setup (no unseen situations) exceed at least 90% for all three tasks. The experimental settings for the three tasks are illustrated in Figure [0] We also illustrate the ten unseen instances of mug, bowl, and bottle in Figures 11 12 and 13 respectively.
28

---

Published as a conference paper at ICLR 2023
Algorithm 2 Pick-and-place algorithm
Input: Pick-model Piick ( (x) Place-model Pllace (T Y), Number of iterations N D Observe the point cloud of the scene b Sample N poses from the pick-model Sort samples by their probabilities (descending order)
X <- observe )
sample( P pick (- (x1x,; N) T 1-l sort ([T.) P pick (-1x))
T
for T in [T:];= do
if feasible(T)) then
(Tc
Pick if the configuration feasible
break
end
end for
X t observe ssceree
D Observe the point cloud the scene Observe the point cloud of the gripper > Sample poses from the place-model 1 < sort ([T]] Ppaace (-XX, Y)) & Sort samples by their probabilities (descending order)
Y observe-gripper(
T < -amplee (pplaca (-1x, N),N)
for T in [T:];=m do
feasible(T) then
pTace(T)
Place if the configuration is feasible
break end if end for
For EDFs, we use a single query point for the pick inference and three query points for the place inference. For the ablated model, for the place inference. The reason we use more query points for the ablated model that the type- descriptors cannot encode orientations alone. Unlike EDFs, type-0 descriptor fields (the ablated model) require at least three non-collinear query points and much more in practice to determine orientations. This is in direct comparison with EDFs, which can determine the orientations even with single point. The computational benefit of using only type-0 descriptors is compensated by the increased number of query points. We the number of query points make the inference time of the ablated model to be similar to or slightly longer than EDFs. We run all the experiments on an Nvidia RTX3090 GPU and an Intel 19-1 1990k CPU with 16Gb RAM. We turned off all the E-cores of the CPU and only used P-cores with fixed clock of 5100Mhz. We found that turning off the E-core is crucial for the inference speed.
query points for the pick inference and five query points
The models were trained for 600 steps (60 epochs) using Adam optimizer (Kingma & Ba 2014) where the learning rates range from 0.005 to 0.001. We randomly perturb the target pose and apply jitters input point clouds to augment the training data. It takes around 5.5 hours to train the pick-model and 8.5 hours to train the place-model, where most of the time is spent on MCMC sampling. We 10000 iterations of the MH and 3000~6000 iterations (linearly increasing as training proceeds) of the Langevin algorithm to draw negative samples for the training.
A)
B)
Figure 9: A) Downsampled point cloud the scene. B) Downsampled point cloud of the gripper with grasped object.
29

---

Published as a conference paper at ICLR 2023
A) Demonstrations B) Unseen Instances C) Unseen Poses D) Unseen Distractors E) Unseen Instances, Poses, Distractors
Figure 10: Test environment for the mug-hanging task, the bowl pick-and-place task, and the bottle pick-and-place task with various unseen setups.
Figure 11: The ten mug instances that were used as unseen mug instances are illustrated.
Figure 12: The ten bowl instances that were used unseen bowl instances illustrated.
Figure 13: The ten bottle instances that were used as unseen bottle instances are illustrated.
30

---

Published as a conference paper at ICLR 2023
I ADDITIONAL EXPERIMENTAL RESULTS
Table 3: Success rate of EDFs for mug-hanging task with different demonstrations
Low Var. & Unimodal Demo
Mixed Grasp (Handle & Rim)
Setup
Pick
Place
Total
Pick
Place
Total
Unseen Poses (P)
1.00
0.96
0.96
1.00
0.99
0.99
Unseen Instances (I)
0.99
0.90
0,89
1.00
0.92
0.92
Unseen Distractors (D)
1.00
1.00
1.00
0.96
0.99
0.95
Unseen P+I+D
0.99
0.83
0.82
0.90
0.89
0.80
In Table we list the success rates of EDFs for (1) low variance and unimodal demonstrations and (2) highly multimodal demonstrations for the mug-ham mug-hanging task. In the highly multimodal demonstrations, the mug is picked both using the rim grasp and the handle grasp. The experimental results indicate that EDFs are both robust to low variance or high variance demonstrations, whereas (33) can only be trained from low variance ones.
Table 4: Success rate and inference time the ablated model and EDFs. All the evaluations are done in the unseen instances, poses & distracting objects setting.
Descriptor Type
Mug
Bowl
Bottle
Pick Query Place Query)
Pick
Place
Total
Pick
Place
Total
Pick
Place
Total
Type-O Only (10/5)
Inference Time
5.7s
8.6s
14.3s
6.1s
9.9s
16.0s
5.8s
17.3s
23.0s
Success Rate
0.84
0.77
0.65
0.60
0.95
0.57
0.66
0.95
0.63
Type-O Only (30/ 10)
Inference Time
10.0s
14.5s
24.5s
13.9s
19.4s
33.4s
10.9s
24.2s
35.1s
Success Rate
0.90
0.90
0.81
0.72
0.99
0.71
0,76
0,96
0.73
EDFs (1 (3)
Inference Time
5.1s
8.3s
13.4s
5.2s
10.4s
15.6s
5.2s
11.5s
16.7s
Success Rate
1.00
0.95
0.95
0.95
1.00
0.95
0.95
1.00
0.95
In T we compare EDFs with the ablated models with only type-0 descriptors. We evaluate the ablated model for different query point numbers. The result shows that EDFs still outperform ablated model even though much more query points and longer inference time is used.
Table 5: Success rate of experimented methods in the trained setups
Mug
Bowl
Bottle
Method
Pick
Place
Total
Pick
Place
Total
Pick
Place
Total
EDFs
1.00
0.99
0.99
1.00
1.00
1.00
0.98
1.00
0.98
SE(3)-TNs (Zeng et al. 2020)
1.00
0.91
0.91
1.00
1.00
1.00
0.99
0.93
0.92
Type-0 Only (Fast)
1.00
0.98
0.98
1.00
1.00
1.00
1.00
0.97
0.97
Type-O Only (Slow)
1.00
0.98
0.98
1.00
1.00
1.00
1.00
1.00
1.00
Lastly, in Table 5] we list the success rate of all the methods in the trained setup. Note that all the methods used in the experiments were sufficiently trained to achieve least 91 % total success rate.
31

---

Published as a conference paper at ICLR 2023
J EXPERIMENTAL RESULTS ON NEURAL DESCRIPTOR FIELDS WITH UNSEGMENTED POINT CLOUD INPUTS
In this section, show by experiment that object segmentation is critical to the performance of Neural Descriptor Fields (NDFs) (Simeonov et al. 2021). We compare the success rate of NDFs with unsegmented point cloud input the same NDFs with segmented point cloud input. All the experiments are done using the official implementations of Simeonov et al. (202T). The results are summarized in Table As can be seen in Table the performances significantly drop for both tasks when NDFs are provided with unsegmented inputs. We provide qualitative examples in Figure [4 Therefore, can be concluded that object segmentation is essential for NDFs.
Table Success rate NDFs with and without object segmentation
Bottle
Mug
Object Segmentation
Pick
Place
Total
Pick
Place
Total
With Segmentation
0.94
0.86
0.81
0.97
0.72
0.70
Without Segmentation
0.00
0.00
0.00
0.37
0.00
0.00
Difference
0.94
0.86
0.81
0.60
0.72
0.70
A)
3)
D)
Figure 14: A) NDFs successfully infer the pick position of well-segmented mug point cloud. B) NDFs fail to successfully infer the pick position of an unsegmented mug point cloud, C) NDFs suc- cessfully infer the pick position of well-segmented bottle point cloud. D) NDFs fail successfully infer the pick position of an unsegmented bottle point cloud. The black dots are the query points attached to the gripper.
32


Reviews:
[{"review_id": "wC9uL-kG1Dq", "reviewer": "Reviewer_W127", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the author explores the direction of how to utilize SE(3)-equivariance as a novel inductive bias to facilitate sample efficiency. Specifically, it focuses on the same robotic manipulation task as previous work, Neural Descriptor Field. The authors derive an energy function based on the representation of the Lie group, which can be used to characterize the distance/discrepancy between SE(3) pose in the demonstration and the current state. By approximating the gradient of the function with sampling, the SE(3)-equivariance demonstration following problems are formulated as an energy minimization problem. Experiments on the object placement task show that the proposed method outperforms the modified-Transporter Network and Invariant Descriptor Fields method. \n", "summary_of_the_review": "I feel that the problem solved in this paper is well-motivated, and the problem formulation looks good to me. However, the implementation choice can be improved to better support the theoretical power of EBM. The difficulty level of the experiments is on-par with previous work in this field. So I recommend a weak acceptance for this paper but the rating may be lower or higher after the rebuttal phase. ", "strength_and_weaknesses": "This paper is trying to solve a very important problem: how to encode the SE-(3) equivariance property into the network as a powerful inductive bias. It is an important technique for many 3D vision and robotics applications. Although the TransporterNetwork shows that such kind of network design is helpful for manipulation tasks, it still relies on the translational invariance of CNN but does not explore this problem in a more general SE(3) space and does not deal with the hard rotation part. The authors leverage the representation theory of third-order Lie groups to define the bi-equivariant energy using the probability distribution on SE(3). The formulated equivariant descriptor field can then be used in a key-query system, as previous NDF. I feel that the energy-based model in 4.2 makes much sense. But the implementation choice of using Dirac delta functions to shape the query density makes this design not so useful in many applications where continuous parameterization is more preferred.\n\nOne weakness of this paper is the gradient of EBM, which requires a special sampling strategy described in Section 5. It seems that this design makes the overall complexity of the method very high in applications and leads to a very long inference time. As the author discussed in the final section, this method cannot solve problems at the trajectory level. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "uFLfL4cyFfc", "reviewer": "Reviewer_8RUM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a SE3-equivariant EBM with high sample efficiency and a competent generalization capability. ", "summary_of_the_review": "This work introduces a SE3 equivariant formulation for robot manipulations that is fully differentiable, and empirically verifies the sample efficiency, generalization capability and robustness of the proposed method. The tasks being evaluated, however, are highly restricted. The work has a nice theoretical guidance and more empirical evidence that it works well in practice would improve the contribution of this work. \n\n--- Post Rebuttal ---\n\nI believe the formulation of this work can inspire further research, and it has empirical support following the evaluation protocol from the literature. I've raised my score accordingly. ", "strength_and_weaknesses": "Strength\n* The proposed method is well-motivated theoretically, and its practical instantiation incorporatoins several components to accomendate the innate instability and intractability in the theoretical formulation. \n* The paper offers an analysis on the generalization capability and robustness of the proposed method under different unseen conditions. \n\nWeaknesses\n* The main concern is that the empirical verification is very limited. The tasks being evaluated are restricted to one bug-hanging task and one stick-to-tray task. It is unclear whether the proposed method will encounter practical problems in general.\n* It will be helpful if the paper can provide an analysis on how much demonstration is needed for the method to succeed, and how the learning curve saturates with respect to the amount of demonstration data available.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mTKQXFWIzb6", "reviewer": "Reviewer_y6Vo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents SE(3) equivariant energy-based models for sample-efficient visual robotic manipulation from point clouds. It also provides theoretical conditions for bi-equivariant energy-based models, along with sampling strategies over the SE(3) manifold. Experiments on simple simulated pick-place and stick-in-tray tasks suggest the feasibility of  generalization to unseen target poses and unseen target object instances of the same category.", "summary_of_the_review": "SE(3) equivariant energy-based models are an important direction of research for robot manipulation. The paper adequately formalizes the problem. Initial results are promising, and would need more experiments to ground the claims of the paper and establish its usability for visual robotic manipulation.", "strength_and_weaknesses": "- The paper deals with an important problem in robot manipulation. Learning equivariant and background invariant policies is key to visual robotic manipulation. Some recent work has addressed this problem empirically, but formal treatment of the problem has been widely neglected.\n\n- The work builds upon Neural Descriptor Fields (NDFs) work in Simeonov et al., 2021. The authors claim that point cloud input needs to be segmented from the background in NDFs, and requires pre-training on an excessive dataset of objects. But do not validate these claims with grasping in clutter and/or multiple objects or challenging objects.\n\n- Comparison with Dense object descriptors and grasping methods for visual robotic planning methods would further improve the quality of the paper. At this point, it is hard to comment on the efficacy of the approach in comparison to the state-of-the-art in visual robotic manipulation.\n\n- Grasping in clutter and/or challenging objects would further develop the limitation of the presented sampling methods.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hNlP4QUNcJF", "reviewer": "Reviewer_R4yz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed Equivariant Descriptor Fields (EDF), a generalized, incredibly low-data-needed, fully end-to-end-training model which is SE(3)-equivariant (this means apply 3D translation and rotation to input will results to the same as it is applied to the output).\n\nAuthors utilized the representation theory of Lie groups and designed a novel SE(3)-equivariant energy-based model. It has high sample efficiency. It can also be learned from scratch using a few data (5-10 demonstrations). The EBM used MCMC-sample-based MLE to train.\n\nThey experiment EDF on 6-DoF robotic manipulation tasks (pick-and-place cups and etc.). Compared with TN (an SE(2)-equivariant method) and IDF (an SE(3)-invariant method, similar to NDF), EDF showed its sample efficiency and generalizability.", "summary_of_the_review": "This paper designed a good method to solve an important problem. Multiple advantages are presented but not validated. It lacks supportive discussions and experiments.\n\nSince I do not have a very strong math background, I am not able to access the correctness and novelty of this paper from the perspective of math. If a math guy evaluated this paper as perfect to be accepted. Just ignore my weakness assessment. \n", "strength_and_weaknesses": "- Strength\n  - The paper is well written, at least non-math part. \n  - The problem formulation is good and the designed method seems technically sound.\n- Weaknesses\n  - As an non-math background reader, I cannot find supportive discussion on why this new approach can achieve such advantages (very few training examples and good generalization method). \n  - Even though it is normal practice as a theoretical paper, the experiment is too naive.\n    - Only two tasks are tested, it is hard to prove anything.\n    - With that number of data (5-10) and smaller than 5% difference, I do not think any conclusion can be made. The author should at least do the same experiments multiple times to provide confidential interval for accuracy.\n    - You should provide more background research related to EBM, especially EBM on controlling. Such as [1] and its following paper; [2].\n\n--- after rebuttal --- \nreadability improved. Experiments added. Background added. Problem addressed. I raise the score.\n\n- Question:\n  - Time cost is not provided but EBM must be incredibly slow. Is it correct? Is efficiency a bottleneck for scalability? Both in training and applying.\n  - IDF is SE(3)-invar. , NDF is SE(3)-equivar. Why IDC can  be understood as an end-to-end trainable version? Appendix E do not have a clear explanation.\n\n[1] Xie, Jianwen, et al. \"A theory of generative convnet.\" International Conference on Machine Learning. PMLR, 2016.\n[2] Y. Xu, et al.  \"Energy-Based Continuous Inverse Optimal Control,\" in IEEE Transactions on Neural Networks and Learning Systems, doi: 10.1109/TNNLS.2022.3168795.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "-5Ca6o1HYEd", "reviewer": "Reviewer_tB6e", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an end-to-end visual learning method for 3D robotic manipulation. The authors construct an SE(3)-equivariant energy-based model to learn end-to-end from limited demonstrations without prior knowledge. The proposed Equivariant Descriptor Fields could be generalized to unseen poses, instances, and target objects.", "summary_of_the_review": "This paper has proposed an EBM to learn end-to-end 3D robotic manipulation from limited visual demonstrations. The major concern from me about the effectiveness of the proposed method is the experiment section, where the author mainly conducts the experiment in a simulation environment and merely in the pick&place task of a mug. Since the author has claimed no pre-segmented points cloud info is required in the proposed method, the author needs to show the generalization on different manipulation objects. Furthermore, in the compared methods, i.e., Transporter Networks (TN) or Neural Descriptor Fields (NDF), the authors conduct the experiments in multiple situations to demonstrate the effectiveness of their proposed methods, such as 10 different manipulation tasks in TN and real-world experiment in NDF. Please refer to the previous comments for my concerns.\n\n---Post Rebuttal----\n\nFirst, I'd like to move my rating from 3->5 due to the significant revision of the paper and the updated experiments. However, as I am not an expert in the field of robotics, I still have concerns about the experiment part. Although the author has updated the experiments with more objects, I am not sure about the meaning of the comparison result. Are there any other metrics other than success rate and computation time to compare different models? Are they any other methods to compare in the pick-and-place task except for SE(3)-TN? \n", "strength_and_weaknesses": "This paper proposes a data-efficient model that could learn end-to-end for robotic manipulation from visual demonstrations. Compared with previous SE(3) - equivariant models, the proposed method could learn from only a few demonstrations without pre-training. The authors have proposed an interesting solution to solve the sample efficiency problem, yet there are still concerns about the experiment part to demonstrate the proposed method. Please refer to the below comments for my questions about this paper.\n\nMethod:\n1. According to the paper, the major contribution is that the authors propose an EBM to model the descriptor field. However, the related content only takes a small part, and most materials in section 2 are about the description of SE(3)-equivalent. The author should emphasize more on the proposed contribution, while the less relevant concept and formulas could be left in citations.\n\n2. For the proposed EDF, there are related works about MCMC sampling [1][2] and relevant learning methods for point clouds, such as [3]. It is essential to introduce and compare these related works since the EDF is the major contribution in this paper.\n\nExcept for the proposed method, I also have some concerns about the experiments.\n\n1. The author has only shown the experiment in the task of mug hanging. However, as the author has mentioned about Neural Descriptor Fields (NDF), there are additional manipulation targets used in the simulated environment, i.e., bowl and bottle. Why not the author conduct these experiments to show the generalization of the proposed method?\n\n2. Although the author has claimed that the model could learn without prior segmentation from the background, the experiment is conducted in a very simple environment. This raises concerns if the EDFs (or IDFs) could perform as expected when the environment becomes complicated, such as introducing random distracting objects.\n\n3. Although the author has mentioned that NDF requires pre-training, it is self-supersized in an auto-encoding way that requires no expert labels. Therefore it is of great interest to see the comparison between NDF and the proposed method since both models could learn from a few demonstrations. This should be listed in the experiment section.\n\n4. In the first paragraph in section 6, the author should mention the computation architecture when mentioning the processing time. Additionally, since the author has claimed the learning efficiency of NDF, it is worthwhile to compare the time efficiency in the experiments.\n\n\nWriting:\n1. In the experiment section, the author mix uses different tenses in the experiment settings and descriptions (also found in other sections). \n2. In the last paragraph of section 3, Transporter networks ->  Transporter Networks\n3. Background related to EBM is missing in section 2.\n\n[1] Zhu, Song Chun, and David Mumford. \"Grade: Gibbs reaction and diffusion equations.\" Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271). IEEE, 1998.\n\n[2] Xie, Jianwen, et al. \"A theory of generative convnet.\" International Conference on Machine Learning. PMLR, 2016.\n\n[3] Xie, Jianwen, et al. \"Generative pointnet: Deep energy-based learning on unordered point sets for 3d generation, reconstruction and classification.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster