Fourier Transporter: Bi-Equivariant Robotic Manipulation in 3D
Authors: Haojie Huang
Owen Lewis Howell
Dian Wang
Xupeng Zhu
Robert Platt
Robin Walters
Venue: ICLR 2024 poster (2024)
Keywords: Robot Learning
Geometric Deep Learning
Robotic Manipulation
Equivariant deep learning
Primary area: applications to robotics, autonomy, planning
TL;DR: Sample efficient policy learning for manipulation in 3D

Abstract: Many complex robotic manipulation tasks can be decomposed as a sequence of pick and place actions. Training a robotic agent to learn this sequence over many different starting conditions typically requires many iterations or demonstrations, especially in 3D environments. In this work, we propose Fourier Transporter ($\text{FourTran}$), which leverages the two-fold $\mathrm{SE}(d)\times\mathrm{SE}(d)$  symmetry in the pick-place problem to achieve much higher sample efficiency. $\text{FourTran}$ is an open-loop behavior cloning method trained using expert demonstrations to predict pick-place actions on new configurations. $\text{FourTran}$ is constrained by the symmetries of the pick and place actions independently. Our method utilizes a fiber space Fourier transformation that allows for memory-efficient computation. Tests on the RLbench benchmark achieve state-of-the-art results across various tasks.

Published as a conference paper at ICLR 2024
FOURIER TRANSPORTER:
BI- BI-EQUIVARIANT ROBOTIC MANIPULATION IN 3D
Haojie Huang, Owen L. Howell", Dian Wang", Xupeng Zhu", Robert Platt Robin Walters Equal Contribution, i Equally Advising Northeastern University, Boston, MA 02115, USA
Tauang.haoj) howell.c; weng.dian; zhu.xup; s.welters; r.platt/@northhesstenn.
ABSTRACT
Many complex robotic manipulation tasks can be decomposed as a sequence of pick and place actions. Training robotic agent to learn this sequence over many different starting conditions typically requires many iterations or demonstrations, especially in 3D environments. In this work, we propose Fourier Transporter (FOURTRAN) which leverages the two-fold SE(d) x SE(d) symmetry in the pick-place problem achieve much higher sample efficiency. FOURTRAN is an open-loop behavior cloning method trained using expert demonstrations to predict pick-place actions on new configurations. FOURTRAN is constrained by the sym- metries of the pick and place actions independently. Our method utilizes a fiber space Fourier transformation that allows for memory-effiiiet computation. Tests on the RLbench benchmark achieve state-of-the-arr results across various tasks.
INTRODUCTION
Imitation learning for manipulation tasks in SE(3) has emerged as key topic in robotic learn- ing. Imitation learning attractive because of its practicality. In contrast to reinforcement learning wherein training happens via a period of autonomous interaction with the environment (which can potentially damage the robot and the environment), imitation learning requires only human demon- strations of the task to be performed which is often safer and easier to provide. Sample efficiency is critical here; the robot needs to learn to perform a task without requiring the human to provide an un- due number of demonstrations. Unfortunately, many current state-of-the-a methods are not sample efficient. For example, even after training with one hundred demonstrations, methods like PerAct and RVT still struggle to solve standard RLBench tasks like STACK WINE STACK CUPS (Shrid- har et al., 2023; Goyal et al., 2023).
Why is sample efficiency such challenge in three dimensions? A big reason is that these tasks are defined over SE(3) action spaces where the policy must output both a 3D position and SO(3) orientation. The orientation component here is a particular challenge for robot learning because SO(3) is not Euclidean and standard convolutional layers, which lack geodesic properties, are not well-adapted for SO(3) convolutions. Discretization of the group SO(3) also difficult. For exam- ple, a grid sampling of 1000 different orientations in SO(3) still only realizes an angular resolution of 36", which is insufficient for many manipulation tasks. Instead, existing methods, e.g. Shridhar et al. (2023) or Goyal et al. (2023), often fall back on generic self-attention layers that not take advantage of the geometry of SO(3).
One approach to this problem is to leverage neural-network policy models that are symmetric in SO(2) or SO(3). This has been explored primarily in SO(2) and SE(2) settings, e.g. Wang et al. (2021) and Jia et al. (2023), where significant gains in sample efficiency have been made using policy networks that incorporate steerable convolution kernels (Cohen & Welling, 2017). In 8O(3), symmetric models have been mainly applied to pose or descriptor inference, e.g. Klee et al. (2023) and Ryu al. (2022), but not directly policy learning. Moreover, most works do not address the dual symmetry present in many pick and place problems, sometimes referred to as bi-equivariance (Ryu et al., 2022), where the pick-place action distribution transforms symmetrically (equivariantly)
Project website:
page

---

Published as a conference paper at ICLR 2024
when transformation is applied independently to either the pick or the place pose. This is illustrated in Figure 1. Independent rotations of the gear (91) and the slot (g2) result a change (a' = I2agi ') in the requisite action needed to perform the insertion. While bi-equivariance in policy learning has been studied in S0(2) (Huang et al., 2023a), models for encoding SO(3) bi-equivariance in a general policy learning setting have not yet been developed.
This paper proposes Fourier Transporter (FOURTRAN), an approach to modeling SE(3) bi- equivariance using 3D convolutions and Fourier representation of rotations. Unlike existing meth- ods, e.g. Equivariant Descriptor Fields (Ryu et al., 2022) and TAX-Pose (Pan et al., 2023), our method encodes (3)3) bi-equivariance inside a general purpose policy learning model rather than relying on point descriptors, which often require sample and optimization during inference. Our key innovation is to parameterize action distributions over SO(3) in the Fourier domain as coefficients of Wigner D-matrix entries. We embed this representation within a 3D translational convolution, thereby enabling us to do convolutions directly SF)(3) without excessive computational cost and with minimal memory requirements. The end result is policy learning model for imitation learning with high sample efficiency and high angular resolution that can outperform existing SE(3) methods by significant margins (Table 1). Our contributions are:
We analyze problems with bi-equivariant symmetry and provide general theoretical solu- tion to leverage the coupled symmetries.
We propose Fourier Transporter (FOURTRAN) for leveraging bi-equivariant structure in manipulation pick-place problems in 2D and 3D.
We achieve state-of the-ar performance on several RLbench tasks (James al. (2020)). Specifically, FOURTRAN outperforms baselines by margin of between six percent (STACK-WINR) and Iwo-hundred percent (STACK-CUPS).
RELATED WORK
Action-centric manipulation. Traditionally, vision-based manipulation policies (Zhu et al., 2014; Zeng al., 2017: Deng al., 2020; Wen al., 2022) often require pretrained vision models to conduct object detection, segmentation and pose estimation, and may struggle with deformable or granular objects. Action-centric manipulation associates each pixel, voxel, or point with a target position of the end-effector, providing an efficient framework that evaluates a large amount of ac- tion with dense output map. Transporter Networks (Zeng et al., 2021; Seita et al., 2021; Shridhar et al., 2022) combine action-centric representations with end-to-end learning, showing fast conver- gence speed and strong generalization ability. However, these end-to- and-to-end learning methods are often limited to 2D tow-down settings and cannot efficiently or accurately solve 3D pick and place tasks.
scene
scere
91 90
45
d' 0200
92
180'
0200
80ane
(a) 2D Bi-Equivariance
(b) 3D Bi-Equivariance
Figure 1: Ilusrtationn of bi-equivariance in 2D (left) and 3D (right). The place action, a' 02a9] symmetric with respect to both the orientation of the object to be picked, g1, and the orientation of the place target, 92-
2

---

Published as a conference paper at ICLR 2024
Recent works have made great progress in extending action-centric manipulation to the full SE(3) action space. Lin et al. (2023) synthesize different views and apply 2D Transporter Networks on those views to realize 3D pick-place. C2FARM (James et al., 2022) represents the workspace with multi-resolution voxel grids and infers the translation in a coarse-to-fine fashion. Transformer based methods like PerAct (Shridhar al., 2023) first tokenize the voxel grids together with a language description of the task and learns a Language-conditionn al., 2021). RVT (Goyal et al., 2023) projects the 3D observation onto orthographic images and uses the generated dense feature map of each image to generate 3D actions. However, they all require a large number of expert demonstrations to learn even simple 3D pick and place skills. Compared with the previous works, our proposed architecture is an end-to-end method that demonstrates a significant increase in sample efficiency in both 2D and 3D pick-place tasks.
policy with Perceiver Transformer (Jaegle
Symmetries and Robot Learning. Robot learning methods can benefit significantly by leveraging the underlying symmetry the task. (Zeng al., 2018; Morrison et al., 2018) show the transla- tional equivariance of Fully Convolutional Network can improve learning efficiency for manipula- tion tasks. Recent works explore the use of equivariant networks (Cohen & Welling, 2016; 2017; Weiler et al., 2018; Weiler & Cesa, 2019; Cesa et al., 2021) to embed 2D rotational symmetries, sulting in a dramatic improvement in sample efficiency. Wang et al. (2021); : Zhu et al. (2022) encode (0( rotational symmetry in manipulation policies and demonstrate better on-robot grasp learning- (Wang et al., 2022b;a; Jia et al., 2023; Nguyen et al., 2023) exploit S0(2) and ((2) symmetries to solve multi-step manipulation tasks with a closed-loop policies. (Huang et al., 2022; 2023a) analyze the bi-equivariant symmetry of pick and place on the 2D rotation groups. However, they are limited to 2D action spaces.
Several works have attempted to utilize SO(3) symmetry in robot learning. Neural Descriptor Fields (NDF) (Simeonov et al., 2022) uses Vector Neurons (Deng et al., 2021) to generate SO(3)- equivariant key point descriptors to define the object's ' pose for pick and place tasks. However, NDF and variations (Chun et al., 2023; Huang et al., 2023b) require well-segmented point clouds and pre-trained descriptor networks. TAX-Pose (Pan et al., 2023) generates -3i3)-invariant dense corre- spondences for pick-place tasks, but is limited to reasoning over two objects. Equivariant Descriptor point features
Field (Ryu et al., 2022) achieves bi-equivariance by encoding the S0(3) with Tensor Field Network (Thomas et al., 2018) and SE(3) Transformer (Fuchs et al., 2020). How- ever, it requires many samples in SE(3) to train and test the energy- model. In comparison, our proposed method generalizes bi-equvariance to both 2D and 3D manipulation pick-and-place problems, infers the pick-and-place distribution over the entire action space with single pass, and utilizes convolution in Fourier space to improve computation efficiency.
BACKGROUND
We provide some background on symmetry and groups, which are used in our method. Please see Appendix C for a more thorough mathematical introduction.
Groups and Representations. In this work, we are primarily interested in the special Euclidean group SE(d) (d) x IRd which includes all rotations and translations of Rd. The discrete SO(2) subgroup Cn [Rot : : 2wi/n, O < i < n. contains rotations by angles which are multiples of 2TT nn. The icosahedral rotation group 16o and octohedral rotation group O2t are finite subgroups of the group SO(3) which give the orientation preserving symmetries of the icosohedron and octohedron and contain 60 rotations and 24 rotations, respectively.
An n-dimensional representation p: G + GL, of a group G assigns to each element E G an invertible n x n-matrix p(g) where for all g.g' E the matrices satisfy p(gg') = p(g)) Different representations of SO(d) or its subgroups describe how different signals are transformed under rotations. We consider several examples. The trivial representation Po: SO(d) + GL, assigns Po(g) = 1 for all g G. i.c. there is transformation under rotation. The standard representation p1 of (d) assigns each group element its standard d x d rotation matrix. For finite groups G, the regular representation Preg acts on RIo|, Label a basis of [G] by {cg : E G}, then Preg (g)) Egh- Both S0(2) and SÓ(3) have irreducible representations, i.e. representations with no non-trivial fixed subspaces, indexed by non-negative integers k E Zzn- The irreducible representations of S3(3) are known as Wigner D-matrices, have dimension (2k 1) (2k 1)
3

---

Published as a conference paper at ICLR 2024
and are denoted D The irreducible representation Pt: of SO(2) is given by 2 x 2 rotation matrices (o) = Rotge.
Steerable Feature Maps. We consider all features to be stecrable feature vector fields f: R R which assign a feature vector f(x) E R to each position x E R An element g E SO(d) acts on steerable p-field f by acting on both the base space Rd by rotating the pixel voxel positions and on the fiber space R (a.k.a,, channel space) by some representation p. We define the action of p(g) (pi (g) (g)- We denote the base space action alone by
g via p on f by (g))())(f(x)) (z())f)(x) f() xx). Note that B Indpp
G-Equivariant Mappings. A mapping F: X + Y is equivariant to a group acting on X by Ind px and Y by Ind if it intertwines the two group actions Ind py (g) px (g)ff. Ag E G.f E X. Equivariant linear mappings, i.e. intertwiners, between spaces of steerable feature fields are given by convolution with G-steerable kernels (Weiler et al., 2018; Jenner & Weiler, 2021). Assume the input field type transforms as Ind and the out- put field type Ind Then by Cohen al. (2019), Theorem 3.3, convolution with the ker- nel K: Rd do Xdin is G-equivariant if and only if its satisfies the steerability constraint, (x) (g) (g)- A complete characterization and explicit parametrization of steerable kernels is given in Lang & Weiler (2021).
Kgg x) = Pout (g)
(O(d) Fourier Transformation. Signals defined over the group SO(d) can be decomposed as limits of linear combinations of complex exponential functions (for SO(2)) or Wigner D-matrices (for SO(3). We refer to the Fourier transform that maps (Oid)asignals to the coefficients of the basis functions as F+ and the inverse Fourier transform as F-1, For a more in depth discussion of the Fourier transform, we refer the reader to Appendix A.3.
METHOD
4.1 PROBLEM STATEMENT
This paper focuses on behavior cloning for robotic pick-and-place problems. Given a set of expert demonstrations that contain a sequence of one or more observation-action pairs (or, ae), the objective is to learn a policy ((ae(or) where the action a, = (apick. aplace) has pick and place components and the observation or describes the current state of the workspace. In 2D manipulation tasks, o, is in the format of a top-down image, and in 3D manipulation tasks, or is a voxel grid. Our model factors the policy p(ac, 0z)
p(ay or) = p(aplace lo,, apick (ppcc))((ppceko))
where p(apick (o)) and p(a lot; (pick) are parameterized as the output of two separate neural networks. The pick action Opick SE(d) and place action Oplace SE(d) are decomposed in terms of translation and orientation (T, R) E SE(d), where T is pixel coordinates (u, v) in 2D or voxel coordinates (i, j,,j, k) in 3D The rotational part of the action R E SO(d) denotes the gripper orientation, which is a planar rotation Rot, in 2D tasks and three-dimensional rotation R E SO(3) in 3D tasks. The pick rotation Rpick SO(d) is defined with respect the world frame and Rplace E (O(d) is the relative rotation between the pick pose and place pose.
4.2 SE(d)-EQUIVARIANT PICK
We first analyze the symmetry of the robotie pick task in SE(d), where (2, indicates 2D or 3D picking. Then, we present our pick network which realizes this symmetry.
Symmetry of the Pick Action fpick. The pick network takes input observation o, and outputs the pick pose probability distribution over (E)(d), i.e., fpick: o + p(apick lot). We represent p(Apick (or) as a steerable field Rd + (30(3) + R} in which the base space determines the pick location and the fiber space (a.k.a,, channel space) encodes the distribution over pick orientations. The distri- bution (3) - R transforms by (pr.gg)p)(s p/g where the subscript L denotes the left-action of g corresponding post-composing with the rotation. A consistent pick network
Fach pixel or voxel corresponds unique (a, y, 2) spatial coordinate in the workspace
4

---

Published as a conference paper at ICLR 2024
sorrelation
10
crop
S:V
Ft
x(c)
RoV(c)
plick
f placc
Figure 2: Architecture of FOURTRAN. Spick first detects task-appropriate pick pose. The crop centered at the pick location is fed to network tb. The lift operation generates stack of rotated features and Fourier transformation F+ is applicd to the channel space of the feature to output the dynamic kernel n(c). The cross correlation is conducted in Fourier space.
should satisfy the pick symmetry relation,
Vgg E SF(d), [gick(g (.)) Indpe (p)ck
Equation illustrates the underlying symmetry of robotic picking problem, i.e., if there trans- formation g E SF(d)
(RHS of Equation 1), the pick pose distribution Ppick should transform accordingly by Ind,z- Specifically, the action (g)) on the base space rotates the pick location and the fiber action p(g) transforms the pick orientation.
Pick Symmetry Constraints. To construct fpick satisfying Equation 1, we parameterize with equivariant convolutional layers. The pick network fpick takes the input observation or pu field and outputs a Pirrep field where Pirrep represents the direct sum of irreducible representations giving the truncated Fourier space representation of functions over SO(d). To be specific, fpick can be encoded as an equivariant U-Net that takes the observation and generates the coefficients of (O basis functions. As result, fpick generates
un-normalized distribution over (0(3) above each voxel coordinate or over SO(2) above each pixel coordinate. The pick pose distribution over SE(d) calculated by jointly normalizing the signal on translation and orientation. The best pick pose apick can be evaluate by "pick = arg max ltt)
4.3 SEEd) x SE(())EQUIVARIANY PLACE
We first clarify the coupled symmetries inside place tasks. Then, we present how FOURTRAN realizes the solution.
Symmetry of the Place Action fplace. After the pick action apick determined, the place network Splace infers the place action Oplace to transport the object to be grasped to the target placement. We assume the object does not move or deform during grasping so that Opick may geometrically represented by an image or voxel patch centered on the pick location?. as shown in Figure Our place network is described
Splace : (c, or) + p(aplace lo:, apick) where p(aplace I0,, apick) denotes the probability that the object grasped at apick in scene should be placed Oplace.
Since the place action is conditioned on the pick action, consistent place network fplace should sat- isfy the following bi-equivariance constraint. For f:: SO(3) - RR, denote (pR(g)s)(h) = s(hg the right action on orientation distributions corresponding to pre-rotation by g The place network fplace should be SE(d) x SE(()-equivariint:
92 SLE(). (glace(91 C, o,) Indpz (92)Pn(9) (placo
Equation states that if 9i E SE(d) on the picked object and g2 SE(d) acts on the observa- tion that contains the placement of interest?, the desired place action will transform accordingly.
"The pick pose can also he geometrically represented by occupied voxel. Strictlyy g2 acts o, 'c the observation excluding the patch, since the picked object is transformed by 91-
5

---

Published as a conference paper at ICLR 2024
Recalling Indp,, acts on the location and orientation, the place orientation Rplace will be transformed p1 (1()) Rplaceo (9r (or and the place location T place will be transformed to pr (92)Tplace-
Realizing place symmetry: Solution. In order to design a network architecture fplace which satisfies Equation 2, we follow the previous works (Zeng et al., 2021; Huang et al., 2022) which treat the placing as a template matching problem and encode Splace with two separate functions ø and x to process or and e respectively. Then ((Aplaco|Ots apick) is computed as the cross-correlation between k(c) and (b(r),
fplace oz) x(c) ((z))
(4)
where * denotes the cross correlation. Specifically. x(c): Rd + Ro Xd dynamic kernel and (o,)): Rd did is a feature map generated from the workspace observation
schematic of our proposed method FOURTRAN is shown in Figure 2. In the top branch, an encoder e uses convolutional layers to map the input observation on to a dense feature map. Both or and ((or) are considered po-fields. the bottom branch, the crop e processed by v, which has the same architecture as é. to generate dense feature map. Then, we lift (h() with finite number to generate stack of rotated feature maps. Specifically, of rotations G = (9. 19: {m define LT as
Vx E EERR D: & (1(((x) = (U(o))=((((T"))) x),, f(9
, [mm x))
(5)
stack of fully rotated signals above each pixel or voxel. We then apply Fourier transform the channel-space. Our dynamic kernel generator ń. is summarized as
<(c) = [+(()((())))
(6)
where F+ denotes the Fourier transform in the channel space. Finally, the cross-correlation between k(() and ((oz) is performed in the Fourier space. Appendix.A.1 shows the pseudocode of the
inference step of FOURTRAN.
Representing (d)d) rotations in Fourier space can achieve high angular resolution and enormously save the computation load. In contrast, directly corseccorg between number of rotated (c)) and (o(o)) independently is impossible when the action space contains a large number of rotations, especially in 3D. For example, 10" discretization along each axis in 3D will result in 36 rotations, which is 46,656 in total.
Realizing place symmetry: Theory. We present general solution to achieve the bi-equivariant symmetries of Equation 3 and analyze how our porposed FOURTRAN satisfies it.
Proposition 1 Equation 4 satisfies the bi-equivariani symmetry stated in Equation 3 if the following constraints hold:
1. (c)) satisfies the equivariant property that (bgg ((g)((().
2. ((oi) satisfies the equivariant property that ō(g 0z) ((g)
3. x(c): Rdd_
Rddo deeiviall is steerable convolational kernel with po-type imput.
The full proof of Proposition I is in Appendix A.2. Intuitively, rotations on the crop and the placement o, result in consistent transformations of the kernel kc(c) and feature map Φ(o.). The steerability of the kernel bridges the two transformations during the cross-correlation.
FOURTRAN satisfies the three constraints listed in Proposition The first and the second constraints hold since va and á are implemented using equivariant convolution layers Cesa et al. (2021). The third constraint is shown in Proposition 2.
Proposition 2 Let
R: R3 LT generates Tso-steerable kernel with regular-tppe oulput and trivial input. The fiber space Fourier transformation [+) approximately an -3tte--tterablle kernel with trivial imput
R be an
feature field. Then, the lifting operator
and outpul type + Enna
== (2l + 1)D
Proposition 2 states that x(c) in Equation is a dynamic steerable kernel generator, which satisfies the third constraint in Proposition 1. The proof of Proposition is derived in Appendix B.

---

Published as a conference paper at ICLR 2024
4.4 SAMPLING ROTATIONS IN COARSE- TO-FINE FASHION
Coarse to fine sampling methods (James et al., 2022) are numerically efficient importance sampling schemes that only sample in regions of dense signal. The pick network Spick and place network f place output the (O(d) distribution for each pixel voxel in the format of coefficients of (O(d) basis functions. When taking the inverse Fourier transform, we can sample a finite number of rotations for each element. However, the memory requirements for the voxel grid are cubic in the resolution of the grid and therefore limit the the number of sampled rotations for each voxel. To solve this, we first sample a small number of rotations and evaluate the best coarse pick action (Tpick: Rpick) and best place action pTaace, R. Then, we locate the best pick and place location Tpick and place and generate higher-order fourier signals with equivariant layers from the corresponding hidden features. Finally, we conduct second sampling to generate a large number of fine SO(3) rotations for Tpick and T place and calculate the best fine orientation pick and Ricck The robot will receive commands of (Tpick> Ritck): (Tplecer lfgl) in the same time step and execute the actions.
EXPERIMENTS
5.1 MODEL ARCHITECTURE DETAILS
In FOURTRAN, fpick is a single convolutional network and Splace is composed of two equivariant convolutional networks, è and sb We implement them 18-layer residual networks with a U- Net (Ronneberger et al., 2015b) as the backbone. The U-Net has 8 residual blocks and each block contains two equívariant convolution layers and one skip connection. The first layer maps the trivial- representation O, to regular representation, and the last equivariant layer transforms the Preg -type feature to trivial representation for & and ó and to a direct sum of irredueible representations for Spick. We use pointwise ReLU activations (Nair & Hinton, 2010) inside the network.
For 3D FOURTRAN, we use O24 regular representations in the hidden layers and lift the 3D crop features with a set of 384 approximately uniformly sampled rotations of SO)3). We show that this random lifting approximately generates a steerable kernel in the Appendix B.3. To decode the Pirrop feature, we first coarsely sample 384 rotations for every voxel and later conduct a finer sampling of 26244 rotations on a subregion. The maximum order Emaz of the Winger-D matrices in the coarse and fine sampling levels is mma 2 and Emaz 4. respectively. In 2D FOURTRAN, C4 group in the intermediate layers of the three 2D convolution networks and use the Coo group to lift (c). After the Fourier transform, the frequency is truncated with the max order of 37.
select the
5.2 3D PICK-PLACE
3D pick-place tasks are difficult due to the large observation and action spaces. We conduct our primary experiments on five tasks shown in Figure 3 from RLbench (James et al., 2020) and compare with three strong baselines (James al., 2022; Shridhar et al., 2023; Goyal et al., 2023).
3D Task Description. We choose the five most difficult tasks from James et al. (2020) to test our proposed method. Stack-hlocks: It consists of stacking two blocks of the red color the green platform. Stack-cups: In stack-cups, the agent must stack two blue cups on top of the red color cup. Stack-wine: The agent must grab the wine bottle and put it on the wooden rack at one of three specified locations. Place-cups: The agent must place one mug on the mug holder. This is a very high precision task where the handle of the mug has be exactly aligned with the spoke of the holder for the placement to succeed. Put-plate: The agent is asked to pick up the plate and insert it between the red spokes on the dish rack. This is also a high- sigecpreciiion task. The different 3D tasks are shown graphically in Figure 3. Note that the object poses are randomly sampled at the beginning of each episode and the agent needs to learn to generalize to novel object poses.
3D Baselines. Our method is compared against three state-of-the-aa baselines: C2FARM-BC (James et al., 2022) represents the scene with multi-resolution voxels and infers the next key-frame action using coarse-to-fie totnse-t--fneeeeee scheme. PerAct (Shridhar et al., 2023) is the sfate-of-the-ard multi-task behavior cloning agent that tokenizes the voxel grids together with a language description of the task and learns a waaguuge-conditinne policy with Perceiver Transformer (Jaegle et al., 2021). RVT (Goyal et al., 2023) projects the 3D observation onto five orthographic images and uses the dense feature map of each image generate 3D) actions.
7

---

Published as a conference paper at ICLR 2024
Figure 3: 3D pick and place tasks. From left to right the tasks are: Stack-blocks, Stack- Suck- Stack- Wine, Place- Cups, and Put-Plate. The top row shows the initial scene and the bottom shows the completion state.
Model
# demos stack-blocks stack-cups stack wine place-cups put-plate
FOURTRAN (ours)
4
2.6
89.3
8
16
FOURTRAN (ours)
76
88
100
10.6
32
FOURTRAN (ours)
10
80
92
100
26.6
66.6
RVT-Single
10
9.3
13.3
33.3
1.3
54.6
PerAct-Single
10
52
1.3
12
1.3
8
C2FARM BC Bi-gie
10
36
0
1.3
1.3
RVT-Multi
100
28.8
26.4
91.0
4.0
PerAct-Muus
100
36
0
12
0
STFARM-BC-MuL
100
0
0
0
Discrete Expert
100
92
100
90.6
65.3
Table 1: Performance comparisons on RL benchmark. Success rate (%) on 25 tests v.s. the number of demonstration episodes (1, 5, 10) used in training. Even with only demos, our method is able to outperform existing baselines by significant margin.
Training and Metrics. We train method with {1, 10) demonstrations and train the baselines with 10 demonstrations on each task individually. The single-task versions of the baselines are denoted as 'Single'. All methods unseen configurations every 5K steps. Each evaluation is averaged over evaluation seeds, and we report the best evaluation across the training process. In favor of the baselines, results of multi-task versions of the baselines trained on 16 different tasks of RLbench with 100 demonstrations per task from Goyal et al. (2023), denoted '-Multi', Please note put-plate task is not covered in Goyal et al. (2023). To measure the effects of discretization error and path planning, we also report the expert performance in the discrete action space used by our method.
trained for 15K SGD steps, and evaluate them 25
also include the
3D Results. We report the results of all methods in Table 1. Several conclusions can be drawn from Table 1: 1) FOURTRAN significantly outperforms all baselines trained with 10 demos on all the tasks. 2) For tasks with high-precision requirement, e.g., stack-cups, FOURTRAN keeps high success rate while all the basclines fail to learn good policy. 3) FOURTRAN achieves better sample efficiency, and with {1,5) 5} demonstrations, it can outperform baselines trained with hundreds of demonstrations.
5.3 2D
We further evaluate the ability of FOURTRAN to solve precise pick-place tasks in 2D where the action space is (u, v, 0), i.c., æ, y translations and top-down rotation. adopt three tasks shown in Figure 4 from the Ravens Benchmark (Zeng et al., 2021).
2D Task Description. block-insertion: The agent must pick up an L-shape block and place into an L-shaped fixture; assembling-kits The agent needs to pick 5 shaped objects (randomly sampled with replacement from set of 20) and fit them to corresponding silhouettes on a board. sweeping- piles: The agent must push piles of small objects (randomly initialized) into desired target goal
8

---

Published as a conference paper at ICLR 2024
-onec-inseetion-- assembling-kin kits-10
seeeping-piles--
Model
15"
7.5
15"
7.5"
FOURTRAN (ours)
100
100
86,2
78.0
99.8
Equivariant Transporter
100
98.0
85.0
76.0
100
Transporter Net
100
88.0
80.0
64.0
90.4
Table 2: Performance comparisons on 2D tasks. Success rate (%) on 100 tests. Results for both low-resolution (15") reward and high-resolution reward (7.5").
zone on the tabletop marked with green boundaries. Detailed task settings and descriptions can be found in Appendix A.8.
2D Baselines. We com- pare our method against two strong baselines.
Trans- porter Net (Zeng et al., 2021) implements and v with ResNet-43 without equivari- ant convolutional layers. It lifts the image crop to Cn before feeding it the v net- work. Equivariant Transporter (Huang et al., 2022) may be considered variation of our proposed method with Pu-input and Output steerable kernel. is Cn x C, equivariant. Since tasks of Ravens Benchmark use symmetric suction gripper, all outputs of picking angle 0 are equivalent. The picking angle 0 is thus a nuisance variable and our pick network outputs trivial-type features (instead of regular or higher-order irreducible features).
Figure 2D pick and place task descriptions. Left: Block-insertin
task. Center: Assembling kits task. Right: Sweeping-pite task.
Training and Metrics. We train each model with 10 expert demonstrations and measure the per- formance with the two reward functions. The low-resolution reward function credits the agent for translation and rotation errors relative to the target pose within high-resolution reward function tightens the threshold to = 0.5cm and 7.5
Iem and w = 15" and the
Results. Table shows the performance of all models trained with 10 demonstrations for 10K steps. All tests are evaluated on 100 unseen configurations. First, FOURTRAN and Equivariant Transporter realize bi-equivariance and achieve higher success rate than Transporter Net on single- step tasks and multi-step tasks. Second, as the criteria tightens from a 15" rotation threshold 7.5" rotation threshold, FOURTRAN maintains performance better than others. This indicates that the SO(2) SO(2) equivariance of FOURTRAN is more precise.
CONCLUSION
In this work, we propose the FOURTRAN architecture for pick and place problems. Similar to previous pick and place methods (Ryu et al., 2022: Huang et al., 2023a), FOURTRAN leverages coupled SE(d) x SE(d)-symmetries inherent in pick-place tasks. FOURTRAN uses novel fiber space Fourier transform method to construct a bi-equivarient architecture in a memory efficient manner. The of Fourier space convolutions allows architecture to process high resolution features without the need for extensive sampling. We evaluate our proposed architecture on various tasks and empirically demonstrate that our method significantly improves sample efficiency and success rate. Specifically, in the two-dimensional case, FOURTRAN achieves better performance than existing SOTA methods (Huang al., 2023a; Zeng al., 2021) on the Ravens benchmark. For citre-ddimesiooe pick and place, our method achieves SOTA results on a subset of tasks in the James al. (2019) benchmark by significant (in some cases up two-hundred percent) margin. This empirical success establishes FOURTRAY as a powerful architecture for pick-place tasks.
One limitation of the formulation manipulation tasks this paper that relies on open-loop control and does not take path planning and collision awareness into account. Moreover, the cur- rent model is limited to a single-task setting, and extending to a multi-task, language-conditioned equivariant agent is an important future direction. Lastly, this paper considers only robotic ma- nipulation problems, while the bi-equivariant architecture proposed here may have uses outside of robotic manipulation. Specifically, various binding tasks in biochemistry, like rigid protein-ligant interaction (Ganea et al., 2022), and point cloud registration (Huang et al., 2021) have the same bi-equivariant symmetry as pick-place problems.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
Haojie Huang, Dian Wang. Xupeng Zhu, Rob Platt and Robin Walters were supported in part by NSF 1724257, NSF 1724191, NSFI763878, NSF 1750649, NSF 2107256, NSF 2134178 and NASA 80N5SC19K1474, Owen Howell was supported by the NSF-GRFP. Dian Wang was also funded by the JPMorgan Chase PhD fellowship.
REFERENCES
T. Ceccherini -itheritiii,l,,rteiin,, F. Scarabotti, and F. Tolli. Harmonic Analysis on Finite Groups: Rep- resentation Theory. Gelfand Pairs and Markov Chains. Cambridge Studies in Advanced Math- ematics. Cambridge University Press, 2008. ISBN 9780521883368. URL https: /books.
google homm/bookse cccm///okkks26fff/77nnAAACA 6
Gabriele Cesa, Leon Lang, and Maurice Weiler. program to build e (n)-equivariant steerable cnns. In International Conference on Learning Representations, 2021.
Ethan Chun, Yilun Du, Anthony Simeonov, Tomas Lozano-Peree and Leslie Kaelbling. Local ral descriptor fields: Locally conditioned object representations for manipulation. arXiv preprint arXiv:2302.03553. 2023.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- machine learning, pp. 2990-2999. PMLR, 2016.
Taco Cohen and Max Welling. Steerable cnns. In International Conference on Learning Repre- sentations, 2017.
Taco Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns homo- geneous spaces. Advances in neural information processing systems, 32, 2019.
Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J Guibas. Vector neurons: A general framework for so (3)-equivariant networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12200-12209, 2021.
Xinke Deng, Yu Xiang, Arsalan Mousavian, Clemens Eppner, Timothy Bretl, and Dieter Fox. Self- supervised 6d object pose estimation for robot manipulation. In 2020 IEEE International Confer- ence on Roboties and Automation (ICRA), pp. 3665-3671. IEEE, 2020.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se (3)-transformers: 3d roto- translation equivariant attention networks. Advances in Neural Information Processing Systems, 33:1970-1981, 2020.
Octavian-Eugen Ganea, Xinyuan Huang, Charlotte Bunne, Yatao Bian, Regina Barzilay, Tommi Jaakkola, and Andreas Krause. Independent se(3)-equivariant models for end-to-end rigid protein docking, 2022.
Ankit Goyal, Jie Xu. Yijie Guo, Valts Blukis, Yu-Wei Chao, and Dieter Fox. RVT: Robotic view transformer for 3d object manipulation. In 7th Annual Conference on Robor Learning, 2023. URL /ttpps:/ooet //tp::////ppeeevieee net / forum? doOtkkt.
Owen Howell, David Klee, Ondrej Biza, Linfeng Zhao, and Robin Walters. Equivariant single view pose prediction via induced and restricted representations, 2023.
Haojie Huang, Dian Wang, Robin Walters, and Robert Platr. Equivariant Transporter Network. In Proceedings of Robotics: Science and Systems, New York City, NY, USA, June 2022. doi: 216607/RSS.2022XX1111
Haojie Huang, Dian Wang, Arsh Tangri, Robin Walters, and Robert Platt. Leveraging symmetries in pick and place. arXiv preprint arXiv:2308 07948, 2023a.
Xiaoshui Huang, Guofeng Mei. Jian Zhang, and Rana Abbas. A comprehensive survey on point cloud registration. arXiv preprint arXiv:2103.02669 2021.
10

---

Published as a conference paper at ICLR 2024
Zeyu Huang, Juzhan Xu, Sisi Dai, Kai Xu, Hao Zhang, Hui Huang, and Ruizhen Hu. Nift: Neural interaction field and template for object manipulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp.
IEEE, 2023b.
Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin lonescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795, 2021.
Stephen James, Marc Freese, and Andrew J Davison. Pyrep: Bringing v-rep to deep robot learning. arXiv preprint arXiv: 176, 2019.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 2020.
Stephen James, Kentaro Wada, Tristan Laidlow, and Andrew J Davison. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 19739-13748, 2022.
Erik Jenner and Maurice Weiler. Steerable partial differential operators for equivariant neural net-
works. arXiv preprint arXiv:2106. 1X11:2,06.10163, 2021.
Mingxi Jia, Dian Wang, Guanang Su, David Klee, Xupeng Zhu, Robin Walters, and Robert Platt. Seil: cimulation-augmenteee equivariant imitation learning. In 2023 IEEE International Confer- ence on Robotics and Automation (ICRA), pp. 1845-1851. IEEE, 2023.
David Klee, Ondrej Biza, Robert Platt, and Robin Walters. Image to sphere: Learning equivariant features for efficient pose prediction. In The Eleventh International Conference on Learning Representations, 2023. URL https: httllhttt/ppppreeeeeeeeeee //wp//ppwee.w....c//eee net/ forum? serrii=__boe @ede==pppprrPtt1
Leon Lang and Maurice Weiler. A wigner-eckart theorem for group equivariant convolution kernels, 2021.
Yen-Chen Lin, Pete Florence, Andy Zeng, Jonathan Barron, Yilun Du, Wei-Chiu Ma, Anthony Simeonov, Alberto Rodriguez Garcia, and Phillip Isola. Mira: Mental imagery for robotic affor- dances. In Conference on Robot Learning, pp. 1916-1927. PMLR, 2023.
Douglas Morrison, Juxi Leitner, and Peter Corke. Closing the loop for robotic grasping: A real- time, generative grasp synthesis approach. In Proceedings of Robotics: Science and Systems, Pittsburgh, Pennsylvania. June 2018. doi: 2.5607/RSS.2018.XIV.2
Kieran Murphy, Carlos Esteves, Varun Jampani, Srikumar Ramalingam, and Ameesh Makadia. Implicit-pdf: con-parametric representation of probability distributions on the rotation manifold, 2022.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In leml, 2010.
Hai Huu Nguyen, Andrea Baisero, David Klee, Dian Wang, Robert Platt, and Christopher Am- ato. Equivariant reinforcement learning under partial observability, 7th Annual Conference on Robot Learning, 2023. URL. https:: openneviee net forum? d=AnDDMQgM77-
Chuer Pan, Brian Okorn, Harry Zhang, Ben Eisner, and David Held. Tax-pose: Task-specific cross-pose estimation for robot manipulation. In Conference on Robot Learning, pp. 1783-1792. PMLR, 2023.
Eric Rohmer, Surya PN Singh, and Mare Freese. V-rep: A versatile and scalable robot simulation framework. In 2013 IEEE/RSJ international conference on intelligent robots and systems, pp. 1321-1326. IEEE, 2013.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation, 2015a.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241, Springer, 2015b.

---

Published as a conference paper at ICLR 2024
Hyunwoo Ryu, Hong-in Lee, Jeong- Lee, and Jongeun Choi. Equivariant descriptor fields: Se equiivarai energy- based models for end-to-end visual robotic manipulation learning. arXiv preprint arXiv:2206.0883 08321, 2022.
Daniel Seita, Pete Florence, Jonathan Tompson, Erwin Coumans, Vikas Sindhwani, Ken Gold- berg, and Andy Zeng. Learning rearrange deformable cables, fabrics, and bags with goal- conditioned transporter networks. In 2021 IEEE International Conference on Robotics and Au- tomation (ICRA), pp. 4568 4575. IEEE, 2021.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference Robot Learning, pp. 894-906. PMLR, 2022.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pp. 785-799. PMLR, 2023.
Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se (3)-vquvvariant object rep- resentations for manipulation. In 2022 International Conference on Robotics and Automation (ICRA), pp. 6394 6400. IEEE, 2022.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivariaat neural networks for 3d point clouds. arXiv preprint arXiv: 2018.
Dian Wang, Robin Walters, Xupeng Zhu, and Robert Platt. Equivariant learning in spatial action spaces. In 5th Annual Conference on Robot Learning, 2021.
Dian Wang, Mingxi Jia, Xupeng Zhu, Robin Walters, and Robert Platt. On-robot learning with equivariant models. In 6th Annual Conference on Robot Learning, 2022a.
Dian Wang, Robin Walters, and Robert Platt. SO(2)-equivariant reinforcement learning. In The Tenth International Conference on Learning Representations, 2022b.
Maurice Weiler and Gabriele Cesa. General e (2)-equivariant steerable cnns. Advances in Neural Information Processing Systems, 32, 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. Advances in Neural Informa- fion Processing Systems, 31, 2018.
Bowen Wen, Wenzhao Lian, Kostas Bekris, and Stefan Schaal. You only demonstrate once: Category-level manipulation from single visual demonstration. arXiv preprint arXiv:2201.12716, 2022.
A. Zee. Group Theory in Nutshell for Physicists. In a Nutshell. Princeton University Press, 2016. ISBN 9780691162690. URL https:://// httpp:://boottt //bookss///ok....// google. com/books?id- FWKujGEACAAJ.
Andy Zeng, Kuan-Ting Yu, Shuran Song, Daniel Suo, Ed Walker, Alberto Rodriguez, and Jianxiong Xiao. Multi-view self-supervised deep learning for 6d pose estimation in the amazon picking challenge. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 1386- 1383. IEEE, 2017.
Andy Zeng, Shuran Song, Kuan-Ting Yu, Elliott Donlon, Francois R Hogan, Maria Bauza, Daolin Ma. Orion Taylor, Melody Liu, Eudald Romo, et al. Robotic pick-and-place of novel objects in clutter with aulti-affordance grasping and cross cross-domain image matching. In 2018 IEEE interna- tional conference robotics and automation (ICRA), pp. 3750-3757. IEEE, 2018.
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani. et al. Transporter networks: Re- arranging the visual world for robotic manipulation. In Conference on Robot Learning, pp. 726- 747. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2024
Menglong Zhu, Konstantinos G Derpanis, Yinfei Yang, Samarth Brahmbhatt, Mabel Zhang, Cody Phillips, Matthieu Lecce, and Kostas Danilidis. Single image 3d object detection and pose esti- mation for grasping. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 3936-3943. IERE, 2014.
Xupeng Zhu, Dian Wang, Ondrej Biza, Guanang Su, Robin Walters, and Robert Platt. Sample efficient grasp learning using equivariant models. Proceedings of Robotics: Science and Systems (RSS), 2022.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 PSEUDOCODE
Algorithm FourTran inference
1: Given the observation og, pick neural network fpick. observation and crop encoder o, v. coarse discrete SO(3) rotations G., fine discrete SO(3) rotations G
2: if -qquivarrrt pick then
3:
Calculate pick logits in Fourier space: Spick - (pick((s)
4: Calculate coarse pick distribution in spacial: Pe(a Pe(apicklor) Fā. [pick]
5: The coarse pick pose Spick (T, Re) can be evaluated by ppcc arg max pe (Apick)
6: The fine pick orientation R can be evaluated at T by Ry arg max Fa, pick [T]]
apick (T, (y)
8:
end if
9:
if (E) x (EEdd--qquivrian place then
10:
Crop the observation at pick location: c= crop(or, apick)
Encode observation and crop: o(o,), vb(e
12:
Calculate lifted crop encoding Fourier domain: k(c) = F+ FFFF (v(c))
(v((((()))
Calculate place logits in fourier: fplace x(()) d(o))
14:
Calculate coarse place distribution in spacial: Po(Aplace loe) = Follfil
The coarse place pose Slace (T, R.) can be evaluate by dflace = arg max Po (a(ullan))
16:
The fine place orientation R, can be evaluated at T by R arg max Fa, fpace [T]
17:
aplace (T, Ry)
18:
end if
19:
return (apick: aplace)
A.2 PROOF OF PROPOSITION
To prove proposition 1, we begin with 2 lemmas.
Lemma A steerable kernel K; R= - idu x desiviia satisfies
((g) = Pout (g ()
proof Recall that po(g) is an identity mapping. Substituting pm with pu(g) and g with in the steerability constraint K(g x) Pout (1) completes the proof.
((g) K(g K(g-²x)
Pout (g (x))pta)g)
Pout (g 1 (xx)
Lemma cross correlation satisfies that
(9(g) ())(T) (((l))
(8)
proof 2 We evaluate the left-hand side of Equation 8:
(Kg)(K + f) = Σ -(
JEZ
Re-indexing the sum with T gui,
Σ
f(g 7 G
gEZZ
14

---

Published as a conference paper at ICLR 2024
is by definition
E(AG) JEZR
((B(g) * (B(1))))(
as desired.
We first prove that if there is rotation u acting on the observation os, we have x(c) o(u oz) Indpr (u)x(()) * Φ(o,) and the desired place location changed from 7 to Pı (u)T and the action of orientation is changed from ua.
First, consider the no-rotation case without channel-space Fourier transform.
k(c) ((o)) = [3(g) (e)(), ((g2))(()))) (((m))(())) <(o.)
where R(e) L (eb(c)), each rotated crop feature is cross-correlated with o(o,) independently and the entire output can be considered as an m-channel feature map. Assume that the best place action found in i-th channel, i.e., (c) matches the dense feature map of the placement best and thus 9i.
proof Then, consider a rotation acting on og and assume the new best place action is found in the k-th channel
k(c) * puu 0+) = k(c) (B(w))(o:)) Equiv. of
= [(() * B(u)
())() * ((o)) lemma
Bu ((gm)) Φ(o,))
(()))))( lemma
(())) (R(c))
= Indpr (u) o(o.))
Note ((-) acting on the base domain while p(g) acting on the fiber space. Assume that m infinite and the best place action is found in the k-th channel, i.e, ((u=))8(g)))) (((c) produces the best match. We have 3(u )((g))) = ((g:) (b(()) and we can get:
9k 94 since B(-) is bijective mapping
Multiplying from the left realizes that gk ua. It shows that after rotation the crop, the orientation component of the best place action is changed to = ua.
Then, we prove that if rotation h acting on the crop c, the desired place action changed from to ah
proof Consider rotation h acting on the crop and assume the the best place action found in the j-th channel. The place network can be evaluated as R(3(h)c) ((o,) = (B(g)) (B())))) (1. * )(o,)
(ll.))))((()))))))) ((a)))c))) ((0 * Φ(ot) equiv. of
= (pp(gg"))))
Φ(v.) lemma and definition of pR(g)
Assume that m infinite and the best place action is found the j-th channel, i,e, (()))))))) produces the best match. We have B(g, 3(h)p((c) = 3(g:))(()) and since B(-) is bijective map- ping, we can get:
g gi
Multiplying h- from the right realizes that gj = ah It shows after a rotation on the crop, the best place action is changed to 93 = ah-1
Combining proof 4 and proof 3 finishes the proofs of proposition 1.
15

---

Published as a conference paper at ICLR 2024
A.3 SO(3) (0(3)-FOURIER FOURIER TRANSFORM
Let ʃ SO(3) - R be a real valued function defined on SO(3). Then, by the Peter-Weyl theorem can be decomposed
Vg E SO(3), f()) =
fkk Das(())
t=0
k,
where D' are the Wigner D-matrices. Wigner D-matrices of order & are irreducible representations
of dimension 26 + 1. The Fourier transform over 5O(3) is defined F(J) (f') where each of
the fe is matrix of size (2 1) (26 1). The inverse Fourier transform can be computed using the orthogonality of the Wigner D-matrices,
fku.
/ gESO(
dg f() (T
where dg denotes the SO(3) Haar measure. practice, we will truncate the index at some maxi- mum value maxx.
A.4 TRAINING DETAILS
We evaluate our method on both 2D and 3D manipulation pick-place tasks. Specifically, we train single-task policy for each task with dataset of n experiment demonstrations. Each demonstration contains more dbservation-acttinn pairs (ot, dpick where @pick denotes the expert pick action and &place is the expert place action. We use expert actions to generate one-hot maps as the ground-truth labels for picking model and placing model. Due to the computation load of equiv- ariant convolutional layers on 3D) voxel grids, we slightly lift the second constraint of Proposition by encoding our & with traditional U-net (Ronneberger al., 2015a). U-net with the long skip connection also maintains a certain amount of the + Po equivariance. Both models are trained end end using a cross-entropy loss. The model is trained using the Adam optimizer with fixed learning rate=le" We report the training time and GPU memory requirement of 3D FOURTRAN in Table 4.
A.5 ABLATION STUDIES
We perform two ablation studies to explore the functionality of our proposed architecture. We first replace the equivariant U-net of th with traditional U-net. This modification reduces the architecture to satisfy the first constraint in Proposition is that remove the lifting and Fourier transform and directly generate the irredueible features for each element, i.c., the model is forced to learn the third constraint of Proposition without prior knowledge. Table shows the performances of all ablations. Comparing the first row with the second row, we find that the results are consistent. We hypothesize that the reasons are: 1) traditional U-net with the skip connections also captures the trivial + trival equivariance and the equivariant constraints of the equivariant layers with O24 group in FOURTRAN Data augmentation is applied both models to learn the equivariance. Comparing the first row to the third row, the ablated model attempting to learn the coefficients of the basis function is not as well as our proposal to generate dynamic steerable kernel. On the other hand, given the fact that lifting with a fixed number of SO(3) rotations approximates the steerability of 3D kernel, this ablated version avoids lifting reduce the computation load. However, without representing SO(3) rotations in Fourier space, it is way more expensive to evaluate fine discrete SO(3) distribution.
The second ablated version of our model
not strong. 2)
A.6 MODEL COMPARISONS
We train a single-task agent for each 3D baseline on the five tasks shown in Figure 3 with the same settings presented in their paper. Note that we have {pre-pick, pick, post-pick] and fpre-place, place, post-place) where the pre-action and post-action action while the baseline predicts the action for the next keyframe without a clear line between pick and place.
defined as relative to the pick and place
16

---

Published as a conference paper at ICLR 2024
Model
stack-blocks
phone-on-base stack-wine
put-plate
FOURTRAN (ours)
76
100
96
32
Fourier Transporter w.o equiv, of
76
96
96
24
Fourier Transporter w.o lifting
72
88
96
28
Table 3: Ablation Study: Performance of three variants of FOURTRAN on different RLbench tasks
James al. (2019) Each model trained with 5 demonstrations and evaluated on tests. Best performances are highlighted in bold.
Model
Parameters (M)
Memory(Gb)
Training Time (secs/sgd step)
FOURTRAN SENTTRAN-PPON pick
3.1 M
8.5
1.5
FOURTRAN-
1.6
13.6
1.6
RVT
36 M
12.5
0.46
PerAct
33 M
12
1.5
C2FARM
3.6 M
2
0.07
Table 4: Comparison FOURTRAN (ours) architecture and existing pick-place methods. Tests were performed on NVIDIA 3090 GPU.
Table compares FOURTRAN with the baseline architectures. Note that our model has about same number of parameters C2FARM James et al. (2020) significantly fewer parameters than the transformer-based methods PerActJaegle et al. (2021) (33 M) and RVTGoyal et al. (2023) (36 M). The equivariant 3D convolution implemented in FOURTRAN requires relatively large GPU memory. However, FOURTRAN can produce the action distribution over the entire action space directly instead of a of discrete rotations along each axis. As result, FOURTRAN easily query the backup actions if the action selected cannot be reached.
A.7 DETAILED 3D TASK SETTINGS AND METRICS
The RLbench James al. (2020) is implemented in CoppelaSim (Rohmer et al., 2013) and inter- faced through PyRep (James et al., 2019). All experiments of RLbeneg use a Franka Panda robot with parallel gripper. Our input observations tured point could is parameterized as x 96 x 56 voxel grid and each voxel represents a 0.94cm² cube in our settings.
captured from four RGB-D cameras. The cap-
Figure shows the expert SO(3) rotation action distributions in stack-wing and put-plate tasks. The agent needs to reason about the SO(3) rotation space finish the tasks. For more information of plotting elements of SO(3), please see Murphy et al. (2022).
A.8 DETAILED 2D TASK SETTINGS AND METRICS
At the beginning of each episode, the poses of objects and placements in each task are randomly sampled the workspace without collision. The visual observation o, is a top-down projection of the workspace with 3 simulated RGB-D cameras pointing towards the workspace. Our pixel resolution is 320 160 for the Im x 0.5m workspace. We measure performance the same way as it was measured in Transporter Net Zeng et al. (2021) using metric in the range of 0 (failure) to 100 (success). Partial scores are assigned to multiple-action tasks. For example, in the assembling kit task where the agent needs to assemble 5 objects, each successful rearrangement is credited with a score of 0.2. We report the highest validation performance during training, averaging over 100 unseen tests for each task.
17

---

Published as a conference paper at ICLR 2024
182
270
270
Figure 5: Visualization of expert S3(3) actions from 10 demonstrations. First column: expert pick action. Second column: expert place action. First row: stack-wine. Second row: put-plate. The orientation visualization follows "YXY" convention. For more detail on plot formatting, please see Murphy et al. (2022)
B PROOF OF PROPOSITION 2
B. THE JOF-LIGNING MAP IS A STEERABLE KERNEL WITH TRIVIAL INPUT TYPE AND REGULAR OUTPUT TYPE
The lifting map (f) generates Ite-rteee kernel with regular-type output. To this, note that, by the definition of Lt,
Vz Rd. (f(())) {gr f(x), 92 f(x), ... 9m f(x)} where each 9i are elements of loo- Thus, Vgg E Lso. we have that (f =)) (g f(x)) f(gx),.-.. [(g)) ( ( [ ( gx))
Now, let g be fixed element of the icosahedral group. Left multiplication by an element of g is group homeonorphism of 160. Let define 9g(1) to be the element of the sampled set y: so that
9g(1) 9
Let the identity element on SO(d). Then, note that
(1()) 9i
The map Iat() satisfies an additional composition property. Let g.g' be elements of SO(d), then,
(()) 9a(s'(9)
holds exactly. Thus, the map g(-) E The expression for (f( can be rewritten as
L' r)) (f 2). x)} ((g)))(f()))
where the matrix has elements given by
p(g);z 8ig
Using (1(3) = 9t. we have that the matrix p satisfies.
p(1) Ilo
where Loo is the identity matrix in 60 dimensions. Using the identity I(gs = Ig(9"( the matrix p satisfies the relation
p(g)p(g') p(gg')
which is the definition of group representation. Thus, the matrix p is exactly the permutation representation of Igo- Thus, the lifting operator (f) is an steerabee kernel with trivial input type and regular output type.
18

---

Published as a conference paper at ICLR 2024
B.2 FIBER SPACE FOURIER TRANSFORM
Let us suppose that the map (f) is exactly -to-Sreeeblle kernel with trivial input type and regular output type. We can then compute the fiber space (0(3)-Fourier transform of LT)
Emaaee
Pout (g)
my D (g)
t=0
where Df is the 2-th irreducible SO(3).
Let F+ be the Fourier transform in the fiber space given by
Emar
(t))
E (z))
t=
where the irreducible coefficients are given by the fiber space Fourier transform
(x))= = /
dR (R(((
(RESO(3)
B.3 THE STOCIISTTCCLLYY SAMPLED LIFTING MAP IS APPROXIMATELY A (TEE)-STEERABLE KERNEL
The stochastically sampled lifting map (J) approximately generates a steerable kernel with regular-type output. To see this, note that, by the definition of L
Vo E ER", (T((()) = (gr f(x),92 f(z)...., Im f(x)}
where each ys sampled iid from (d))). Thus, Vy E (O(d), we have that
(f(g z)) {gr f(g)) f(gr), 9m f(gr)} = (f(fss z), f(gz
f gz)}
Now, let g be fixed element of (O(d). Left multiplication by an element of g is group homeo- morphism of (O(d). Let us define 9o(1) to be the closest element of the sampled set 9; so that
( argmax j=1.2...m 1lg 9. gill
where the norm is the geodesic distance on SO(d). Let us assume that we work in the regime where the number of samples m is large and
max 122 [g
9:
95
m
For iid uniform Is. this property holds generically for large values m. This can be proved rigor- ously using concentration of measure phenomena.
Let the identity element on SO(d). Then, note that
91(3) argmax
Ilg, -- m.
gill 9i
The map Ia(t) satisfies an additional composition property. Let g be elements of SO(d), then,
(6) argmax (gg')) =7ll -= argmax mlg(gl g:)) 9;
Thus,
(() (9(s((()
holds approximately.
We may decompose the expression for (g)(())
(f(ll x)) = (f(g fl f(g (og())))))))))))) 2)} + error
19

---

Published as a conference paper at ICLR 2024
where the error term can be written as
error (f I [( ( (0x"g gz) - /( x). ... (m gx) - I gg(mm))))
we assume that the function f is L-Lisschitz continuous, then,
Le
19 ga) I (g)de))) Ellge' 9 go(k)]
m
and long is large, the error term is small (roughly O )). Ignoring the emror term, the expression for L (f) can be rewritten
k (3 x)) - (o 145 (() (.. = plg)
where the matrix p (which implicitly depends on the sampled g:) has elements given by
p(g) 6tg(s)
Using (1(3) Dt. have that the matrix p satisfies,
((1) Tm
where In is the m-dimensional identify matrix. Using the identity ((a)')()) (s)a'()) the matrix p approximately satisfies the relation
p(g))(g') p(gg')
which is the definition of group representation. Thus, up to approximation, the matrix is the permutation representation of (Od).. The di sampled at some numerical resolution with corre- sponding bandwidth kmaz. Ergo, for large m, the matrix is good approximation to the SO(d) permutation representation at bandwidth L Lmax-
C MATHEMATICAL BACKGROUND
We establish some notations and review some elements of group theory and representation theory. For comprehensive review of representation theory, please see Zee (2016); Ceccherini-Silberrtiii et al. (2008).
C.I GROUP THEORY
At high level, a group is the mathematical description of symmetry. Formally, a group G is non-empty set combined with a associative binary operation G G + G that satisfies the following properties
existence of identity: E G, s.t. Ag G, e-g=g.c.
existence of inverse: Vg G, 3g G, g g.g = g g=e
The identity element of any group G will be denoted as Note that the set consisting of just the identity element e a group.
C.2 REPRESENTATION THEORY
Let V be vector space over the field representation (p. V) of a group G consists of V and group homomorphism p G + Hom V]. By definition, the p map satisfies
Vg.g'= E G, V plg)p(g')v = p(gg')v
Two representations (p, V) and (a, W) are said to be equivalent representations there exists an invertable matrix U
bg G, Up(o) o
20

---

Published as a conference paper at ICLR 2024
The linear map U is said to be G-intertwiner of the (p. V) and (a, W) representations. A repre- sentation is said to be reducible it breaks into a direct sum of smaller representations. Specifically, unitary representation is reducible if there exists unitary matrix U such that
Ag NEG. G, p(g) UI (.(l))t 1=1
where k 2 and o; smaller irreducible representations of G. The set of all non-equivalent representations of group G will be denoted G. All representations of compact groups G can be decomposed into direct sums of irreducible representations. Specifically, if (a, () is G- representation.
(o, V) = UU θ
m (p, Vp)]Ui
where U is a unitary matrix and the integers my denote the number of copies of the irreducible (p, vp) in the representation (o, V).
C.3 WIGNER D-MATRICES
The fundamental representation of the rotation group in three-dimensions is given by
SO(3) =
e R333, R R 13, det(Rt =
It should be noted that a group is an abstract mathematical object and that the standard parameteri- zation is a choice. There are multiple non-equivalent representations of the group SO(3). The irre- ducible representations of SO(3) are known as Wigner D-matrices (D V*). The Wigner D-matrix of order is real representation dimension (2f + 1) x (2f + 1). Although Wigner D-matrices are difficult to visualize, the l representation is just the standard 3 x 3 matrix representation of (3()).
C.4 PETER WEYL THEOREM AND FIBER SPACE FOURIER TRANSFORM
The Peter- Weyl theorem Cecchhrrii-Silbeestenn et al., 2008) states that all representations of com- pact groups can be decomposed into a countably infinite sets of irreducible representations. Consider the set of functions
F=(S1S:G-C)
of all complex valued function defined on G. The set F forms a vector space over the field C. The group G' acts vector space F. Specifically, define the group action À A G
E F, Vg.g's E G, (ig') = f(g EF
The action satisfies >giag = >gg and is group homeomorphism. The left-regular representation of group is defined (A, F). The Peter-Weyl theorem Ceccherini-Silleertthh al. (2008) states that
=) (A. = UUD dolp, Vo)lur
DEG
where is the unitary matrix. Thus, the left-regular representation decomposes into d, copies of each (p, Vo) irreducible. In other words, the Weyl theorem states that matrix elements of irreducible G-representations form an orthonormal base of the space of square- -quare-integrable functions on G.
C.5 IRREDUCIBLE REPRESENTATION ORTHOGONALITY RELATIONS
Matrix elements of irreducible representations satisfy a set of orthogonality relations Zee (2016). Specifically, let p and be irreducible representations of the group G. Then,
|G| So.dkrdōw
Σ BEG
Pku (g)o
dp
21

---

Published as a conference paper at ICLR 2024
where |G] is the cardinality of the group. The orthogonality relations of different irreducible repre- sentations is an analogue of harmonics of different frequencies in Fourier analysis. As an example, consider the case with G (2) Irreducible representations of SO(2) are all one dimensional and labeled by an integer. They are all of the form
px( = exp(ike)
The orthogonality relations applied to G SO(2) then state that
J.
2m
døpx (
dø dx - k')Ø) 2nT
which is the standard orthogonality relation for different Fourier harmonics.
C.6 INDUCED REPRESENTATIONS
The induced representation is way to construct representations of larger group G out of repre- sentations subgroup _ G. Let (p, V) be a representation of H. The induced representation of (p. V) from to G denoted as Ind ((,, V)]. Define the space of functions
F F= f f: V, E H, f(gh) p(h-t)f(g)
Then the induced representation defined as (r, = Ind V)] where the induced action æ acts on the function space FF via
Vg.g's E G, VFEF, (T(2) A)0) = f(g
The induced representation was originally used Cohen & Welling (2017) to design networks that equivariant with respect both rotations and translations. Induced representations to change the underling group equivariance Cesa et al. (2021); Weiler & Cesa (2019); Howell et al. (2023). Specifically, induced representations can be used design SE(3) -qquivarrra maps from (3 maps.
be used
22


Reviews:
[{"review_id": "EY3K2bopfd", "reviewer": "Reviewer_DeU9", "summary": "The paper introduces a method called Fourier Transporter (FOURTRAN) to enhance the efficiency of training robotic agents in performing pick and place actions in 3D environments. By incorporating the bi-equivariant symmetry of the problem into a behavior cloning model, FOURTRAN utilizes a Fourier transformation to encode the symmetries of these actions independently, which enables memory-efficient construction and improves sample efficiency.", "questions": "- What is **fiber space** Fourier transformation?\n- In Figure 2, how do you crop the object in the scene? What if there are multiple objects?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper proposes FOURTRAN for leveraging bi-equivariant structure in manipulation pick-place problems in 2D and 3D.\n- The paper presents a theoretical framework for exploiting bi-equivariant symmetry. It contains proofs for propositions that address the symmetry constraints and properties of the model.", "weaknesses": "- The current model is limited in a single-task setting, while the baseline methods are designed for multi-task purposes. I'm concerned that the comparisons may not be fair.\n- It relies solely on open-loop control, disregarding path planning and collision awareness.\n- The paper is not well-written and some of the terms are difficult to understand. It uses a lot of notations, but many of them are not explained.\n- There are no real robot experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FVH7iATNrt", "reviewer": "Reviewer_n4JL", "summary": "The paper presents a method for taking advantage of bi-equivariance found in some manipulation problems (equivariance with respect to both the pick and the place pose) for representing distributions over pick-place actions, which exist in $\\textrm{SE}(3) \\times \\textrm{SE}(3)$ and pose sample-efficiency challenges when represented without taking advantage of symmetry. The proposed method demonstrates very strong performance on a variety of imitation learning benchmarks, particularly those requiring fine-grained control.", "questions": "Is it possible that $\\textrm{SO}(2)x\\mathbb{R}^3$-equivariance (2D rotational+translational) is actually more general if grasp dynamics are dependent on the object's orientation with respect to gravity?\n\nIt seems like $I_{60}$ is used in the lifting operation, but as far as I can tell there's no reason the set of rotations has to form a subgroup. Is this correct, e.g. could the granularity be increased by simply sampling more rotations (roughly evenly spaced in $\\textrm{SO}(3)$) in this step?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The argument for a bi-equivariant policy is compelling. The use of Wigner D-matrices to represent an output distribution is very clever and (to my limited knowledge of the literature) seems novel. Their use in the place network to generate fast cross-correlations for bi-equivariance is definitely novel. All theory is well presented and seems well-backed, if a little dense at times to readers less versed in differential geometry and representation theory.\n\nEmpirical results are extremely compelling. The proposed method seems to strongly outperform some relatively strong baselines on very low-data BC tasks.", "weaknesses": "Weaknesses mostly center around presentation: the paper contains a lot of dense jargon, which is understandable given the material but could be improved:\n - Given that the Wigner D-matrix representation and corresponding 3D Fourier transform is the key insight that allows this action representation to work, it would be worth spending some more time to describe them in more detail\n - Some pseudocode/method description would be welcome\n\nOtherwise, further analysis of the representations introduced would be nice:\n - $\\ell$\n - the number of rotations in the lifting operation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5ljy2MbeMJ", "reviewer": "Reviewer_sjsh", "summary": "This paper introduces a novel approach for solving 2D and 3D pick and place tasks. The key innovation lies in leveraging Fourier transformation in fiber space to create a memory-efficient and sample-efficient bi-equivariant model. The paper provides theoretical analyses of the method and evaluates it in 2D and 3D simulation benchmarks. When compared to other methods on the RLBench (James et al. (2020)), this approach achieves a substantially higher success rate, and on the Ravens benchmark (Zeng et al., 2021), it demonstrates some improvements.", "questions": "In section 5.3 2D Pick-Place results, the last line: ”It indicates that the\nSO(2) × SO(2) equivariance of FOURTRAN is more sensitive to rotations.\n”. Does “sensitive” means more precise or prone to noise? It would be interesting to conduct separate tests with high-resolution thresholds to distinguish the impact of position error and rotation error. For example, considering parameters like τ = 1cm and ω = 7.5&deg; as well as τ = 0.5cm and ω = 15&deg;. Additionally, a box plot of the rotation error would also provide more insight into the effect of the method.\n\n\nMinor issues and typos\n\n* The last line in 3 Background: Appendix C\n* The last line on page 4: “Here the action on the base space rotates the pick location and the fiber action transforms the pick orientation.” should it be: “Here the action on the base space transforms the pick location and the fiber action rotates the pick orientation”?\n* Page 7:  “The different 3D tasks are shown graphically in Figure 4” should be Figure 3\n* Table 3: Success rate(%) of three......\n* Page 15: icosohedral -> icosahedral", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper shows novelty in the use of Fourier transformation in fiber space, leading to memory efficiency and enhanced sample efficiency for 3D pick and place tasks. Additionally, the proposed methods demonstrate superior performance compared to baseline approaches in select RLBench tasks.", "weaknesses": "While the paper demonstrates strong results on RLBench tasks, it's important to note that some tasks like \"stack-blocks\" and \"stack-cups\" primarily operate in 2D space, which may not fully reveal the strengths of the methods in 3D. It would be valuable to include additional\ntasks that involve more 3D rotation angles, such as “put books on bookshelf”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fcy2uvPYyR", "reviewer": "Reviewer_vhc8", "summary": "This paper introduces the fourier transporter, a neural architecture that explicitly captures the bi-equivariant relationships implicit in many pick and place tasks, that is, an object may exhibit rotational symmetry in both picking actions, and placing actions. The architecture first selects an appropriate pick pose using a network that outputs a distribution over picking actions (positions and orientations), with positions used to crop a region about the object to be placed.  This region is then lifted to form a stack of rotated features (a steerable filter) by a network to capture the rotational symmetry present in the picking action. The fourier transform of these features is applied, and used to perform cross-correlation (in the fourier domain, to allow for more efficient computation) with a feature map generated over the workspace observation to determine a distribution over placing positions.   A coarse to fine approach is used to refine the resolution of the pick and place actions, by sampling more rotations as required to refine the pick and place actions. Behaviour cloning experiments are conducted on a range of simulated 3D and 2D manipulation tasks (RLBench, Raven) and show improved success in terms of success rate as a function of training demonstrations. Ablations appear to indicate that most of the heavy work is done by the lifting operation.", "questions": "What level of rotational variation is present in the demonstrations and experiments? Is it possible to share some data on the typical distributions and tolerances in pick/place angles the tasks here require?\n\nCould you explain table 3 in more detail - the  ablation here appears to undermine many of the choices this work makes. If a unet + data augmentation can capture many of the equivariance relations, and the lifting operation is the big contributor, then why do we need bi-equivariance? Is this simply an artifact of the test scenarios not adequately evaluating 3D equivariance symmetries?\n\nIn terms of the extension to 3D, it seems that the lifting operator introduces challenges around partial occlusions, that may be hard to learn regardless of the bi-equivariance structure. Could you comment on the general performance/ potential limitations in this regard?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well written and motivated, and does an excellent job of formalising the equivariance in robotic pick/place tasks, nicely mapping theory to practice.\n\nThe core contribution (applying the cross-correlation in the fourier domain) is a great way to reduce complexity and allow more lifting angles and finer resolution pick/place, particularly when combined with coarse to fine sampling.\n\nBi-equivariant networks provide a seemingly impressive boost in performance when compared to prior models that do not consider these symmetries.", "weaknesses": "The core weakness of this work is the strength of the contribution when compared to the equivariant transporter network proposed in Huang 2022. As far as I can tell key differences include generalising to 3D, more empirical experiments in this domain, and the implementation of the cross correlation in the fourier domain. As mentioned in this work, it is true that the Huang 2022 paper only considers SO(2), and is a subset of the general theory presented here, but more needs to be done to justify why the extension to 3D is non-trivial, particularly when it comes to the major claims of this work, greater angular resolution, computational benefits of fourier implementation, and sample efficiency. \n\nAlong these lines, I would have liked to see an explicit experiment showing clear evidence of higher angular resolution performance (beyond the 15/7.5 degree results in table 2).\n\nNo error bars are provided in experiments (Tables 1/2), so we have no indication that the results are significant. I am sure they are, but this is important for the table 2 comparison with Huang 22.\n\nThe mapping between Figure 2 and the equations in Section 4 is incomplete, and not easily followed. Not all notation is clearly defined (eg. $Ind_{\\rho l}, \\rho_{irrep}, h$ etc.) and equations don't immediately use the network notations ($\\psi, \\phi$). I recognise that much of this notation is standard in group theory, but it is not in robot learning, so there would be value in defining this. This forces the reader to make assumptions/ spend significant time interpreting the mappings between text and figures, and hurts readability. I also recognise that this is out of a desire to formalise and explain the general problem before introducing the specifics of the architecture and approach taken to address this, but the current structure of this section/ group theory jargon made this difficult to follow.\n\nHaojie Huang, Dian Wang, Robin Walters, and Robert Platt. Equivariant Transporter Network. In Proceedings of Robotics: Science and Systems, New York City, NY, USA, June 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)